feat(ai): dedicated wiki-embed endpoint for the Notes ranker

Second bge-m3 llama.cpp Deployment in ns ai so wiki queries never queue
behind Paperless on llama-embed. Shares the llama-server-models PVC, so
the weights are not downloaded again.

Exposed LAN-only as wiki-embed.renner.uno (no dns.renner.uno/public
label) for bin/wiki_rank.py on the workstation and the nightly wiki jobs.
The client is a script, and the Authentik gate is browser-only, so the
Ingress uses Traefik BasicAuth backed by an ExternalSecret, the same
shape as loki.renner.uno. A pod-scoped CiliumNetworkPolicy admits Traefik
to wiki-embed only; llama-server stays reachable solely via its gate.

Measured on 40 past wiki-synth queries: embeddings picking 150 candidates
plus one TypeSafe Choice puts a cited note in the top 5 for 95% of
queries at a sixth of the cost of ranking all 1700 notes with TypeSafe.

Needs before first use: OpenBao key apps/ai/wiki-embed-basic-auth with
field `users` (htpasswd line), and the same credential as user:password
in ~/.config/wiki-embed/basic-auth on the workstation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.woodpecker/alert-monitor.yml

@@ -42,7 +42,7 @@ when:
 
 steps:
   - name: fetch-alerts
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/build-ci-images.yml

@@ -19,12 +19,13 @@
 # the admin account could (#330).
 #
 # The token is still OWNED by gitea_admin, because Gitea scopes packages to
-# a user or org namespace and these images live under `gitea_admin/*` — a
-# different account cannot publish there. Moving them to a dedicated org is
-# the remaining cleanup; it is a 38-reference migration across pipelines and
-# Flux-deployed CronJobs, so it is tracked separately in #330 rather than
-# bundled here. Note this pipeline runs no LLM agent, so the exfil path that
-# motivated the other cutovers does not apply to it.
+# a user or org namespace and most of these images live under `gitea_admin/*`.
+# The rebuild-critical ones additionally publish to `konairius/` — an org
+# created for the ghcr mirror path, with gitea_admin as a member so this same
+# token can push there. The full gitea_admin/* -> org migration is a
+# 38-reference change tracked separately in #330 rather than bundled here.
+# Note this pipeline runs no LLM agent, so the exfil path that motivated the
+# other cutovers does not apply to it.
 when:
   - event: push
     path:
@@ -87,6 +88,30 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
+  - name: build-vmtest
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: ci/images/vmtest
+    depends_on: []
+    when:
+      - event: deployment
+      - event: push
+        path: ["ci/images/vmtest/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/ci-vmtest:$TAG \
+          --destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
+
   - name: build-tofu
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
@@ -147,6 +172,28 @@ steps:
   # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
+  #
+  # Published THREE times, and the third one is not a typo. Its consumers — the
+  # zigbee2mqtt seed-data init container and the trmnl renderer — start while
+  # the cluster is still being rebuilt, and at that moment gitea.renner.uno does
+  # not exist, because Gitea runs inside the cluster being built. A cold-rebuild
+  # rehearsal caught both in ImagePullBackOff.
+  #
+  # So there is a copy on ghcr.io. But the manifests must NOT point at it:
+  # normal operation pulls from the local registry, and an external reference
+  # would make every steady-state pull leave the network. The recovery source
+  # and the everyday source are different things, exactly as they are for Flux's
+  # git source.
+  #
+  # Talos resolves that with machine.registries.mirrors, which takes an ordered
+  # endpoint list — gitea first, ghcr second (bootstrap.sh). Containerd keeps
+  # the repository path when it falls through, so the two registries have to
+  # agree on it: `gitea_admin/alpine-jq` cannot be served by ghcr, where the
+  # namespace is fixed to the account name. Hence the second Gitea destination
+  # under konairius/, which the manifests reference and which matches the ghcr
+  # path exactly.
+  #
+  # The gitea_admin/ copy stays until nothing references it any more.
   - name: build-alpine-jq
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
@@ -159,23 +206,47 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
+      GHCR_USER: konairius
+      GHCR_TOKEN:
+        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
+          --destination gitea.renner.uno/konairius/alpine-jq:$TAG \
+          --destination gitea.renner.uno/konairius/alpine-jq:latest \
+          --destination ghcr.io/konairius/alpine-jq:$TAG \
+          --destination ghcr.io/konairius/alpine-jq:latest
 
   # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
   # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
   # node-side fetch from GitHub that had NO checksum verification and sat on the
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
+  #
+  # This one ALSO publishes to ghcr.io. A cold-rebuild rehearsal proved why: the
+  # DaemonSet could not pull from gitea.renner.uno, because Gitea runs inside the
+  # very cluster being rebuilt. Without the plugins there is no macvlan, so
+  # Multus cannot attach ceph-san, so Rook's network canary never starts, so the
+  # CephCluster sits at "Configuring Ceph Mons" — and OpenBao waits forever for a
+  # PVC that will never bind. Moving the download here fixed the integrity gap
+  # but quietly swapped github.com on the critical path for something strictly
+  # worse: a registry that by definition does not exist during a recovery. The
+  # GHCR copy is public, so a bare node can pull it with no credential and no
+  # Gitea.
+  #
+  # The test that proved this also proved the rule is not specific to this
+  # image. alpine-jq and alpine-openssl mirror for the same reason — see the
+  # note on build-alpine-jq. Anything an app needs while the cluster is still
+  # being built cannot live only in that cluster's own registry.
   - name: build-cni-plugins
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
@@ -188,23 +259,36 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
+      GHCR_USER: konairius
+      GHCR_TOKEN:
+        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
+          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
+          --destination gitea.renner.uno/konairius/cni-plugins:$TAG \
+          --destination gitea.renner.uno/konairius/cni-plugins:latest \
+          --destination ghcr.io/konairius/cni-plugins:$TAG \
+          --destination ghcr.io/konairius/cni-plugins:latest
 
   # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
   # needing s_client/x509. Named for what it is; first consumer is the paperless
   # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
+  #
+  # Mirrored to ghcr.io for the reason given on build-alpine-jq: it is an init
+  # container, so paperless does not start without it, and during a rebuild
+  # gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
+  # paperless in Init:ErrImagePull on exactly this image.
   - name: build-alpine-openssl
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
@@ -217,17 +301,25 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
+      GHCR_USER: konairius
+      GHCR_TOKEN:
+        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
+          --destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
+          --destination gitea.renner.uno/konairius/alpine-openssl:latest \
+          --destination ghcr.io/konairius/alpine-openssl:$TAG \
+          --destination ghcr.io/konairius/alpine-openssl:latest
 
   # Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
   # run as two services on edge-1. Replaces pip-into-venv on the host, where an
@@ -257,3 +349,56 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
           --destination gitea.renner.uno/gitea_admin/heating-can:latest
+
+  # netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
+  # services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
+  # Runtime artifact like heating-can; same registry so the audit tracks it.
+  - name: build-netbird-proxy-services
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: ci/images/netbird-proxy-services
+    depends_on: []
+    when:
+      - event: deployment
+      - event: push
+        path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
+          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
+
+  # bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
+  # artifact like heating-can; same registry so the audit tracks it.
+  - name: build-bird
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: ci/images/bird
+    depends_on: []
+    when:
+      - event: deployment
+      - event: push
+        path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/bird:$TAG \
+          --destination gitea.renner.uno/gitea_admin/bird:latest

.woodpecker/ci-manifests-test.yml

@@ -60,7 +60,7 @@ steps:
   # run that never touched them. Tracked in #461.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     commands:
       - |
         set -eu
@@ -158,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
     commands:
       - |
         set -eu
@@ -246,7 +246,7 @@ steps:
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     commands:
       - |
         set -eu

.woodpecker/e2e-test.yml

@@ -4,7 +4,7 @@ when:
 
 steps:
   - name: e2e
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/e2e.yml

@@ -31,7 +31,7 @@ steps:
       - sh scripts/ci/test-model-selection.sh
 
   - name: e2e-test
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/issue-fix.yml

@@ -49,7 +49,7 @@ steps:
     # (the issue-pick logic needs jq, which only the ci-claude image has);
     # minting a 1h read-only token per run is cheap. Degrades to
     # snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

ansible/roles/heating-can/defaults/main.yml

@@ -26,8 +26,16 @@ edge_open3e_poll_seconds: 30
 # entries when added), so the multi-frame read is cheap at a 30 s poll.
 # Records faults/info events only (NormalOperation, LagDeviceDtcReported,
 # LegionellaProtectionTemperatureReached) — NOT setpoint or program changes.
+# onebase.*: the 29 paired ZigBee slots on the One Base gateway (ECU 0x6c3) --
+# 8 radiator valves, 9 climate sensors, 9 underfloor actuators, 3 floor
+# thermostats. Per-device room temperature, active setpoint, humidity, valve
+# opening/demand, battery and RSSI; the local replacement for the ViCare cloud
+# room sensors. Needs the onebase overlay (files/open3e-onebase.py) -- the stock
+# codec declares 57 bytes and a paired slot answers with 68.
+# Only CurrentValues are polled. The room and device registries (1884.., 2084..)
+# are static configuration and are read on demand, not every 30 s.
 edge_open3e_datapoints: >-
-  vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791
+  vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791,onebase.2086,onebase.2089,onebase.2092,onebase.2095,onebase.2098,onebase.2101,onebase.2104,onebase.2107,onebase.2110,onebase.2113,onebase.2116,onebase.2119,onebase.2122,onebase.2125,onebase.2128,onebase.2131,onebase.2134,onebase.2137,onebase.2140,onebase.2143,onebase.2262,onebase.2265,onebase.2268,onebase.2271,onebase.2274,onebase.2277,onebase.2280,onebase.2283,onebase.2286
 # ECU addresses; -dev selects codec set, tx selects the UDS address.
 # vcmu (refrigerant-circuit controller) has no dedicated codec set —
 # omitting dpList makes open3e fall back to the generic datapoint list,
@@ -38,6 +46,9 @@ edge_open3e_devices:
   # files/open3e-vx3plus.py) — die Stock-vx3-Liste filtert sie sonst weg.
   vx3: { tx: "0x6a1", dpList: "/config/vx3plus.py" }
   vcmu: { tx: "0x68c" }
+  # One Base ZigBee gateway. Overlay re-declares the paired ZigBee slots at
+  # their real 68-byte length (see files/open3e-onebase.py).
+  onebase: { tx: "0x6c3", dpList: "/config/onebase.py" }
 
 edge_e3oncan_gap_seconds: 10
 edge_e3oncan_mqtt_format: "{device}_{didNumber:04d}_{didName}"

ansible/roles/heating-can/files/open3e-onebase.py

@@ -0,0 +1,54 @@
+# Managed by ansible (roles/heating-can). Datapoint overlay for the One Base
+# ZigBee gateway on ECU 0x6c3 (BACKENDGATEWAY).
+#
+# Upstream declares ZigBee<N>DeviceCurrentValues as 57 bytes. A PAIRED device
+# slot on this gateway answers with 68: the 57 known bytes plus the parent
+# device serial and three trailing bytes. udsoncan reads only the declared
+# length and then treats the next two bytes -- the first half of that serial --
+# as a foreign data identifier, so every paired slot fails to decode with
+# "Server returned values for data identifier 0x.... that was not requested".
+# The bogus identifier differs per device because the serial does.
+#
+# This overlay re-declares the 29 paired slots at 68 bytes, reusing upstream's
+# sub-type list so the per-device-type O3ESwitch (climate sensor / TRV / floor
+# thermostat / actuator, incl. DeviceTemperatureSetpoint) is inherited rather
+# than copied.
+#
+# EMPTY slots answer with 57 and decode against the stock codec, so they are
+# left alone and are not polled. A slot that is later unpaired therefore starts
+# reporting a decode error on its topic instead of a value.
+#
+# Verified on ECU 0x6c3, 2026-09-17: 29 paired slots, all 68 bytes.
+from open3e.Open3Ecodecs import O3EComplexType, RawCodec
+from open3e.Open3Edatapoints import dataIdentifiers as _master
+
+# ZigBee<N>DeviceCurrentValues of the paired slots: devices 1-20 run 2086..2143
+# in steps of 3, devices 21-29 continue at 2262..2286.
+_PAIRED_CURRENT_VALUES = [2086 + 3 * n for n in range(20)] + [
+    2262 + 3 * n for n in range(9)
+]
+
+
+def _with_parent_serial(did):
+    stock = _master["dids"][did]
+    return O3EComplexType(
+        68,
+        stock.id,
+        list(stock.subTypes)
+        + [
+            RawCodec(8, "ParentDeviceSerialnumber"),
+            RawCodec(2, "Unknown_65_66"),
+            RawCodec(1, "Unknown_67"),
+        ],
+        desc=stock.desc,
+        info=stock.info,
+        acc=stock.acc,
+    )
+
+
+dataIdentifiers = {
+    "dids": {
+        **_master["dids"],
+        **{did: _with_parent_serial(did) for did in _PAIRED_CURRENT_VALUES},
+    }
+}

ansible/roles/heating-can/tasks/main.yml

@@ -129,6 +129,13 @@
     mode: "0644"
   notify: Restart heating-can
 
+- name: Deploy open3e One Base datapoint overlay (68-byte ZigBee slots)
+  ansible.builtin.copy:
+    src: open3e-onebase.py
+    dest: "{{ heating_can_base_dir }}/onebase.py"
+    mode: "0644"
+  notify: Restart heating-can
+
 # --- Compose project ---------------------------------------------------------
 - name: Deploy the broker credential env file
   ansible.builtin.template:

ansible/roles/heating-can/templates/docker-compose.yml.j2

@@ -17,6 +17,7 @@ services:
     volumes:
       - ./devices.json:/config/devices.json:ro
       - ./vx3plus.py:/config/vx3plus.py:ro
+      - ./onebase.py:/config/onebase.py:ro
     env_file:
       # Broker credential, 0600 -- see heating-can.env.j2.
       - heating-can.env

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -32,8 +32,13 @@ netbird_watchdog_fail_threshold: 3
 
 # Minimum seconds between watchdog-issued restarts, so a genuinely broken
 # backend can't put the proxy into a restart loop. Shared by both the URL-probe
-# watchdog and the flood watchdog below.
-netbird_watchdog_cooldown_seconds: 600
+# watchdog and the flood watchdog below. One minute: the route-manager wedge
+# recurs on every routing-peer loss, and rolling node reboots lose one peer
+# every few minutes; a suppressed restart there leaves the wedge standing
+# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
+# measured 2026-09-07). A minute still caps a loop at one restart per
+# probe interval.
+netbird_watchdog_cooldown_seconds: 60
 
 # Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
 # substring in the proxy's docker logs by the flood watchdog. On this build the

ci/images/bird/Containerfile

@@ -0,0 +1,22 @@
+FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
+
+# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
+# step — the image is the binary plus its runtime.
+#
+# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
+# 3.22/community.
+#
+# Config is read from /etc/bird/bird.conf, not the package default
+# /etc/bird.conf, so the mount cannot shadow what the package ships.
+#
+# kubectl renders the init container's per-node config: go-template-file for
+# the templating, a field selector for excluding the node itself, in-cluster
+# config for the auth. From the distro, so there is no version literal to drift
+# and no download to trust. It is only ever used to read core/v1 Nodes.
+RUN apk add --no-cache bird2 kubectl
+
+# Fails the build rather than the first pod if a binary or its path is wrong.
+RUN /usr/sbin/bird --version \
+ && kubectl version --client > /dev/null
+
+ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]

ci/images/k8s/Containerfile

@@ -1,9 +1,9 @@
 FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
 
-ARG TALOS_VERSION=v1.12.6
+ARG TALOS_VERSION=v1.13.9
 ARG VELERO_VERSION=v1.16.0
-ARG KUBECTL_VERSION=v1.36.2
-ARG FLUX_VERSION=2.9.2
+ARG KUBECTL_VERSION=v1.36.3
+ARG FLUX_VERSION=2.9.5
 
 RUN apk add --no-cache \
       aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
 
 RUN curl -sfL -o /usr/local/bin/talosctl \
       "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
  && chmod +x /usr/local/bin/talosctl
 
 RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \
 
 RUN curl -sfL -o /tmp/flux.tar.gz \
       "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
- && echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3  /tmp/flux.tar.gz" | sha256sum -c - \
+ && echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473  /tmp/flux.tar.gz" | sha256sum -c - \
  && tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
  && chmod +x /usr/local/bin/flux \
  && rm -f /tmp/flux.tar.gz

ci/images/netbird-proxy-services/Containerfile

@@ -0,0 +1,12 @@
+# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
+# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
+# public label. Standard library only, so the image is the interpreter plus one
+# file; runs unprivileged as its own user.
+FROM python:3.13-slim
+
+RUN useradd --system --no-create-home --shell /usr/sbin/nologin controller
+COPY controller.py /opt/controller.py
+
+ENV PYTHONUNBUFFERED=1
+USER controller
+ENTRYPOINT ["python3", "/opt/controller.py"]

ci/images/netbird-proxy-services/controller.py

@@ -0,0 +1,275 @@
+#!/usr/bin/env python3
+"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
+
+Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
+For each host on such an Ingress the NetBird management gets one reverse-proxy
+service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
+subnet resource that covers the cluster's service CIDR. SNI and Host reach
+Traefik untouched, the proxy terminates nothing, and no domain resource is
+involved -- the routing peers stay ordinary IP routers. The target sends a
+PROXY protocol v2 header so Traefik (and CrowdSec behind its access log) sees
+the WAN client address instead of the routing peer's pod IP.
+
+Ownership is derived from NetBird itself, not from local state: a service is
+managed iff its mode is "tls" and its target is this controller's subnet
+resource. Services that do not match (the hand-made HTTP/domain ones) are never
+touched; a desired host held by such a service is reported and skipped until
+that service is removed by hand.
+
+Credentials: the NetBird PAT is read from OpenBao on every pass through the
+Kubernetes auth role, exactly like the setup-key rotator; it never lands in a
+Kubernetes Secret or in a log line.
+"""
+
+import json
+import logging
+import os
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+LOG = logging.getLogger("netbird-proxy-services")
+
+NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
+BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
+BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
+BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
+BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
+SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
+NETWORK_NAME = os.environ.get("NETWORK_NAME", "k8s public ingress")
+ROUTER_GROUP = os.environ.get("ROUTER_GROUP_NAME", "k8s-ingress-router")
+SOURCE_GROUP = os.environ.get("SOURCE_GROUP_NAME", "All")
+TRAEFIK_NS, TRAEFIK_SVC = os.environ.get("TRAEFIK_SERVICE", "ingress/traefik").split("/", 1)
+PUBLIC_LABEL = os.environ.get("PUBLIC_LABEL", "dns.renner.uno/public")
+LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "443"))
+INTERVAL = int(os.environ.get("INTERVAL_SECONDS", "60"))
+HTTP_TIMEOUT = 20
+
+K8S_API = "https://kubernetes.default.svc"
+K8S_TOKEN_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+K8S_CA_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+
+
+class ApiError(Exception):
+    pass
+
+
+def http(method, url, body=None, headers=None, cafile=None):
+    data = None if body is None else json.dumps(body).encode()
+    req = urllib.request.Request(url, data=data, method=method)
+    req.add_header("Accept", "application/json")
+    if data is not None:
+        req.add_header("Content-Type", "application/json")
+    for k, v in (headers or {}).items():
+        req.add_header(k, v)
+    ctx = None
+    if cafile:
+        import ssl
+        ctx = ssl.create_default_context(cafile=cafile)
+    try:
+        with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT, context=ctx) as resp:
+            raw = resp.read()
+            return json.loads(raw) if raw else None
+    except urllib.error.HTTPError as e:
+        detail = e.read().decode(errors="replace")[:300]
+        raise ApiError(f"{method} {url} -> {e.code}: {detail}") from None
+
+
+# --- Kubernetes -------------------------------------------------------------
+
+def k8s(path):
+    with open(K8S_TOKEN_FILE) as f:
+        token = f.read().strip()
+    return http("GET", K8S_API + path, headers={"Authorization": f"Bearer {token}"}, cafile=K8S_CA_FILE)
+
+
+def desired_hosts():
+    sel = urllib.parse.quote(f"{PUBLIC_LABEL}=true")
+    ingresses = k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}")
+    hosts = {}
+    for ing in ingresses.get("items", []):
+        ref = f'{ing["metadata"]["namespace"]}/{ing["metadata"]["name"]}'
+        for rule in ing.get("spec", {}).get("rules", []):
+            host = rule.get("host")
+            if not host or host.startswith("*"):
+                continue
+            hosts.setdefault(host, ref)
+    return hosts
+
+
+def traefik_cluster_ip():
+    svc = k8s(f"/api/v1/namespaces/{TRAEFIK_NS}/services/{TRAEFIK_SVC}")
+    for ip in svc["spec"].get("clusterIPs", []):
+        if ":" not in ip:
+            return ip
+    raise ApiError(f"{TRAEFIK_NS}/{TRAEFIK_SVC} has no IPv4 ClusterIP")
+
+
+# --- OpenBao ----------------------------------------------------------------
+
+def netbird_pat():
+    with open(K8S_TOKEN_FILE) as f:
+        jwt = f.read().strip()
+    login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
+    bao_token = login["auth"]["client_token"]
+    kv = http("GET", f"{BAO_ADDR}/v1/{BAO_PAT_PATH}", headers={"X-Vault-Token": bao_token})
+    pat = kv["data"]["data"].get(BAO_PAT_KEY)
+    if not pat:
+        raise ApiError(f"OpenBao {BAO_PAT_PATH} has no key {BAO_PAT_KEY}")
+    return pat
+
+
+# --- NetBird management -----------------------------------------------------
+
+class NetBird:
+    def __init__(self, pat):
+        self.h = {"Authorization": f"Token {pat}"}
+
+    def get(self, path):
+        return http("GET", f"{NB}/api/{path}", headers=self.h)
+
+    def list(self, path):
+        # An empty collection comes back as an empty body / null, not [].
+        return self.get(path) or []
+
+    def post(self, path, body):
+        return http("POST", f"{NB}/api/{path}", body, headers=self.h)
+
+    def put(self, path, body):
+        return http("PUT", f"{NB}/api/{path}", body, headers=self.h)
+
+    def delete(self, path):
+        return http("DELETE", f"{NB}/api/{path}", headers=self.h)
+
+
+def group_id(nb, name):
+    for g in nb.list("groups"):
+        if g["name"] == name:
+            return g["id"]
+    raise ApiError(f"NetBird group {name!r} not found")
+
+
+def ensure_network(nb):
+    for n in nb.list("networks"):
+        if n["name"] == NETWORK_NAME:
+            return n["id"]
+    n = nb.post("networks", {"name": NETWORK_NAME,
+                             "description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
+    LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
+    return n["id"]
+
+
+def ensure_resource(nb, net):
+    for r in nb.list(f"networks/{net}/resources"):
+        if r.get("address") == SERVICE_CIDR:
+            return r["id"]
+    r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
+                                              "description": "managed by netbird-proxy-services",
+                                              "enabled": True, "groups": []})
+    LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
+    return r["id"]
+
+
+def ensure_router(nb, net, router_group):
+    for r in nb.list(f"networks/{net}/routers"):
+        if router_group in (r.get("peer_groups") or []):
+            return r["id"]
+    r = nb.post(f"networks/{net}/routers", {"peer_groups": [router_group], "metric": 9999,
+                                            "masquerade": True, "enabled": True})
+    LOG.info("created router on group %s (%s)", ROUTER_GROUP, r["id"])
+    return r["id"]
+
+
+def ensure_policy(nb, resource, source_group):
+    name = f"{NETWORK_NAME} access"
+    for p in nb.list("policies"):
+        if p["name"] == name:
+            return p["id"]
+    p = nb.post("policies", {"name": name, "enabled": True, "rules": [{
+        "name": name, "enabled": True, "action": "accept", "protocol": "tcp",
+        "ports": [str(LISTEN_PORT)], "bidirectional": False, "sources": [source_group],
+        "destinationResource": {"id": resource, "type": "subnet"}}]})
+    LOG.info("created policy %s (%s)", name, p["id"])
+    return p["id"]
+
+
+def service_body(host, resource, target_ip):
+    # proxy_protocol: PROXY v2 header carries the WAN client address to
+    # Traefik (proxyProtocol.trustedIPs on websecure); TCP/TLS modes only.
+    return {"name": host, "domain": host, "mode": "tls", "listen_port": LISTEN_PORT, "enabled": True,
+            "targets": [{"target_type": "subnet", "target_id": resource, "protocol": "tcp",
+                         "host": target_ip, "port": 443, "enabled": True,
+                         "options": {"proxy_protocol": True}}]}
+
+
+def is_managed(svc, resource):
+    return svc.get("mode") == "tls" and any(t.get("target_id") == resource for t in svc.get("targets") or [])
+
+
+def target_matches(svc, target_ip):
+    if svc.get("listen_port") != LISTEN_PORT or not svc.get("enabled"):
+        return False
+    ts = svc.get("targets") or []
+    return len(ts) == 1 and ts[0].get("host") == target_ip and ts[0].get("port") == 443 \
+        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled") \
+        and (ts[0].get("options") or {}).get("proxy_protocol") is True
+
+
+def reconcile():
+    hosts = desired_hosts()
+    target_ip = traefik_cluster_ip()
+    nb = NetBird(netbird_pat())
+
+    router_group = group_id(nb, ROUTER_GROUP)
+    source_group = group_id(nb, SOURCE_GROUP)
+    net = ensure_network(nb)
+    resource = ensure_resource(nb, net)
+    ensure_router(nb, net, router_group)
+    ensure_policy(nb, resource, source_group)
+
+    by_domain = {s["domain"]: s for s in nb.list("reverse-proxies/services")}
+    created = updated = deleted = held = 0
+
+    for host, ref in sorted(hosts.items()):
+        svc = by_domain.get(host)
+        if svc is None:
+            nb.post("reverse-proxies/services", service_body(host, resource, target_ip))
+            LOG.info("created %s (from %s) -> %s:443", host, ref, target_ip)
+            created += 1
+        elif not is_managed(svc, resource):
+            LOG.warning("%s is held by unmanaged service %s (mode %s); remove it to migrate",
+                        host, svc["id"], svc.get("mode"))
+            held += 1
+        elif not target_matches(svc, target_ip):
+            nb.put(f"reverse-proxies/services/{svc['id']}", service_body(host, resource, target_ip))
+            LOG.info("updated %s -> %s:443", host, target_ip)
+            updated += 1
+
+    for domain, svc in by_domain.items():
+        if is_managed(svc, resource) and domain not in hosts:
+            nb.delete(f"reverse-proxies/services/{svc['id']}")
+            LOG.info("deleted %s (no public Ingress carries it any more)", domain)
+            deleted += 1
+
+    LOG.info("reconciled: %d hosts desired, %d created, %d updated, %d deleted, %d held by unmanaged services",
+             len(hosts), created, updated, deleted, held)
+
+
+def main():
+    logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"), stream=sys.stdout,
+                        format="%(asctime)s %(levelname)s %(message)s")
+    if os.environ.get("ONCE") == "1":
+        reconcile()
+        return
+    while True:
+        try:
+            reconcile()
+        except Exception as e:  # keep running; the next pass retries
+            LOG.error("reconcile failed: %s", e)
+        time.sleep(INTERVAL)
+
+
+if __name__ == "__main__":
+    main()

ci/images/vmtest/Containerfile

@@ -0,0 +1,84 @@
+FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
+
+# Everything `bootstrap.sh vmtest` needs, so the rebuild test can run as a Job in
+# the cluster instead of on a workstation. The workstation has 31G and the test
+# wants ~14.5G for four VMs — a single run left under 1G headroom, and two
+# overlapping runs OOM-killed the session. The nodes have ~94G with 70G free.
+#
+# Separate from ci-k8s rather than built on it: that image pins talosctl v1.12.6,
+# and this test creates a cluster at the version talos/patches/cluster.yaml
+# installs. A talosctl older than the Talos it provisions is exactly the kind of
+# skew the test exists to catch, so it must not be baked in here.
+#
+# KEEP TALOS_VERSION IN STEP with the installer image tag in
+# kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
+# file at run time and passes it to the provisioner; this pin only decides which
+# client talks to it.
+ARG TALOS_VERSION=v1.13.9
+ARG KUBECTL_VERSION=v1.36.2
+ARG FLUX_VERSION=2.9.2
+ARG KOPIA_VERSION=0.23.1
+
+# qemu-system-x86_64 and qemu-img are the provisioner's actual backend, and ovmf
+# supplies the UEFI firmware those VMs boot from — without it the provisioner
+# stops with "the required flash image was not found in any of the expected
+# paths", having already downloaded the ISO and generated the PKI. Note the
+# firmware comes from ovmf specifically: qemu-system-x86_64 ships an edk2 CODE
+# image but no VARS, which is the half the provisioner needs.
+#
+# iptables is shelled out to by the CNI plugins that build the VM network; the
+# provisioner fails with "error accessing iptables" without it. The rest is
+# what the phase shells out to: helm for the Cilium install, yq to read the
+# HelmRelease and the installer image, jq for OpenBao's seal status.
+#
+# Versions come from the digest-pinned base rather than being pinned one by one,
+# the same way ci-ansible does it.
+RUN apk add --no-cache \
+      bash \
+      ca-certificates \
+      curl \
+      git \
+      helm \
+      iptables \
+      jq \
+      openssl \
+      ovmf \
+      qemu-img \
+      qemu-system-x86_64 \
+      yq
+
+RUN curl -sfL -o /usr/local/bin/talosctl \
+      "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
+ && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
+ && chmod +x /usr/local/bin/talosctl
+
+RUN curl -sfL -o /usr/local/bin/kubectl \
+      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
+ && curl -sfL -o /tmp/kubectl.sha256 \
+      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl.sha256" \
+ && echo "$(cat /tmp/kubectl.sha256)  /usr/local/bin/kubectl" | sha256sum -c - \
+ && rm -f /tmp/kubectl.sha256 \
+ && chmod +x /usr/local/bin/kubectl
+
+RUN curl -sfL -o /tmp/flux.tar.gz \
+      "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
+ && echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3  /tmp/flux.tar.gz" | sha256sum -c - \
+ && tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
+ && chmod +x /usr/local/bin/flux \
+ && rm -f /tmp/flux.tar.gz
+
+# kopia reads the Velero backup repository directly, which is what the OpenBao
+# restore does when there is no Velero yet to do it: the credentials Velero
+# needs live in OpenBao, so OpenBao cannot be restored by Velero. Its absence
+# was found by a rebuild rehearsal reaching the restore for the first time and
+# stopping at "kopia not found".
+#
+# Same checksum-verified download as the tools above; the tarball unpacks into
+# a versioned directory, so the binary is pulled out by name.
+RUN curl -sfL -o /tmp/kopia.tar.gz \
+      "https://github.com/kopia/kopia/releases/download/v${KOPIA_VERSION}/kopia-${KOPIA_VERSION}-linux-x64.tar.gz" \
+ && echo "416d0f84a3dbb321a8b2d8f0997b1a0a6e915babe79ee76fa6e4d2bd1e1c5178  /tmp/kopia.tar.gz" | sha256sum -c - \
+ && tar -xzf /tmp/kopia.tar.gz -C /tmp \
+ && mv "/tmp/kopia-${KOPIA_VERSION}-linux-x64/kopia" /usr/local/bin/kopia \
+ && chmod +x /usr/local/bin/kopia \
+ && rm -rf /tmp/kopia.tar.gz "/tmp/kopia-${KOPIA_VERSION}-linux-x64"

kubernetes/MIGRATION-sops-openbao.md

@@ -50,15 +50,18 @@ pass insert homelab/openbao/root-token
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```
 
-**Manual unseal is required after every pod restart** (node drain, upgrade,
-crash), **per replica** — the server runs three raft peers (see
-`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
-voter nor a standby, so it stays out of the cluster until unsealed:
-`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
-all three. Three peers mean one sealed replica is survivable; two are not.
-Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
-explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
-Secrets keep their last value (they are real k8s Secrets — pods keep running).
+**Unseal is automatic since 2026-09-08** (static seal, PR #825, issue #805):
+the seal key lives in the `openbao-seal-key` Secret (created by `bootstrap.sh`
+from `pass show homelab/openbao/seal-key`) and every restarted replica unseals
+itself. The shamir key above became the **recovery key** in the migration; it
+is kept in `pass` under its old name, `homelab/openbao/unseal-key`. The server
+runs three raft peers (see `infrastructure/openbao/helmrelease.yaml`); a sealed
+one is neither a voter nor a standby, so a replica that stays sealed after a
+restart — wrong or missing seal-key Secret — costs redundancy, two cost
+quorum. The trade-off taken with the static seal: the key that unlocks the
+store sits in the same etcd as the Secrets ESO already syncs there in
+plaintext. Symptom of sealed: ESO stores go NotReady, synced Secrets keep
+their last value (they are real k8s Secrets — pods keep running).
 
 ## Phase 2 — configure + ClusterSecretStore (follow-up PR)
 
@@ -234,16 +237,19 @@ one full reconcile + one OpenBao pod restart/unseal cycle:
 - **Backups:** ✅ nightly raft-snapshot CronJob (03:00, keeps 7 on a
   dedicated PVC; see infrastructure/openbao/snapshot-cronjob.yaml) + velero
   `openbao-daily` Schedule (06:15) ships that PVC offsite, 30 d retention.
-  A raft snapshot restore needs the matching unseal key — which lives in
+  A raft snapshot restore needs the matching seal key — which lives in
   `pass`, outside the cluster (GPG key backed up externally 2026-07-14).
   A failed snapshot Job is the sealed-after-restart alarm. Still owed:
   one tested restore before phase 4.
-- **Cluster rebuild order:** Flux brings up openbao (sealed, empty) → restore
-  raft snapshot → unseal (key from `pass`) → ClusterSecretStore Ready → ESO
-  materializes Secrets → apps start. Until unseal, secret-consuming apps
-  simply stay Pending on missing Secrets — same as any missing dependency.
-- **Break-glass:** `pass` holds unseal key + root token (GPG, offline-capable).
-  With those two, everything is recoverable from a raft snapshot alone.
+- **Cluster rebuild order:** seal-key Secret from `pass` → Flux brings up
+  openbao (empty) → init → restore raft snapshot → replicas unseal themselves
+  → ClusterSecretStore Ready → ESO materializes Secrets → apps start
+  (`bootstrap.sh flux` does all of it; RUNBOOK-cold-rebuild.md Step 4 by
+  hand). Until then, secret-consuming apps simply stay Pending on missing
+  Secrets — same as any missing dependency.
+- **Break-glass:** `pass` holds seal key, recovery key + root token (GPG,
+  offline-capable). With those, everything is recoverable from a raft
+  snapshot alone.
 - **Ansible Vault password:** `ansible/vault-password.sh` reads
   `ANSIBLE_VAULT_PASSWORD` from the environment with no offline backup as of
   2026-07-14. It gates decryption of several Ansible-Vault files (PBS

kubernetes/NETWORK-traffic-classes.md

@@ -0,0 +1,117 @@
+# Traffic classes on the k8s nodes
+
+Issue #576. Three of the four classes are separated; management is not.
+
+## Where each class runs
+
+| class | link | mechanism |
+|---|---|---|
+| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
+| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
+| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
+| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
+| management | `mgmt0`, VLAN 40 | not started |
+
+Hard constraint, set by the owner: inter-node and pod traffic never traverses
+the UDM. Everything above stays on the 10G switch.
+
+## Why policy routing rather than moving the node IP
+
+Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
+and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
+`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
+Both were established by test (#837, #838), not by reading.
+
+So the InternalIP decides where pod traffic goes — unless the lookup happens in
+a different table. The rules do that, and nothing about the node's identity
+moves. The routing decision runs before Cilium's masquerade hook, so a rule
+matching the pod source sees the pod address, and the masquerade then rewrites
+it to the egress link's.
+
+## Rule order
+
+Lower priority runs first.
+
+| prio | match | table |
+|---|---|---|
+| 980 | src pod, dst pod | 48 |
+| 981 | src pod, dst pod (v4) | 48 |
+| 990 | dst `fd00::/8` | main |
+| 991 | dst `10.0.0.0/8` | main |
+| 1000 | src pod | 47 |
+| 1001 | src pod (v4) | 47 |
+
+980/981 match source **and** destination. Source alone would also catch the
+node's own traffic to a pod, which would then leave with a `pod0` source — an
+address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
+`world` rather than `host`.
+
+990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
+cluster-internal traffic out of table 47. Without them a pod's packet to a
+service matches the source rule, finds only a default route, and leaves for the
+UDM — measured, in-cluster DNS stopped resolving.
+
+Traffic to a pod on the same node falls through on its own: table 48 holds only
+the peers' CIDRs, so the lookup misses and 990/991 send it to main.
+
+## Where each table comes from
+
+- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
+  The IPv6 gateway is the UDM's configured address on the segment, not the
+  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
+  default does not land in main.
+- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
+  per peer pod CIDR with the peer's `pod0` address as next hop.
+
+## The bird mesh
+
+`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
+and installs what it learns into table 48. Cilium's routes in main are
+untouched, which is why the two coexist.
+
+No address is written into this repo twice. Each node publishes its `pod0`
+addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
+pod CIDRs come from the Node objects. An init container renders both from the
+API with `kubectl -o go-template-file`, excludes the node itself with a field
+selector, and ends with `bird -p` so BIRD's own parser gates startup.
+
+The node's own CIDR is **originated**, not read back from the kernel. Reading
+it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
+rejected outright: its next hop sits inside the prefix it describes, which KRT
+refuses with `strange next-hop`. The IPv6 route is a device route and passed,
+so the failure was silent on one family only.
+
+## Failure behaviour
+
+Benign at every layer. An empty or absent table 48 makes 980/981 fall through
+to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
+holds if the DaemonSet cannot start, which is the case during a cold rebuild
+until Gitea serves its image.
+
+## Management, the remaining class
+
+Not started. The levers differ per service and none of them is the node IP:
+
+| service | can leave the node IP? | lever |
+|---|---|---|
+| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
+| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
+| Talos API | n/a | `apid` answers on every node address; a firewall question |
+| kubelet | no | it *is* the node IP |
+
+Two approaches were rejected by the owner and are not to be proposed again:
+moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
+cluster VLAN while moving etcd and the API VIP to a new management VLAN.
+Replacing the CNI (#839, Antrea) was shelved.
+
+Note that the reasoning which produced that list also concluded pod↔pod was
+blocked, which policy routing disproved. Treat it as a record of decisions, not
+as an exhaustive analysis.
+
+## Sources
+
+Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
+`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
+`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
+`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
+#855; closed unmerged: #830, #838, #839.

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -1,194 +0,0 @@
-# Proposal — separate the four traffic classes onto their own VLANs
-
-Status: draft. The question this started with is **answered** — from the Cilium
-source and flag reference, not from a test — and the answer changes the shape
-of the proposal rather than confirming it.
-
-Supersedes the framing in issue #576, which described this as "give node
-management its own NIC, and Ceph public its own VLAN". That was two of four
-classes and one of them turned out already solved.
-
-## The four classes
-
-| class | what it is | where it goes today |
-|---|---|---|
-| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
-| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
-| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
-| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
-
-Three of four share one interface. That is not a design decision — until the
-bare-metal conversion there was one NIC to work with.
-
-Ceph client I/O living on the pod network is correct rather than a leftover:
-that traffic *is* cluster-internal. Ceph is done.
-
-## Principle
-
-**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
-separate, changeable decision.**
-
-Defining classes by VLAN means the separation survives recabling, port failures
-and capacity changes, and it does not require one port per class. Defining them
-by interface would freeze a capacity decision into the architecture.
-
-## The finding that shapes everything else
-
-The original open question was whether `nodePort.directRoutingDevice` moves the
-next-hop of Cilium's direct routes, or only selects an interface. It is
-answerable from the source, and the answer is: **only the interface.**
-
-`installDirectRoute` takes the next-hop as a parameter:
-
-```go
-func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
-    skipUnreachable bool) (routeSpec *netlink.Route, err error)
-```
-
-and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
-in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
-same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
-
-`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
-an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
-`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
-cluster there is no path by which Cilium's node IP differs from the kubelet's
-`InternalIP`.
-
-`--direct-routing-device` is documented as "Device name used to connect nodes in
-direct routing mode" — a device, not an address.
-
-**So cluster-internal traffic is welded to the node IP, and the node IP is the
-kubelet InternalIP.** Two consequences:
-
-- The class cannot be moved off whatever VLAN the node IP lives on.
-- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
-  from the node IP; native mode uses it as the direct-route next-hop. Native
-  routing buys nothing here, so the disruptive, non-rolling step this proposal
-  previously contained is simply gone.
-
-## Consequence: draw the line the other way
-
-If cluster-internal cannot leave the node IP, then separation has to come from
-moving *management services* off it — and each one has to be checked
-individually, because they do not share a mechanism.
-
-| management service | can it leave the node IP? | lever |
-|---|---|---|
-| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
-| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
-| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
-| kubelet | **no** | it *is* the node IP, by definition |
-
-So four VLANs are reachable, but the boundary does not fall where the class
-names suggest: **kubelet stays with cluster-internal.** That is the honest
-result, and it is worth stating plainly rather than filing kubelet under
-management and hoping.
-
-The remaining classes keep the levers established earlier:
-
-- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
-  For eBGP the session source is the next-hop, so pinning the session pins
-  where return traffic arrives. The session to the UDM has been running for
-  months.
-- **cluster-external egress** — eBPF masquerading follows the output device, so
-  this is a default-route decision plus `enable-masquerade-to-route-source`
-  (today `false`).
-- **Ceph replication** — Multus macvlan on VLAN 49, done.
-
-## The general escape hatch
-
-Ceph is separated because its OSDs got a second interface and stopped using the
-pod network for replication. That is the *only* mechanism that fully detaches a
-workload from the node IP, and it generalises: anything that must not ride the
-node IP gets its own `NetworkAttachmentDefinition`.
-
-It is also expensive per workload — an attachment, an IPAM pool, a pod spec
-change. Worth it for Ceph replication. Not worth it for east-west traffic in
-general.
-
-## Recommended mapping
-
-Two ways to arrange this, and they differ only in which VLAN gets relabelled:
-
-**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
-and Talos API. Cleanest match to the class names, but it changes every node's
-identity, which is the one address the whole cluster is administered through.
-
-**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
-VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
-Identical end state, and no node ever changes its InternalIP.
-
-Under B:
-
-| VLAN | carries |
-|---|---|
-| 49 | Ceph replication |
-| 40 (relabelled *cluster*) | pod east-west, kubelet |
-| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
-| new *external* | LoadBalancer ingress, egress default route |
-
-The property the user asked for — management carries only management traffic —
-holds under B. The label moves; the property is what matters.
-
-## Migration shape
-
-Every step is rolling and reversible. The routing-mode change is gone.
-
-1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
-2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
-   Still unused; this only proves the L2 works.
-3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
-   waiting for a healthy quorum between each.
-4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
-   `k8s.renner.uno` follows.
-5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
-   Reversible by reverting the override; failure is visible immediately in
-   `cilium-dbg shell -- bgp/peers`.
-6. Verify VLAN 40 by what is *absent* from it, not by what still works.
-
-Step 3 is the one that deserves care — etcd is the only step where a mistake
-costs quorum rather than connectivity.
-
-## What this does not buy
-
-Management currently shares a 10 Gb link and is not contended. Nothing observed
-today is slow because of this. The case for the split is that traffic classes
-with different failure modes and security postures should not be
-indistinguishable — not that something is broken.
-
-## Sources and measurements
-
-Source, Cilium v1.20.1 (the running version):
-
-- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
-  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
-  a next-hop that is not directly reachable.
-- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
-- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
-  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
-  Kubernetes Node.
-- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
-  to connect nodes in direct routing mode".
-
-Talos:
-
-- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
-  defaults to it. kube-apiserver and Talos API listen addresses are not
-  covered there.
-- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
-  `10.2.40.60` with `link: mgmt0`.
-
-Measured on the cluster:
-
-- `apid` answered on a node address that is not the InternalIP, so the Talos
-  API is not bound to one address.
-- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
-  InternalIP had changed since the last session, which is why nothing here
-  names an address as fixed.
-- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
-  unset and no direct-routing device configured.
-
-Not verified: whether the kube-apiserver's `--advertise-address` follows the
-node IP independently of the VIP. It does not affect the recommendation, since
-clients reach the API through the VIP either way.

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -432,8 +432,8 @@ is why it belongs here.
   among its devices but not `ens19`, because `ens19` is deliberately
   address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
   untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
-  on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
-  VLAN carrying pod traffic will need the same treatment.
+  on the shared parent `uplink`, still tagged. Any future address-less VLAN
+  carrying pod traffic will need the same treatment.
 
   The symptom does not look like a VLAN problem. A tc-layer drop reaches an
   AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space

kubernetes/RUNBOOK-cold-rebuild.md

@@ -67,10 +67,12 @@ These must exist outside the cluster or the rebuild cannot complete:
       `flux` phase creates the `sops-age` Secret, and any not-yet-migrated sops
       files decrypt with it. (Post-OpenBao-phase-4 this may be retired — see
       `MIGRATION-sops-openbao.md`.)
-- [ ] **OpenBao unseal key + root token** — in the GPG `pass` store
-      (`homelab/openbao/unseal-key`, `homelab/openbao/root-token`), and the GPG
-      private key backed up off-machine. Without the unseal key, no OpenBao
-      snapshot can be restored and no secret-consuming app can start.
+- [ ] **OpenBao seal key, recovery key + root token** — in the GPG `pass`
+      store (`homelab/openbao/seal-key`, `homelab/openbao/unseal-key` — the
+      former shamir key, now the recovery key — and `homelab/openbao/root-token`),
+      and the GPG private key backed up off-machine. The seal key is the one
+      the snapshots are encrypted under: without it, no OpenBao snapshot can be
+      restored and no secret-consuming app can start.
 - [ ] **OpenBao raft snapshot** reachable — nightly PVC snapshots
       (`infrastructure/openbao/snapshot-cronjob.yaml`) plus the Velero
       `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
@@ -294,42 +296,65 @@ output** — a green step over a still-forming Ceph is a known trap.
 > Steady-state (non-rebuild) bootstrap is just `kubernetes/bootstrap.sh flux`
 > with `GITEA_TOKEN` set; `FLUX_SOURCE` defaults to `gitea`.
 
-## Step 4 — openbao: restore the snapshot + unseal
+## Step 4 — openbao: restore the snapshot
 
-After Step 3, OpenBao comes up **sealed and empty**. Until it is restored and
-unsealed, the `ClusterSecretStore` stays NotReady, ESO cannot materialize
-Secrets, and every secret-consuming app stays Pending. This step is **manual**
-(the unseal key lives in `pass`, off-cluster by design).
+`bootstrap.sh flux` does this step itself (`restore_openbao`): it creates the
+`openbao-seal-key` Secret from `pass` before Flux brings OpenBao up, initialises
+the empty server, restores the newest raft snapshot straight out of the Velero
+kopia repository, and waits until every replica has unsealed itself from the
+static seal. What follows is the same procedure by hand, for when the script
+is not an option.
+
+OpenBao unseals itself from the static seal (`openbao-seal-key` Secret, key
+`key`, the 32-byte base64 key from `pass show homelab/openbao/seal-key`). The
+snapshots are encrypted under that key, so it must be the same key the backed
+up cluster used — a different one restores nothing. Until the snapshot is
+restored, the `ClusterSecretStore` stays NotReady, ESO cannot materialize
+Secrets, and every secret-consuming app stays Pending.
 
 ```sh
-# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
-kubectl -n openbao get pods
+# 1. The seal key, before the pods start (a pod without it never starts).
+kubectl create namespace openbao --dry-run=client -o yaml | kubectl apply -f -
+kubectl -n openbao create secret generic openbao-seal-key \
+  --from-literal=key="$(pass show homelab/openbao/seal-key)"
 
-# 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
+# 2. Wait for openbao-0 (uninitialised: 0/1 Ready is expected), then
+#    initialise it. With an auto-seal, init hands out recovery keys and the
+#    server unseals itself. These keys are throwaway: the restore replaces them.
+kubectl -n openbao exec -it openbao-0 -- bao operator init \
+  -recovery-shares=1 -recovery-threshold=1
+
+# 3. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
-#    from the Velero `openbao-daily` offsite copy. Copy it into the pod, then:
+#    from the Velero `openbao-daily` offsite copy. Root token from the init
+#    output. No -force: OpenBao checks the snapshot was written under this
+#    server's seal key and refuses otherwise ("different autoseal key") — that
+#    refusal is the signal the Secret is wrong. -force would skip the check and
+#    leave a server that can never unseal.
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
-kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
+kubectl -n openbao exec -it openbao-0 -- \
+  env BAO_TOKEN=<init root token> bao operator raft snapshot restore /tmp/restore.snap
 
-# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
-#    A sealed server is neither a raft voter nor a standby, so unsealing only
-#    openbao-0 leaves the three-peer cluster one node short of quorum.
-kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
-kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
-kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
+# 4. The server seals against the snapshot's keyring and unseals itself from
+#    the static seal; openbao-1 and openbao-2 follow (OrderedReady). Nothing
+#    to type. Afterwards the tokens and recovery key from pass apply again.
+kubectl -n openbao get pods -w
 ```
 
 Verify the store recovers before moving on:
 
 ```sh
-kubectl get clustersecretstore openbao        # want: Valid / Ready
-kubectl -n <ns> get externalsecret            # want: SecretSynced
+kubectl -n openbao exec openbao-0 -- bao status   # want: Seal Type static, Sealed false
+kubectl get clustersecretstore openbao            # want: Valid / Ready
+kubectl -n <ns> get externalsecret                # want: SecretSynced
 ```
 
-> **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds, and for *each* replica that restarts — a rolling node
-> drain seals all three in turn. Details + rationale:
-> `MIGRATION-sops-openbao.md`.
+> A replica that stays **sealed** after a restart could not use the static
+> seal — the Secret is missing or does not match the key the data was
+> written under. Fix the Secret and restart the pod; `bao operator unseal`
+> with the recovery key (`pass show homelab/openbao/unseal-key`) does not
+> unseal an auto-sealed server and is only for recovery-mode operations.
+> Details + rationale: `MIGRATION-sops-openbao.md`.
 
 ## Step 5 — apps: let the rest reconcile
 
@@ -393,14 +418,15 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
 - **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
   Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
   Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
-  It has also never had an `openbao` phase — Step 4 is manual because the
-  unseal key must not live in CI; see issue #187 on whether to wire a
-  semi-automated OpenBao restore.
-- **OpenBao seals on every restart and each replica must be unsealed
-  individually.** Three raft replicas mean the service survives losing one, but
-  a sealed server is neither a voter nor a standby — unsealing only one leaves
-  the cluster short of quorum. Step 4 covers this; it is repeated here because
-  it is the step most likely to be half-done under pressure.
+  Its `flux` phase carries the OpenBao restore (Step 4); the seal key and the
+  other secrets are read from `pass` or streamed in at run time and never live
+  in CI.
+- **OpenBao unseals itself, but only under the seal key the snapshot was
+  written with.** Three raft replicas mean the service survives losing one; a
+  sealed server is neither a voter nor a standby. A pod that stays sealed after
+  a restart has the wrong `openbao-seal-key` Secret — Step 4 covers this; it is
+  repeated here because it is the step most likely to be half-done under
+  pressure.
 - Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
   IPAM, not host networking. Nothing in the rebuild needs special handling for
   it — the nodes hold no address on that segment — but a rebuilt cluster will

kubernetes/apps/audiobookshelf/deployment.yaml

@@ -21,6 +21,9 @@ metadata:
   namespace: audiobookshelf
   labels:
     app.kubernetes.io/name: audiobookshelf
+  annotations:
+    # envFrom does not restart the pod when the ConfigMap changes; Reloader does.
+    configmap.reloader.stakater.com/reload: audiobookshelf-config
 spec:
   replicas: 1
   strategy:
@@ -67,7 +70,7 @@ spec:
               mountPath: /ca
       containers:
         - name: audiobookshelf
-          image: ghcr.io/advplyr/audiobookshelf:2.36.0 # {"$imagepolicy": "flux-system:audiobookshelf"}
+          image: ghcr.io/advplyr/audiobookshelf:2.36.1 # {"$imagepolicy": "flux-system:audiobookshelf"}
           ports:
             - name: http
               containerPort: 80

kubernetes/apps/authentik/kustomization.yaml

@@ -16,7 +16,6 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
-  - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/authentik/ldap-outpost-lb.yaml

@@ -40,7 +40,7 @@ metadata:
     # so this Service produces no public record — LDAP stays
     # LAN-internal, which matches the threat model of the cleartext
     # bind on 389.
-    external-dns.alpha.kubernetes.io/hostname: ldap.renner.uno
+    external-dns.kubernetes.io/hostname: ldap.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/external-dns/kustomization.yaml

@@ -3,3 +3,4 @@ kind: Kustomization
 resources:
   - externalsecret.yaml
   - helmrelease.yaml
+  - networkpolicy.yaml

kubernetes/apps/external-dns/networkpolicy.yaml

@@ -0,0 +1,22 @@
+# The apps namespace is default-deny-ingress, so the ServiceMonitor added in
+# monitoring-rules/external-dns-servicemonitor.yaml scraped into a timeout.
+# The selector matches both releases — the Cloudflare one deployed here and
+# external-dns-unifi from infrastructure/ — since they share the chart's
+# app.kubernetes.io/name and land in the same namespace.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: external-dns-allow-metrics-scrape
+  namespace: apps
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: external-dns
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            k8s:io.kubernetes.pod.namespace: monitoring
+      toPorts:
+        - ports:
+            - port: "7979"
+              protocol: TCP

kubernetes/apps/home-assistant/bgp-service.yaml

@@ -18,7 +18,7 @@ metadata:
     lb-pool: bgp
   annotations:
     lbipam.cilium.io/ips: "10.2.99.87,fd99::87"
-    external-dns.alpha.kubernetes.io/hostname: ha.renner.uno
+    external-dns.kubernetes.io/hostname: ha.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/home-assistant/deployment.yaml

@@ -95,7 +95,7 @@ spec:
               memory: 32Mi
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.9.3 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/home-assistant/mosquitto-iot-service.yaml

@@ -19,7 +19,7 @@ metadata:
     # LAN can connect with `mqtt.renner.uno` (matches the broker cert
     # SAN) instead of the bare IP. No Cloudflare label — kept off the
     # public DNS; UniFi serves a split-DNS answer for LAN clients.
-    external-dns.alpha.kubernetes.io/hostname: mqtt.renner.uno
+    external-dns.kubernetes.io/hostname: mqtt.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -57,7 +57,14 @@ spec:
           # and leaves this one untouched (tags are not deleted), so this keeps
           # resolving — but the image then came from the branch build rather than
           # master. Re-point it at a master-built tag when convenient.
-          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
+          #
+          # konairius/, not gitea_admin/: this init container runs during a cold
+          # rebuild, when Gitea does not exist because it runs inside the
+          # cluster being rebuilt. The node then falls back to a ghcr.io copy
+          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
+          # keeps the repository path when it falls through — so the path has to
+          # exist on both registries. Steady state still pulls from Gitea.
+          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
           command:
             - sh
             - -c

kubernetes/apps/immich/machine-learning.yaml

@@ -32,7 +32,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: immich-machine-learning
-          image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
+          image: ghcr.io/immich-app/immich-machine-learning:v3.2.2 # {"$imagepolicy": "flux-system:immich-machine-learning"}
           ports:
             - name: http
               containerPort: 3003

kubernetes/apps/immich/server.yaml

@@ -35,6 +35,9 @@ metadata:
   namespace: immich
   labels:
     app.kubernetes.io/name: immich-server
+  annotations:
+    # envFrom does not restart the pod when the ConfigMap changes; Reloader does.
+    configmap.reloader.stakater.com/reload: immich-config
 spec:
   # Scaled to 0 alongside immich-machine-learning during single-node
   # maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -84,7 +87,7 @@ spec:
               mountPath: /ca
       containers:
         - name: immich-server
-          image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
+          image: ghcr.io/immich-app/immich-server:v3.2.2 # {"$imagepolicy": "flux-system:immich-server"}
           ports:
             - name: http
               containerPort: 3001

kubernetes/apps/libation/cronjob.yaml

@@ -42,9 +42,11 @@ spec:
               type: RuntimeDefault
           initContainers:
             # Seed the libation-config PVC from the bootstrap secret on
-            # first run only — sentinel file gates re-runs so token
-            # refreshes in AccountsSettings.json and DB updates from
-            # subsequent liberate passes are preserved.
+            # first run only — sentinel file gates re-runs so DB updates
+            # from subsequent liberate passes are preserved. Refreshed
+            # Audible tokens are preserved by the container args, not by
+            # this sentinel: the entrypoint copies AccountsSettings.json
+            # into ephemeral /config-internal rather than symlinking it.
             - name: seed-config
               image: busybox:1.37.0
               command: ["/bin/sh", "-eu", "-c"]
@@ -66,6 +68,11 @@ spec:
                   install -m 660 /bootstrap/AccountsSettings.json /libation-state/AccountsSettings.json
                   install -m 660 /bootstrap/Settings.json          /libation-state/Settings.json
                   install -m 660 /bootstrap/LibationContext.db     /libation-state/LibationContext.db
+                  # Without this key the seeded tokens cannot be decrypted:
+                  # TokenStorageMethod is Encrypted and the container has no
+                  # OS secret store, so a missing key file makes the
+                  # entrypoint mint a throwaway one on every run.
+                  install -m 660 /bootstrap/libation-master.key   /libation-state/libation-master.key
                   touch /libation-state/.bootstrapped
                   ls -la /libation-state
               volumeMounts:
@@ -85,7 +92,23 @@ spec:
                   type: RuntimeDefault
           containers:
             - name: libation
-              image: docker.io/rmcrackan/libation:latest
+              # Pinned by digest: Spegel caches mutable tag resolutions, so
+              # :latest stayed on 13.7.11 despite imagePullPolicy Always.
+              image: docker.io/rmcrackan/libation:14.2.1@sha256:6f9d8ba19dc3803dd6ba295ab5a7abfb6ff3bc06286eebf4ecd8ccbe43245888
+              # The entrypoint stages config into ephemeral /config-internal
+              # by copy (only the DB is symlinked to the PVC), so Audible
+              # tokens refreshed during a run are lost at container exit.
+              # Passing args makes the entrypoint set up the environment and
+              # then exec this instead of its own loop; the trap copies the
+              # refreshed file back to the PVC on every exit path.
+              args:
+                - /bin/sh
+                - -c
+                - |
+                  trap 'cp /config-internal/AccountsSettings.json /libation-state/AccountsSettings.json' EXIT
+                  set -e
+                  /libation/LibationCli scan
+                  /libation/LibationCli liberate
               env:
                 - name: LIBATION_CONFIG_DIR
                   value: /libation-state

kubernetes/apps/llama-server/kustomization.yaml

@@ -6,3 +6,5 @@ resources:
   - embed.yaml
   - gate-service.yaml
   - ingress.yaml
+  - wiki-embed.yaml
+  - wiki-embed-ingress.yaml

kubernetes/apps/llama-server/wiki-embed-ingress.yaml

@@ -0,0 +1,57 @@
+---
+# wiki-embed.renner.uno — LAN-only (no dns.renner.uno/public label), consumed by
+# bin/wiki_rank.py on the workstation and the nightly wiki jobs. Machine client,
+# so Traefik BasicAuth instead of the browser-only Authentik gate; same pattern
+# as kubernetes/apps/monitoring/loki-ingress.yaml.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: wiki-embed-basic-auth
+  namespace: ai
+spec:
+  basicAuth:
+    secret: wiki-embed-basic-auth
+    realm: wiki-embed
+---
+# OpenBao key apps/ai/wiki-embed-basic-auth, field `users` = htpasswd line.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-embed-basic-auth
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-embed-basic-auth
+  dataFrom:
+    - extract:
+        key: apps/ai/wiki-embed-basic-auth
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wiki-embed
+  namespace: ai
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+    traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-embed-basic-auth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - wiki-embed.renner.uno
+      secretName: wiki-embed-tls
+  rules:
+    - host: wiki-embed.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: wiki-embed
+                port:
+                  number: 8080

kubernetes/apps/llama-server/wiki-embed.yaml

@@ -0,0 +1,117 @@
+---
+# Embedding endpoint for the ~/Notes wiki ranker (bin/wiki_rank.py). Own
+# Deployment so wiki traffic never queues behind Paperless on llama-embed.
+# Same model and PVC as llama-embed, so no second download.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wiki-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: wiki-embed
+spec:
+  replicas: 1
+  # Above the 600s startupProbe budget; see embed.yaml.
+  progressDeadlineSeconds: 900
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: wiki-embed
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: wiki-embed
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: wiki-embed
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            - --hf-repo
+            - gpustack/bge-m3-GGUF:Q8_0
+            - --embeddings
+            - --alias
+            - bge-m3
+            # bge-m3 is trained for CLS pooling; the GGUF default is not.
+            - --pooling
+            - cls
+            - --n-gpu-layers
+            - "99"
+            - --threads
+            - "2"
+            # wiki_rank.py caps each note at 1500 chars and sends 2 per request.
+            - --batch-size
+            - "1536"
+            - --ubatch-size
+            - "1536"
+            - --ctx-size
+            - "4096"
+            - --no-webui
+          env:
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          # Same sizing as llama-embed: 1.6 GiB measured peak, headroom for slots.
+          resources:
+            requests:
+              cpu: 250m
+              memory: 2Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "2"
+              memory: 8Gi
+              gpu.intel.com/i915: "1"
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 60
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wiki-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: wiki-embed
+spec:
+  selector:
+    app.kubernetes.io/name: wiki-embed
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -0,0 +1,46 @@
+# Cilium's agent ServiceMonitor, declared here rather than by the chart.
+#
+# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
+# taken field for field off the live object so scraping is unchanged: same name,
+# same namespace, same selector, same 10s interval and node relabeling.
+#
+# It lives here because the chart's template is gated on monitoring.coreos.com/v1
+# existing at install time, and on an empty cluster it does not — those CRDs
+# arrive with kube-prometheus-stack, which reconciles after infrastructure.
+# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
+# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
+#
+# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
+# why; this is the same move for the component where it mattered most.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: cilium-agent
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: cilium-agent
+    app.kubernetes.io/part-of: cilium
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cilium-agent
+  namespaceSelector:
+    matchNames:
+      - kube-system
+  # The chart's Service keeps rendering (prometheus.enabled), so the target and
+  # its `metrics` port are there regardless of who declares the ServiceMonitor.
+  targetLabels:
+    - k8s-app
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      relabelings:
+        # Agent metrics are per-node; without this every series lands without a
+        # node label and BPF map pressure cannot be attributed to a machine.
+        - action: replace
+          sourceLabels:
+            - __meta_kubernetes_pod_node_name
+          targetLabel: node
+          replacement: ${1}

kubernetes/apps/monitoring-rules/external-dns-servicemonitor.yaml

@@ -0,0 +1,25 @@
+# Both external-dns releases (Cloudflare and the UniFi webhook) carry
+# app.kubernetes.io/name: external-dns and differ only by instance, so one
+# monitor covers both. jobLabel keeps them apart as separate job values.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: external-dns
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: external-dns
+    release: kube-prometheus-stack
+spec:
+  jobLabel: app.kubernetes.io/instance
+  namespaceSelector:
+    matchNames:
+      - apps
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: external-dns
+  endpoints:
+    # The controller's own metrics. Port 8080 on the unifi release is the
+    # webhook sidecar and exports no external_dns_* series.
+    - port: http
+      path: /metrics
+      interval: 60s

kubernetes/apps/monitoring-rules/intel-gpu-exporter-servicemonitor.yaml

@@ -0,0 +1,20 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: intel-gpu-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: intel-gpu-exporter
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: intel-gpu-exporter
+  endpoints:
+    - port: http-metrics
+      path: /metrics
+      interval: 30s
+      relabelings:
+        # The series would otherwise carry only the exporter pod's IP.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -21,11 +21,31 @@ resources:
   # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
   # endpoint, which proves reachability, not that anything was written.
   - prometheusrule-cnpg-backup.yaml
+  # DNS publication. Nothing scraped external-dns, so it deleted six LAN
+  # records on 2026-09-12 and stayed silent for five days.
+  - prometheusrule-external-dns.yaml
+  - external-dns-servicemonitor.yaml
+  # Outcome check for the same failure: the names themselves, asked of the
+  # UDM. Catches a cause external-dns cannot see, e.g. records removed there.
+  - prometheusrule-dns-resolution.yaml
+  - probes-dns.yaml
   - alloy-servicemonitor.yaml
+  - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
+  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
+  # kube-prometheus-stack, which reconciles after infrastructure, so these
+  # cannot live beside the Velero HelmRelease.
+  - velero-podmonitor.yaml
+  - velero-servicemonitor.yaml
+  - authentik-outpost-servicemonitors.yaml
+  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
+  # when it cannot create these, which held the CephCluster in Progressing and
+  # deadlocked the whole apps layer. See the file for the cycle.
+  - rook-servicemonitors.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml
   - smartctl-exporter-servicemonitor.yaml
+  - intel-gpu-exporter-servicemonitor.yaml
   - probes-blackbox.yaml

kubernetes/apps/monitoring-rules/probes-dns.yaml

@@ -0,0 +1,143 @@
+# DNS probes for the names external-dns publishes to the UDM from
+# LoadBalancer Services. One Probe per name because the dns prober's
+# query_name is module-scoped; the target is the resolver, queried on both
+# families so a single-stack blip does not read as a missing record.
+#
+# The UDM is queried directly rather than through the cluster resolver:
+# k8s-gateway caches these with `serve_stale 1h immediate`, which would keep
+# a deleted record green for an hour.
+#
+# `release` is load-bearing — probeSelector matches on it.
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-ha
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_ha
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      # instance is the resolver and identical across these Probes, so the
+      # queried name has to come from a label.
+      labels:
+        dns_name: ha.renner.uno
+---
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-mqtt
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_mqtt
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      labels:
+        dns_name: mqtt.renner.uno
+---
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-ldap
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_ldap
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      labels:
+        dns_name: ldap.renner.uno
+---
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-syslog
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_syslog
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      labels:
+        dns_name: syslog.renner.uno
+---
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-ftp-scanner
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_ftp_scanner
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      labels:
+        dns_name: ftp.paperless.renner.uno
+---
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: dns-lan-syncthing
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-dns
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: dns_lan_syncthing
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        - "[fd00::1]:53"
+        - "10.2.40.1:53"
+      labels:
+        dns_name: syncthing.paperless.renner.uno

kubernetes/apps/monitoring-rules/prometheusrule-ci-openbao.yaml

@@ -34,7 +34,9 @@ spec:
           # query parameter, and an exact match would break silently if the
           # instance label ever encodes it differently.
           expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
-          for: 5m
+          # Two consecutive failed probes: enough to skip a single lost probe
+          # through Traefik, not enough to sit on a sealed store.
+          for: 1m
           labels:
             severity: critical
           annotations:

kubernetes/apps/monitoring-rules/prometheusrule-dns-resolution.yaml

@@ -0,0 +1,49 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: dns-resolution
+  namespace: monitoring
+  labels:
+    # ruleSelector matches on this label; without it the rule is created but
+    # never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: dns-resolution.rules
+      rules:
+        # min by dns_name: both resolver families are probed, and the record
+        # is only gone when neither answers. A single-family failure is the
+        # UDM's dnsmasq refusing connections in bursts (k8s-gateway carries
+        # the same note), which is why the window is 30m rather than the 10m
+        # used for HTTP probes.
+        - alert: LanDnsRecordMissing
+          expr: min by (dns_name) (probe_success{job="blackbox-dns"}) == 0
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.dns_name }} no longer resolves to its LAN address"
+            description: >-
+              The UDM stopped answering for a name external-dns publishes from
+              a LoadBalancer Service, or answered with the public
+              *.renner.uno wildcard instead of the record. Clients that use
+              the name rather than the address lose the service while
+              everything in the cluster stays healthy — nothing else in the
+              stack reports this. Check the external-dns alerts first, then
+              the record on the UDM.
+
+        # Dead-man's-switch, mirroring BlackboxProbesAbsent: without probe
+        # results the rule above cannot fire.
+        - alert: LanDnsProbesAbsent
+          expr: absent(probe_success{job="blackbox-dns"})
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "No LAN DNS probe results are reaching Prometheus"
+            description: >-
+              The blackbox-dns Probes stopped being discovered or the exporter
+              rejects the modules. A 400 from /probe yields no probe_* series
+              at all, which looks the same as no Probe — check that every
+              dns_lan_* module the Probes reference exists in the
+              blackbox-exporter config.

kubernetes/apps/monitoring-rules/prometheusrule-external-dns.yaml

@@ -0,0 +1,87 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: external-dns
+  namespace: monitoring
+  labels:
+    # ruleSelector matches on this label; without it the rule is created but
+    # never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: external-dns.rules
+      rules:
+        # The signal that was missing on 2026-09-12: the sources stopped
+        # producing endpoints they had produced before, and policy=sync
+        # deleted the records. 10% below the 24h peak clears a single
+        # retired Ingress (2 of ~48) and catches a whole class of names.
+        - alert: ExternalDNSSourceEndpointsDropped
+          expr: >-
+            external_dns_source_endpoints_total
+              < 0.9 * max_over_time(external_dns_source_endpoints_total[24h])
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.job }} is publishing {{ $value }} endpoints, below its 24h peak"
+            description: >-
+              external-dns stopped seeing endpoints it had before. With
+              policy=sync the matching DNS records are deleted, so names stop
+              resolving without anything else failing. Compare
+              external_dns_source_endpoints_total against the last 24h, then
+              check whether the resources still carry a hostname annotation
+              the running version understands — a changed annotation prefix
+              default empties the source silently. Deliberate removals also
+              fire this and resolve themselves 24h later.
+
+        # Source errors mean the Kubernetes side failed; registry errors mean
+        # the provider did. Either way desired state is computed from partial
+        # input, which policy=sync acts on.
+        - alert: ExternalDNSSourceErrors
+          expr: increase(external_dns_source_errors_total[15m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.job }} is failing to read its sources"
+            description: >-
+              external-dns could not list the Kubernetes resources it derives
+              records from. Check the pod log and its RBAC.
+
+        - alert: ExternalDNSRegistryErrors
+          expr: increase(external_dns_registry_errors_total[15m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.job }} is failing to reach its DNS provider"
+            description: >-
+              external-dns could not read or write records at the provider.
+              For external-dns-unifi that is the UniFi webhook sidecar and the
+              controller behind it; for external-dns it is the Cloudflare API.
+
+        - alert: ExternalDNSDown
+          expr: up{job=~"external-dns.*"} == 0
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.job }} is not being scraped"
+            description: >-
+              The external-dns instance is down or unreachable. Records already
+              at the provider keep resolving, but nothing reconciles them any
+              more and every other alert in this group is blind.
+
+        # Dead-man's-switch for the group, mirroring BlackboxProbesAbsent: a
+        # dropped ServiceMonitor makes every rule above evaluate to nothing.
+        - alert: ExternalDNSMetricsMissing
+          expr: absent(external_dns_source_endpoints_total)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "No external-dns metrics are reaching Prometheus"
+            description: >-
+              Either both external-dns instances are gone or the ServiceMonitor
+              stopped being discovered. Verify it still carries
+              release=kube-prometheus-stack.

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -28,24 +28,28 @@ spec:
         # scheduling would also fire this — but both want a human too, so the
         # imprecision costs nothing.
         - alert: OpenBaoReplicaNotReady
-          # A replica that is sealed never recovers on its own, so the
-          # duration is only there to ignore ordinary rescheduling: a healthy
-          # pod moving between nodes is Ready again in well under a minute.
-          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          # A replica unseals itself from the static seal, so a restart is
+          # Ready again in well under a minute; the duration only ignores that
+          # ordinary rescheduling. A replica still sealed after it never
+          # recovers on its own: the seal failed (openbao-seal-key Secret
+          # missing or wrong).
+          # max by: kube-state-metrics stamps instance/pod on the series, so a
+          # rescheduled exporter starts a new series and resets the timer.
+          expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
           for: 10m
           labels:
             severity: warning
           annotations:
             summary: "OpenBao has {{ $value }} of 3 replicas ready"
             description: >-
-              An OpenBao replica has been unavailable for 10 minutes. The usual
-              cause is a sealed replica after a node drain — every restart
-              seals it, and it stays out of the raft cluster until unsealed by
-              hand. The service is still up on the remaining replicas, so
-              nothing is broken yet, but redundancy is reduced and a second
-              drain would cost quorum. Check `kubectl -n openbao get pods` and
-              `bao status` in the not-ready pod; unseal with the key from
-              `pass` (homelab/openbao/unseal-key).
+              An OpenBao replica has been unavailable for 10 minutes. A restart
+              unseals itself from the static seal, so this is not the ordinary
+              drain case. The service is still up on the remaining replicas,
+              so nothing is broken yet, but redundancy is reduced and a second
+              failure would cost quorum. Check `kubectl -n openbao get pods`,
+              the pod log and `bao status` in the not-ready pod. If it is
+              sealed, the static seal failed: verify the openbao-seal-key
+              Secret matches `pass show homelab/openbao/seal-key`.
 
         - alert: OpenBaoQuorumLost
           # Raft quorum of a three-node cluster is two. At one ready replica
@@ -53,11 +57,10 @@ spec:
           # the ClusterSecretStore goes InvalidProviderConfig and no
           # ExternalSecret can refresh.
           #
-          # Shorter window than the warning above because this is an outage,
-          # not a degradation — but not zero, so a rolling restart that
-          # briefly dips does not page.
-          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
-          for: 5m
+          # No window: below quorum the secret store is down and every
+          # ExternalSecret stalls with it, so waiting only delays the page.
+          expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
+          for: 0m
           labels:
             severity: critical
           annotations:
@@ -67,9 +70,11 @@ spec:
               quorum and the secret store is unavailable. ExternalSecrets stop
               refreshing; existing Secrets keep their last values, so failures
               surface only when something rotates or a new secret is needed.
-              Unseal the sealed replicas (key in `pass`,
-              homelab/openbao/unseal-key) — each one individually, since a
-              sealed server cannot be a standby.
+              Replicas unseal themselves from the static seal; one that stays
+              sealed could not. Check its log and the openbao-seal-key Secret
+              against `pass show homelab/openbao/seal-key`, fix the Secret and
+              restart the pod. The recovery key cannot unseal an auto-sealed
+              server. Procedure: kubernetes/RUNBOOK-cold-rebuild.md, Step 4.
 
         - alert: OpenBaoMetricsMissing
           # Guards the two alerts above. Both are threshold comparisons, so if

kubernetes/apps/monitoring-rules/rook-servicemonitors.yaml

@@ -0,0 +1,93 @@
+# Rook's two ServiceMonitors, declared here rather than by the operator.
+#
+# Reproduced field for field off the live objects in the production cluster, so
+# scraping is unchanged: same names, same namespace, same selectors, same 10s
+# interval and `cluster` relabeling. The only thing dropped is the CephCluster
+# ownerReference, which Rook set and Flux cannot.
+#
+# They live here because CephCluster.spec.monitoring.enabled is now false, and
+# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
+# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
+# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
+# reconcile.go, v1.20.7):
+#
+#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
+#         // exporter reconciled; its errors are logged and reconcile continues
+#         if cephCluster.Spec.Monitoring.Enabled {
+#             if err := EnableCephExporterServiceMonitor(...); err != nil {
+#                 return errors.Wrap(err, "failed to enable service monitor")
+#             }
+#         }
+#     }
+#
+# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
+# fails with "the server could not find the requested resource (post
+# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
+# aborts, and the CephCluster never leaves status.phase Progressing. That is
+# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
+# closes on itself:
+#
+#     infrastructure-ceph-cluster → CephCluster phase Ready
+#                                 → Rook reconcile settles
+#                                 → monitoring.coreos.com CRDs exist
+#                                 → kube-prometheus-stack (monitoring)
+#                                 → grafana-db
+#                                 → infrastructure-ceph-cluster
+#
+# Nothing outside that loop can break it, so a cold rebuild stopped here with
+# every one of the twenty app Kustomizations behind it.
+#
+# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
+# get, and for the same reason: the metrics stay where they are, only the
+# scrape wiring moves to the layer where its CRDs exist.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: rook-ceph-exporter
+  namespace: rook-ceph
+  labels:
+    team: rook
+spec:
+  selector:
+    matchLabels:
+      app: rook-ceph-exporter
+      rook_cluster: rook-ceph
+  namespaceSelector:
+    matchNames:
+      - rook-ceph
+  endpoints:
+    - port: ceph-exporter-http-metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      # Rook stamps the cluster name so series from several Ceph clusters stay
+      # separable. One cluster here, but dropping it would rename every series.
+      relabelings:
+        - action: replace
+          replacement: rook-ceph
+          targetLabel: cluster
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: rook-ceph-mgr
+  namespace: rook-ceph
+  labels:
+    team: rook
+spec:
+  selector:
+    matchLabels:
+      app: rook-ceph-mgr
+      rook_cluster: rook-ceph
+  namespaceSelector:
+    matchNames:
+      - rook-ceph
+  endpoints:
+    - port: http-metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      relabelings:
+        - action: replace
+          replacement: rook-ceph
+          targetLabel: cluster

kubernetes/apps/monitoring/dashboards/intel-gpu.json

@@ -0,0 +1,368 @@
+{
+  "title": "Intel iGPU",
+  "uid": "intel-gpu",
+  "tags": ["gpu", "intel", "i915"],
+  "schemaVersion": 39,
+  "editable": true,
+  "timezone": "",
+  "refresh": "1m",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "Node",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": {
+          "query": "label_values(gpumon_power, node)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "definition": "label_values(gpumon_power, node)",
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "sort": 1,
+        "current": { "selected": true, "text": ["All"], "value": ["$__all"] }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Render/3D busy now",
+      "gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "decimals": 0,
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "GPU power now",
+      "gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "decimals": 1,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "GT frequency now",
+      "gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "suffix: MHz",
+          "decimals": 0,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Render/3D busy",
+      "description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Video busy",
+      "description": "Fixed-function decode/encode (QuickSync).",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "VideoEnhance busy",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Blitter busy",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "GT frequency (actual)",
+      "description": "0 while the GT sits in RC6.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "suffix: MHz",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU power",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Package power",
+      "description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Memory controller bandwidth",
+      "description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "MiBs",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_imc_read{node=~\"$node\"}",
+          "legendFormat": "{{node}} read",
+          "refId": "A"
+        },
+        {
+          "expr": "gpumon_imc_write{node=~\"$node\"}",
+          "legendFormat": "{{node}} write",
+          "refId": "B"
+        }
+      ]
+    }
+  ]
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -77,6 +77,9 @@ configMapGenerator:
   - name: dashboard-infrastructure-overview
     files:
       - infrastructure-overview.json
+  - name: dashboard-intel-gpu
+    files:
+      - intel-gpu.json
   - name: dashboard-netbird-overview
     files:
       - netbird-overview.json

kubernetes/apps/monitoring/helmrelease-alloy-syslog.yaml

@@ -232,7 +232,7 @@ metadata:
     lb-pool: bgp
   annotations:
     io.cilium/lb-ipam-ips: "10.2.99.84,fd99::84"
-    external-dns.alpha.kubernetes.io/hostname: syslog.renner.uno
+    external-dns.kubernetes.io/hostname: syslog.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml

@@ -72,3 +72,73 @@ spec:
           tcp:
             preferred_ip_protocol: ip6
             ip_protocol_fallback: true
+        # One module per name: the dns prober takes the RESOLVER as its
+        # target and reads query_name from the module, so a name cannot be
+        # passed per scrape. The Probes in monitoring-rules/probes-dns.yaml
+        # point these at the UDM.
+        #
+        # Both answer checks are load-bearing. valid_rcodes alone passes a
+        # deleted record two labels deep (NOERROR, no answer), and the public
+        # *.renner.uno wildcard answers one label deep with a CNAME to
+        # proxy.renner.uno — NOERROR with the WAN proxy behind it. Requiring
+        # an A and rejecting a CNAME separates a real record from both.
+        dns_lan_ha:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: ha.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']
+        dns_lan_mqtt:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: mqtt.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']
+        dns_lan_ldap:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: ldap.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']
+        dns_lan_syslog:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: syslog.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']
+        dns_lan_ftp_scanner:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: ftp.paperless.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']
+        dns_lan_syncthing:
+          prober: dns
+          timeout: 5s
+          dns:
+            query_name: syncthing.paperless.renner.uno
+            query_type: A
+            valid_rcodes: [NOERROR]
+            validate_answer_rrs:
+              fail_if_not_matches_regexp: ['\sIN\s+A\s']
+              fail_if_matches_regexp: ['\sIN\s+CNAME\s']

kubernetes/apps/monitoring/intel-gpu-exporter.yaml

@@ -0,0 +1,81 @@
+# Intel iGPU utilisation, frequency and power for every node (#752).
+#
+# Not privileged: /dev/dri comes from the device plugin's monitoring resource
+# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
+# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
+# hostNetwork.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-gpu-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: intel-gpu-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: intel-gpu-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: intel-gpu-exporter
+    spec:
+      # Every node is a control plane; without this the exporter covers nothing.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      securityContext:
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: intel-gpu-exporter
+          image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
+          args:
+            # No -device: intel_gpu_top picks the only GPU the plugin injects.
+            - -interval=5s
+          ports:
+            - name: http-metrics
+              containerPort: 9100
+              protocol: TCP
+          securityContext:
+            # Root so the PERFMON capability is effective without file
+            # capabilities on the binary.
+            runAsUser: 0
+            runAsNonRoot: false
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["PERFMON"]
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+              gpu.intel.com/monitoring: 1
+            limits:
+              memory: 128Mi
+              gpu.intel.com/monitoring: 1
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: intel-gpu-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: intel-gpu-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: intel-gpu-exporter
+  ports:
+    - name: http-metrics
+      port: 9100
+      targetPort: http-metrics
+      protocol: TCP

kubernetes/apps/monitoring/kustomization.yaml

@@ -16,6 +16,7 @@ resources:
   - prometheus-gate-ingress.yaml
   - ci-diagnostics-rbac.yaml
   - smartctl-exporter.yaml
+  - intel-gpu-exporter.yaml
   - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the

kubernetes/apps/music-assistant/deployment.yaml

@@ -28,7 +28,7 @@ spec:
         # net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
         # worker, though it is no longer the same kind of device everywhere:
         # on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
-        # of enp2s0f0np0 rather than a NIC, which is why the datapath needs
+        # of uplink rather than a NIC, which is why the datapath needs
         # bpf.vlanBypass to carry it (issue #592).
         k8s.v1.cni.cncf.io/networks: music-assistant-iot
     spec:
@@ -55,7 +55,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -65,7 +65,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/apps/paperless/ftp-scanner.yaml

@@ -43,7 +43,7 @@ spec:
               mountPath: /consume
       containers:
         - name: vsftpd
-          image: gitea.renner.uno/k.renner/vsftpd:build.1784731469 # {"$imagepolicy": "flux-system:vsftpd"}
+          image: gitea.renner.uno/k.renner/vsftpd:build.1789630783 # {"$imagepolicy": "flux-system:vsftpd"}
           ports:
             - name: ftp-ctrl
               containerPort: 21
@@ -70,24 +70,19 @@ spec:
             - name: consume
               mountPath: /home/scanner
               subPath: scanner
-          # FTP has no health verb, so the control port accepting a connection
-          # is what can be asserted. It is enough for the failure that matters:
-          # if vsftpd dies the scanner cannot deposit anything and nothing else
-          # in the chain notices, because paperless only ever sees an empty
-          # consume directory — which is indistinguishable from nobody having
-          # scanned.
-          #
-          # Only the control port is probed. The passive range is negotiated
-          # per transfer and is not listening at rest.
+          # Asserts the control socket is listening without opening an FTP
+          # connection: vsftpd 3.0.5 segfaults its listener on a fraction of
+          # the connections it accepts. The passive range is negotiated per
+          # transfer and is not listening at rest, so it is not probed.
           livenessProbe:
-            tcpSocket:
-              port: ftp-ctrl
+            exec:
+              command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
             initialDelaySeconds: 15
             periodSeconds: 30
             timeoutSeconds: 5
           readinessProbe:
-            tcpSocket:
-              port: ftp-ctrl
+            exec:
+              command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
             initialDelaySeconds: 5
             periodSeconds: 10
             timeoutSeconds: 5
@@ -124,7 +119,7 @@ metadata:
     # label — FTP must not be exposed via the cargo tunnel (clear-text
     # protocol, public exposure not appropriate). The brother scanner
     # uses this hostname instead of the LB IP.
-    external-dns.alpha.kubernetes.io/hostname: ftp.paperless.renner.uno
+    external-dns.kubernetes.io/hostname: ftp.paperless.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/paperless/gotenberg.yaml

@@ -22,7 +22,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: gotenberg
-          image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
+          image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
           args:
             - "gotenberg"
             - "--chromium-disable-javascript=true"

kubernetes/apps/paperless/kustomization.yaml

@@ -25,18 +25,15 @@ resources:
 configMapGenerator:
   # vsftpd config for the ftp-scanner deployment. The content-hash name
   # suffix is intentionally LEFT ENABLED (unlike home-assistant's nginx
-  # config): vsftpd templates pasv_address from PASV_ADDRESS at container
-  # start, so a changed value must roll the pod. With the hash suffix,
-  # editing any value here renames the generated ConfigMap, which mutates
-  # the Deployment's pod template and triggers an automatic rolling
-  # restart. Without it, an IP change (e.g. an LB re-allocation) silently
-  # leaves the running pod advertising the stale PASV address, which
-  # breaks classic-PASV clients (older Brother scanners) while EPSV
-  # clients keep working.
+  # config): vsftpd reads these into vsftpd.conf at container start, so a
+  # changed value must roll the pod, which the renamed ConfigMap does.
+  #
+  # No PASV_ADDRESS: the LB address is declared once, on the Service
+  # (io.cilium/lb-ipam-ips). The image default 0.0.0.0 tells PASV clients
+  # to reuse the control-connection address.
   - name: ftp-scanner-config
     namespace: paperless
     literals:
-      - PASV_ADDRESS=10.2.99.82
       - PASV_MIN_PORT=21100
       - PASV_MAX_PORT=21104
       - FTP_USER=scanner

kubernetes/apps/paperless/paperless.yaml

@@ -143,7 +143,20 @@ spec:
           # on every paperless pod start: tools belong in the image, not
           # installed at runtime. Dated tag so a change is a reviewable diff
           # here, matching the ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
+          #
+          # konairius/, not gitea_admin/, and that is not cosmetic. This is an
+          # INIT container, so paperless does not start without it — and during
+          # a cold rebuild gitea.renner.uno does not exist, because Gitea runs
+          # inside the cluster being rebuilt. A rehearsal left this pod in
+          # Init:ErrImagePull on exactly this image.
+          #
+          # The node falls back to a ghcr.io copy in that window
+          # (machine.registries.mirrors, applied by bootstrap.sh). Containerd
+          # keeps the repository path when it falls through, so the path has to
+          # exist on both registries — hence konairius/, which matches the ghcr
+          # namespace. Steady state still pulls from Gitea and never leaves the
+          # network.
+          image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04
           command:
             - sh
             - -c
@@ -170,7 +183,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.3 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.2.0 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000

kubernetes/apps/paperless/syncthing.yaml

@@ -145,7 +145,7 @@ metadata:
     lb-pool: bgp
   annotations:
     io.cilium/lb-ipam-ips: "10.2.99.85,fd99::85"
-    external-dns.alpha.kubernetes.io/hostname: syncthing.paperless.renner.uno
+    external-dns.kubernetes.io/hostname: syncthing.paperless.renner.uno
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

kubernetes/apps/trmnl/renderer.yaml

@@ -177,7 +177,14 @@ spec:
         # webhook start failing with "expired JWT access token" about
         # thirty minutes after seeding.
         - name: render-schedule
-          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
+          # konairius/, not gitea_admin/: this container starts during a cold
+          # rebuild, when Gitea does not exist because it runs inside the
+          # cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff.
+          # The node falls back to a ghcr.io copy then
+          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
+          # keeps the repository path, so it has to exist on both registries.
+          # Steady state still pulls from Gitea.
+          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
           command:
             - sh
             - -c

kubernetes/apps/trmnl/terminus.yaml

@@ -134,7 +134,7 @@ spec:
         # is a device-bricking class of risk. Bumps are a reviewed, manual
         # act — see kubernetes/infrastructure/image-automation/README.md.
         - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
           ports:
             - name: http
               containerPort: 2345
@@ -197,7 +197,7 @@ spec:
             capabilities:
               drop: ["ALL"]
         - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
           command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
           envFrom:
             - configMapRef:

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/apps/woodpecker/agent.yaml

@@ -34,7 +34,7 @@ spec:
           # Image automation, range 3.x (see image-automation/woodpecker.yaml).
           # MUST stay version-matched with woodpecker-server (gRPC parity) — both
           # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
               value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,18 +1,3 @@
-# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
-# isolated from the apps namespace.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    # Image-build steps run a privileged docker/buildx plugin pod (see the
-    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
-    # without this the privileged step pods are rejected at admission. The
-    # namespace is dedicated to ephemeral CI step pods, so scoping the
-    # exemption here keeps it off apps/.
-    pod-security.kubernetes.io/enforce: privileged
----
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/apps/woodpecker/server.yaml

@@ -40,7 +40,7 @@ spec:
           # minor + patch auto-update within 3.x; only the 4.0 major is held for
           # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
           # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
           env:
             - name: WOODPECKER_HOST
               value: https://ci.renner.uno

kubernetes/apps/zone-configurator/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       # Service; nothing on this path needs the host's network namespace.
       containers:
         - name: zone-configurator
-          image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.4 # {"$imagepolicy": "flux-system:mmwave-configurator"}
+          image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.5 # {"$imagepolicy": "flux-system:mmwave-configurator"}
           env:
             - name: HA_BASE_URL
               value: "http://home-assistant:80"

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,6 +28,19 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
+    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
+    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
+    # kind is unknown when this path is first applied, and a server-side
+    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
+    #
+    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
+    #   no matches for kind "ObjectStore" in version ...
+    #
+    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
+    # woodpecker — already declares it. This was the only one that did not, and
+    # it took speech-to-phrase, zone-configurator and matter-server down with
+    # it, since all three wait on home-assistant.
+    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,8 +122,20 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  dependsOn:
-    - name: infrastructure
+  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
+  # existing cluster could ever hit and no new one could ever escape:
+  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
+  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
+  # defined in this very Kustomization. A cold rebuild stopped here for good,
+  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
+  #
+  # The edge was never needed. infrastructure runs wait: false, so it applies
+  # the rook-ceph namespace and the operator immediately and only its readiness
+  # waits on the healthChecks — everything this path needs exists long before
+  # infrastructure reports Ready. On an empty API server this Kustomization
+  # retries on retryInterval until the operator installs its CRDs, the same
+  # self-healing shape post-crds already relies on; on a live cluster the CRDs
+  # are always present and nothing changes.
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/infrastructure/bird/bird.conf

@@ -0,0 +1,103 @@
+# Identical on every node and free of addresses. defines.conf and peers.conf
+# are rendered per node by the init container; the first is included above the
+# templates that reference it, the second below them.
+include "/etc/bird/defines.conf";
+
+router id from "pod0";
+log stderr all;
+
+protocol device { scan time 10; }
+
+# Everything a pod may need to reach, in one table, resolved by longest prefix:
+# the node's attached networks, the peers' pod CIDRs, and a default for the
+# rest. No rule has to carve out exceptions, because the specific routes win on
+# their own.
+#
+# The table is BIRD's alone. If this daemon stops, it empties, the rule finds
+# nothing and falls through to main — pod traffic returns to VLAN 40,
+# unseparated rather than misrouted. That is why the default route lives here
+# and not in the Talos config: a default that outlived the specific routes
+# would send pod traffic to the UDM.
+
+# The node's attached networks, taken from the interfaces rather than written
+# down. The usual advice is not to export these to the kernel because it
+# maintains them itself — true of main, not of a table the kernel never fills.
+protocol direct attached {
+  interface "mgmt0", "ingress0", "pod0";
+  ipv4 { table master4; };
+  ipv6 { table master6; };
+}
+
+# Pod egress. GW4/GW6 are the UDM on the ingress segment, published as node
+# annotations beside the address that defines them.
+protocol static egress4 {
+  ipv4 { table master4; };
+  route 0.0.0.0/0 via GW4;
+}
+protocol static egress6 {
+  ipv6 { table master6; };
+  route ::/0 via GW6;
+}
+
+# This node's own pod CIDR, via the interface Cilium serves it on. Originated
+# rather than read back out of the kernel: reading it back made BIRD's view
+# depend on the shape of Cilium's route, and the IPv4 one is rejected outright,
+# its next hop sitting inside the prefix it describes.
+#
+# It belongs in the table, not just in the BGP announcement. A reply from a pod
+# on another node is forwarded traffic with a pod source, so the rule matches it
+# too; without this route it would find only the default and leave for the UDM.
+# Measured — cross-node pod traffic, and with it cluster DNS, stopped.
+protocol static own4 {
+  ipv4 { table master4; };
+  route OWN_POD4 via "cilium_host";
+}
+protocol static own6 {
+  ipv6 { table master6; };
+  route OWN_POD6 via "cilium_host";
+}
+
+protocol kernel k4 {
+  kernel table 47;
+  metric 0;
+  ipv4 { table master4; import none; export all; };
+}
+protocol kernel k6 {
+  kernel table 47;
+  metric 0;
+  ipv6 { table master6; import none; export all; };
+}
+
+# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
+# from ingress0 and is unrelated.
+#
+# direct is explicit because iBGP defaults to multihop, which makes the gateway
+# recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
+# keeps the listener off mgmt0 and ingress0.
+#
+# Only the node's own CIDR is offered: the export filter drops everything the
+# static protocols and the direct protocol produce.
+template bgp pod4 {
+  local as 64514;
+  source address MY4;
+  direct;
+  strict bind;
+  ipv4 {
+    next hop self;
+    import all;
+    export filter { if net = OWN_POD4 then accept; reject; };
+  };
+}
+template bgp pod6 {
+  local as 64514;
+  source address MY6;
+  direct;
+  strict bind;
+  ipv6 {
+    next hop self;
+    import all;
+    export filter { if net = OWN_POD6 then accept; reject; };
+  };
+}
+
+include "/etc/bird/peers.conf";

kubernetes/infrastructure/bird/daemonset.yaml

@@ -0,0 +1,115 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: bird
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: bird
+rules:
+  # The peer list is read from the nodes' pod0 annotations, so no address is
+  # written into this repo twice.
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: bird
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: bird
+subjects:
+  - kind: ServiceAccount
+    name: bird
+    namespace: kube-system
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: bird
+  namespace: kube-system
+  labels:
+    tier: node
+    app: bird
+spec:
+  selector:
+    matchLabels:
+      app: bird
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: bird
+    spec:
+      # Carries pod east-west once the routing rules land. Without it table 48
+      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
+      priorityClassName: system-node-critical
+      # Writes host routes and peers on host addresses.
+      hostNetwork: true
+      serviceAccountName: bird
+      # Every node here is a control plane node, so this is not optional.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      initContainers:
+        - name: render-config
+          # By digest, not by the dated tag: the tag is rebuilt within the day
+          # it names, and the cluster's image path keeps resolving it to the
+          # first build of that day — imagePullPolicy: Always does not help,
+          # because the tag itself resolves stale. Update this when the image
+          # is rebuilt.
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -eu
+              cp /config/bird.conf /etc/bird/bird.conf
+              kubectl get node "$NODE_NAME" \
+                -o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
+              kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
+                -o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
+              # BIRD's own parser is the check: a node whose annotation is
+              # missing renders as <no value> and fails here rather than
+              # crashlooping the daemon or forming a mesh with a hole.
+              bird -p -c /etc/bird/bird.conf
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          volumeMounts:
+            - name: config
+              mountPath: /config
+            - name: etc-bird
+              mountPath: /etc/bird
+      containers:
+        - name: bird
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
+          resources:
+            requests:
+              cpu: "50m"
+              memory: "32Mi"
+            limits:
+              cpu: "200m"
+              memory: "128Mi"
+          securityContext:
+            capabilities:
+              drop: ["ALL"]
+              # NET_ADMIN writes the routes; NET_BIND_SERVICE binds 179.
+              add: ["NET_ADMIN", "NET_BIND_SERVICE"]
+          volumeMounts:
+            - name: etc-bird
+              mountPath: /etc/bird
+      volumes:
+        - name: config
+          configMap:
+            name: bird-config
+        - name: etc-bird
+          emptyDir: {}

kubernetes/infrastructure/bird/defines.tmpl

@@ -0,0 +1,9 @@
+define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
+define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
+define GW6 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v6" }};
+define GW4 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v4" }};
+{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
+     IPv6 prefix into an ipv4 protocol, which fails the parse in the init
+     container rather than passing silently. */}}
+define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
+define OWN_POD6 = {{ index .spec.podCIDRs 1 }};

kubernetes/infrastructure/bird/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - daemonset.yaml
+configMapGenerator:
+  # Hashed name, so editing any of these rolls the DaemonSet.
+  - name: bird-config
+    namespace: kube-system
+    files:
+      - bird.conf
+      - defines.tmpl
+      - peers.tmpl

kubernetes/infrastructure/bird/peers.tmpl

@@ -0,0 +1,7 @@
+{{- range $i, $n := .items -}}
+{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
+{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
+# {{ $n.metadata.name }}
+protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
+protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
+{{ end -}}

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -70,9 +70,23 @@ spec:
       # map only shows up as "no space left on device" in the log while pod
       # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
       # alertable beforehand.
+      #
+      # That reasoning is about the metrics, which this line provides. The
+      # ServiceMonitor is only the scrape wiring and is declared standalone in
+      # apps/monitoring-rules instead — the same treatment velero/helmrelease
+      # already gives its own, and for the same reason: the chart's template is
+      # gated on monitoring.coreos.com/v1 existing at install time.
+      #
+      # Here it is worse than for velero. Cilium's chart does not skip the
+      # template, it aborts rendering — "Service Monitor requires
+      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
+      # healthChecks the infrastructure Kustomization waits on. So on an empty
+      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
+      # blocked monitoring, which is where those CRDs come from. The lowest
+      # layer in the cluster was waiting on the highest.
       enabled: true
       serviceMonitor:
-        enabled: true
+        enabled: false
     bpf:
       # Per-endpoint policy map size: one entry per (identity, direction,
       # protocol, port) allowed. When it fills, endpoint regeneration fails
@@ -166,3 +180,17 @@ spec:
     # the supernets here cover the LAN as well, which is the point.
     ipv4NativeRoutingCIDR: 10.0.0.0/8
     ipv6NativeRoutingCIDR: fd00::/8
+    # Pod east-west leaves the node on pod0 (VLAN 48) rather than the
+    # management VLAN. Native routing hands forwarding to the kernel;
+    # autoDirectNodeRoutes installs the remote PodCIDR routes, which is
+    # allowed because all three nodes share pod0's L2 segment.
+    #
+    # directRoutingDevice is explicit on purpose: with several addressed
+    # devices Cilium falls back to the one carrying the Kubernetes InternalIP,
+    # which is mgmt0.
+    #
+    # Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
+    routingMode: native
+    autoDirectNodeRoutes: true
+    nodePort:
+      directRoutingDevice: pod0

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -99,7 +99,7 @@ spec:
     # required.
     #
     # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
     # so there is no live impact, but re-enabling it will require pointing the
     # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
     tls:

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -1,7 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - bouncer-key-externalsecret.yaml
-  - edge-bouncer-key-externalsecret.yaml
   - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -31,22 +31,12 @@
 # serves read-only decision streams to a valid bouncer key and nothing else.
 # The admin surface is cscli, in-cluster. If the source address ever becomes
 # deterministic, an ipAllowList can be added on top.
----
+#
 # The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
 # fallback and no second port; 8080 is the TLS port. Its certificate comes
 # from the chart's own self-signed CA, so Traefik needs that CA to verify the
-# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
-# what the Service resolves to.
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: crowdsec-lapi
-  namespace: crowdsec
-spec:
-  serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret
----
+# backend; the ServersTransport carrying it is a traefik.io CRD consumer and
+# lives in post-crds/crowdsec-lapi-serverstransport.yaml.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:

kubernetes/infrastructure/external-dns-unifi/kustomization.yaml

@@ -1,5 +1,4 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - externalsecret.yaml
   - helmrelease.yaml

kubernetes/infrastructure/image-automation/ceph.yaml

@@ -9,9 +9,26 @@ spec:
 ---
 # v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
 # (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
-# filter pins us to the bare three-component release. Range stays at
-# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
-# upgrade tied to the Rook operator version.
+# filter pins us to the bare three-component release.
+#
+# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
+# upgrade tied to the Rook operator version. That coupling had already come
+# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
+# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
+# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
+# the mons with a client a whole major ahead of them.
+#
+# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
+# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
+# Nothing broke only because the four CSI users predate the cipher change and
+# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
+# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
+# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
+# because a fresh cluster has no legacy keys at all.
+#
+# So the cutover happens now, with the version the client already is. Rook
+# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
+# upstream reports data corruption there when read affinity is enabled.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -25,4 +42,4 @@ spec:
     extract: '$ver'
   policy:
     semver:
-      range: "19.x"
+      range: ">=20.2.1 <21.0.0"

kubernetes/infrastructure/intel-gpu-plugin/daemonset.yaml

@@ -72,6 +72,9 @@ spec:
             # they contend for it. Sized for an inference server plus room for
             # a transcoder and a benchmark pod alongside it.
             - -shared-dev-num=4
+            # Advertises gpu.intel.com/monitoring: one resource that grants a
+            # pod every GPU on the node instead of a slot. The exporter needs it.
+            - -enable-monitoring
           securityContext:
             seLinuxOptions:
               type: "container_device_plugin_t"

kubernetes/infrastructure/kustomization.yaml

@@ -7,6 +7,7 @@ resources:
   - cluster-settings/
   - cilium/
   - multus/
+  - bird/
   - cert-manager/
   - snapshot-controller/
   - metrics-server/
@@ -18,6 +19,7 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
+  - netbird-proxy-services/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -30,6 +30,11 @@ spec:
       labels:
         app: cni-plugins-installer
     spec:
+      # Same reasoning as multus-daemonset.yaml. This one installs the macvlan
+      # binary the ceph-san attachment is made of, so a node that cannot
+      # schedule it has no plugin to delegate to — the failure the init
+      # container comment below already describes, reached by a different road.
+      priorityClassName: system-node-critical
       hostNetwork: true
       tolerations:
         - operator: Exists
@@ -44,7 +49,32 @@ spec:
           # at all and put github.com on the critical path of every node rebuild.
           # Dated tag so a version change is a reviewable diff here, matching the
           # ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
+          #
+          # konairius/, not gitea_admin/, and the difference is what makes a
+          # rebuild possible. Gitea runs inside this cluster, so during a rebuild
+          # it does not exist: a rehearsal sat here in ImagePullBackOff with "no
+          # route to host", and because the plugins never installed there was no
+          # macvlan, so Multus could not attach ceph-san, so Rook's network
+          # canary never started, so the CephCluster hung at "Configuring Ceph
+          # Mons" and OpenBao waited forever on a PVC that could not bind. The
+          # failure surfaced four layers from its cause.
+          #
+          # This used to point straight at ghcr.io, which fixed the rebuild and
+          # broke the normal case: every steady-state pull then left the network
+          # for an image the local registry already holds. The fallback belongs
+          # to the node, not to the manifest — machine.registries.mirrors names
+          # Gitea first and ghcr.io second (bootstrap.sh), so nothing here has to
+          # know about the recovery path.
+          #
+          # Containerd keeps the repository path when it falls through, so the
+          # path must resolve on both registries; the build pushes one digest to
+          # konairius/ on each, verified equal. Bumping the tag means building
+          # first — an unpublished tag is unpullable on both.
+          #
+          # No imagePullSecret: the ghcr namespace is private and its credential
+          # is in machine.registries too, because a Secret would have to come
+          # from OpenBao, which needs Ceph, which needs these binaries.
+          image: gitea.renner.uno/konairius/cni-plugins:2026-09-04
           command:
             - sh
             - -c

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -154,6 +154,40 @@ spec:
         app: multus
         name: multus
     spec:
+      # Upstream ships this DaemonSet with no priority class, so it runs at 0
+      # and can preempt nothing. Any event that makes the pod need a fresh
+      # scheduling decision — a rollout, a node restart, an eviction — then has
+      # to fit it onto a node that may since have filled up, and if it does not
+      # fit it simply stays Pending. The node has no multus after that, and
+      # nothing reports it: pods still start, they just never get their
+      # secondary attachment, and the annotation everything waits on never
+      # appears.
+      #
+      # Seen on a cold rebuild: two of four nodes sat at 99% memory requests
+      # when the DaemonSet rolled, both multus pods went Pending, and Rook's
+      # network canary hung in Init:0/1 waiting for a network-status annotation
+      # that could never come. The CephCluster stayed in Progressing, which is
+      # what infrastructure-ceph-cluster gates on, so the whole apps layer
+      # stopped behind a scheduling decision three layers away.
+      #
+      # node-critical, the same class cilium and cni-dhcp-daemon carry, because
+      # that is what this is: without multus a node cannot attach a secondary
+      # network, and only that node is affected. Nothing cluster-wide fails.
+      #
+      # This does outrank kube-apiserver, which Talos runs as a static pod at
+      # system-cluster-critical (2000000000, against 2000001000 here), and on a
+      # control plane that has run out of memory the kubelet will pick the API
+      # server as the pod to evict. That was observed twice, at 6GiB and at
+      # 8GiB.
+      #
+      # It is not a reason to demote this DaemonSet. Kubernetes puts the CNI
+      # above the control plane deliberately — a node without its CNI cannot run
+      # anything at all — and a class name is a statement about what the pod is,
+      # not a dial for picking a number. The eviction says the control plane is
+      # starved, and that is where it has to be fixed: enough allocatable
+      # memory, or kubeReserved/systemReserved so the static pods keep what they
+      # need. Demoting this to cluster-critical would only move the victim.
+      priorityClassName: system-node-critical
       hostNetwork: true
       hostPID: true
       tolerations:

kubernetes/infrastructure/multus/whereabouts-daemonset.yaml

@@ -146,6 +146,11 @@ spec:
         app: whereabouts
         name: whereabouts
     spec:
+      # Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
+      # secondary-network path, so a node without it cannot hand out an address
+      # on ceph-san and the attachment fails just as completely as if multus
+      # itself were missing. Upstream leaves the priority class unset.
+      priorityClassName: system-node-critical
       hostNetwork: true
       serviceAccountName: whereabouts
       # Every node here is a control plane node, so the NoSchedule toleration

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -190,3 +190,31 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
+---
+# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
+# from `apps`. Distinct from `woodpecker`, which holds the service itself.
+#
+# Here rather than in apps/woodpecker for the reason spelled out above that
+# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
+# ../network-policies/, and infrastructure-crds runs before every app KS. With
+# the namespace declared in the workload KS the order inverted, and a rebuild
+# stopped with
+#
+#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
+#   namespaces "woodpecker-ci" not found
+#
+# which held infrastructure-crds, and with it all twenty app Kustomizations
+# that depend on it. The same reasoning was already written down for
+# `woodpecker` and simply not carried across to its sibling.
+#
+# Image-build steps run a privileged docker/buildx plugin pod (see the server's
+# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
+# label the privileged step pods are rejected at admission. The namespace holds
+# nothing but ephemeral CI step pods, so the exemption stays off apps/.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    pod-security.kubernetes.io/enforce: privileged

kubernetes/infrastructure/netbird-ingress/kustomization.yaml

@@ -1,5 +1,4 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - externalsecret.yaml
   - daemonset.yaml

kubernetes/infrastructure/netbird-proxy-services/deployment.yaml

@@ -0,0 +1,108 @@
+# Reconciles NetBird reverse-proxy services from Ingresses that carry
+# dns.renner.uno/public: one TLS-passthrough service per host on :443,
+# targeting Traefik's ClusterIP through a subnet resource over the service
+# CIDR (ci/images/netbird-proxy-services/controller.py).
+#
+# openbao namespace, like the rotators: its default-deny already allows egress
+# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
+# NetBird PAT is read from OpenBao on every pass through the
+# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
+# once by hand like the rotator roles. Until that role exists the pod logs an
+# auth error each minute and does nothing.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+---
+# Read-only, cluster-wide: public Ingresses live in many namespaces, the
+# Traefik Service in ingress/.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: netbird-proxy-services-read
+rules:
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: netbird-proxy-services-read
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: netbird-proxy-services-read
+subjects:
+  - kind: ServiceAccount
+    name: netbird-proxy-services
+    namespace: openbao
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+spec:
+  replicas: 1
+  # One writer against the management API at a time.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: netbird-proxy-services
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: netbird-proxy-services
+    spec:
+      serviceAccountName: netbird-proxy-services
+      securityContext:
+        runAsNonRoot: true
+        # Numeric, or the kubelet refuses the image's named user with
+        # "cannot verify user is non-root". The script needs no home and no
+        # writable path, so nobody/nogroup is enough.
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: controller
+          # Pinned by digest: the date tag is rebuilt on every push of the
+          # image directory (branch and master alike) and the registry keeps
+          # only the newest manifest per tag, so the digest must come from
+          # the MASTER build of the code change.
+          image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: NB_MANAGEMENT_URL
+              value: https://netbird.renner.uno
+            - name: BAO_ADDR
+              value: http://openbao.openbao.svc.cluster.local:8200
+            - name: BAO_ROLE
+              value: netbird-proxy-services
+            # Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
+            - name: SERVICE_CIDR
+              value: "10.96.0.0/12"
+            - name: TRAEFIK_SERVICE
+              value: ingress/traefik
+            - name: INTERVAL_SECONDS
+              value: "60"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi

kubernetes/infrastructure/netbird-proxy-services/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -126,3 +126,25 @@ spec:
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
             - { type: 143, family: IPv6 }
+---
+# wiki-embed is the one ai pod Traefik may reach: it sits behind BasicAuth
+# (apps/llama-server/wiki-embed-ingress.yaml), not behind the Authentik gate.
+# Scoped to this pod so the default-deny above keeps Traefik off llama-server.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wiki-embed-from-traefik
+  namespace: ai
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: wiki-embed
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP

kubernetes/infrastructure/openbao/helmrelease.yaml

@@ -25,10 +25,10 @@ spec:
         name: openbao
         namespace: openbao
   install:
-    # A fresh OpenBao pod is 0/1 Ready until `bao operator init` + unseal
-    # (readiness probe reports sealed). helm --wait would time out and
-    # remediation would uninstall the release, so don't wait here; the
-    # init/unseal procedure is manual (see kubernetes/MIGRATION-sops-openbao.md).
+    # A fresh OpenBao pod is 0/1 Ready until `bao operator init` (readiness
+    # reports sealed). helm --wait would time out and remediation would
+    # uninstall the release, so don't wait here; init is manual (see
+    # kubernetes/MIGRATION-sops-openbao.md). Unseal is automatic (static seal).
     disableWait: true
     remediation:
       retries: 3
@@ -42,6 +42,11 @@ spec:
       # the agent-injector mutating webhook is unused surface.
       enabled: false
     server:
+      # The static seal key, from the openbao-seal-key Secret (bootstrap.sh).
+      extraSecretEnvironmentVars:
+        - envName: BAO_SEAL_KEY
+          secretName: openbao-seal-key
+          secretKey: key
       # Raft (not standalone/file) for two reasons: `bao operator raft
       # snapshot` as a consistent backup primitive, and real HA across three
       # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
@@ -53,16 +58,8 @@ spec:
         # the moment one pod is sealed or rescheduled, which is worse than
         # the single replica it replaced. Three tolerates one.
         #
-        # What this buys, and what it does not: a node drain seals the replica
-        # on it, and raft elects a new leader from the remaining standbys
-        # within seconds, so the service does not go down. It does NOT remove
-        # the manual unseal — a sealed server can neither serve requests nor
-        # act as standby, so that replica stays out until someone unseals it.
-        #
-        # The consequence for rolling node maintenance is worth stating: a
-        # Talos upgrade drains all three nodes in turn, sealing each replica
-        # as it goes. Unseal each one as it comes back, or the third drain
-        # takes the last unsealed replica with it and quorum is lost anyway.
+        # A restarted replica unseals itself from the static seal and rejoins
+        # raft, so a rolling reboot only ever costs one replica at a time.
         replicas: 3
         raft:
           enabled: true
@@ -111,6 +108,14 @@ spec:
 
             service_registration "kubernetes" {}
 
+            # Auto-unseal. The key is a 32-byte AES key in the openbao-seal-key
+            # Secret, created from pass at bootstrap before this release. The
+            # former shamir key is the recovery key (#805).
+            seal "static" {
+              current_key_id = "1"
+              current_key    = "env://BAO_SEAL_KEY"
+            }
+
             audit "file" "stdout" {
               description = "audit trail to pod stdout -> Loki"
               options {

kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl

@@ -0,0 +1,23 @@
+# OpenBao policy: netbird-proxy-services
+#
+# The reverse-proxy-services controller (Deployment netbird-proxy-services in
+# the openbao namespace) reads the NetBird PAT on every reconcile pass and
+# calls the management API with it. Read-only on that one path; it shares the
+# bootstrap credential with rotator-netbird but not that role's PATCH rights.
+#
+# NOT applied by Flux. Apply ONCE with an admin token:
+#
+#   bao policy write netbird-proxy-services kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl
+#   bao write auth/kubernetes/role/netbird-proxy-services \
+#     bound_service_account_names=netbird-proxy-services \
+#     bound_service_account_namespaces=openbao \
+#     policies=netbird-proxy-services ttl=15m
+#
+# The PAT itself is the one rotator-netbird already uses
+# (homelab/infrastructure/secret-rotators-bootstrap/netbird, key `pat`).
+#
+# Do NOT set token_no_default_policy=true (kv-v2 preflight needs the default policy).
+
+path "homelab/data/infrastructure/secret-rotators-bootstrap/netbird" {
+  capabilities = ["read"]
+}

kubernetes/infrastructure/openbao/rotation/flux-git.yaml

@@ -10,33 +10,8 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
-# own it) so a sync failure leaves the last-good value in place; Flux's
-# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
-# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
-# provisioned by hand (see policies/rotator-gitea-flux.hcl).
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    creationPolicy: Merge
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token
+# The ExternalSecret that publishes the rotated token to Flux lives in
+# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
 ---
 apiVersion: v1
 kind: ServiceAccount

kubernetes/infrastructure/openbao/snapshot-cronjob.yaml

@@ -11,8 +11,8 @@
 # secret is stored anywhere; each run logs in fresh.
 #
 # A SEALED OpenBao fails the login step and the Job goes Failed — that is
-# deliberate signal, not noise: it means unseal was forgotten after a pod
-# restart and backups have stopped.
+# deliberate signal, not noise: it means the static seal could not unseal a
+# restarted pod and backups have stopped.
 #
 # Retention: last 7 snapshots on the PVC; velero adds 30 d offsite.
 apiVersion: batch/v1

kubernetes/infrastructure/post-crds/cilium-bgp.yaml

@@ -25,13 +25,13 @@ spec:
       stop: "fd99::ff"
 ---
 # Cilium BGPv2 control-plane config peering each node with the UDM over the
-# management LAN — never over v30iot, which is the untrusted IoT segment.
+# ingress VLAN 47 — never over v30iot, which is the untrusted IoT segment.
 # The source address comes from the kernel routing decision, so no explicit
 # localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
-#   - v4 session: peer 10.2.40.1, family IPv4 unicast only
-#   - v6 session: peer fd40::1, family IPv6 unicast only
+#   - v4 session: peer 10.2.47.1, family IPv4 unicast only
+#   - v6 session: peer fd02:47::1, family IPv6 unicast only
 # This avoids the "v6 NLRIs over v4 transport" path, which requires
 # bilateral extended-nexthop capability negotiation (RFC 5549/8950).
 # Cilium 1.19 BGPv2 advertises the capability only for the IPv4 Unicast
@@ -74,14 +74,14 @@ spec:
       peers:
         - name: udm-v6
           peerASN: 64512
-          # ULA — stable across ISP renumber events. UDM's br40 has
-          # fd40::1/64; nodes get fd40::<eui-64> via SLAAC on eth0.
-          peerAddress: fd40::1
+          # ULA — stable across ISP renumber events. UDM's br47 has
+          # fd02:47::1/64; nodes get fd02:47::<eui-64> via SLAAC on ingress0.
+          peerAddress: fd02:47::1
           peerConfigRef:
             name: bgp-peer-udm-v6
         - name: udm-v4
           peerASN: 64512
-          peerAddress: 10.2.40.1
+          peerAddress: 10.2.47.1
           peerConfigRef:
             name: bgp-peer-udm-v4
 ---