feat(intel-gpu-plugin): expose the Intel iGPU as gpu.intel.com/i915

Adds the Intel GPU device plugin as a DaemonSet in kube-system, making the on-board Iris Xe of each MS-01 node schedulable as `gpu.intel.com/i915`. Pods can then request the iGPU without a /dev/dri hostPath or privilege escalation — the plugin passes the render node through the kubelet device-plugin API. The resource covers QuickSync transcoding as well as GPU compute. Vendored from upstream v0.36.0 rather than installed from the `intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin` custom resource requiring the Intel Device Plugin Operator, its CRDs and a cert-manager webhook. One device type on three identical nodes gives the operator nothing to manage that the DaemonSet does not. Runs in kube-system because the plugin mounts hostPaths; the cluster enforces PodSecurity `baseline`, which forbids those, and kube-system is the only namespace in the API server's exemption list. NFD is not deployed, so upstream's nfd_labeled_nodes overlay and its nodeSelector are omitted. `-shared-dev-num=4` lets several containers share the single iGPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chore(image-automation): bump valkey/valkey:9.1.2-alpine
2026-09-01 12:38:42 +02:00 · 2026-09-01 02:21:46 +00:00 · 2026-08-29 09:20:10 +00:00 · 2026-08-29 11:09:04 +02:00 · 2026-08-29 10:56:05 +02:00 · 2026-08-29 10:43:08 +02:00
35 changed files with 1205 additions and 4069 deletions
@@ -0,0 +1,111 @@
+# Cilium: one owner
+
+## The rule
+
+Cilium is owned by **Flux**, through `infrastructure/cilium/helmrelease.yaml`.
+Talos provides the conditions for a CNI and nothing else. Never both.
+
+Talos-only would also be a valid answer — it is the simpler one, and it keeps a
+cold rebuild self-forming. It is rejected here only because it costs Flux's
+reconciliation, image automation and value ergonomics for the one component
+most likely to need a careful, reviewed change.
+
+## Where we are
+
+Both own it, which is the state this document exists to end.
+
+- 22 Cilium objects are listed in the `talos-bootstrap-manifests-inventory`
+  ConfigMap: both DaemonSets, the operator Deployment, both ConfigMaps, the
+  ServiceAccounts, the ClusterRoles/Bindings and the `cilium-secrets`
+  namespace. That is the whole installation, not a fragment.
+- The same objects carry `meta.helm.sh/release-name: cilium`.
+- `cilium-config` has three field managers: `helm-controller`, `talos` and
+  `kubectl-patch`.
+
+The consequences are not theoretical. The seed sat at Cilium 1.19.2 for twelve
+days while the chart was on 1.20.1, and `upgrade-k8s --dry-run` shows that a
+Kubernetes upgrade in that state downgrades the CNI and switches off socket
+load balancing. Separately, `enable-bgp-control-plane` and the two
+native-routing CIDRs exist in no file at all — applied by hand, so a cold
+rebuild would have come up with BGP off while the peering CRDs in git made it
+look configured.
+
+Nothing fails when these drift. That is the whole problem.
+
+## Target state
+
+**Talos** keeps exactly this, in `talos/patches/cluster.yaml`, and gains nothing
+else:
+
+```yaml
+cluster:
+  network:
+    cni:
+      name: none
+  proxy:
+    disabled: true
+```
+
+No `inlineManifests` entry for cilium. `talos/patches/cilium-cni.yaml` is
+deleted.
+
+**Flux** owns everything else. `helmrelease.yaml` is the single source for
+every Cilium setting, including the three that were hand-patched.
+
+**A cold rebuild** installs Cilium once, by hand, between `talosctl bootstrap`
+and the Flux phase, from the same HelmRelease values. This is the cost of the
+decision and it is deliberate: the cluster no longer forms itself from the
+machine config alone. Given `bootstrap.sh talos` is already dead and Step 2 of
+RUNBOOK-cold-rebuild.md is already a manual procedure, the marginal cost is one
+documented command.
+
+## Transition
+
+The order matters, because Talos prunes what its inventory used to list and no
+longer does — removing the manifest naively deletes the CNI.
+
+1. **Track the untracked settings first.** `bgpControlPlane.enabled`,
+   `ipv4NativeRoutingCIDR`, `ipv6NativeRoutingCIDR` into the HelmRelease. Note
+   this is not a pure change of ownership: enabling bgpControlPlane through the
+   chart also emits five companion keys the hand-patch never set, so Cilium has
+   been running on its own defaults for router-id allocation. Verify the BGP
+   sessions after Flux applies it.
+
+2. **Remove cilium from `cluster.inlineManifests`.** This has no immediate
+   effect. Evidence: Talos was upgraded to 1.13.9 four days ago and Cilium
+   stayed at 1.20.1 — an OS upgrade does not reconcile bootstrap manifests.
+   Only `upgrade-k8s` does.
+
+3. **Prove the prune before running it.** `upgrade-k8s --dry-run
+   --manifests-no-prune` and confirm the 22 objects are not listed for
+   deletion. `NoPrune` is honoured by the dry run as well as the apply
+   (`pkg/cluster/kubernetes/talos_managed.go`), so this is a real test and not
+   a reading of the flag description.
+
+4. **Run `upgrade-k8s --manifests-no-prune`.** The inventory is rewritten
+   without Cilium and the objects survive. Helm is then sole owner. This is the
+   same run that takes Kubernetes 1.35.2 → 1.36.3.
+
+5. **Verify** the inventory lists no Cilium, the agents did not restart, and
+   the BGP sessions are still established.
+
+6. **Delete `talos/patches/cilium-cni.yaml`** and add the install step to
+   RUNBOOK-cold-rebuild.md.
+
+## Not part of this, and not a defect either
+
+The cluster runs `routingMode: tunnel` (the chart default, set nowhere) while
+also advertising pod CIDRs over BGP. That is not one layer too many — the two
+carry different directions:
+
+- **VXLAN** carries pod-to-pod between nodes, directly, node to node.
+- **BGP** gives the UDM routes into the pod CIDRs so traffic from the LAN can
+  reach pods. That path goes through the router by definition.
+
+Dropping the encapsulation in favour of native routing would not simplify
+anything; it would push inter-node pod traffic onto the underlay path and
+through the UDM, replacing a direct node-to-node hop with a router hop.
+
+`routingMode` being unset is still worth knowing — the current behaviour is a
+default nobody chose rather than a decision anyone recorded — but the default
+is the right one here, and this document is not the place to revisit it.
@@ -198,8 +198,7 @@ talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
 talosctl gen config homelab https://k8s.renner.uno:6443 \
  --with-secrets secrets.yaml --output-types controlplane \
  --config-patch @talos/patches/cluster.yaml \
-  --config-patch @<cilium additive half> \
-  --config-patch @talos/patches/registry-mirror.yaml \
+  --config-patch @talos/patches/containerd-layer-retention.yaml \
  --config-patch @talos/patches/k8s-N.yaml -o install.yaml
 ```

@@ -208,9 +207,11 @@ Confirm `cluster.id` matches the running cluster before applying.
 Strip the generated `HostnameConfig` document — it collides with
 `machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.

-`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
-against a base with no `inlineManifests` (issue #581). Use only the additive
-second document.
+There is no CNI patch to pass any more. Talos supplies only `cni: none` and
+`proxy: disabled`; Cilium is installed from its HelmRelease values, which is
+Step 2b of RUNBOOK-cold-rebuild.md. A node converted into an existing cluster
+gets its CNI from the Cilium DaemonSet already running there, so nothing extra
+is needed here.

 Validate:

@@ -136,11 +136,18 @@ in `talos/patches/k8s-N.yaml`.

 ## Step 2 — talos: form the cluster + get kubeconfig

-**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
-config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
-`$patch: delete` document fails against a base with no `inlineManifests`
-(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
-and 6; only config generation is broken.
+`kubernetes/bootstrap.sh talos` does this, including the CNI install that used
+to be a hand-typed step. It was unusable for months because it passed
+`cilium-cni.yaml` whole and the file's `$patch: delete` document failed against
+a fresh config (#581); that file no longer exists, so the failure is gone with
+it.
+
+Verified as far as it can be without wiping a cluster: `talosctl gen config`
+with exactly the patches the script passes produces a config carrying
+`cni: none` and `proxy: disabled` that passes `talosctl validate --mode metal`.
+The apply-and-bootstrap half has not been exercised since the rewrite. Read the
+manual procedure below before running the script, so you know what it is doing
+and can take over at any step if it stops.

 The procedure below is what actually works, and is the same one
 `RUNBOOK-baremetal-conversion.md` uses.
@@ -160,17 +167,17 @@ for N in 1 2 3; do
  talosctl gen config homelab https://k8s.renner.uno:6443 \
    --with-secrets secrets.yaml --output-types controlplane \
    --config-patch @talos/patches/cluster.yaml \
-    --config-patch @<cilium additive half> \
-    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/containerd-layer-retention.yaml \
    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
 done
 ```

-Two things the generator gets wrong for this repo, both of which will bite
-silently:
+There is no CNI patch here any more. Talos supplies `cni: none` and
+`proxy: disabled` from `cluster.yaml` and nothing else; Cilium is installed in
+Step 2b below. See PROPOSAL-cilium-single-owner.md.
+
+One thing the generator gets wrong for this repo, and it bites silently:

- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
-  second document. See issue #581.
 - **Strip the generated `HostnameConfig`** — it collides with
  `machine.network.hostname` from the per-node patch.

@@ -209,8 +216,53 @@ talosctl -n k8s-1.srv.renner.uno bootstrap
 talosctl -n k8s-1.srv.renner.uno kubeconfig .
 ```

-Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
-supported way to get the kubeconfig into place afterwards.
+`scripts/import-cluster-access.sh` is the supported way to get the kubeconfig
+into place afterwards.
+
+The nodes will **not** reach `Ready` here. Talos is configured with `cni: none`,
+so there is no pod network until Step 2b — expect `NotReady` with
+`container runtime network not ready` and carry on.
+
+## Step 2b — cilium: install the CNI
+
+Nothing schedules without it, Flux included, so this comes before Step 3. Talos
+no longer carries a rendered copy of Cilium; `helmrelease.yaml` is the only
+place it is configured, and this installs from those same values so the cluster
+comes up on what git describes rather than on a second, drifting copy. See
+PROPOSAL-cilium-single-owner.md for why it works this way.
+
+`bootstrap.sh talos` already did this. What follows is the same thing by hand,
+for when the script stopped partway or you are not using it.
+
+Every input is read out of `helmrelease.yaml`. Nothing here is a version or a
+URL you have to keep in step with anything — copy the block as it stands:
+
+```sh
+HR=kubernetes/infrastructure/cilium/helmrelease.yaml
+
+helm repo add cilium "$(yq -r 'select(.kind == "HelmRepository") | .spec.url' "$HR")"
+helm repo update cilium
+
+yq 'select(.kind == "HelmRelease") | .spec.values' "$HR" > /tmp/cilium-values.yaml
+
+helm install cilium cilium/cilium \
+  --version "$(yq -r 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "$HR")" \
+  --namespace kube-system \
+  -f /tmp/cilium-values.yaml \
+  --set prometheus.serviceMonitor.enabled=false
+```
+
+`--version` takes the HelmRelease's range verbatim; helm resolves it to the same
+release Flux would pick, so the two cannot diverge. That is the point of writing
+it this way — a pinned version in a runbook is a copy nobody maintains, and the
+next rebuild installs something the HelmRelease will immediately fight.
+
+The one literal is `serviceMonitor`, disabled for this install only: its CRD
+arrives with the monitoring stack much later and the object would fail to apply.
+Flux adds the real one once it owns the release. Nothing else is overridden,
+which is what lets helm-controller adopt this release rather than replace it.
+
+Ends when all three nodes are `Ready`.

 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)

@@ -18,15 +18,18 @@ metadata:
 #
 # Opaque-label map (maintainer reference only — NOT served publicly):
 #   services group                   infrastructure group
-#     Web Service 1 = Authentik        Infrastructure 1 = PBS 2 (pbs2.renner.uno)
+#     Web Service 1 = Authentik        Infrastructure 1 = retired (was PBS 2)
 #     Web Service 2 = Gitea            Infrastructure 2 = retired (was Proxmox VE)
 #     Web Service 3 = Immich           Infrastructure 3 = UDM DNS (10.2.40.1)
 #     Web Service 4 = Woodpecker CI    Infrastructure 4 = Mosquitto MQTT broker
 #     Web Service 5 = Grafana
 #     Web Service 6 = Home Assistant
-#     Web Service 7 = Paperless
+#     Web Service 7 = retired (was the NAS file UI; never Paperless, which
+#                     this map claimed and which nothing here checks)
 #     Web Service 8 = Audiobookshelf
 #     Web Service 9 = Music Assistant
+#     Web Service 10 = TRMNL
+#     Web Service 11 = Paperless (gate-level only, see its entry)
 data:
  config.yaml: |
    storage:
@@ -121,18 +124,10 @@ data:
        conditions:
          - "[STATUS] == any(200, 301, 302)"

-      - name: Web Service 7
-        group: services
-        url: https://file.renner.uno
-        interval: 60s
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: the NAS file UI (file.renner.uno). Its Ingress and
+      # ExternalName Service were deleted in #696 because they pointed at
+      # nas-1, which is powered off; this check outlived them and had been
+      # reporting failure on the public page since.

      - name: Web Service 8
        group: services
@@ -178,6 +173,29 @@ data:
        conditions:
          - "[STATUS] == any(200, 301, 302)"

+      # Paperless sits entirely behind the Authentik gate — every path,
+      # including /api/, answers 302 to unauthenticated callers, so there is no
+      # endpoint that proves the app rather than the gate. This check is
+      # therefore reachability only: DNS, TLS, Traefik and the outpost. It
+      # stays green if paperless itself is dead behind a live gate.
+      #
+      # Closing that blind spot needs an in-cluster probe of the paperless
+      # ClusterIP, which crosses a namespace boundary and so needs a
+      # CiliumNetworkPolicy first — the monitoring probes get away without one
+      # only because their targets share a namespace with the prober.
+      - name: Web Service 11
+        group: services
+        url: https://paperless.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
      # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
      # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
      # no longer deployed. Re-add when/if they come back.
@@ -191,20 +209,11 @@ data:
      # NoData). Monitor the offsite PBS from the offsite side or via a
      # check that doesn't depend on this cluster's DNS path.

-      - name: Infrastructure 1
-        group: infrastructure
-        url: https://pbs2.renner.uno:8007
-        interval: 60s
-        client:
-          insecure-skip-verify: true
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: PBS 2 (pbs2.renner.uno:8007) — it has no clients
+      # left. Its two were the PVE cluster, retired to bare-metal Talos, and
+      # nas-1, powered off and dropped from the repo. Checking the liveness of
+      # a backup server that nothing backs up to reports on nothing. Re-add
+      # alongside the first host that is declared to back up to it.

      - name: Infrastructure 3
        group: infrastructure
@@ -5,6 +5,12 @@ metadata:
  namespace: apps
  labels:
    app: gatus
+  annotations:
+    # Gatus reads its config once at startup and never re-reads the file, so
+    # without this a ConfigMap change reconciles cleanly and then does nothing:
+    # git and the running checks drift apart silently, with no failed resource
+    # to notice. Adding a check would not take effect either.
+    configmap.reloader.stakater.com/reload: gatus-config
 spec:
  replicas: 1
  # RWO PVC (`gatus-data`, ceph-block) — RollingUpdate's default 25% surge
@@ -22,9 +28,30 @@ spec:
      labels:
        app: gatus
    spec:
+      # The upstream image is FROM scratch with no USER, so without this the
+      # pod runs as root and trips PodSecurity restricted on four counts.
+      #
+      # The uid is numeric because a scratch image carries no /etc/passwd for
+      # the kubelet to resolve a name against. fsGroup is what makes the switch
+      # survivable: the only write path is the sqlite store on the RWO volume
+      # (/data/gatus.db, see the storage stanza in the ConfigMap), and fsGroup
+      # both chowns the existing volume and leaves the directory group-writable
+      # so sqlite can create its -wal and -journal siblings next to the db.
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: gatus
          image: twinproduction/gatus:v5.36.0 # {"$imagepolicy": "flux-system:gatus"}
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
          ports:
            - containerPort: 8080
              name: http
@@ -210,6 +210,28 @@ spec:
            - name: vscode
              containerPort: 8080
              protocol: TCP
+          # /healthz answers 200 whenever the HTTP server is up; its body
+          # reports session heartbeat state, which is not health — an idle
+          # editor reports "expired" and is perfectly fine. So this asserts
+          # that code-server is serving, not that anyone is using it.
+          #
+          # Liveness matters here more than elsewhere in this pod: code-server
+          # shares the Home Assistant config volume, and a wedged editor is
+          # otherwise invisible until someone tries to open it.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: config
              mountPath: /config
@@ -66,6 +66,11 @@ spec:
        # parameter, 200 means initialized, unsealed and serving, on whichever
        # node answers; 503 sealed and 501 uninitialized both stay red.
        - https://openbao.renner.uno/v1/sys/health?standbyok=true
+        # ical-proxy serves an unauthenticated /health that answers 200 with
+        # {"status":"healthy"}. Unlike the gated hostnames this is real app
+        # health, not reachability — the root path returns 404 by design, since
+        # the service only answers on calendar paths.
+        - https://ical-proxy.renner.uno/health
        # APP-HEALTH probes for the gated monitoring UIs, from the correct
        # in-cluster vantage (monitoring-vantage-point-rule): the gate is
        # only on the ingress, so the ClusterIP /-/healthy endpoints are
@@ -78,6 +83,11 @@ spec:
        # external reachability probe above; it is not the health vantage.)
        - http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/-/healthy
        - http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/-/healthy
+        # Loki has no external vantage at all: its ingress requires auth on
+        # every path, so an outside probe can only ever report 401. In-cluster
+        # /ready answers 200 unauthenticated and is in this namespace, so the
+        # same intra-namespace allowance the two above rely on covers it.
+        - http://loki.monitoring.svc.cluster.local:3100/ready
 # WAN-path probes of the public edge BY PROXY IP. The name-based
 # home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
 # and never leaves the LAN — the vantage-point blind spot that let the
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -24,6 +24,22 @@ spec:
            - name: wyoming
              containerPort: 10400
              protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process. Weaker than an HTTP probe, but the previous state was none
+          # at all: a dead server stayed Ready and kept being routed to.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 15
+            periodSeconds: 10
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 100m
@@ -70,6 +70,27 @@ spec:
            - name: consume
              mountPath: /home/scanner
              subPath: scanner
+          # FTP has no health verb, so the control port accepting a connection
+          # is what can be asserted. It is enough for the failure that matters:
+          # if vsftpd dies the scanner cannot deposit anything and nothing else
+          # in the chain notices, because paperless only ever sees an empty
+          # consume directory — which is indistinguishable from nobody having
+          # scanned.
+          #
+          # Only the control port is probed. The passive range is negotiated
+          # per transfer and is not listening at rest.
+          livenessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 20m
@@ -155,21 +155,33 @@ spec:
            - name: bridge-cert
              mountPath: /shared
              readOnly: true
-          # TCP probes — HTTP probes against /api/ flooded the
-          # `paperless.auth` logger every 10 s with "No authentication
-          # provided from private IP …". With ENABLE_HTTP_REMOTE_USER
-          # the middleware logs every unauth API request, and there's
-          # no public health endpoint to probe. TCP confirms gunicorn
-          # is bound; that's the same liveness signal we'd get from
-          # /api/ minus the noise.
+          # /accounts/login/ rather than /api/ or a TCP connect.
+          #
+          # /api/ is unusable as a probe target: with ENABLE_HTTP_REMOTE_USER
+          # the remote-user middleware logs "No authentication provided from
+          # private IP …" for every unauthenticated API request, so probing it
+          # floods `paperless.auth`. Measured: four requests to /api/ produce
+          # four log lines, four to /accounts/login/ produce none — the
+          # middleware only covers the API surface.
+          #
+          # A TCP connect avoids the noise but only proves gunicorn accepted a
+          # socket. /accounts/login/ renders a real Django view, so URL
+          # routing, settings and templating all have to work. It does not
+          # prove Postgres is healthy; nothing cheap here does.
+          #
+          # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
+          # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
+          # means giving these probes an explicit Host header.
          livenessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
              port: http
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 10
          readinessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
@@ -34,6 +34,32 @@ spec:
            - name: wyoming
              containerPort: 10300
              protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process — weaker than an HTTP probe, but the previous state was
+          # none at all.
+          #
+          # A startupProbe rather than a long liveness delay: this container
+          # runs with --retrain-on-start, and retraining against the Home
+          # Assistant entity list takes as long as it takes. The startup gate
+          # allows up to 10 minutes (60 x 10s) and only then hands over to
+          # liveness, so a slow retrain cannot be mistaken for a hang, and a
+          # genuine hang after startup is still caught within a minute.
+          startupProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: models
              mountPath: /models
@@ -27,7 +27,7 @@ spec:
        # Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
        # is the only consumer; a major bump is a separate migration.
        - name: valkey
-          image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
+          image: valkey/valkey:9.1.2-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
          args:
            - valkey-server
            - --save
@@ -29,6 +29,27 @@ spec:
            - name: wyoming
              containerPort: 10300
              protocol: TCP
+          # Wyoming is a raw TCP protocol with no HTTP surface and no health
+          # verb, so accepting a connection is the only thing that can be
+          # asserted from outside the process. That is weaker than an HTTP
+          # probe but strictly better than the previous state of none at all,
+          # where a dead server stayed Ready and Home Assistant kept routing
+          # speech to it.
+          #
+          # The liveness delay is generous because the model is loaded at
+          # startup from the data volume.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 90
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /data
@@ -94,6 +94,31 @@ spec:
          envFrom:
            - secretRef:
                name: woodpecker-secret
+          ports:
+            # WOODPECKER_HEALTHCHECK defaults to true and serves :3000, so this
+            # port already existed; declaring it just lets the probe name it.
+            - name: health
+              containerPort: 3000
+              protocol: TCP
+          # /healthz answers 500 ONLY when a running workflow has passed its own
+          # timeout plus a one-hour buffer (agent/state.go, State.Healthy). An
+          # idle agent always answers 200.
+          #
+          # So this catches a wedged build holding a slot, which is otherwise
+          # invisible — and it does NOT catch an agent that has lost its server
+          # connection, which reports healthy while quietly taking no work. Do
+          # not read a green probe here as "CI is fine".
+          #
+          # Liveness only: the agent fronts no Service, so readiness would gate
+          # nothing. The upstream condition is already debounced by that hour,
+          # which is why the thresholds here are short.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            initialDelaySeconds: 20
+            periodSeconds: 30
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 25m
@@ -8,6 +8,9 @@ set -euo pipefail
 #   ./bootstrap.sh talos        # Talos apply + K8s bootstrap + kubeconfig (writes access-staging/)
 #   ./bootstrap.sh flux         # reads access-staging/, installs flux, waits for infrastructure
 #   ./bootstrap.sh wait-apps    # waits for apps/authentik/paperless to reconcile
+#   ./bootstrap.sh vmtest       # throwaway QEMU VMs, then the talos phase
+#                               # against them unchanged. Needs root for the
+#                               # provisioner and leaves nothing behind.
 #
 # Flux git source (break-glass): the in-cluster Gitea that Flux normally pulls
 # from is itself Flux-managed, so a cold rebuild cannot pull from it until the
@@ -21,6 +24,13 @@ set -euo pipefail

 TALOS_DIR="$(cd "$(dirname "$0")/talos" && pwd)"
 REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+# Cilium's single source of truth. Talos ships `cni: none` (talos/patches/
+# cluster.yaml) and nothing else, so the CNI has to be installed here, between
+# `talosctl bootstrap` and Flux — Flux's own controllers are pods and cannot
+# reconcile the network they need in order to start. Everything the install
+# needs is read out of this file at run time so there is no second copy of a
+# version or a value to keep in step. See PROPOSAL-cilium-single-owner.md.
+CILIUM_HR="$REPO_ROOT/kubernetes/infrastructure/cilium/helmrelease.yaml"
 ACCESS_STAGING="$REPO_ROOT/kubernetes/access-staging"
 NODES=(k8s-1 k8s-2 k8s-3)
 # No dedicated workers since the k8s-edge VM was retired (2026-07-02) —
@@ -33,6 +43,24 @@ DNS_SERVER="10.2.40.1"  # UDM (old PVE DNS VIP 10.2.40.53 decommissioned)
 # Cluster API endpoint — Talos VIP (10.2.40.60) elected via etcd, fronted by
 # UniFi DNS A record k8s.renner.uno. Defined in talos/patches/k8s-{1,2,3}.yaml.
 K8S_API_HOST="k8s.renner.uno"
+# Which site patch the nodes get: talos/patches/site-$SITE.yaml. The three
+# variables above and this one are everything the vmtest phase has to change in
+# order to run this same code against throwaway VMs — if that list ever grows,
+# the test is drifting away from the thing it claims to exercise.
+SITE="${SITE:-homelab}"
+# Where this script writes the cluster access files. These are the paths
+# talosctl and kubectl read from anyway, so unset they resolve to exactly the
+# old behaviour. Set, the whole run stays out of $HOME — which is what lets the
+# vmtest phase drive a throwaway cluster without replacing the operator's
+# production access with a VM's.
+TALOSCONFIG_PATH="${TALOSCONFIG:-$HOME/.talos/config}"
+KUBECONFIG_PATH="${KUBECONFIG:-$HOME/.kube/config}"
+case "$KUBECONFIG_PATH" in
+    *:*) echo "KUBECONFIG holds multiple paths; set it to a single file" >&2; exit 1 ;;
+esac
+# Addresses to apply to, in the order of NODES. Empty means resolve the node
+# FQDNs, which is right for real hardware on a network that has DNS for it.
+NODE_ADDRS=()
 GITEA_HOST="gitea.renner.uno"
 GITEA_OWNER="gitea_admin"
 GITEA_REPO="homelab"
@@ -44,6 +72,19 @@ FLUX_SOURCE="${FLUX_SOURCE:-gitea}"
 GITHUB_OWNER="${GITHUB_OWNER:-konairius}"
 GITHUB_REPO="${GITHUB_REPO:-Homelab}"
 FLUX_PATH="kubernetes/clusters/homelab"
+# Branch Flux reconciles from, and that `flux bootstrap` commits its own sync
+# manifests into. master for a real rebuild.
+#
+# Overridable so the phase can be rehearsed without writing to master — point it
+# at a throwaway branch, which has to exist already: `flux bootstrap` clones the
+# branch and fails with `couldn't find remote ref` rather than creating it.
+#
+#     git push origin origin/master:refs/heads/rehearsal-1
+#     FLUX_BRANCH=rehearsal-1 ./bootstrap.sh flux
+#
+# Nothing else about the phase changes, which is the point. Delete the branch
+# afterwards.
+FLUX_BRANCH="${FLUX_BRANCH:-master}"
 SOPS_KEY="$HOME/.config/sops/age/keys.txt"

 PHASE="${1:-all}"
@@ -78,43 +119,145 @@ phase_runs() {
    [[ "$PHASE" == "all" || "$PHASE" == "$1" ]]
 }

+# Provision throwaway QEMU VMs, then run the Talos phase below against
+# them unchanged. Deliberately not a copy of that phase: it sets the site
+# variables and falls through, because a rehearsal that runs different code
+# proves less than one that does not. Everything it changes is listed here — if
+# that list grows, the test is drifting away from what it claims to exercise.
+#
+# What it cannot cover, and what therefore still has to be true on hardware:
+# Ceph (the provisioner gives extra disks to workers only), the mgmt and v30iot
+# VLANs and the Layer2 VIP (one flat NIC per VM), and Flux (its own phase).
+#
+# Needs root for the QEMU provisioner (CNI and KVM), and only for that step.
+if [[ "$PHASE" == "vmtest" ]]; then
+    echo -e "${YELLOW}=== Phase: VM rebuild test ===${NC}"
+    command -v talosctl >/dev/null || fail "talosctl not found"
+    command -v yq >/dev/null       || fail "yq not found"
+    # The node patches are the list; the VM count follows from them rather than
+    # being stated twice. vm-1 is the control plane, the rest are the workers
+    # that carry an OSD device — see the comment on NODES below for why.
+    VM_CP_NODES=(vm-1)
+    VM_WORKER_NODES=(vm-2 vm-3 vm-4)
+    VM_CPS=${#VM_CP_NODES[@]}
+    VM_WORKER_COUNT=${#VM_WORKER_NODES[@]}
+    for n in "${VM_CP_NODES[@]}" "${VM_WORKER_NODES[@]}"; do
+        [[ -f "$TALOS_DIR/patches/$n.yaml" ]] || fail "$TALOS_DIR/patches/$n.yaml not found"
+    done
+    [[ -f "$TALOS_DIR/patches/site-qemu.yaml" ]] || fail "site-qemu.yaml not found"
+
+    # Talos version and Image Factory schematic are read out of the installer
+    # image in cluster.yaml. Naming them here would be a second copy to keep in
+    # step, and the day they diverged the test would quietly stop booting the
+    # image a rebuild installs.
+    VM_IMAGE=$(yq -r '.machine.install.image' "$TALOS_DIR/patches/cluster.yaml")
+    VM_SCHEMATIC=${VM_IMAGE#*metal-installer/}
+    VM_SCHEMATIC=${VM_SCHEMATIC%%:*}
+    VM_VERSION=${VM_IMAGE##*:}
+    [[ ${#VM_SCHEMATIC} -eq 64 && "$VM_VERSION" == v* ]] \
+        || fail "could not read a schematic and version out of '$VM_IMAGE'"
+
+    # sun_path caps a unix socket path at 108 bytes and the provisioner puts a
+    # qemu monitor socket under the state directory, so this has to stay short.
+    # A path under the repo or a long mktemp dir silently fails to boot.
+    VM_STATE=/tmp/bootstrap-vmtest
+    VM_CLUSTER=bootstrap-vmtest
+    VM_DIR=$(mktemp -d)
+    TALOSCTL_BIN=$(command -v talosctl)
+
+    # Hard isolation from the production cluster: talosctl and kubectl both
+    # honour these, and the phase below writes to them rather than to $HOME.
+    export TALOSCONFIG="$VM_DIR/talosconfig"
+    export KUBECONFIG="$VM_DIR/kubeconfig"
+    TALOSCONFIG_PATH="$TALOSCONFIG"
+    KUBECONFIG_PATH="$KUBECONFIG"
+
+    cleanup_vmtest() {
+        echo -n "Destroying throwaway VMs... "
+        sudo -E "$TALOSCTL_BIN" cluster destroy --name "$VM_CLUSTER" --state "$VM_STATE" >/dev/null || true
+        # destroy removes the cluster's own directory but leaves the state root.
+        sudo rm -rf "$VM_STATE"
+        rm -rf "$VM_DIR"
+        echo "done"
+    }
+    trap cleanup_vmtest EXIT
+
+    step 0 "Creating $VM_CPS control plane and $VM_WORKER_COUNT workers in maintenance mode..."
+    # maintenance means the provisioner applies no configuration of its own, so
+    # every byte of machine config below comes from talos/patches.
+    sudo -E "$TALOSCTL_BIN" cluster create qemu \
+        --name "$VM_CLUSTER" \
+        --state "$VM_STATE" \
+        --presets iso,maintenance \
+        --controlplanes "$VM_CPS" \
+        --workers "$VM_WORKER_COUNT" \
+        --talos-version "$VM_VERSION" \
+        --schematic-id "$VM_SCHEMATIC" \
+        --memory-controlplanes 2560MiB \
+        --memory-workers 4GiB \
+        --cpus-controlplanes 2 \
+        --cpus-workers 2 \
+        --disks virtio:10GiB,virtio:8GiB \
+        --talosconfig-destination "$TALOSCONFIG" \
+        || fail "could not create the throwaway VMs"
+    sudo chown -R "$(id -u):$(id -g)" "$VM_STATE" "$TALOSCONFIG"
+    ok
+
+    # The provisioner hands out its CIDR sequentially from .2; .1 is the
+    # gateway and the load balancer it puts in front of the API. If that ever
+    # changes, the apply loop below reports which address stopped answering
+    # rather than failing silently.
+    # One control plane and three workers, not three control planes: the
+    # provisioner gives the second disk only to workers ("disks after the first
+    # one are added only to worker machines"), and an OSD needs it. Three of
+    # them because the CephCluster runs three mons with allowMultiplePerNode
+    # false over pools that are replica 3 across failureDomain host.
+    NODES=("${VM_CP_NODES[@]}")
+    WORKERS=("${VM_WORKER_NODES[@]}")
+    ALL_NODES=("${NODES[@]}" "${WORKERS[@]}")
+    NODE_ADDRS=()
+    for i in $(seq 0 $(( ${#ALL_NODES[@]} - 1 ))); do
+        NODE_ADDRS+=("10.5.0.$(( i + 2 ))")
+    done
+    SITE=qemu
+    K8S_API_HOST=10.5.0.1
+    VMTEST=1
+
+    # Run the real phase, not a reimplementation of it.
+    PHASE=talos
+fi
+
 if phase_runs talos; then
    echo -e "${YELLOW}=== Phase: Talos bootstrap ===${NC}"
    command -v talosctl >/dev/null || fail "talosctl not found"
    command -v kubectl >/dev/null  || fail "kubectl not found"
+    command -v helm >/dev/null   || fail "helm not found (needed to install the CNI)"
+    command -v yq >/dev/null     || fail "yq not found (needed to read the Cilium HelmRelease)"
    [[ -f "$TALOS_DIR/patches/cluster.yaml"         ]] || fail "cluster.yaml patch not found"
-    [[ -f "$TALOS_DIR/patches/cilium-cni.yaml"      ]] || fail "cilium-cni.yaml patch not found"
-    [[ -f "$TALOS_DIR/patches/registry-mirror.yaml" ]] || fail "registry-mirror.yaml patch not found"
+    [[ -f "$TALOS_DIR/patches/containerd-layer-retention.yaml" ]] || fail "containerd-layer-retention.yaml patch not found"
+    [[ -f "$CILIUM_HR"                              ]] || fail "cilium helmrelease not found at $CILIUM_HR"

    step 0 "Generating Talos configs..."
    TALOS_GEN_DIR=$(mktemp -d)
    talosctl gen config homelab "https://${K8S_API_HOST}:6443" \
        --output "$TALOS_GEN_DIR" \
        --force
-    # Strip the auto-generated HostnameConfig docs (conflicts with machine.network.hostname in per-node patches)
-    for f in "$TALOS_GEN_DIR"/controlplane.yaml "$TALOS_GEN_DIR"/worker.yaml; do
-        awk '
-          BEGIN { doc="" }
-          /^---[ \t]*$/ {
-            if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
-            doc=$0 "\n"; next
-          }
-          { doc = doc $0 "\n" }
-          END {
-            if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
-          }
-        ' "$f" > "$f.tmp" && mv "$f.tmp" "$f"
-    done
-    mkdir -p ~/.talos
-    cp "$TALOS_GEN_DIR/talosconfig" ~/.talos/config
+    mkdir -p "$(dirname "$TALOSCONFIG_PATH")"
+    cp "$TALOS_GEN_DIR/talosconfig" "$TALOSCONFIG_PATH"

    step 0 "Applying Talos configs to nodes..."
    apply_node_config() {
-        local node=$1 base_config=$2
+        local node=$1 base_config=$2 fixed_addr=${3:-}
        local fqdn="${node}.${DOMAIN}"
-        echo "  $fqdn:"
+        echo "  ${fixed_addr:-$fqdn}:"
        for i in $(seq 1 60); do
-            node_ip=$(resolve_node_ip "$fqdn")
+            # A fixed address skips DNS. Real hardware has a name to resolve;
+            # a throwaway VM has an address and nothing else.
+            if [[ -n "$fixed_addr" ]]; then
+                node_ip="$fixed_addr"
+            else
+                node_ip=$(resolve_node_ip "$fqdn")
+            fi
            if [[ -z "$node_ip" ]]; then
                echo "    attempt $i/60: DNS returned no IP"
                sleep 10
@@ -123,73 +266,216 @@ if phase_runs talos; then
            if err=$(talosctl apply-config --insecure -n "$node_ip" \
                --file "$base_config" \
                --config-patch "@$TALOS_DIR/patches/cluster.yaml" \
-                --config-patch "@$TALOS_DIR/patches/cilium-cni.yaml" \
-                --config-patch "@$TALOS_DIR/patches/registry-mirror.yaml" \
+                --config-patch "@$TALOS_DIR/patches/site-${SITE}.yaml" \
+                --config-patch "@$TALOS_DIR/patches/containerd-layer-retention.yaml" \
                --config-patch "@$TALOS_DIR/patches/${node}.yaml" 2>&1); then
-                echo "    applied via $node_ip ${GREEN}(OK)${NC}"
+                echo -e "    applied via $node_ip ${GREEN}(OK)${NC}"
                return 0
            fi
            echo "    attempt $i/60: $node_ip - ${err//$'\n'/ | }"
            sleep 10
        done
-        fail "Timeout waiting for $fqdn"
+        fail "Timeout waiting for ${fixed_addr:-$fqdn}"
    }

-    for node in "${NODES[@]}"; do
-        apply_node_config "$node" "$TALOS_GEN_DIR/controlplane.yaml"
-    done
-    for node in "${WORKERS[@]}"; do
-        apply_node_config "$node" "$TALOS_GEN_DIR/worker.yaml"
+    # One loop over ALL_NODES rather than one per role, so NODE_ADDRS can be a
+    # single list in the same order. Control planes come first in ALL_NODES,
+    # which is what decides the base config each node gets.
+    for idx in "${!ALL_NODES[@]}"; do
+        if [[ $idx -lt ${#NODES[@]} ]]; then
+            base="$TALOS_GEN_DIR/controlplane.yaml"
+        else
+            base="$TALOS_GEN_DIR/worker.yaml"
+        fi
+        apply_node_config "${ALL_NODES[$idx]}" "$base" "${NODE_ADDRS[$idx]:-}"
    done

    echo -n "Configuring talosctl... "
-    talosctl config endpoint "${NODES[@]/%/.${DOMAIN}}"
-    talosctl config node "${NODES[0]}.${DOMAIN}"
+    # `gen config` leaves talosconfig endpoints empty; they are the Talos API
+    # addresses and unrelated to cluster.controlPlane.endpoint.
+    if [[ ${#NODE_ADDRS[@]} -gt 0 ]]; then
+        talosctl config endpoint "${NODE_ADDRS[@]}"
+        talosctl config node "${NODE_ADDRS[0]}"
+    else
+        talosctl config endpoint "${NODES[@]/%/.${DOMAIN}}"
+        talosctl config node "${NODES[0]}.${DOMAIN}"
+    fi
    ok

-    step 0 "Waiting for Talos cluster formation..."
-    for i in $(seq 1 30); do
-        members=$(talosctl get members 2>/dev/null || true)
-        missing=0
-        for node in "${ALL_NODES[@]}"; do
-            grep -q "$node" <<<"$members" || missing=1
+    step 0 "Waiting for every node to come back configured..."
+    # Asks each machine directly, by address, and expects it to answer with the
+    # hostname its patch assigns. That is what has to be true before etcd is
+    # bootstrapped: the config took, the node rebooted into it, and the machine
+    # at that address is the one we think it is.
+    #
+    # This used to wait on `talosctl get members`, which was wrong twice over.
+    # It queried through the configured endpoint list rather than each node, so
+    # apid refused it with `PermissionDenied: no request forwarding` — the same
+    # thing that broke the bootstrap call. And membership comes from the
+    # discovery service, because the Kubernetes registry is disabled: a cold
+    # rebuild would have been waiting on discovery.talos.dev being reachable,
+    # before the cluster it is trying to rebuild even exists.
+    #
+    # Each node installs to disk and reboots first, so the window is generous.
+    for idx in "${!ALL_NODES[@]}"; do
+        want="${ALL_NODES[$idx]}"
+        addr="${NODE_ADDRS[$idx]:-${want}.${DOMAIN}}"
+        echo -n "  $want ($addr)... "
+        node_err=""
+        for i in $(seq 1 60); do
+            got=$(talosctl -n "$addr" -e "$addr" get hostname \
+                    -o jsonpath='{.spec.hostname}' 2>"$TALOS_GEN_DIR/hostname.err") || true
+            node_err=$(cat "$TALOS_GEN_DIR/hostname.err")
+            [[ "$got" == "$want" ]] && break
+            sleep 10
+            if [[ $i -eq 60 ]]; then
+                echo
+                [[ -n "$node_err" ]] && echo "  last talosctl error: $node_err" >&2
+                fail "$addr never came back as $want (last answer: '${got:-none}')"
+            fi
        done
-        [[ $missing -eq 0 ]] && break
-        sleep 5
-        [[ $i -eq 30 ]] && fail "Talos cluster didn't form"
+        ok
    done

    echo -n "Bootstrapping Kubernetes... "
-    talosctl bootstrap 2>/dev/null || true
+    # -n AND -e on one control plane, deliberately. etcd bootstrap is the one
+    # call apid refuses to forward: reached through any other node it fails with
+    # `PermissionDenied: no request forwarding`. Without them talosctl picks an
+    # endpoint from the configured list — with more than one control plane, or a
+    # list that includes workers, that is a coin flip.
+    if [[ ${#NODE_ADDRS[@]} -gt 0 ]]; then
+        bootstrap_target="${NODE_ADDRS[0]}"
+    else
+        bootstrap_target="${NODES[0]}.${DOMAIN}"
+    fi
+    # Re-running against an already-bootstrapped cluster is not an error and is
+    # the reason for `|| true` — but that used to swallow every other reason as
+    # well, and the run carried on to fail somewhere further downstream with no
+    # trace of the actual cause. Anything that is not "already exists" is shown.
+    if ! bootstrap_err=$(talosctl bootstrap -n "$bootstrap_target" -e "$bootstrap_target" 2>&1); then
+        grep -qi 'already\|AlreadyExists' <<<"$bootstrap_err" \
+            || echo -e "\n${YELLOW}bootstrap reported:${NC} ${bootstrap_err//$'\n'/ | }"
+    fi
    ok

    echo -n "Getting kubeconfig... "
+    kubeconfig_err=""
    for i in $(seq 1 30); do
-        if talosctl kubeconfig ~/.kube/config --force 2>/dev/null; then
-            sed -i "s|server:.*|server: https://${K8S_API_HOST}:6443|" ~/.kube/config
+        mkdir -p "$(dirname "$KUBECONFIG_PATH")"
+        if kubeconfig_err=$(talosctl kubeconfig "$KUBECONFIG_PATH" --force 2>&1); then
+            sed -i "s|server:.*|server: https://${K8S_API_HOST}:6443|" "$KUBECONFIG_PATH"
            break
        fi
        sleep 5
-        [[ $i -eq 30 ]] && fail "Couldn't get kubeconfig"
+        # Held back while retrying — the first attempts fail by design — but
+        # printed if it never works, because that message is the diagnosis.
+        [[ $i -eq 30 ]] && { echo; echo "$kubeconfig_err" >&2; fail "Couldn't get kubeconfig"; }
    done
    ok

+    echo -n "Waiting for the Kubernetes API to answer... "
+    # Having a kubeconfig is not the same as having an apiserver. `talosctl
+    # kubeconfig` succeeds as soon as Talos can hand out credentials, which is
+    # before kube-apiserver is serving on the endpoint the file points at — so
+    # the very next command could fail with an EOF against a port nothing is
+    # listening on yet. On hardware the gap usually closes on its own; on a
+    # scratch cluster it does not, which is how this was found.
+    # `get --raw /version` needs the server; `kubectl version` would answer from
+    # the client alone and pass against nothing at all.
+    api_err=""
+    for i in $(seq 1 60); do
+        api_err=$(kubectl get --raw /version 2>&1) && break
+        sleep 5
+        if [[ $i -eq 60 ]]; then
+            echo "$api_err" >&2
+            fail "Kubernetes API at https://${K8S_API_HOST}:6443 never answered"
+        fi
+    done
+    ok
+
+    step 0 "Installing Cilium (CNI)..."
+    # Must run before the node-Ready wait below: with `cni: none` the kubelet
+    # reports NotReady until a CNI is present, so that loop would time out.
+    #
+    # Chart repo, version constraint and values all come out of the HelmRelease
+    # so this cannot drift from what Flux will reconcile minutes later. helm
+    # accepts the semver range verbatim and resolves it the same way Flux does.
+    #
+    # serviceMonitor is the one override, and it is a bootstrap constraint
+    # rather than a value: its CRD arrives with the monitoring stack much
+    # later, and the object would fail to apply. Nothing else is overridden,
+    # which is what lets helm-controller adopt this release instead of
+    # replacing it.
+    CILIUM_VALUES=$(mktemp)
+    yq 'select(.kind == "HelmRelease") | .spec.values' "$CILIUM_HR" > "$CILIUM_VALUES"
+    grep -q 'policyMapMax' "$CILIUM_VALUES" \
+        || fail "extracted Cilium values look wrong (no policyMapMax) — check $CILIUM_HR"
+
+    helm repo add cilium "$(yq -r 'select(.kind == "HelmRepository") | .spec.url' "$CILIUM_HR")" >/dev/null
+    helm repo update cilium >/dev/null
+    helm upgrade --install cilium cilium/cilium \
+        --version "$(yq -r 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "$CILIUM_HR")" \
+        --namespace kube-system \
+        -f "$CILIUM_VALUES" \
+        --set prometheus.serviceMonitor.enabled=false \
+        --wait --timeout 10m
+    rm -f "$CILIUM_VALUES"
+    ok
+
    step 0 "Waiting for K8s nodes..."
    expected=${#ALL_NODES[@]}
+    nodes_err=""
    for i in $(seq 1 60); do
-        ready=$(kubectl get nodes --no-headers 2>/dev/null | grep -c "Ready" || true)
+        # The STATUS column, matched whole. `grep -c Ready` counted NotReady
+        # too — the substring is in it — so this waited for nodes to register
+        # rather than to become ready, and passed while every one of them was
+        # still NotReady.
+        nodes_out=$(kubectl get nodes --no-headers 2>"$TALOS_GEN_DIR/nodes.err") || true
+        nodes_err=$(cat "$TALOS_GEN_DIR/nodes.err")
+        ready=$(awk '$2 == "Ready"' <<<"$nodes_out" | grep -c . || true)
        if [[ "$ready" -eq "$expected" ]]; then
            break
        fi
        sleep 5
-        [[ $i -eq 60 ]] && fail "Not all nodes ready (got $ready/$expected)"
+        if [[ $i -eq 60 ]]; then
+            echo
+            [[ -n "$nodes_err" ]] && echo "last kubectl error: $nodes_err" >&2
+            echo "${nodes_out:-  (no nodes listed)}" >&2
+            fail "Not all nodes ready (got $ready/$expected)"
+        fi
    done

-    # Stash configs for subsequent phases / export-access step
-    mkdir -p "$ACCESS_STAGING"
-    cp ~/.talos/config "$ACCESS_STAGING/talosconfig"
-    cp ~/.kube/config  "$ACCESS_STAGING/kubeconfig"
-    echo -e "${GREEN}Talos phase complete.${NC} Access files in $ACCESS_STAGING/"
+    # Stash configs for subsequent phases / export-access step. Skipped under
+    # vmtest: those are a throwaway cluster's credentials, and writing them here
+    # would leave the flux phase pointed at VMs that no longer exist.
+    if [[ "${VMTEST:-0}" == "1" ]]; then
+        # Nodes being Ready says nothing about storage. The CephCluster picks
+        # OSD nodes by a label out of the machine config and their device by the
+        # partition label RawVolumeConfig creates, so assert both here — running
+        # it by hand afterwards races the teardown, and a test that does not
+        # check its own premise is worth little.
+        step 0 "Checking the OSD prerequisites Ceph selects on..."
+        for idx in $(seq "${#NODES[@]}" $(( ${#ALL_NODES[@]} - 1 ))); do
+            vm_node="${ALL_NODES[$idx]}"
+            vm_addr="${NODE_ADDRS[$idx]}"
+            vols=$(talosctl -n "$vm_addr" -e "$vm_addr" get discoveredvolumes 2>&1) \
+                || fail "could not read volumes from $vm_node ($vm_addr): ${vols//$'\n'/ | }"
+            awk '$NF == "r-ceph-osd" {found=1} END {exit !found}' <<<"$vols" \
+                || fail "$vm_node has no r-ceph-osd partition — RawVolumeConfig did not take"
+            kubectl get node "$vm_node" \
+                -o jsonpath='{.metadata.labels.storage\.renner\.uno/ceph-osd}' \
+                | grep -q true \
+                || fail "$vm_node is missing the storage.renner.uno/ceph-osd label"
+            echo -e "  $vm_node: partition and label ${GREEN}(OK)${NC}"
+        done
+
+        echo -e "${GREEN}VM rebuild test passed.${NC} ${#ALL_NODES[@]} nodes Ready on the repo's own config."
+    else
+        mkdir -p "$ACCESS_STAGING"
+        cp "$TALOSCONFIG_PATH" "$ACCESS_STAGING/talosconfig"
+        cp "$KUBECONFIG_PATH"  "$ACCESS_STAGING/kubeconfig"
+        echo -e "${GREEN}Talos phase complete.${NC} Access files in $ACCESS_STAGING/"
+    fi
 fi

 # Helper: load staged access files into home dir (for later CI steps)
@@ -228,7 +514,7 @@ if phase_runs flux; then
        flux bootstrap github \
            --owner="$GITHUB_OWNER" \
            --repository="$GITHUB_REPO" \
-            --branch=master \
+            --branch="$FLUX_BRANCH" \
            --path="$FLUX_PATH" \
            --components-extra=image-reflector-controller,image-automation-controller \
            --personal \
@@ -239,7 +525,7 @@ if phase_runs flux; then
            --hostname="$GITEA_HOST" \
            --owner="$GITEA_OWNER" \
            --repository="$GITEA_REPO" \
-            --branch=master \
+            --branch="$FLUX_BRANCH" \
            --path="$FLUX_PATH" \
            --components-extra=image-reflector-controller,image-automation-controller \
            --personal \
@@ -33,6 +33,13 @@ spec:
    remediation:
      retries: 3
  values:
+    # THIS IS THE ONLY PLACE CILIUM IS CONFIGURED. Talos used to carry a
+    # rendered copy as a bootstrap inline manifest, so every setting existed
+    # two or three times and nothing enforced agreement — see
+    # PROPOSAL-cilium-single-owner.md for what that cost. Talos now provides
+    # only `cni: none` and `proxy: disabled`; a cold rebuild installs Cilium
+    # from these values with helm before Flux takes over.
+    #
    # Rolls the agent DaemonSet when cilium-config changes. Cilium reads
    # datapath options once at agent start and the chart puts no config
    # checksum on the pod template, so without this a changed ConfigMap leaves
@@ -50,8 +57,6 @@ spec:
    # and every identity costs one entry per allowed (direction, proto, port)
    # in every endpoint's policy map, cluster-wide. Nothing selects on these
    # labels in policy, so they buy identity churn and nothing else.
-    # Keep in lockstep with kubernetes/talos/patches/cilium-cni.yaml and
-    # kubernetes/talos/cilium.yaml.
    labels: '!woodpecker-ci\.org/.* !step$'
    ipv6:
      enabled: true
@@ -76,10 +81,6 @@ spec:
      # cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
      # restart.
      #
-      # THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
-      # and talos/cilium.yaml. The latter two are what a cold rebuild installs
-      # from. Nothing enforces agreement; grep bpf-policy-map-max before
-      # changing it.
      policyMapMax: 65536
      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
@@ -140,3 +141,28 @@ spec:
      enabled: true
    externalIPs:
      enabled: true
+    # BGP to the UDM. The peering itself has always been in git as CRDs in
+    # post-crds/cilium-bgp.yaml — but the switch that starts the control plane
+    # was not, and neither were the CIDRs below. They were applied by hand and
+    # held by the `kubectl-patch` field manager on cilium-config. Server-side
+    # apply left them alone, so nothing broke and nothing noticed; a cold
+    # rebuild would have come up with BGP off while the CRDs said otherwise.
+    #
+    # This is not a free change of ownership. Enabling bgpControlPlane through
+    # the chart also emits companion keys the hand-patch never set —
+    # bgp-secrets-namespace, bgp-router-id-allocation-mode and -ip-pool,
+    # enable-bgp-control-plane-status-report, enable-bgp-legacy-origin-attribute
+    # — so Cilium has been running on its own defaults for those. Router-id
+    # allocation is the one to watch: if the chart's default disagrees with
+    # Cilium's, the sessions re-establish.
+    bgpControlPlane:
+      enabled: true
+    # Masquerading exception, not a routing statement: pod traffic to these
+    # ranges leaves unmasqueraded, so LAN services see real pod IPs. That only
+    # works because BGP advertises the pod CIDRs, which is what gives the UDM a
+    # route back. The two settings are one decision and have to move together.
+    #
+    # Pod CIDRs are 10.244.0.0/16 and fd10:244::/48 (talos/patches/cluster.yaml);
+    # the supernets here cover the LAN as well, which is the point.
+    ipv4NativeRoutingCIDR: 10.0.0.0/8
+    ipv6NativeRoutingCIDR: fd00::/8
@@ -33,3 +33,16 @@ spec:
    crds: CreateReplace
    remediation:
      retries: 3
+  values:
+    # The chart ships these disabled, so the controller ran with no probe of
+    # any kind: a wedged reconciler stayed Ready and every ExternalSecret would
+    # simply stop refreshing, which surfaces only when a consumer needs a
+    # credential that has since rotated.
+    #
+    # Enabling them also starts the health server — the chart derives
+    # --health-probe-bind-address from these values, and without them the
+    # controller exposes nothing but its metrics port.
+    livenessProbe:
+      enabled: true
+    readinessProbe:
+      enabled: true
@@ -0,0 +1,118 @@
+---
+# Advertises the on-board Intel iGPU as the schedulable resource
+# `gpu.intel.com/i915`, so pods can request it without a /dev/dri hostPath or
+# any privilege escalation — the plugin hands the render node to the container
+# through the kubelet device-plugin API. The same resource covers both GPU
+# compute (LLM inference) and QuickSync video transcoding; it is not
+# compute-specific.
+#
+# All three nodes are MS-01 with an i9-12900H (Alder Lake-P, PCI 8086:46a6,
+# Iris Xe 96 EU) at 0000:00:02.0. Talos carries the driver in the `i915` system
+# extension, already present in the schematic, so no machineconfig change is
+# needed.
+#
+# Vendored from upstream rather than installed via the
+# `intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
+# custom resource, which only works alongside the Intel Device Plugin Operator
+# and its CRDs, cert-manager webhook and controller. With one device type on
+# three identical nodes, the operator manages nothing this DaemonSet does not.
+#
+# Source: intel/intel-device-plugins-for-kubernetes, v0.36.0,
+# deployments/gpu_plugin/base/intel-gpu-plugin.yaml
+# Upgrading means re-syncing this file against that path at the new tag and
+# bumping the image tag below.
+#
+# Namespace is kube-system because the plugin mounts hostPaths (/dev/dri,
+# /sys/class/drm, the kubelet device-plugin socket and the CDI path). The
+# cluster enforces PodSecurity `baseline`, which forbids hostPath volumes, and
+# kube-system is the only namespace in the API server's exemption list.
+#
+# Node Feature Discovery is not deployed, so upstream's `nfd_labeled_nodes`
+# overlay is omitted along with its `intel.feature.node.kubernetes.io/gpu`
+# nodeSelector. Every node carries the same iGPU, leaving nothing to discover;
+# the plugin exits cleanly on a node without /dev/dri.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-gpu-plugin
+  namespace: kube-system
+  labels:
+    app: intel-gpu-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-gpu-plugin
+  updateStrategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+  template:
+    metadata:
+      labels:
+        app: intel-gpu-plugin
+    spec:
+      containers:
+        - name: intel-gpu-plugin
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+            - name: HOST_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.hostIP
+          image: intel/intel-gpu-plugin:0.36.0
+          imagePullPolicy: IfNotPresent
+          args:
+            # Slots the single iGPU is handed out as, letting that many
+            # containers share it concurrently. A slot count, not fractional
+            # accounting — nothing partitions the device between sharers, so
+            # they contend for it. Sized for an inference server plus room for
+            # a transcoder and a benchmark pod alongside it.
+            - -shared-dev-num=4
+          securityContext:
+            seLinuxOptions:
+              type: "container_device_plugin_t"
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+            seccompProfile:
+              type: RuntimeDefault
+          resources:
+            requests:
+              memory: "45Mi"
+              cpu: 40m
+            limits:
+              memory: "90Mi"
+              cpu: 100m
+          volumeMounts:
+            - name: devfs
+              mountPath: /dev/dri
+              readOnly: true
+            - name: sysfsdrm
+              mountPath: /sys/class/drm
+              readOnly: true
+            - name: kubeletsockets
+              mountPath: /var/lib/kubelet/device-plugins
+            - name: cdipath
+              mountPath: /var/run/cdi
+      volumes:
+        - name: devfs
+          hostPath:
+            path: /dev/dri
+        - name: sysfsdrm
+          hostPath:
+            path: /sys/class/drm
+        - name: kubeletsockets
+          hostPath:
+            path: /var/lib/kubelet/device-plugins
+        - name: cdipath
+          hostPath:
+            path: /var/run/cdi
+            type: DirectoryOrCreate
+      nodeSelector:
+        kubernetes.io/arch: amd64
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - daemonset.yaml
@@ -10,6 +10,7 @@ resources:
  - cert-manager/
  - snapshot-controller/
  - metrics-server/
+  - intel-gpu-plugin/
  - crowdsec/
  - traefik/
  - rook-ceph/
@@ -118,43 +118,76 @@ spec:
    # simultaneously — every PG inactive until they return. Binds any change
    # that rewrites the OSD deployment spec, not just version upgrades.
    osdMaxUpdatesInParallel: 1
-    useAllNodes: false
+    # Which machines carry storage is declared by the machines, as the node
+    # label storage.renner.uno/ceph-osd (talos/patches/k8s-*.yaml), and read
+    # here through placement.osd.nodeAffinity below. Listing hostnames here
+    # instead said the same thing in a second place and tied this file to three
+    # specific boxes: a cluster built anywhere else got no OSDs at all, however
+    # its machines were equipped.
+    #
+    # useAllNodes only means "do not keep a node list here"; nodeAffinity is
+    # what restricts OSDs, and without it this would be every node.
+    useAllNodes: true
    useAllDevices: false
-    nodes:
-      - name: k8s-1
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
-      - name: k8s-2
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
-      - name: k8s-3
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
+    # Two forms while the production drives are migrated:
+    #
+    #   by-partlabel/r-ceph-osd  the target. Talos names the drive in the
+    #                            machine config (RawVolumeConfig ceph-osd), so
+    #                            this file needs to know nothing about models,
+    #                            serials or hostnames.
+    #   by-id/nvme-SAMSUNG…      the three MS-01 drives as they are today:
+    #                            bluestore written straight onto the raw disk,
+    #                            with no partition table and therefore no
+    #                            partition label to match. Matching by model
+    #                            rather than by the three individual serials it
+    #                            replaced — a drive swap is no longer a repo
+    #                            edit. It cannot match the Kingston system disk
+    #                            or the Crucial that carries EPHEMERAL.
+    #
+    # The second half goes when those three are repartitioned, one node at a
+    # time. Until then it is what keeps the running OSDs claimed.
+    devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
    config:
      osdsPerDevice: "1"
  # Highest built-in class: kubelet evicts these last and the kernel gives
  # them the lowest oom_score_adj.
  priorityClassNames:
    all: system-node-critical
-  # requests == limits gives Guaranteed QoS. CPU limits are deliberately NOT
-  # set — Ceph daemons burst during recovery and a limit would throttle them.
+  # Requests are what a daemon actually uses; limits stay where they were, as
+  # burst headroom. The two were equal, for Guaranteed QoS, and the gap was
+  # large enough to matter on machines where memory is the scarce resource:
+  # measured against the running cluster, mon sat at ~420Mi against a 2Gi
+  # request, mgr at ~650Mi against 4Gi, and the OSDs at ~2.2Gi against 4Gi —
+  # about 17GiB reserved cluster-wide and never used. Only the requests moved.
+  #
+  # Guaranteed QoS is given up deliberately. What it bought was eviction order,
+  # and priorityClassNames below already pins these to system-node-critical,
+  # which the kubelet weighs first.
+  #
+  # CPU limits are deliberately NOT set — Ceph daemons burst during recovery and
+  # a limit would throttle them.
  resources:
    mon:
      requests:
        cpu: 500m
-        memory: 2Gi
+        memory: 512Mi
      limits:
        memory: 2Gi
    osd:
+      # The OSD request is not simply "what it uses": an OSD grows its cache
+      # towards Ceph's own osd_memory_target, which defaults to 4Gi, so the
+      # measured 2.2Gi is a floor and the 4Gi limit is what that tunable implies.
+      # Lower the limit without lowering osd_memory_target and the OSD gets
+      # OOM-killed once the cache fills.
      requests:
        cpu: 500m
-        memory: 4Gi
+        memory: 2560Mi
      limits:
        memory: 4Gi
    mgr:
      requests:
        cpu: 250m
-        memory: 4Gi
+        memory: 1Gi
      limits:
        memory: 4Gi
  placement:
@@ -162,3 +195,16 @@ spec:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
+    # The other half of selecting OSD nodes by label: with useAllNodes: true
+    # this is the only thing keeping OSDs off machines that carry no OSD drive.
+    # Removing it does not fall back to the old node list — it puts OSDs
+    # everywhere.
+    osd:
+      nodeAffinity:
+        requiredDuringSchedulingIgnoredDuringExecution:
+          nodeSelectorTerms:
+            - matchExpressions:
+                - key: storage.renner.uno/ceph-osd
+                  operator: In
+                  values:
+                    - "true"
@@ -27,7 +27,9 @@ machine:
    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
  features:
    rbac: true
-    stableHostname: true
+    # stableHostname is NOT here: it is the v1alpha1 spelling of what the
+    # generated HostnameConfig document already says with `auto: stable`.
+    # Asking for the same thing in both schemas is what Talos rejects.
    apidCheckExtKeyUsage: true
    diskQuotaSupport: true
    kubePrism:
@@ -39,13 +41,12 @@ machine:
  nodeLabels:
    node.kubernetes.io/exclude-from-external-load-balancers: ""
  kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without
-    # this, kubelet picks the first non-loopback IP, which could land on
-    # the v30iot NIC (ens19, 10.2.30.0/24). Control-plane traffic +
-    # hostNetwork pod IPs must stay on mgmt LAN.
-    nodeIP:
-      validSubnets:
-        - 10.2.40.0/24
+    # nodeIP.validSubnets is NOT here: it names a network this cluster happens
+    # to sit on, not something true of every cluster built from this file, so it
+    # lives in the site patch (site-homelab.yaml). Here it forced every machine
+    # onto 10.2.40.0/24 whether or not it has an interface there, which is what
+    # made this file unusable anywhere but on the three production boxes.
+    #
    # Lower than the kubelet defaults (85/80) to bound containerd's image
    # cache, the largest single consumer of /var.
    #
@@ -57,8 +58,11 @@ machine:
      imageGCHighThresholdPercent: 70
      imageGCLowThresholdPercent: 60
 cluster:
-  controlPlane:
-    endpoint: https://k8s.renner.uno:6443
+  # controlPlane.endpoint is NOT here: `talosctl gen config` takes it as its
+  # second argument, and bootstrap.sh already passes https://k8s.renner.uno:6443
+  # there. Setting it again in this patch restated the same value in a second
+  # place and overrode whatever endpoint a caller had generated for, which is the
+  # only reason a config generated for a different cluster could not be used.
  apiServer:
    certSANs:
      - k8s.renner.uno
@@ -75,4 +79,14 @@ cluster:
    serviceSubnets:
      - 10.96.0.0/12
      - fd10:96::/112
+    # Talos ships no CNI and no kube-proxy; Cilium provides both. These two
+    # keys used to live in the deleted cilium-cni.yaml alongside the rendered
+    # manifest, which is why they moved here when Cilium became Flux-only —
+    # without them a freshly generated config comes up on Talos's default CNI
+    # with kube-proxy running, which collides with the chart's
+    # kubeProxyReplacement: true. See PROPOSAL-cilium-single-owner.md.
+    cni:
+      name: none
+  proxy:
+    disabled: true
  allowSchedulingOnControlPlanes: true
@@ -2,8 +2,6 @@
 # machine.network.interfaces. Keep every link in one schema: the docs do not
 # define what happens when both configure the same link.
 machine:
-  network:
-    hostname: k8s-1
  sysctls:
    # ens19 (VLAN 30, v30iot) is LINK ONLY — it is the macvlan parent for the
    # v30iot pod attachments and must never get host addressing. A connected
@@ -26,9 +24,28 @@ machine:
    - k8s-1.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: jupiter
+    # This machine holds a Ceph OSD drive. The CephCluster selects OSD nodes by
+    # this label instead of listing hostnames, so which boxes carry storage is
+    # stated once, here, by the machine that knows.
+    storage.renner.uno/ceph-osd: "true"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
+# The hostname, in the schema talosctl generates rather than in
+# machine.network.hostname. `talosctl gen config` always emits a HostnameConfig
+# document, and naming the host in both schemas is what Talos rejects — at apply
+# time, with `static hostname is already set in v1alpha1 config`, not merely in
+# validate. bootstrap.sh used to cut the generated document back out with awk to
+# keep the v1alpha1 spelling working; setting the name here instead means there
+# is nothing to cut.
+#
+# `auto: off` switches off the generated `auto: stable`, which is otherwise
+# mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: k8s-1
+---
 # EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
 # contents are reconstructible, so a worn disk is consumable here in a way it
 # is not as an OSD. Note /var/lib/etcd lives here too, which is the one thing
@@ -2,8 +2,6 @@
 # to k8s-1.yaml — see that file for the reasoning behind each choice; only the
 # serials, MACs and addresses differ.
 machine:
-  network:
-    hostname: k8s-2
  sysctls:
    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It is the
    # macvlan parent for the v30iot pod attachments; host addresses on that L2
@@ -23,9 +21,18 @@ machine:
    - k8s-2.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: saturn
+    # Holds a Ceph OSD drive — see k8s-1.yaml.
+    storage.renner.uno/ceph-osd: "true"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
+# Hostname in the generated schema, not machine.network.hostname — see
+# k8s-1.yaml for why `auto: off` is set alongside it.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: k8s-2
+---
 # EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
 # contents are reconstructible, so a worn disk is consumable in this role.
 #
@@ -1,8 +1,6 @@
 # Structurally identical to k8s-1.yaml — see that file for the reasoning
 # behind each choice; only the serials and addresses differ.
 machine:
-  network:
-    hostname: k8s-3
  sysctls:
    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
@@ -19,9 +17,18 @@ machine:
    - k8s-3.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: neptune
+    # Holds a Ceph OSD drive — see k8s-1.yaml.
+    storage.renner.uno/ceph-osd: "true"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
+# Hostname in the generated schema, not machine.network.hostname — see
+# k8s-1.yaml for why `auto: off` is set alongside it.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: k8s-3
+---
 # EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
 # contents are reconstructible, so a worn disk is consumable in this role.
 #
@@ -0,0 +1,16 @@
+# Facts about the network this cluster is wired into, as opposed to facts about
+# the cluster (cluster.yaml) or about one machine (k8s-1.yaml, vm-1.yaml).
+#
+# Applied to all three production nodes and to nothing else. A cluster brought up
+# somewhere else — a test VM on the provisioner's network — takes its own site
+# patch instead, which is the whole point of separating this out: it is the only
+# part of the config that could not be true anywhere but here.
+machine:
+  kubelet:
+    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
+    # kubelet picks the first non-loopback IP, which could land on the v30iot
+    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
+    # must stay on mgmt LAN.
+    nodeIP:
+      validSubnets:
+        - 10.2.40.0/24
@@ -0,0 +1,15 @@
+# The site patch for a cluster on the QEMU provisioner's network, used in place
+# of site-homelab.yaml when a rebuild is tried on a VM rather than on the three
+# MS-01 boxes.
+#
+# This file is the complete list of what a test run assumes about its network.
+# Everything else it uses — cluster.yaml and a node patch — is the same file a
+# real rebuild reads. If this file ever grows beyond network facts, the test has
+# started proving less than it appears to.
+machine:
+  kubelet:
+    # The provisioner's default CIDR (talosctl cluster create --cidr). One NIC,
+    # no VLANs, so there is nothing to keep kubelet away from.
+    nodeIP:
+      validSubnets:
+        - 10.5.0.0/24
@@ -0,0 +1,45 @@
+# Node patch for a virtual test node, alongside k8s-1/2/3 and in the same shape.
+#
+# It exists so a rebuild can be tried against the real cluster.yaml instead of a
+# hand-picked extract of it. An extract tests the extract: an earlier attempt
+# dropped allowSchedulingOnControlPlanes and every Flux controller went
+# unschedulable, which says nothing about the config a recovery would use.
+#
+# No chassis-serial marker, unlike the k8s-* patches. Virtual machines report no
+# serial at all — `talosctl get systeminformation` has no serialnumber field —
+# so bootstrap.sh's patch_for_machine cannot identify one. This file is named
+# explicitly by the test harness.
+#
+# Everything k8s-1.yaml carries that is missing here is missing because the
+# hardware is:
+#
+#   no LinkConfig / VLANConfig  the VM has one NIC. There is no X710 trunk to
+#                               tag, no v30iot macvlan parent to keep
+#                               addressless, and no Ceph cluster network.
+#   no Layer2VIPConfig          the shared control-plane VIP is elected between
+#                               three machines; one node has nothing to share it
+#                               with.
+#   no VolumeConfig EPHEMERAL   that moves /var onto a retired OSD drive. One
+#                               disk here, so /var stays on the system disk.
+machine:
+  install:
+    # Named directly rather than selected by serial. The serial selector on the
+    # real nodes guards against the installer taking the Ceph OSD drive; with a
+    # single disk there is nothing to pick wrongly.
+    disk: /dev/vda
+  certSANs:
+    - vm-1
+cluster:
+  allowSchedulingOnControlPlanes: true
+---
+# The hostname, in the schema talosctl generates rather than in
+# machine.network.hostname. `talosctl gen config` always emits a HostnameConfig
+# document, and naming the host in both schemas is what Talos rejects at
+# apply time with `static hostname is already set in v1alpha1 config`.
+#
+# `auto: off` switches off the generated `auto: stable`, which is otherwise
+# mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: vm-1
@@ -0,0 +1,37 @@
+# Virtual test worker carrying a Ceph OSD device.
+#
+# A worker rather than a control plane for one reason: the QEMU provisioner
+# hands out only the first disk to control planes ("disks after the first one
+# are added only to worker machines"), and an OSD needs a second one. Ceph does
+# not care which role a node has — mons and OSDs schedule on workers fine.
+machine:
+  install:
+    disk: /dev/vda
+  certSANs:
+    - vm-2
+  nodeLabels:
+    # This machine carries a Ceph OSD device. The CephCluster selects OSD nodes
+    # by this label rather than by hostname, so which machines hold storage is
+    # stated once, by the machine that has it.
+    storage.renner.uno/ceph-osd: "true"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: vm-2
+---
+# The OSD device, named by the machine that has it. Ceph then consumes
+# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
+# serials or hostnames — the same CephCluster works here and on the MS-01s.
+#
+# The selector is a CEL expression, so each machine says whatever identifies its
+# own drive. Here that is "the virtio disk that is not the system disk"; on the
+# real nodes it will name the Samsung.
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: disk.transport == "virtio" && !system_disk
+  minSize: 5GiB
+  grow: true
@@ -0,0 +1,23 @@
+# Virtual test worker carrying a Ceph OSD device. Identical to vm-2.yaml — see
+# that file for the reasoning; only the hostname differs.
+machine:
+  install:
+    disk: /dev/vda
+  certSANs:
+    - vm-3
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: vm-3
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: disk.transport == "virtio" && !system_disk
+  minSize: 5GiB
+  grow: true
@@ -0,0 +1,27 @@
+# Virtual test worker carrying a Ceph OSD device. Identical to vm-2.yaml — see
+# that file for the reasoning; only the hostname differs.
+#
+# Three OSD workers because the CephCluster runs three mons with
+# allowMultiplePerNode: false and its pools are replica 3 over failureDomain
+# host — two nodes would never reach quorum or place a third replica.
+machine:
+  install:
+    disk: /dev/vda
+  certSANs:
+    - vm-4
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: vm-4
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: disk.transport == "virtio" && !system_disk
+  minSize: 5GiB
+  grow: true