fix(monitoring): unconfine seccomp so smartctl can open the devices

The exporter deployed cleanly, ran 3/3 with no restarts, answered its liveness
probe, and produced no drive metrics at all. Only smartctl_devices and
smartctl_exporter_build_info reached Prometheus — a green DaemonSet measuring
nothing, which is the failure mode the dashboard's own caveats panel warns
about.

The logs said why: it found all three drives per node and could open none of
them, with 'Smartctl open device: /dev/nvme0 failed: Operation not permitted'.

EPERM rather than EACCES is the tell — it is the default seccomp action for a
syscall a profile does not list. Rather than guess between that and the
capabilities, I read /proc/1/status in the running container: CapEff was
0x220000, so SYS_ADMIN and SYS_RAWIO were both effective, while Seccomp was 2
with one filter loaded. The capabilities were never the problem. The kubelet
runs with seccompDefault: true, so every pod that does not say otherwise gets
RuntimeDefault.

SELinux was ruled out the same way: /sys/fs/selinux/enforce reads 0.

This stays narrower than privileged: true, which would additionally grant every
capability, all host devices and write access to /sys. The tighter option is a
custom seccomp profile allowing exactly the ioctl smartctl issues; that is worth
building if this set of exceptions ever grows, and is not worth it for one pod.

Refs #549

.woodpecker/alert-monitor.yml

@@ -101,7 +101,7 @@ steps:
         bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/build-ci-images.yml

@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:
 
 steps:
   - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/ansible
     depends_on: []
     # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
 
   - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/k8s
     depends_on: []
     when:
@@ -77,7 +88,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
     depends_on: []
     when:
@@ -108,7 +119,7 @@ steps:
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
   - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: secret-rotator
     depends_on: []
     when:
@@ -137,7 +148,7 @@ steps:
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
   - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
     depends_on: []
     when:
@@ -166,7 +177,7 @@ steps:
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
   - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
     depends_on: []
     when:
@@ -195,7 +206,7 @@ steps:
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
   - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
     depends_on: []
     when:
@@ -224,7 +235,7 @@ steps:
   # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
   # builds the same way and lands in the same registry so the audit sees it.
   - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/heating-can
     depends_on: []
     when:

.woodpecker/ci-claude-refresh.yml

@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
       - |
         set -eu
         mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
         NEW=$$(npm view @anthropic-ai/claude-code version)
         case "$$NEW" in
           [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
   #    --build-arg to the resolved version (not "latest") guarantees the tag and
   #    the image content agree even if a new release lands mid-run.
   - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/claude
     depends_on: [decide]
     environment:
@@ -176,13 +186,10 @@ steps:
           echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
           exit 0
         fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
         # Exact-string swap of the current tag for the new one, consumers only.
         for f in $$CONSUMERS; do
           sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
         {
           printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
           printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
           printf 'run the image so CI stops running a stale Claude Code.\n\n'
           printf 'Merging is a human decision — review, then merge to roll it out.\n'
         } > /tmp/pr-body.md

.woodpecker/ci-scripts-test.yml

@@ -17,6 +17,6 @@ when:
 
 steps:
   - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh

.woodpecker/e2e.yml

@@ -25,7 +25,7 @@ steps:
     # fetch-models.sh response mapping. No cluster, no secrets, no network — the
     # ci-claude image just provides sh + jq. Keeps the flag-injection regression
     # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh
 

.woodpecker/issue-fix.yml

@@ -62,7 +62,7 @@ steps:
       - bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/pr-review.yml

@@ -65,7 +65,7 @@ when:
 
 steps:
   - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/security-scan.yml

@@ -42,7 +42,7 @@ steps:
     # this step is skipped — otherwise we'd fork-bomb PRs.
     when:
       - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/update-audit.yml

@@ -30,7 +30,7 @@ when:
 
 steps:
   - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

kubernetes/MIGRATION-sops-openbao.md

@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```
 
 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.

kubernetes/RUNBOOK-cold-rebuild.md

@@ -1,10 +1,25 @@
 # Cold-rebuild runbook — homelab cluster from scratch
 
 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
+in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
+gone).
 
 Order: **tofu → talos → flux → openbao → apps.**
 
+All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
+are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
+Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
+resources, and physically booting three machines from install media.
+
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
+
 ## The circular-bootstrap problem this runbook exists to break
 
 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -61,12 +76,21 @@ These must exist outside the cluster or the rebuild cannot complete:
       `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
       copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
+      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
+      Proxmox credentials — there is no Proxmox any more.
+- [ ] **Install media** written and to hand: the Image Factory ISO named in
+      Step 1, on a USB stick, for all three machines.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.
 
-## Step 1 — tofu: provision the VMs + supporting infra
+## Step 1a — tofu: DNS, object storage, UniFi
 
-Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
-UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
+OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
+the tree but nothing instantiates it, so `tofu apply` here creates only the
+platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
+records, the Hetzner object storage, and the NAT hairpin rules.
 
 ```sh
 cd opentofu
@@ -74,44 +98,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```
 
-In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
-which also pre-seeds the Talos image onto every Proxmox node. Wait until the
-node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
-Step 2 — `bootstrap.sh` polls DNS but will not create the records.
+Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
+FQDNs to resolve — the control-plane endpoint is baked into every generated
+config, and the nodes are addressed by name.
 
-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
+## Step 1b — boot all three machines from install media
 
 ```
-https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```
 
-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
+Write it to a USB stick and boot each MS-01 from it. Each comes up in
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.
 
-- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
-- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
-- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
+The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
+interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
+against do not exist, and the node ends up with no address on any interface.
+Beyond that it carries:
+
+- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
+  today; present so wanting hardware transcoding later is a config change
+  rather than another rolling reinstall
+- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
+  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
+  rule auto-authorises any Thunderbolt device attached once those modules load
+- no `qemu-guest-agent`, which has nothing to talk to on bare metal
+
+Verify the disks before going further. Talos enumerates NVMe in an order that
+is not stable across boots, and the per-node configs select by serial for
+exactly that reason — the failure mode is the installer taking the Ceph OSD
+drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
+NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
+in `talos/patches/k8s-N.yaml`.
 
 ## Step 2 — talos: form the cluster + get kubeconfig
 
+**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
+config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
+`$patch: delete` document fails against a base with no `inlineManifests`
+(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
+and 6; only config generation is broken.
+
+The procedure below is what actually works, and is the same one
+`RUNBOOK-baremetal-conversion.md` uses.
+
+### Generate the configs
+
+This is a genuinely new cluster — nothing survives to copy secrets from, so
+generate them fresh. (In a *partial* rebuild, where one node still runs, pull
+secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
+--from-controlplane-config`, or the new node forms its own cluster rather than
+joining.)
+
 ```sh
-kubernetes/bootstrap.sh talos
+talosctl gen secrets -o secrets.yaml
+
+for N in 1 2 3; do
+  talosctl gen config homelab https://k8s.renner.uno:6443 \
+    --with-secrets secrets.yaml --output-types controlplane \
+    --config-patch @talos/patches/cluster.yaml \
+    --config-patch @<cilium additive half> \
+    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
+done
 ```
 
-Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
-writes `~/.talos/config` + `~/.kube/config`, and stages both into
-`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
+Two things the generator gets wrong for this repo, both of which will bite
+silently:
+
+- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
+  second document. See issue #581.
+- **Strip the generated `HostnameConfig`** — it collides with
+  `machine.network.hostname` from the per-node patch.
+
+Validate each before applying:
+
+```sh
+talosctl validate --config install-k8s-$N.yaml --mode metal
+```
+
+### Apply in two stages
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
+and nothing times out. Concretely, the `ens19` VLAN document must be removed
+from the first-stage config — its parent exists, but the install stalls anyway.
+
+```sh
+# Stage 1: minimal config, from the maintenance-mode address off the console
+talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
+# Stage 2: once the node is up on management, apply the complete config
+talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
+```
+
+`--mode=try` rolls back on its own if the node becomes unreachable, which is the
+only safety net there is for a network change that can strand a machine with no
+IPMI. Use it for every network-affecting apply, then make it permanent once
+proven.
+
+### Bootstrap etcd, then collect access
+
+Bootstrap **exactly one** node:
+
+```sh
+talosctl -n k8s-1.srv.renner.uno bootstrap
+talosctl -n k8s-1.srv.renner.uno kubeconfig .
+```
+
+Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
+supported way to get the kubeconfig into place afterwards.
 
 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)
 
@@ -149,8 +248,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).
 
 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods
 
 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +257,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
 
-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```
 
 Verify the store recovers before moving on:
@@ -170,7 +273,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```
 
 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.
 
 ## Step 5 — apps: let the rest reconcile
 
@@ -225,9 +330,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
   in-cluster commits made **after** Flux/Gitea broke are not on the mirror
   (Gitea couldn't push them if Gitea itself was down). This is the intended
   recovery point, but be aware of it when reconciling drift.
-- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
-  key must not live in CI. See issue #187 for the open question of whether to
-  wire a semi-automated OpenBao restore.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
+- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
+  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
+  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
+  It has also never had an `openbao` phase — Step 4 is manual because the
+  unseal key must not live in CI; see issue #187 on whether to wire a
+  semi-automated OpenBao restore.
+- **OpenBao seals on every restart and each replica must be unsealed
+  individually.** Three raft replicas mean the service survives losing one, but
+  a sealed server is neither a voter nor a standby — unsealing only one leaves
+  the cluster short of quorum. Step 4 covers this; it is repeated here because
+  it is the step most likely to be half-done under pressure.
+- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
+  IPAM, not host networking. Nothing in the rebuild needs special handling for
+  it — the nodes hold no address on that segment — but a rebuilt cluster will
+  not have the whereabouts allocations from before, which is harmless since
+  they are assigned fresh.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
   Ceph/infra bring-up; always read the phase output rather than trusting the
   exit code.

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -6,12 +6,14 @@ kind: Kustomization
 resources:
   - prometheusrule-ceph.yaml
   - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
   - prometheusrule-mariadb-backup.yaml
   - prometheusrule-netbird-availability.yaml
   - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
   # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -23,4 +25,5 @@ resources:
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
   - probes-blackbox.yaml

kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml

@@ -0,0 +1,121 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # THE GAP THESE RULES CLOSE
+        #
+        # Drive wear is invisible in every other signal this cluster has.
+        # `ceph -s` reads active+clean at any wear level, no Ceph health check
+        # fires until a device actually fails, and the block-layer metrics show
+        # throughput and latency but never endurance. That blind spot is how a
+        # set of OSD drives reached 81-87% before anyone looked (#549, #434).
+        #
+        # The numbers below are wear percentages reported by the drive itself:
+        # 100 means the manufacturer's rated endurance is used up. Drives do
+        # not stop at 100 — they typically keep working and start consuming
+        # spare blocks — so treat these as planning thresholds, not failure
+        # predictions.
+        - alert: NvmeDriveWearHigh
+          # 80% with months of runway left is the point where ordering a
+          # replacement is still a calm decision rather than an outage.
+          # `for` is long because wear is monotonic and slow: it cannot
+          # flap, and a short window would only make the alert noisier
+          # after a restart.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive has used {{ $value }}% of its rated write endurance.
+              Nothing is failing and no data is at risk yet. Check what the
+              drive is carrying before deciding urgency: an OSD is replaceable
+              with a rebalance, whereas the EPHEMERAL volume has no such
+              escape — a node simply stops when it goes. The Disk Health
+              dashboard resolves which drive holds which role.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # A drive that has consumed its spare area is no longer masking media
+        # defects, which is a materially different state from being worn:
+        # wear is a budget, spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          # The NVMe critical warning byte is a bitfield: spare below
+          # threshold, temperature out of range, degraded reliability,
+          # read-only mode, volatile-memory backup failed. Any non-zero value
+          # is the drive itself asking for attention.
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # Unlike wear, this counter should never move. Alert on the increase
+          # rather than the absolute value, so a drive that recorded errors
+          # long ago does not alert forever while a drive that starts failing
+          # today does.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # Without this the whole group is silently useless: no series means no
+        # alert fires, and a dashboard of empty panels reads much like a
+        # healthy one. This is the check that the checks are running.
+        - alert: SmartctlExporterIncomplete
+          # Compared against node-exporter's own NVMe inventory rather than a
+          # written-down fleet size. Adding or removing a drive then moves both
+          # sides together, so this cannot rot into a permanent false alarm the
+          # way a hardcoded count would.
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.

kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml

@@ -40,17 +40,31 @@ spec:
             description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
 
         - alert: VeleroBackupStale
-          # Daily schedule with no successful backup in over 30h: covers
-          # the case where multiple consecutive runs failed (Garage flapping
-          # over the backup window) without the per-event alerts being
-          # acted on.
+          # Daily schedule with no successful backup in over 60h: covers the
+          # case where consecutive runs failed without the per-event alerts
+          # being acted on.
+          #
+          # 60h, not 30h. At 30h a single missed cycle pages, and a single
+          # missed cycle is the normal cost of touching the nodes: Velero moves
+          # data through pods, so draining a node kills whatever upload is in
+          # flight and the schedule is simply lost for that day — it is not
+          # retried. Rolling node work between 03:15 and 08:45 UTC therefore
+          # produced a guaranteed critical alert for every schedule it
+          # overlapped, which is noise about planned work rather than a signal
+          # about backups.
+          #
+          # 60h still catches what this is for. Two consecutive failures is no
+          # longer explicable by one maintenance window, and the per-event
+          # VeleroBackupFailed alert fires on the first failure regardless — so
+          # nothing is hidden, only the escalation to critical is deferred
+          # until a second cycle confirms it.
           expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
             and
             velero_backup_last_successful_timestamp{schedule!=""} > 0
           for: 10m
           labels:
             severity: critical
           annotations:
-            summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
-            description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
+            summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
+            description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."

kubernetes/apps/monitoring-rules/smartctl-exporter-servicemonitor.yaml

@@ -0,0 +1,29 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl itself polls the drives every 60s, so scraping faster than that
+    # returns the same numbers with a new timestamp. Wear moves on a scale of
+    # weeks; 60s is already far finer than the signal.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this every series is labelled with the exporter pod's IP and
+        # nothing else. A DaemonSet pod's IP says nothing about which machine
+        # the drive is in, and it changes on every reschedule — so the wear
+        # figure would be unattributable, which is most of its value gone.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node

kubernetes/apps/monitoring/dashboards/disk-health.json

@@ -0,0 +1,382 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
+      ],
+      "title": "OSDs down / out",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
   - name: dashboard-gitea-overview
     files:
       - gitea-overview.json

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -275,8 +275,11 @@ spec:
           - alertmanager-smtp
       # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
       # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
       # monitoring ns is not blocked by any CNP today.
       config:
         global:

kubernetes/apps/monitoring/kustomization.yaml

@@ -15,6 +15,7 @@ resources:
   - prometheus-gate-service.yaml
   - prometheus-gate-ingress.yaml
   - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
   - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the

kubernetes/apps/monitoring/smartctl-exporter.yaml

@@ -0,0 +1,142 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# WHY THIS HOLDS ELEVATED CAPABILITIES, and why that is a deliberate exception
+# rather than an oversight:
+#
+# NVMe reports health over an admin ioctl on the character device
+# (NVME_IOCTL_ADMIN_CMD), which the kernel gates on CAP_SYS_ADMIN. There is no
+# sysfs path for wear, media errors or written-bytes totals — hwmon publishes
+# the controller temperature and nothing else. Upstream smartctl_exporter
+# documents `privileged: true` and `user: root` for exactly this reason.
+#
+# The alternative that needs no privileges — reading Ceph's devicehealth
+# archive, which does hold full smartctl output — covers OSD drives only. In
+# this cluster that is three of nine, and the six it misses include the drives
+# that were retired from Ceph for wear and now carry EPHEMERAL. Covering the
+# healthy drives and not the worn ones was judged not worth a new service; the
+# owner chose fleet-wide coverage with the privilege exception instead.
+#
+# The exception is kept as small as the kernel allows: no `privileged: true`,
+# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are
+# added, everything else is dropped, and the only host mount is /dev — which
+# is required because the exporter must open the NVMe character devices and
+# Kubernetes has no way to pass individual character devices to a pod without
+# a device plugin.
+#
+# If this ever needs revisiting: the thing to check is whether the kernel has
+# grown a sysfs interface for NVMe health, not whether some other exporter
+# needs fewer privileges. The requirement comes from the ioctl gate, not from
+# the tool.
+#
+# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
+# The repository is not archived; the pin is simply its newest tag.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device on purpose: the exporter scans for devices
+          # itself and rescans every 10m by default. Device names are not
+          # stable across these machines, so a hardcoded list would quietly
+          # stop covering a drive the next time one is replaced.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            # Root plus two capabilities, NOT privileged. See the header.
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: false
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["SYS_ADMIN", "SYS_RAWIO"]
+            # The capabilities alone are not enough, and this was measured
+            # rather than assumed. With the default profile the exporter found
+            # all three drives and could open none of them:
+            #   "Smartctl open device: /dev/nvme0 failed: Operation not
+            #   permitted"
+            # EPERM is the default seccomp action for a syscall the profile
+            # does not list. Reading /proc/1/status in the running container
+            # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN
+            # and SYS_RAWIO were both effective — while Seccomp was 2 with one
+            # filter loaded. The kubelet sets seccompDefault: true, so every
+            # pod that does not say otherwise gets RuntimeDefault.
+            #
+            # This is still narrower than `privileged: true`, which would also
+            # grant every capability, all devices and write access to /sys.
+            # Narrowing further would mean shipping a custom seccomp profile
+            # that allows exactly the ioctl smartctl needs — worth doing if
+            # this list of exceptions ever grows, not worth it for one pod.
+            seccompProfile:
+              type: Unconfined
+          volumeMounts:
+            - name: dev
+              mountPath: /dev
+              readOnly: true
+            # smartctl is not known to need scratch space, but the root
+            # filesystem is read-only and a tool that turns out to want /tmp
+            # would crashloop instead of degrading. Cheaper to provide it than
+            # to find out from a restart counter.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP

kubernetes/apps/music-assistant/deployment.yaml

@@ -55,7 +55,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -65,7 +65,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml

@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.

kubernetes/infrastructure/openbao/helmrelease.yaml

@@ -42,12 +42,28 @@ spec:
       # the agent-injector mutating webhook is unused surface.
       enabled: false
     server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
       ha:
         enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
         raft:
           enabled: true
           setNodeId: true
@@ -65,8 +81,32 @@ spec:
               cluster_address = "[::]:8201"
             }
 
+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
             storage "raft" {
               path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
             }
 
             service_registration "kubernetes" {}

kubernetes/infrastructure/openbao/snapshot-cronjob.yaml

@@ -53,7 +53,13 @@ spec:
               command: ["/bin/sh", "-ec"]
               args:
                 - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                   BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                     role=openbao-snapshot \
                     jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -203,16 +203,16 @@ metadata:
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
+  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
+  # the home-assistant namespace regardless of which app/ dir owns
   # their manifest (matter-server, otbr, whisper, speech-to-phrase,
   # zone-configurator all set `namespace: home-assistant` via their
   # Kustomization, not per-resource): home-assistant-config (.storage
   # entity registry, recorder DB, secrets.yaml), esphome-data,
   # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
+  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
+  # zwavejs-store, zone-configurator-config. On restore,
+  # home-assistant-config has its
   # own path (the HA pod's `seed` initContainer pulling the native HAOS
   # backup from Garage, see the home-assistant Kustomization comment in
   # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
@@ -229,6 +229,20 @@ spec:
     storageLocation: default
     ttl: 720h0m0s
     snapshotMoveData: true
+    # The ha-pg CNPG cluster now runs in this namespace (it did not when
+    # this schedule was written — the old comment here still claimed
+    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
+    # cnpg.io/cluster label and was being swept in: a crash-consistent
+    # snapshot of a running Postgres data directory is not a restorable
+    # backup, the same trap the wordpress-daily comment below describes.
+    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
+    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
+    # continuous WAL archiving — so this exclusion removes a redundant
+    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -8,6 +8,17 @@ spec:
     image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
   mon:
     count: 3
     allowMultiplePerNode: false
@@ -36,26 +47,27 @@ spec:
   # (paperless + audiobookshelf + immich + ... all clone at once).
   cephConfig:
     mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
       #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
       # Stop compacting the whole RocksDB prefix on every paxos trim.
       #
       # Upstream default is true — "Compact a certain prefix (including paxos)
@@ -147,19 +159,65 @@ spec:
     ssl: false
   monitoring:
     enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
   network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
   # Nodes are listed explicitly rather than useAllNodes, so that a node can be
   # taken out of the storage fleet by removing it here. Under useAllNodes the
   # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +251,15 @@ spec:
   #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
   #   S437NF0M501176    PM983     — this OSD
   storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
     useAllNodes: false
     useAllDevices: false
     nodes:

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
   - cephcluster.yaml
   - toolbox.yaml
   - storageclass.yaml

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -0,0 +1,84 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }

kubernetes/talos/patches/cluster.yaml

@@ -42,8 +42,17 @@ machine:
   # Only read at install and upgrade time. Changing it here does nothing to a
   # running node until that node is upgraded. Bump the tag with the Talos
   # version — it must match the running version, or the upgrade is a downgrade.
+  #
+  # Upgrades follow the documented path: migration is only tested between
+  # adjacent minor releases, so go to the latest patch of each intermediate
+  # minor rather than skipping. 1.12.11 -> 1.13.9 satisfies that — one minor
+  # step from the last 1.12 patch, no intermediate needed.
+  #
+  # Kubernetes is a separate decision and is not forced by this. Talos 1.13
+  # supports Kubernetes 1.36, 1.35, 1.34, 1.33, 1.32 and 1.31, so the cluster
+  # stays on 1.35.2 until `talosctl upgrade-k8s` moves it deliberately.
   install:
-    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.12.11
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
   features:
     rbac: true
     stableHostname: true

kubernetes/talos/patches/k8s-1.yaml

@@ -20,6 +20,29 @@ machine:
     # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
     # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are ordinary block devices to Talos here: the Kingston
     # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -117,13 +164,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29

kubernetes/talos/patches/k8s-2.yaml

@@ -9,6 +9,15 @@ machine:
     # macvlan parent for the v30iot pod attachments; host addresses on that L2
     # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -89,10 +98,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29

kubernetes/talos/patches/k8s-3.yaml

@@ -8,6 +8,15 @@ machine:
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29