fix(monitoring): unconfine seccomp so smartctl can open the devices

The exporter deployed cleanly, ran 3/3 with no restarts, answered its liveness probe, and produced no drive metrics at all. Only smartctl_devices and smartctl_exporter_build_info reached Prometheus — a green DaemonSet measuring nothing, which is the failure mode the dashboard's own caveats panel warns about. The logs said why: it found all three drives per node and could open none of them, with 'Smartctl open device: /dev/nvme0 failed: Operation not permitted'. EPERM rather than EACCES is the tell — it is the default seccomp action for a syscall a profile does not list. Rather than guess between that and the capabilities, I read /proc/1/status in the running container: CapEff was 0x220000, so SYS_ADMIN and SYS_RAWIO were both effective, while Seccomp was 2 with one filter loaded. The capabilities were never the problem. The kubelet runs with seccompDefault: true, so every pod that does not say otherwise gets RuntimeDefault. SELinux was ruled out the same way: /sys/fs/selinux/enforce reads 0. This stays narrower than privileged: true, which would additionally grant every capability, all host devices and write access to /sys. The tighter option is a custom seccomp profile allowing exactly the ioctl smartctl issues; that is worth building if this set of exceptions ever grows, and is not worth it for one pod. Refs #549
Merge pull request 'feat(monitoring): expose NVMe endurance, with the privilege exception it needs' (#669 ) from smartctl-exporter into master
2026-08-27 15:03:34 +02:00 · 2026-08-27 12:56:00 +00:00 · 2026-08-27 14:42:18 +02:00 · 2026-08-27 12:25:39 +00:00 · 2026-08-27 14:23:36 +02:00 · 2026-08-27 14:18:11 +02:00
48 changed files with 2447 additions and 195 deletions
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:

 steps:
  - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/ansible
    depends_on: []
    # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-ansible:latest

  - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/k8s
    depends_on: []
    when:
@@ -77,7 +88,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest

  - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/tofu
    depends_on: []
    when:
@@ -108,7 +119,7 @@ steps:
  # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
  # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
  - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: secret-rotator
    depends_on: []
    when:
@@ -137,7 +148,7 @@ steps:
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
  - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-jq
    depends_on: []
    when:
@@ -166,7 +177,7 @@ steps:
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/cni-plugins
    depends_on: []
    when:
@@ -195,7 +206,7 @@ steps:
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-openssl
    depends_on: []
    when:
@@ -224,7 +235,7 @@ steps:
  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
  # builds the same way and lands in the same registry so the audit sees it.
  - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/heating-can
    depends_on: []
    when:
@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
      - |
        set -eu
        mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
        NEW=$$(npm view @anthropic-ai/claude-code version)
        case "$$NEW" in
          [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
  #    --build-arg to the resolved version (not "latest") guarantees the tag and
  #    the image content agree even if a new release lands mid-run.
  - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/claude
    depends_on: [decide]
    environment:
@@ -176,13 +186,10 @@ steps:
          echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
          exit 0
        fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
        # Exact-string swap of the current tag for the new one, consumers only.
        for f in $$CONSUMERS; do
          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
        {
          printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
          printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
          printf 'run the image so CI stops running a stale Claude Code.\n\n'
          printf 'Merging is a human decision — review, then merge to roll it out.\n'
        } > /tmp/pr-body.md
@@ -17,6 +17,6 @@ when:

 steps:
  - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh
@@ -25,7 +25,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -65,7 +65,7 @@ when:

 steps:
  - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -42,7 +42,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -30,7 +30,7 @@ when:

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -0,0 +1,13 @@
+$ANSIBLE_VAULT;1.1;AES256
+32373134633930343261336362373335623735656634313034393731393661643361346634623439
+3561363963333533313264643036663731656465343162330a333964383462373934303766323533
+63613065313063626261383236396363656565623538666565646661353766643133306334396561
+3961376234623236620a616339326238366538333666316538396465666365373834313064653963
+38623233616633383435373836613438333763343932636239386263306238376137643531373161
+35386437313465643437343832623935353238316565366537646530323233363837656631383962
+32653332333162623966366163633732356637323461613966313263313066643966373335313162
+30306362346561633566316166313164323635356363643034663933363464663562343136646137
+66643536363734316533663032643637356136363237376365613330633961616665323430633135
+36323733633464306330666232663433616533373466633939353061663136666665323432373666
+36393930333435333238306238373931393232393861333334643139306138653336633534626362
+61323038303838396438
@@ -1,42 +1,29 @@
 $ANSIBLE_VAULT;1.1;AES256
-39333932383736353162346462663539633763663565366639646463316636333364386365363863
-3034356438663432363366633537373738363061633764300a393236373834323237623434353361
-32346331666234616335663432386163316161616230396265373337613035323863393663333665
-6463623335626238640a306666616366363130363534623032353730313865343537373931343433
-61636634306565356566303431623533623734343466343636376664646539363363383733666638
-39316362393066666435326438323036353735666630306532353433663163303231376565313838
-33616137303261366330653862613934303138653634323063363232303636346336303337333537
-33643030613662343765303638303339336564633063626462656265646536616563303565323266
-36373765663732666436363866623133366337646265616333633536666630396465363438353865
-37383130373532303131646530353236383637613034616433383333333234353137633166663866
-30366265323537623863336461313666656662383535623435306231363763383364306335623938
-34376363633762303664356262383837313866633362663037616335626135376366613336356132
-39323166366437386236386237396236356631623133373866663831656330313936363361656465
-31373663653736653438643165383338333034613139303333373937626666376432613665656564
-61616365663033633438666562363530323862613436663964356339373635356366386264373437
-36646138313838666561646331316265383637313237616632643831303835313663363034633862
-62623064383539366366386634313938323833323163316365356430633531306134366630623261
-35656461306566666531366364323834663737313835666264316132376638333436343136656233
-39343338653264636435383736613763633531356438643464633164633766383435323331633630
-63316436343863383561653631333434613761343863643465366139353262333463646263633566
-38306266393439346265666133343566353164353765366331343631666163613262363135643938
-31643538363136376462633431643135373932373932303435303365383063646236656532643263
-34663637616535373235336537633339303263383035323264303665643562336431303436653435
-37613162353133636361623431356361646433313161316163333766336531363164316334343061
-38303263643263386662383731656537393731356663353861316630353464363932346666323037
-31343933346530306461363665383235303239373234336631316531353862643937393235643134
-32303265306331336665653162383639353634373462666461643463633531373366633438353538
-31376437386538613531323231396437363434663236663635666638373137353434373733633234
-66363732393166633534316439343561383234636661316439366362326631656134346264623432
-35626332646162386236336434393533626435313462303635313439326163636330636133613335
-38636664376166613461316466323161386436396235376534393063663532333538336164643730
-30636137633663666532666466363730633137396363653832643837366266626461376636333736
-33343832326566353264353266376664643061306630633038313534386563616236636438393434
-61643030616531363963313337646263613366663462326561393261383732376231383036303633
-66633164623232343537316163636666653932646266643561303561323536313532666538643031
-61653663613139383234656465393637323031666462663165323138303164313461623832323236
-32373866636432653135623330333464626661396465353133373965376436353138313866633138
-30376339626165386262396139393737343036333437623365313534636239373830663437643531
-63643233663137646335343338663661623431636162306363363839306631356638356462643336
-62643861336131616433666265316538346264376236386434343539343839313130633535616236
-32336632306165376635313130346436663939303432353966323662326335666564
+30353730633631376539373334376531353263396339313464373538396231616166653738383161
+6438656132616365646463653464396236313437323537620a363632643061333036353537626461
+34393065613836343536383565303066326133363937386263653961663666643430663139653961
+6462363235343063350a636165376232366636363233376637636231363839343738346134623562
+63663839326435303964313730336436396365343163343366613966323938656263303166326435
+65303931393233623064343763633161353161643635396365316362333137353732346366643162
+36386132386638623333623839373161386330353635346331393735366662393935353264646439
+36386539393432373765396562613033656339646532323864366230393638313161383363636636
+32353266376563393239393734316461313566386233333164623135303132346164353165323338
+62306333363838393236346435303339393639623231353232643966613163313134336265633636
+64366539326639303362393933303465376532626663346431613563353032383037363532646637
+38376266323537343437643635343634636164393230313632663533666434306134366261383230
+66383135303565383763646331383338646339613539623835366438393635363832366134316531
+63636466653239383231616139663231623636376564376234383762383066613030313937616533
+33396135616236643034303236353230376434313365326633303961303232363766633161353434
+64333832623535343733353462313862343934633363616237353331386264386632643538333939
+34636437393934326131363236346636663665613537343134313636623439356665636663613938
+37346634356164343537396363663730306232623964636635333333653738383538343236383535
+66326462373964333234343435336630643930333433393034303439366631386666393330343238
+61336237373236313533333732343064336637326636663433396362346631343165656163313561
+30383331326231383539343639343130323731333432306630353835343338613861376631646264
+66383539616663376237663234313431313064663566313566366664386439316161353135306338
+35373962636537653535333935306564303235356232313463386237353737313736386463396164
+62666165336539316565623634326562356434313833643833353635303361353937666663356263
+35653665323736386263333437666432623332333936373938376637633463653930326466383139
+38336239646135316565356564373731386662383234386433633632393165343331353835623437
+30663138633430356334353832633434633939373337663437326263336537356633313763353337
+64373764666430643433
@@ -15,7 +15,14 @@
 #
 #   ansible-vault edit inventory/group_vars/all/vault.yml
 #   # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
-#   #        vault_netbird_proxy_token
+#   #        and netbird_turn_secret
+#
+# That last one is easy to miss and breaks the rotation silently. vault.yml
+# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
+# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
+# a group var outranks a role default — so the bare key wins, and rotating only
+# the vault_-prefixed one changes nothing that reaches the template. The check
+# below matches both spellings for that reason.
 #
 # The first rotation then populates netbird-rotatable.yml with fresh values, so
 # nothing needs copying across.
@@ -29,10 +36,24 @@
 #                               cannot disagree across a run.
 #   vault_netbird_relay_secret  management.json on the server and relay.env on
 #                               the proxies. Two hosts, both managed here.
-#   vault_netbird_proxy_token   proxy.env on the server and the proxies.
+#
+# The test for "safe to rotate here" is whether the value appears in
+# management.json.j2. If the management server does not read it from a file
+# Ansible writes, Ansible does not own both ends, whatever else it templates.
 #
 # WHAT IS NOT, and must not be added:
 #
+#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on both
+#                                the server and the proxies, all Ansible
+#                                templates — but the management server does not
+#                                validate it from any of them. It is registered
+#                                in the management DATABASE. Rotating it changes
+#                                only what the proxies present, so every proxy
+#                                is locked out: "proxy auth failed: too many
+#                                failed authentication attempts", and the
+#                                proxies stop receiving their domain
+#                                configuration. Learned by doing it.
+#
 #   vault_netbird_datastore_key  encrypts NetBird's datastore at rest. Changing
 #                                it without re-encrypting the store makes the
 #                                existing data unreadable. Rotating this needs a
@@ -114,7 +135,7 @@
          ansible-vault view
          --vault-password-file {{ playbook_dir }}/../vault-password.sh
          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
-          | grep -cE '^vault_netbird_(turn_secret|relay_secret|proxy_token):'
+          | grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
      register: _stale_keys
      changed_when: false
      # grep exits 1 on "no matches", which is the good case here.
@@ -130,8 +151,8 @@
          rotates. group_vars files merge alphabetically and vault.yml sorts
          last, so it would override netbird-rotatable.yml on the next deploy
          and revert the secrets. Remove vault_netbird_turn_secret,
-          vault_netbird_relay_secret and vault_netbird_proxy_token from
-          vault.yml first (one-time migration), then re-run.
+          netbird_turn_secret and vault_netbird_relay_secret from vault.yml
+          first (one-time migration), then re-run.
        success_msg: vault.yml does not shadow the rotated keys.

  tasks:
@@ -150,7 +171,6 @@
      ansible.builtin.set_fact:
        new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
        new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
-        new_proxy_token: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
      no_log: true

    - name: Render the new values to a temporary file
@@ -163,7 +183,6 @@
          # the next rotation overwrites this file wholesale.
          vault_netbird_turn_secret: "{{ new_turn_secret }}"
          vault_netbird_relay_secret: "{{ new_relay_secret }}"
-          vault_netbird_proxy_token: "{{ new_proxy_token }}"
      no_log: true

    # Encrypting the whole file rather than inline !vault strings: these three
@@ -198,7 +217,6 @@
  vars:
    netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"

  roles:
    - netbird-server
@@ -213,7 +231,6 @@

  vars:
    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"

  roles:
    - netbird-proxy
@@ -227,7 +244,7 @@
    - name: Remaining manual steps
      ansible.builtin.debug:
        msg:
-          - "Rotated: turn_secret, relay_secret, proxy_token."
+          - "Rotated: turn_secret and relay_secret."
          - "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
          - "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
          - "OpenBao is unaffected by design — its NetBird credentials are separate ones."
@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```

 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).
@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.
@@ -31,8 +31,8 @@ all in state that outlived the VM.
      two hosts every PG is undersized with min_size 2, so one further OSD loss
      blocks IO on every CephFS mount cluster-wide.
 - [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      `0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
+      (`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
      Verify by comparing partition offsets and sizes against the ISO with
      `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
@@ -1,10 +1,25 @@
 # Cold-rebuild runbook — homelab cluster from scratch

 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
+in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
+gone).

 Order: **tofu → talos → flux → openbao → apps.**

+All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
+are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
+Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
+resources, and physically booting three machines from install media.
+
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
+
 ## The circular-bootstrap problem this runbook exists to break

 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -61,12 +76,21 @@ These must exist outside the cluster or the rebuild cannot complete:
      `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
      copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
+      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
+      Proxmox credentials — there is no Proxmox any more.
+- [ ] **Install media** written and to hand: the Image Factory ISO named in
+      Step 1, on a USB stick, for all three machines.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.

-## Step 1 — tofu: provision the VMs + supporting infra
+## Step 1a — tofu: DNS, object storage, UniFi

-Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
-UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
+OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
+the tree but nothing instantiates it, so `tofu apply` here creates only the
+platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
+records, the Hetzner object storage, and the NAT hairpin rules.

 ```sh
 cd opentofu
@@ -74,44 +98,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```

-In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
-which also pre-seeds the Talos image onto every Proxmox node. Wait until the
-node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
-Step 2 — `bootstrap.sh` polls DNS but will not create the records.
+Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
+FQDNs to resolve — the control-plane endpoint is baked into every generated
+config, and the nodes are addressed by name.

-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
+## Step 1b — boot all three machines from install media

 ```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```

-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
+Write it to a USB stick and boot each MS-01 from it. Each comes up in
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.

- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
+The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
+interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
+against do not exist, and the node ends up with no address on any interface.
+Beyond that it carries:
+
+- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
+  today; present so wanting hardware transcoding later is a config change
+  rather than another rolling reinstall
+- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
+  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
+  rule auto-authorises any Thunderbolt device attached once those modules load
+- no `qemu-guest-agent`, which has nothing to talk to on bare metal
+
+Verify the disks before going further. Talos enumerates NVMe in an order that
+is not stable across boots, and the per-node configs select by serial for
+exactly that reason — the failure mode is the installer taking the Ceph OSD
+drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
+NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
+in `talos/patches/k8s-N.yaml`.

 ## Step 2 — talos: form the cluster + get kubeconfig

+**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
+config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
+`$patch: delete` document fails against a base with no `inlineManifests`
+(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
+and 6; only config generation is broken.
+
+The procedure below is what actually works, and is the same one
+`RUNBOOK-baremetal-conversion.md` uses.
+
+### Generate the configs
+
+This is a genuinely new cluster — nothing survives to copy secrets from, so
+generate them fresh. (In a *partial* rebuild, where one node still runs, pull
+secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
+--from-controlplane-config`, or the new node forms its own cluster rather than
+joining.)
+
 ```sh
-kubernetes/bootstrap.sh talos
+talosctl gen secrets -o secrets.yaml
+
+for N in 1 2 3; do
+  talosctl gen config homelab https://k8s.renner.uno:6443 \
+    --with-secrets secrets.yaml --output-types controlplane \
+    --config-patch @talos/patches/cluster.yaml \
+    --config-patch @<cilium additive half> \
+    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
+done
 ```

-Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
-writes `~/.talos/config` + `~/.kube/config`, and stages both into
-`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
+Two things the generator gets wrong for this repo, both of which will bite
+silently:
+
+- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
+  second document. See issue #581.
+- **Strip the generated `HostnameConfig`** — it collides with
+  `machine.network.hostname` from the per-node patch.
+
+Validate each before applying:
+
+```sh
+talosctl validate --config install-k8s-$N.yaml --mode metal
+```
+
+### Apply in two stages
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
+and nothing times out. Concretely, the `ens19` VLAN document must be removed
+from the first-stage config — its parent exists, but the install stalls anyway.
+
+```sh
+# Stage 1: minimal config, from the maintenance-mode address off the console
+talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
+# Stage 2: once the node is up on management, apply the complete config
+talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
+```
+
+`--mode=try` rolls back on its own if the node becomes unreachable, which is the
+only safety net there is for a network change that can strand a machine with no
+IPMI. Use it for every network-affecting apply, then make it permanent once
+proven.
+
+### Bootstrap etcd, then collect access
+
+Bootstrap **exactly one** node:
+
+```sh
+talosctl -n k8s-1.srv.renner.uno bootstrap
+talosctl -n k8s-1.srv.renner.uno kubeconfig .
+```
+
+Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
+supported way to get the kubeconfig into place afterwards.

 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)

@@ -149,8 +248,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).

 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods

 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +257,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap

-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```

 Verify the store recovers before moving on:
@@ -170,7 +273,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```

 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.

 ## Step 5 — apps: let the rest reconcile

@@ -225,9 +330,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
  in-cluster commits made **after** Flux/Gitea broke are not on the mirror
  (Gitea couldn't push them if Gitea itself was down). This is the intended
  recovery point, but be aware of it when reconciling drift.
- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
-  key must not live in CI. See issue #187 for the open question of whether to
-  wire a semi-automated OpenBao restore.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
+- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
+  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
+  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
+  It has also never had an `openbao` phase — Step 4 is manual because the
+  unseal key must not live in CI; see issue #187 on whether to wire a
+  semi-automated OpenBao restore.
+- **OpenBao seals on every restart and each replica must be unsealed
+  individually.** Three raft replicas mean the service survives losing one, but
+  a sealed server is neither a voter nor a standby — unsealing only one leaves
+  the cluster short of quorum. Step 4 covers this; it is repeated here because
+  it is the step most likely to be half-done under pressure.
+- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
+  IPAM, not host networking. Nothing in the rebuild needs special handling for
+  it — the nodes hold no address on that segment — but a rebuilt cluster will
+  not have the whereabouts allocations from before, which is harmless since
+  they are assigned fresh.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
  Ceph/infra bring-up; always read the phase output rather than trusting the
  exit code.
@@ -37,6 +37,13 @@ resources:
  - esphome-service.yaml
  - esphome-gate-service.yaml
  - esphome-ingress.yaml
+  # mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
+  # unicast, which is the only direction the CNI will move them. Without it HA
+  # never learns that a device's DHCP lease moved. See the header comment in
+  # mdns-relay-deployment.yaml.
+  - mdns-relay-network-attachment.yaml
+  - mdns-relay-service.yaml
+  - mdns-relay-deployment.yaml
  # Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
  # runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
  # `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
@@ -0,0 +1,190 @@
+---
+# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
+# without putting a macvlan on the HA pod itself.
+#
+# THE PROBLEM
+#
+# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
+# ff02::fb — and never hears anything on either.
+# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
+# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
+# it does not replicate multicast between pods either. Both halves of the path
+# are dead, which is why an avahi-style reflector — receive on one interface,
+# re-emit as multicast on the other — cannot work here: the re-emitted packets
+# would be dropped by the CNI.
+#
+# Consequence for HA: the ESPHome integration dials devices at the address
+# cached in its config entry. When a lease rotates, every attempt fails with
+# `[Errno 113] Connect call failed` and the device stays unreachable, because
+# the mDNS announcement that would carry the new address never arrives. New
+# devices are never discovered at all.
+#
+# THE MECHANISM
+#
+# This relay holds the multicast membership on the IoT VLAN through its own
+# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
+# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
+# the CNI will carry.
+#
+# It works because python-zeroconf does not filter on how a packet arrived:
+# its datagram handler validates size, deduplicates, and parses, but performs
+# no check on destination address, source address, or source port. A unicast
+# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
+# unicast responses are part of RFC 6762 — the QU bit — so this is within the
+# protocol, not a trick.)
+#
+# WHAT THIS DOES AND DOES NOT RESTORE
+#
+# Restored: passive reception. RFC 6762 requires a responder to re-announce
+# when its address changes, so a device that moves to a new lease announces
+# the move, HA sees it, and the ESPHome integration updates the config entry.
+# That is the failure this exists to fix. New devices announcing themselves
+# are likewise seen.
+#
+# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
+# multicast and die there, so HA cannot actively solicit a response. Anything
+# depending on a query rather than an announcement stays broken. Making that
+# work needs a querier on the VLAN side, which this relay deliberately is not.
+#
+# EXPOSURE
+#
+# The receive sockets carry so-bindtodevice=net1, so only datagrams that
+# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
+# address instead is not possible — it is DHCP-leased and unknown at exec time
+# — but binding to the device achieves the same confinement without needing to
+# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
+# unprivileged uid here.
+#
+# Without it socat would bind the wildcard, and since the namespace CNP grants
+# every pod in home-assistant unrestricted access to every other pod on every
+# port, any sibling pod could reach :5353 over the cluster network and have
+# records forwarded into HA's zeroconf cache.
+#
+# What remains, and cannot be closed here: any host on the IoT VLAN can send
+# this port a datagram. That is the same trust already extended to genuine
+# mDNS from that segment, which any device there can equally forge — the relay
+# widens who can reach HA's listener, not what can be claimed. macvlan traffic
+# bypasses Cilium policy entirely, so the namespace CNP does not constrain
+# this interface either way.
+#
+# Note that the relay is not what makes HA's zeroconf reachable from inside
+# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
+# and the namespace CNP already permits any sibling pod to reach it directly.
+# Confining the relay removes a second route to that listener; it does not
+# remove the first.
+#
+# Nothing else is exposed: the containers run socat and only socat, with no
+# second listener to reach.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mdns-relay
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: mdns-relay
+spec:
+  # One replica. Extra replicas would deliver a duplicate of every announcement
+  # — harmless, since zeroconf deduplicates, but they buy nothing: this is a
+  # cache-refresh path, and a missed window is corrected by the next
+  # announcement.
+  replicas: 1
+  # Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
+  # on v30iot for one workload.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mdns-relay
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mdns-relay
+      annotations:
+        k8s.v1.cni.cncf.io/networks: mdns-relay-iot
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      # One container per address family. mDNS is two protocols — 224.0.0.251
+      # and ff02::fb — and a socket joined to one hears nothing of the other.
+      # HA listens on both (its zeroconf holds both groups on eth0), so a
+      # v4-only relay would silently drop every IPv6 announcement, and devices
+      # that announce only over v6 would stay invisible exactly as before.
+      #
+      # Two processes rather than one: socat carries a single address pair per
+      # invocation, and the families need different receive and send addresses.
+      # They are independent — one crashing does not blind the other — and each
+      # is restarted by the kubelet on its own.
+      containers:
+        # -u is load-bearing on both: unidirectional, left to right. Without it
+        # socat would also carry HA's replies back out, and this must stay a
+        # one-way tap.
+        #
+        # UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
+        # forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
+        # a fork per packet would be a process storm.
+        #
+        # The group is joined on net1 specifically, so the relay subscribes on
+        # the IoT VLAN rather than on the pod network.
+        #
+        # Both send targets are the same ClusterIP name, resolved once at start;
+        # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
+        # mdns-relay-service.yaml for why that indirection is required and why
+        # the Service is dual-stack.
+        - name: socat-ipv4
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
+            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          # No probes. socat is a single process with no health surface: if it
+          # exits, the container exits and the kubelet restarts it, which is the
+          # entire liveness story. A probe here could only re-assert that PID 1
+          # is running.
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+        # Joining ff02::fb needs no address configuration on net1: the group is
+        # link-local and the kernel's automatic link-local address on the
+        # macvlan is sufficient to receive on it. Whether v30iot hands out
+        # global v6 addresses does not affect reception here.
+        #
+        # ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
+        # pod's network namespace, and a dual-stack v6 socket would overlap the
+        # v4 container's port. Measured behaviour without it was no contention —
+        # the v4 receiver kept taking packets while the v6 one ran — but that
+        # depends on the kernel's bindv6only default rather than on anything
+        # stated here, so the roles are made explicit instead of inherited.
+        - name: socat-ipv6
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
+            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
@@ -0,0 +1,52 @@
+---
+# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
+# presence on the IoT VLAN and can join both mDNS multicast groups
+# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
+# configuration of its own: it is link-local, and the kernel's automatic
+# link-local address on the macvlan is enough to receive on it, independently
+# of whether v30iot hands out global v6 addresses.
+# Multicast is link-local — it does not cross the L3
+# boundary between the cluster pod CIDR and v30iot — so that membership has to
+# be held by a pod sitting physically on that segment.
+#
+# The address is leased from the UniFi DHCP server on v30iot. Nothing
+# references it: the relay only receives on this interface and sends onward
+# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
+#
+# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
+# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
+# via the Image Factory schematic), identical across nodes and independent of
+# driver probe order. This is the same master shelly-manager-iot uses.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: mdns-relay-iot
+  namespace: home-assistant
+spec:
+  # Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
+  # v30iot address by DHCP, then `sbr` confines net1's routes to a
+  # source-based routing table. Without sbr the DHCP-supplied default gateway
+  # (10.2.30.1, option 3) installs a second default route at the same metric
+  # as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
+  # dropped. The relay's forward path to Home Assistant runs over eth0, so
+  # that route has to stay intact.
+  #
+  # Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "mdns-relay-iot",
+      "plugins": [
+        {
+          "type": "macvlan",
+          "master": "ens19",
+          "mode": "bridge",
+          "ipam": {
+            "type": "dhcp"
+          }
+        },
+        {
+          "type": "sbr"
+        }
+      ]
+    }
@@ -0,0 +1,42 @@
+---
+# Stable delivery target for the relay's forwarded mDNS traffic.
+#
+# The relay resolves its send target once, at process start (socat has no
+# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
+# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
+# Cilium re-DNATs to whatever pod currently backs it.
+#
+# Deliberately a separate Service from `home-assistant`, which targets the
+# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
+# This one bypasses the sidecar and addresses the home-assistant container's
+# own port directly — containers in a pod share a network namespace, so
+# targetPort 5353 lands on python-zeroconf's listener regardless of which
+# container opened it. Keeping the two apart means the UI path and the
+# discovery path cannot be changed by accident through one another.
+#
+# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
+# permits intra-namespace traffic on all ports, and the relay is in this
+# namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: home-assistant-mdns
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: home-assistant
+spec:
+  # Dual-stack, because mDNS is two protocols and HA listens on both: its
+  # zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
+  # relay forwards each family over its own transport rather than folding v6
+  # announcements into a v4 hop, so the Service has to answer on both.
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: home-assistant
+  ports:
+    - name: mdns
+      port: 5353
+      protocol: UDP
+      targetPort: 5353
@@ -6,12 +6,14 @@ kind: Kustomization
 resources:
  - prometheusrule-ceph.yaml
  - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
  - prometheusrule-velero.yaml
  - prometheusrule-woodpecker-rotation.yaml
  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -23,4 +25,5 @@ resources:
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
@@ -0,0 +1,121 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # THE GAP THESE RULES CLOSE
+        #
+        # Drive wear is invisible in every other signal this cluster has.
+        # `ceph -s` reads active+clean at any wear level, no Ceph health check
+        # fires until a device actually fails, and the block-layer metrics show
+        # throughput and latency but never endurance. That blind spot is how a
+        # set of OSD drives reached 81-87% before anyone looked (#549, #434).
+        #
+        # The numbers below are wear percentages reported by the drive itself:
+        # 100 means the manufacturer's rated endurance is used up. Drives do
+        # not stop at 100 — they typically keep working and start consuming
+        # spare blocks — so treat these as planning thresholds, not failure
+        # predictions.
+        - alert: NvmeDriveWearHigh
+          # 80% with months of runway left is the point where ordering a
+          # replacement is still a calm decision rather than an outage.
+          # `for` is long because wear is monotonic and slow: it cannot
+          # flap, and a short window would only make the alert noisier
+          # after a restart.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive has used {{ $value }}% of its rated write endurance.
+              Nothing is failing and no data is at risk yet. Check what the
+              drive is carrying before deciding urgency: an OSD is replaceable
+              with a rebalance, whereas the EPHEMERAL volume has no such
+              escape — a node simply stops when it goes. The Disk Health
+              dashboard resolves which drive holds which role.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # A drive that has consumed its spare area is no longer masking media
+        # defects, which is a materially different state from being worn:
+        # wear is a budget, spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          # The NVMe critical warning byte is a bitfield: spare below
+          # threshold, temperature out of range, degraded reliability,
+          # read-only mode, volatile-memory backup failed. Any non-zero value
+          # is the drive itself asking for attention.
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # Unlike wear, this counter should never move. Alert on the increase
+          # rather than the absolute value, so a drive that recorded errors
+          # long ago does not alert forever while a drive that starts failing
+          # today does.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # Without this the whole group is silently useless: no series means no
+        # alert fires, and a dashboard of empty panels reads much like a
+        # healthy one. This is the check that the checks are running.
+        - alert: SmartctlExporterIncomplete
+          # Compared against node-exporter's own NVMe inventory rather than a
+          # written-down fleet size. Adding or removing a drive then moves both
+          # sides together, so this cannot rot into a permanent false alarm the
+          # way a hardcoded count would.
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.
@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.
@@ -40,17 +40,31 @@ spec:
            description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."

        - alert: VeleroBackupStale
-          # Daily schedule with no successful backup in over 30h: covers
-          # the case where multiple consecutive runs failed (Garage flapping
-          # over the backup window) without the per-event alerts being
-          # acted on.
+          # Daily schedule with no successful backup in over 60h: covers the
+          # case where consecutive runs failed without the per-event alerts
+          # being acted on.
+          #
+          # 60h, not 30h. At 30h a single missed cycle pages, and a single
+          # missed cycle is the normal cost of touching the nodes: Velero moves
+          # data through pods, so draining a node kills whatever upload is in
+          # flight and the schedule is simply lost for that day — it is not
+          # retried. Rolling node work between 03:15 and 08:45 UTC therefore
+          # produced a guaranteed critical alert for every schedule it
+          # overlapped, which is noise about planned work rather than a signal
+          # about backups.
+          #
+          # 60h still catches what this is for. Two consecutive failures is no
+          # longer explicable by one maintenance window, and the per-event
+          # VeleroBackupFailed alert fires on the first failure regardless — so
+          # nothing is hidden, only the escalation to critical is deferred
+          # until a second cycle confirms it.
          expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
            and
            velero_backup_last_successful_timestamp{schedule!=""} > 0
          for: 10m
          labels:
            severity: critical
          annotations:
-            summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
-            description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
+            summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
+            description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
@@ -0,0 +1,29 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl itself polls the drives every 60s, so scraping faster than that
+    # returns the same numbers with a new timestamp. Wear moves on a scale of
+    # weeks; 60s is already far finer than the signal.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this every series is labelled with the exporter pod's IP and
+        # nothing else. A DaemonSet pod's IP says nothing about which machine
+        # the drive is in, and it changes on every reschedule — so the wear
+        # figure would be unattributable, which is most of its value gone.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
@@ -0,0 +1,382 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
+      ],
+      "title": "OSDs down / out",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}
@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
  - name: dashboard-blackbox-overview
    files:
      - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
  - name: dashboard-gitea-overview
    files:
      - gitea-overview.json
@@ -275,8 +275,11 @@ spec:
          - alertmanager-smtp
      # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
      # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
      # monitoring ns is not blocked by any CNP today.
      config:
        global:
@@ -15,6 +15,7 @@ resources:
  - prometheus-gate-service.yaml
  - prometheus-gate-ingress.yaml
  - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
  - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -0,0 +1,142 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# WHY THIS HOLDS ELEVATED CAPABILITIES, and why that is a deliberate exception
+# rather than an oversight:
+#
+# NVMe reports health over an admin ioctl on the character device
+# (NVME_IOCTL_ADMIN_CMD), which the kernel gates on CAP_SYS_ADMIN. There is no
+# sysfs path for wear, media errors or written-bytes totals — hwmon publishes
+# the controller temperature and nothing else. Upstream smartctl_exporter
+# documents `privileged: true` and `user: root` for exactly this reason.
+#
+# The alternative that needs no privileges — reading Ceph's devicehealth
+# archive, which does hold full smartctl output — covers OSD drives only. In
+# this cluster that is three of nine, and the six it misses include the drives
+# that were retired from Ceph for wear and now carry EPHEMERAL. Covering the
+# healthy drives and not the worn ones was judged not worth a new service; the
+# owner chose fleet-wide coverage with the privilege exception instead.
+#
+# The exception is kept as small as the kernel allows: no `privileged: true`,
+# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are
+# added, everything else is dropped, and the only host mount is /dev — which
+# is required because the exporter must open the NVMe character devices and
+# Kubernetes has no way to pass individual character devices to a pod without
+# a device plugin.
+#
+# If this ever needs revisiting: the thing to check is whether the kernel has
+# grown a sysfs interface for NVMe health, not whether some other exporter
+# needs fewer privileges. The requirement comes from the ioctl gate, not from
+# the tool.
+#
+# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
+# The repository is not archived; the pin is simply its newest tag.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device on purpose: the exporter scans for devices
+          # itself and rescans every 10m by default. Device names are not
+          # stable across these machines, so a hardcoded list would quietly
+          # stop covering a drive the next time one is replaced.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            # Root plus two capabilities, NOT privileged. See the header.
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: false
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["SYS_ADMIN", "SYS_RAWIO"]
+            # The capabilities alone are not enough, and this was measured
+            # rather than assumed. With the default profile the exporter found
+            # all three drives and could open none of them:
+            #   "Smartctl open device: /dev/nvme0 failed: Operation not
+            #   permitted"
+            # EPERM is the default seccomp action for a syscall the profile
+            # does not list. Reading /proc/1/status in the running container
+            # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN
+            # and SYS_RAWIO were both effective — while Seccomp was 2 with one
+            # filter loaded. The kubelet sets seccompDefault: true, so every
+            # pod that does not say otherwise gets RuntimeDefault.
+            #
+            # This is still narrower than `privileged: true`, which would also
+            # grant every capability, all devices and write access to /sys.
+            # Narrowing further would mean shipping a custom seccomp profile
+            # that allows exactly the ioctl smartctl needs — worth doing if
+            # this list of exceptions ever grows, not worth it for one pod.
+            seccompProfile:
+              type: Unconfined
+          volumeMounts:
+            - name: dev
+              mountPath: /dev
+              readOnly: true
+            # smartctl is not known to need scratch space, but the root
+            # filesystem is read-only and a tool that turns out to want /tmp
+            # would crashloop instead of degrading. Cheaper to provide it than
+            # to find out from a restart counter.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.
@@ -4,3 +4,12 @@ resources:
  - cni-plugins-installer.yaml
  - cni-dhcp-daemon.yaml
  - multus-daemonset.yaml
+  # Whereabouts — cluster-coordinated IPAM, for attachments on segments with
+  # no DHCP server. CRDs first: the DaemonSet's control loop reads them at
+  # startup. See the header in whereabouts-daemonset.yaml for why neither
+  # dhcp nor host-local covers the Ceph SAN.
+  - whereabouts-crd-ippools.yaml
+  - whereabouts-crd-overlappingrangeipreservations.yaml
+  - whereabouts-crd-nodeslicepools.yaml
+  - whereabouts-daemonset.yaml
+  - whereabouts-reconciler.yaml
@@ -0,0 +1,71 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: ippools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: IPPool is the Schema for the ippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec defines the desired state of IPPool
+            properties:
+              allocations:
+                additionalProperties:
+                  description: IPAllocation represents metadata about the pod/container
+                    owner of a specific IP
+                  properties:
+                    id:
+                      type: string
+                    ifname:
+                      type: string
+                    podref:
+                      type: string
+                  required:
+                  - id
+                  - podref
+                  type: object
+                description: |-
+                  Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
+                  IP with the same index/offset for the pool's range.
+                type: object
+              range:
+                description: Range is a RFC 4632/4291-style string that represents
+                  an IP address and prefix length in CIDR notation
+                type: string
+            required:
+            - allocations
+            - range
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,79 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: nodeslicepools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: NodeSlicePool
+    listKind: NodeSlicePoolList
+    plural: nodeslicepools
+    singular: nodeslicepool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NodeSlicePool is the Schema for the nodesliceippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
+            properties:
+              range:
+                description: |-
+                  Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
+                  this refers to the entire range where the node is allocated a subset
+                type: string
+              sliceSize:
+                description: SliceSize is the size of subnets or slices of the range
+                  that each node will be assigned
+                type: string
+            required:
+            - range
+            - sliceSize
+            type: object
+          status:
+            description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
+            properties:
+              allocations:
+                description: Allocations holds the allocations of nodes to slices
+                items:
+                  properties:
+                    nodeName:
+                      description: NodeName is the name of the node assigned to this
+                        slice, empty node name is an available slice for assignment
+                      type: string
+                    sliceRange:
+                      description: SliceRange is the subnet of this slice
+                      type: string
+                  required:
+                  - nodeName
+                  - sliceRange
+                  type: object
+                type: array
+            required:
+            - allocations
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,57 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: overlappingrangeipreservations.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: OverlappingRangeIPReservation
+    listKind: OverlappingRangeIPReservationList
+    plural: overlappingrangeipreservations
+    singular: overlappingrangeipreservation
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: OverlappingRangeIPReservationSpec defines the desired state
+              of OverlappingRangeIPReservation
+            properties:
+              containerid:
+                type: string
+              ifname:
+                type: string
+              podref:
+                type: string
+            required:
+            - podref
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,196 @@
+---
+# Whereabouts: cluster-wide IPAM for Multus attachments.
+#
+# WHY THIS EXISTS
+#
+# The attachments in this cluster lease their addresses by DHCP
+# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
+# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
+# Ceph-internal traffic and nothing else, by design: there is no DHCP server
+# there and there must not be one.
+#
+# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
+# in. It keeps its allocation state per node with no cluster-wide
+# coordination, so on a shared L2 segment two nodes hand out the same address.
+# Whereabouts coordinates through the Kubernetes API instead — allocations
+# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
+# the address assignment touches the segment being addressed. That property is
+# what makes it usable on an isolated SAN.
+#
+# HOST ACCESS
+#
+# The DaemonSet runs with hostNetwork and privileged. That is how upstream
+# ships it, and it is what the three CNI components already here do —
+# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
+# all carry both. The reason is the same in each case: the pod writes a CNI
+# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
+# which is node plumbing rather than a workload, and it must work before the
+# attachment it provides addresses for exists.
+#
+# The reconciler is deliberately NOT in here. It needs neither, and runs as an
+# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
+#
+# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
+# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
+# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
+# upstream diff stays readable.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: whereabouts
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: whereabouts
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: whereabouts-cni
+subjects:
+  - kind: ServiceAccount
+    name: whereabouts
+    namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: whereabouts-cni
+rules:
+  - apiGroups:
+      - whereabouts.cni.cncf.io
+    resources:
+      - ippools
+      - overlappingrangeipreservations
+      - nodeslicepools
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - coordination.k8s.io
+    resources:
+      - leases
+    verbs:
+      - '*'
+  - apiGroups: [""]
+    resources:
+      - pods
+    verbs:
+      - list
+      - watch
+      - get
+  - apiGroups: [""]
+    resources:
+      - nodes
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - network-attachment-definitions
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+      - get
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: whereabouts-config
+  namespace: kube-system
+  annotations:
+    kubernetes.io/description: |
+      Configmap containing user customizable cronjob schedule
+data:
+  # Read by the reconciler (whereabouts-reconciler.yaml), not by this
+  # DaemonSet. Governs how often stale allocations are swept — the case that
+  # matters here is a node lost hard enough that the CNI DEL never ran, which
+  # otherwise leaves an address held by a pod that no longer exists.
+  cron-expression: "30 4 * * *"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: whereabouts
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts
+spec:
+  selector:
+    matchLabels:
+      name: whereabouts
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts
+        name: whereabouts
+    spec:
+      hostNetwork: true
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node, so the NoSchedule toleration
+      # is not optional — without it this schedules nowhere.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts
+          command: ["/bin/sh"]
+          args:
+            - -c
+            - |
+              SLEEP=false source /install-cni.sh
+              /token-watcher.sh &
+              /ip-control-loop -log-level debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: NODENAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: spec.nodeName
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "100Mi"
+            limits:
+              cpu: "100m"
+              memory: "200Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
+            - name: cni-net-dir
+              mountPath: /host/etc/cni/net.d
+      volumes:
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: cni-net-dir
+          hostPath:
+            path: /etc/cni/net.d
@@ -0,0 +1,75 @@
+---
+# Sweeps whereabouts allocations that no live pod holds any more.
+#
+# The normal release path is the CNI DEL when a pod goes away. It does not run
+# when a node is lost hard — power cut, kernel panic, a reboot that never
+# comes back — and the allocation then stays claimed by a pod that no longer
+# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
+# of leaked addresses is the difference between an OSD getting an address and
+# not.
+#
+# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
+# this needs no host access at all — it only talks to the Kubernetes API,
+# where the allocations live.
+#
+# replicas stays at 1: there is no leader election, so a second replica would
+# reconcile the same records concurrently.
+#
+# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
+# image tag pinned; upstream ships `:latest`. The schedule comes from the
+# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: whereabouts-reconciler
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts-reconciler
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: whereabouts-reconciler
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts-reconciler
+        name: whereabouts-reconciler
+    spec:
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts-reconciler
+          command: ["/ip-reconciler"]
+          args:
+            - -log-level
+            - debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          volumeMounts:
+            - name: cron-scheduler-configmap
+              mountPath: /cron-schedule
+      volumes:
+        - name: cron-scheduler-configmap
+          configMap:
+            name: whereabouts-config
+            defaultMode: 0744
+            items:
+              - key: "cron-expression"
+                path: "config"
@@ -42,12 +42,28 @@ spec:
      # the agent-injector mutating webhook is unused surface.
      enabled: false
    server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
      ha:
        enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
        raft:
          enabled: true
          setNodeId: true
@@ -65,8 +81,32 @@ spec:
              cluster_address = "[::]:8201"
            }

+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
            storage "raft" {
              path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
            }

            service_registration "kubernetes" {}
@@ -53,7 +53,13 @@ spec:
              command: ["/bin/sh", "-ec"]
              args:
                - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                  BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                    role=openbao-snapshot \
                    jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")
@@ -203,16 +203,16 @@ metadata:
  labels:
    app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
+  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
+  # the home-assistant namespace regardless of which app/ dir owns
  # their manifest (matter-server, otbr, whisper, speech-to-phrase,
  # zone-configurator all set `namespace: home-assistant` via their
  # Kustomization, not per-resource): home-assistant-config (.storage
  # entity registry, recorder DB, secrets.yaml), esphome-data,
  # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
+  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
+  # zwavejs-store, zone-configurator-config. On restore,
+  # home-assistant-config has its
  # own path (the HA pod's `seed` initContainer pulling the native HAOS
  # backup from Garage, see the home-assistant Kustomization comment in
  # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
@@ -229,6 +229,20 @@ spec:
    storageLocation: default
    ttl: 720h0m0s
    snapshotMoveData: true
+    # The ha-pg CNPG cluster now runs in this namespace (it did not when
+    # this schedule was written — the old comment here still claimed
+    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
+    # cnpg.io/cluster label and was being swept in: a crash-consistent
+    # snapshot of a running Postgres data directory is not a restorable
+    # backup, the same trap the wordpress-daily comment below describes.
+    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
+    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
+    # continuous WAL archiving — so this exclusion removes a redundant
+    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule
@@ -8,6 +8,17 @@ spec:
    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
  mon:
    count: 3
    allowMultiplePerNode: false
@@ -36,26 +47,27 @@ spec:
  # (paperless + audiobookshelf + immich + ... all clone at once).
  cephConfig:
    mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
      #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
      # Stop compacting the whole RocksDB prefix on every paxos trim.
      #
      # Upstream default is true — "Compact a certain prefix (including paxos)
@@ -147,19 +159,65 @@ spec:
    ssl: false
  monitoring:
    enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
  network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
  # taken out of the storage fleet by removing it here. Under useAllNodes the
  # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +251,15 @@ spec:
  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
  #   S437NF0M501176    PM983     — this OSD
  storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
    useAllNodes: false
    useAllDevices: false
    nodes:
@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
  - cephcluster.yaml
  - toolbox.yaml
  - storageclass.yaml
@@ -0,0 +1,84 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }
@@ -20,16 +20,39 @@ machine:
  #                            the MS-01 mesh later needs neither a new image
  #                            nor a reinstall — which is exactly the property
  #                            lost by installing without it.
+  #   siderolabs/i915          Intel GPU microcode and kernel modules for the
+  #                            Alder Lake-P Iris Xe in the MS-01. No workload
+  #                            requests /dev/dri today; it is here for the same
+  #                            reason as thunderbolt, so that wanting hardware
+  #                            transcoding later is a config change rather than
+  #                            another rolling upgrade of every node.
  #
-  # a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
-  # same net.ifnames=1 as the Proxmox one (the predictable interface names the
-  # per-node VLAN parents match against), minus qemu-guest-agent.
+  # Not included, and why, so the absences are not mistaken for oversights:
+  # the X710 (i40e) and I226 (igc) NICs need no firmware extension —
+  # intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
+  # Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
+  # util-linux-tools have no consumer: storage is Ceph RBD, and device health
+  # is reported by Ceph itself rather than a node-level exporter.
+  #
+  # 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
+  # one (the predictable interface names the per-node VLAN parents match
+  # against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
+  # identical but without i915 — see RUNBOOK-cold-rebuild.md.
  #
  # Only read at install and upgrade time. Changing it here does nothing to a
  # running node until that node is upgraded. Bump the tag with the Talos
  # version — it must match the running version, or the upgrade is a downgrade.
+  #
+  # Upgrades follow the documented path: migration is only tested between
+  # adjacent minor releases, so go to the latest patch of each intermediate
+  # minor rather than skipping. 1.12.11 -> 1.13.9 satisfies that — one minor
+  # step from the last 1.12 patch, no intermediate needed.
+  #
+  # Kubernetes is a separate decision and is not forced by this. Talos 1.13
+  # supports Kubernetes 1.36, 1.35, 1.34, 1.33, 1.32 and 1.31, so the cluster
+  # stays on 1.35.2 until `talosctl upgrade-k8s` moves it deliberately.
  install:
-    image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
  features:
    rbac: true
    stableHostname: true
@@ -20,6 +20,29 @@ machine:
    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
    # This disables v6 entirely on that link, including link-local.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -117,13 +164,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29
@@ -9,6 +9,15 @@ machine:
    # macvlan parent for the v30iot pod attachments; host addresses on that L2
    # make node replies bypass the UDM and break every UDM-DNATed flow.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -89,10 +98,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29
@@ -8,6 +8,15 @@ machine:
  sysctls:
    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29