docs(infra): trim the remaining oversized comments

crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp
and the openbao rotation kustomization. Comments only; the diff filtered for
non-comment lines is empty and infrastructure/ still builds.

Every instruction survives, because that is the half worth having: do not
switch crowdsec back to password auth, or a wiped LAPI crashloops the agents
and wedges the Kustomization; do not move netbird-ingress off the rootless
image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not
pin the rotators by digest, or a quarterly job fails to pull a manifest that
was garbage-collected months earlier and looks like it simply did not run.

What went is the incident each was learned from, and one duplicate: the
san-network-attachment header re-derived the provider: host argument that
cephcluster.yaml already makes, so it now points there instead.

Two more stale claims found on the way through:

  - woodpecker-ns.yaml described the namespace as holding wp-pg alone with
    'server and agent arriving in stage 2'. Both have been running there for
    some time.
  - cilium-bgp.yaml said each node sources its BGP session from eth0. The
    management interface is mgmt0; eth0 was the VM-era name.

infrastructure/ is now 2553 comment lines.

.woodpecker/alert-monitor.yml

@@ -101,7 +101,7 @@ steps:
         bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/build-ci-images.yml

@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:
 
 steps:
   - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/ansible
     depends_on: []
     # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
 
   - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/k8s
     depends_on: []
     when:
@@ -77,7 +88,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
     depends_on: []
     when:
@@ -108,7 +119,7 @@ steps:
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
   - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: secret-rotator
     depends_on: []
     when:
@@ -137,7 +148,7 @@ steps:
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
   - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
     depends_on: []
     when:
@@ -166,7 +177,7 @@ steps:
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
   - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
     depends_on: []
     when:
@@ -195,7 +206,7 @@ steps:
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
   - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
     depends_on: []
     when:
@@ -224,7 +235,7 @@ steps:
   # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
   # builds the same way and lands in the same registry so the audit sees it.
   - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/heating-can
     depends_on: []
     when:

.woodpecker/ci-claude-refresh.yml

@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
       - |
         set -eu
         mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
         NEW=$$(npm view @anthropic-ai/claude-code version)
         case "$$NEW" in
           [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
   #    --build-arg to the resolved version (not "latest") guarantees the tag and
   #    the image content agree even if a new release lands mid-run.
   - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/claude
     depends_on: [decide]
     environment:
@@ -176,13 +186,10 @@ steps:
           echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
           exit 0
         fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
         # Exact-string swap of the current tag for the new one, consumers only.
         for f in $$CONSUMERS; do
           sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
         {
           printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
           printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
           printf 'run the image so CI stops running a stale Claude Code.\n\n'
           printf 'Merging is a human decision — review, then merge to roll it out.\n'
         } > /tmp/pr-body.md

.woodpecker/ci-scripts-test.yml

@@ -17,6 +17,6 @@ when:
 
 steps:
   - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh

.woodpecker/e2e.yml

@@ -25,7 +25,7 @@ steps:
     # fetch-models.sh response mapping. No cluster, no secrets, no network — the
     # ci-claude image just provides sh + jq. Keeps the flag-injection regression
     # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh
 

.woodpecker/issue-fix.yml

@@ -62,7 +62,7 @@ steps:
       - bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/pr-review.yml

@@ -65,7 +65,7 @@ when:
 
 steps:
   - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/security-scan.yml

@@ -42,7 +42,7 @@ steps:
     # this step is skipped — otherwise we'd fork-bomb PRs.
     when:
       - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/update-audit.yml

@@ -30,7 +30,7 @@ when:
 
 steps:
   - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

ansible/inventory/group_vars/all/netbird-rotatable.yml

@@ -0,0 +1,13 @@
+$ANSIBLE_VAULT;1.1;AES256
+32373134633930343261336362373335623735656634313034393731393661643361346634623439
+3561363963333533313264643036663731656465343162330a333964383462373934303766323533
+63613065313063626261383236396363656565623538666565646661353766643133306334396561
+3961376234623236620a616339326238366538333666316538396465666365373834313064653963
+38623233616633383435373836613438333763343932636239386263306238376137643531373161
+35386437313465643437343832623935353238316565366537646530323233363837656631383962
+32653332333162623966366163633732356637323461613966313263313066643966373335313162
+30306362346561633566316166313164323635356363643034663933363464663562343136646137
+66643536363734316533663032643637356136363237376365613330633961616665323430633135
+36323733633464306330666232663433616533373466633939353061663136666665323432373666
+36393930333435333238306238373931393232393861333334643139306138653336633534626362
+61323038303838396438

ansible/inventory/group_vars/all/vault.yml

@@ -1,42 +1,29 @@
 $ANSIBLE_VAULT;1.1;AES256
-39333932383736353162346462663539633763663565366639646463316636333364386365363863
-3034356438663432363366633537373738363061633764300a393236373834323237623434353361
-32346331666234616335663432386163316161616230396265373337613035323863393663333665
-6463623335626238640a306666616366363130363534623032353730313865343537373931343433
-61636634306565356566303431623533623734343466343636376664646539363363383733666638
-39316362393066666435326438323036353735666630306532353433663163303231376565313838
-33616137303261366330653862613934303138653634323063363232303636346336303337333537
-33643030613662343765303638303339336564633063626462656265646536616563303565323266
-36373765663732666436363866623133366337646265616333633536666630396465363438353865
-37383130373532303131646530353236383637613034616433383333333234353137633166663866
-30366265323537623863336461313666656662383535623435306231363763383364306335623938
-34376363633762303664356262383837313866633362663037616335626135376366613336356132
-39323166366437386236386237396236356631623133373866663831656330313936363361656465
-31373663653736653438643165383338333034613139303333373937626666376432613665656564
-61616365663033633438666562363530323862613436663964356339373635356366386264373437
-36646138313838666561646331316265383637313237616632643831303835313663363034633862
-62623064383539366366386634313938323833323163316365356430633531306134366630623261
-35656461306566666531366364323834663737313835666264316132376638333436343136656233
-39343338653264636435383736613763633531356438643464633164633766383435323331633630
-63316436343863383561653631333434613761343863643465366139353262333463646263633566
-38306266393439346265666133343566353164353765366331343631666163613262363135643938
-31643538363136376462633431643135373932373932303435303365383063646236656532643263
-34663637616535373235336537633339303263383035323264303665643562336431303436653435
-37613162353133636361623431356361646433313161316163333766336531363164316334343061
-38303263643263386662383731656537393731356663353861316630353464363932346666323037
-31343933346530306461363665383235303239373234336631316531353862643937393235643134
-32303265306331336665653162383639353634373462666461643463633531373366633438353538
-31376437386538613531323231396437363434663236663635666638373137353434373733633234
-66363732393166633534316439343561383234636661316439366362326631656134346264623432
-35626332646162386236336434393533626435313462303635313439326163636330636133613335
-38636664376166613461316466323161386436396235376534393063663532333538336164643730
-30636137633663666532666466363730633137396363653832643837366266626461376636333736
-33343832326566353264353266376664643061306630633038313534386563616236636438393434
-61643030616531363963313337646263613366663462326561393261383732376231383036303633
-66633164623232343537316163636666653932646266643561303561323536313532666538643031
-61653663613139383234656465393637323031666462663165323138303164313461623832323236
-32373866636432653135623330333464626661396465353133373965376436353138313866633138
-30376339626165386262396139393737343036333437623365313534636239373830663437643531
-63643233663137646335343338663661623431636162306363363839306631356638356462643336
-62643861336131616433666265316538346264376236386434343539343839313130633535616236
-32336632306165376635313130346436663939303432353966323662326335666564
+30353730633631376539373334376531353263396339313464373538396231616166653738383161
+6438656132616365646463653464396236313437323537620a363632643061333036353537626461
+34393065613836343536383565303066326133363937386263653961663666643430663139653961
+6462363235343063350a636165376232366636363233376637636231363839343738346134623562
+63663839326435303964313730336436396365343163343366613966323938656263303166326435
+65303931393233623064343763633161353161643635396365316362333137353732346366643162
+36386132386638623333623839373161386330353635346331393735366662393935353264646439
+36386539393432373765396562613033656339646532323864366230393638313161383363636636
+32353266376563393239393734316461313566386233333164623135303132346164353165323338
+62306333363838393236346435303339393639623231353232643966613163313134336265633636
+64366539326639303362393933303465376532626663346431613563353032383037363532646637
+38376266323537343437643635343634636164393230313632663533666434306134366261383230
+66383135303565383763646331383338646339613539623835366438393635363832366134316531
+63636466653239383231616139663231623636376564376234383762383066613030313937616533
+33396135616236643034303236353230376434313365326633303961303232363766633161353434
+64333832623535343733353462313862343934633363616237353331386264386632643538333939
+34636437393934326131363236346636663665613537343134313636623439356665636663613938
+37346634356164343537396363663730306232623964636635333333653738383538343236383535
+66326462373964333234343435336630643930333433393034303439366631386666393330343238
+61336237373236313533333732343064336637326636663433396362346631343165656163313561
+30383331326231383539343639343130323731333432306630353835343338613861376631646264
+66383539616663376237663234313431313064663566313566366664386439316161353135306338
+35373962636537653535333935306564303235356232313463386237353737313736386463396164
+62666165336539316565623634326562356434313833643833353635303361353937666663356263
+35653665323736386263333437666432623332333936373938376637633463653930326466383139
+38336239646135316565356564373731386662383234386433633632393165343331353835623437
+30663138633430356334353832633434633939373337663437326263336537356633313763353337
+64373764666430643433

ansible/plays/rotate-netbird-secrets.yml

@@ -15,7 +15,14 @@
 #
 #   ansible-vault edit inventory/group_vars/all/vault.yml
 #   # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
-#   #        vault_netbird_proxy_token
+#   #        and netbird_turn_secret
+#
+# That last one is easy to miss and breaks the rotation silently. vault.yml
+# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
+# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
+# a group var outranks a role default — so the bare key wins, and rotating only
+# the vault_-prefixed one changes nothing that reaches the template. The check
+# below matches both spellings for that reason.
 #
 # The first rotation then populates netbird-rotatable.yml with fresh values, so
 # nothing needs copying across.
@@ -29,10 +36,24 @@
 #                               cannot disagree across a run.
 #   vault_netbird_relay_secret  management.json on the server and relay.env on
 #                               the proxies. Two hosts, both managed here.
-#   vault_netbird_proxy_token   proxy.env on the server and the proxies.
+#
+# The test for "safe to rotate here" is whether the value appears in
+# management.json.j2. If the management server does not read it from a file
+# Ansible writes, Ansible does not own both ends, whatever else it templates.
 #
 # WHAT IS NOT, and must not be added:
 #
+#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on both
+#                                the server and the proxies, all Ansible
+#                                templates — but the management server does not
+#                                validate it from any of them. It is registered
+#                                in the management DATABASE. Rotating it changes
+#                                only what the proxies present, so every proxy
+#                                is locked out: "proxy auth failed: too many
+#                                failed authentication attempts", and the
+#                                proxies stop receiving their domain
+#                                configuration. Learned by doing it.
+#
 #   vault_netbird_datastore_key  encrypts NetBird's datastore at rest. Changing
 #                                it without re-encrypting the store makes the
 #                                existing data unreadable. Rotating this needs a
@@ -114,7 +135,7 @@
           ansible-vault view
           --vault-password-file {{ playbook_dir }}/../vault-password.sh
           {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
-          | grep -cE '^vault_netbird_(turn_secret|relay_secret|proxy_token):'
+          | grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
       register: _stale_keys
       changed_when: false
       # grep exits 1 on "no matches", which is the good case here.
@@ -130,8 +151,8 @@
           rotates. group_vars files merge alphabetically and vault.yml sorts
           last, so it would override netbird-rotatable.yml on the next deploy
           and revert the secrets. Remove vault_netbird_turn_secret,
-          vault_netbird_relay_secret and vault_netbird_proxy_token from
-          vault.yml first (one-time migration), then re-run.
+          netbird_turn_secret and vault_netbird_relay_secret from vault.yml
+          first (one-time migration), then re-run.
         success_msg: vault.yml does not shadow the rotated keys.
 
   tasks:
@@ -150,7 +171,6 @@
       ansible.builtin.set_fact:
         new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
         new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
-        new_proxy_token: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
       no_log: true
 
     - name: Render the new values to a temporary file
@@ -163,7 +183,6 @@
           # the next rotation overwrites this file wholesale.
           vault_netbird_turn_secret: "{{ new_turn_secret }}"
           vault_netbird_relay_secret: "{{ new_relay_secret }}"
-          vault_netbird_proxy_token: "{{ new_proxy_token }}"
       no_log: true
 
     # Encrypting the whole file rather than inline !vault strings: these three
@@ -198,7 +217,6 @@
   vars:
     netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
     netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
 
   roles:
     - netbird-server
@@ -213,7 +231,6 @@
 
   vars:
     netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
-    netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
 
   roles:
     - netbird-proxy
@@ -227,7 +244,7 @@
     - name: Remaining manual steps
       ansible.builtin.debug:
         msg:
-          - "Rotated: turn_secret, relay_secret, proxy_token."
+          - "Rotated: turn_secret and relay_secret."
           - "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
           - "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
           - "OpenBao is unaffected by design — its NetBird credentials are separate ones."

kubernetes/MIGRATION-sops-openbao.md

@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```
 
 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -31,8 +31,8 @@ all in state that outlived the VM.
       two hosts every PG is undersized with min_size 2, so one further OSD loss
       blocks IO on every CephFS mount cluster-wide.
 - [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      `0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
+      (`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
       Verify by comparing partition offsets and sizes against the ISO with
       `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is

kubernetes/RUNBOOK-cold-rebuild.md

@@ -1,10 +1,25 @@
 # Cold-rebuild runbook — homelab cluster from scratch
 
 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
+in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
+gone).
 
 Order: **tofu → talos → flux → openbao → apps.**
 
+All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
+are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
+Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
+resources, and physically booting three machines from install media.
+
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
+
 ## The circular-bootstrap problem this runbook exists to break
 
 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -61,12 +76,21 @@ These must exist outside the cluster or the rebuild cannot complete:
       `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
       copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
+      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
+      Proxmox credentials — there is no Proxmox any more.
+- [ ] **Install media** written and to hand: the Image Factory ISO named in
+      Step 1, on a USB stick, for all three machines.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.
 
-## Step 1 — tofu: provision the VMs + supporting infra
+## Step 1a — tofu: DNS, object storage, UniFi
 
-Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
-UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
+OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
+the tree but nothing instantiates it, so `tofu apply` here creates only the
+platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
+records, the Hetzner object storage, and the NAT hairpin rules.
 
 ```sh
 cd opentofu
@@ -74,44 +98,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```
 
-In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
-which also pre-seeds the Talos image onto every Proxmox node. Wait until the
-node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
-Step 2 — `bootstrap.sh` polls DNS but will not create the records.
+Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
+FQDNs to resolve — the control-plane endpoint is baked into every generated
+config, and the nodes are addressed by name.
 
-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
+## Step 1b — boot all three machines from install media
 
 ```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```
 
-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
+Write it to a USB stick and boot each MS-01 from it. Each comes up in
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.
 
-- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
-- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
-- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
+The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
+interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
+against do not exist, and the node ends up with no address on any interface.
+Beyond that it carries:
+
+- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
+  today; present so wanting hardware transcoding later is a config change
+  rather than another rolling reinstall
+- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
+  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
+  rule auto-authorises any Thunderbolt device attached once those modules load
+- no `qemu-guest-agent`, which has nothing to talk to on bare metal
+
+Verify the disks before going further. Talos enumerates NVMe in an order that
+is not stable across boots, and the per-node configs select by serial for
+exactly that reason — the failure mode is the installer taking the Ceph OSD
+drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
+NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
+in `talos/patches/k8s-N.yaml`.
 
 ## Step 2 — talos: form the cluster + get kubeconfig
 
+**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
+config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
+`$patch: delete` document fails against a base with no `inlineManifests`
+(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
+and 6; only config generation is broken.
+
+The procedure below is what actually works, and is the same one
+`RUNBOOK-baremetal-conversion.md` uses.
+
+### Generate the configs
+
+This is a genuinely new cluster — nothing survives to copy secrets from, so
+generate them fresh. (In a *partial* rebuild, where one node still runs, pull
+secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
+--from-controlplane-config`, or the new node forms its own cluster rather than
+joining.)
+
 ```sh
-kubernetes/bootstrap.sh talos
+talosctl gen secrets -o secrets.yaml
+
+for N in 1 2 3; do
+  talosctl gen config homelab https://k8s.renner.uno:6443 \
+    --with-secrets secrets.yaml --output-types controlplane \
+    --config-patch @talos/patches/cluster.yaml \
+    --config-patch @<cilium additive half> \
+    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
+done
 ```
 
-Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
-writes `~/.talos/config` + `~/.kube/config`, and stages both into
-`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
+Two things the generator gets wrong for this repo, both of which will bite
+silently:
+
+- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
+  second document. See issue #581.
+- **Strip the generated `HostnameConfig`** — it collides with
+  `machine.network.hostname` from the per-node patch.
+
+Validate each before applying:
+
+```sh
+talosctl validate --config install-k8s-$N.yaml --mode metal
+```
+
+### Apply in two stages
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
+and nothing times out. Concretely, the `ens19` VLAN document must be removed
+from the first-stage config — its parent exists, but the install stalls anyway.
+
+```sh
+# Stage 1: minimal config, from the maintenance-mode address off the console
+talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
+# Stage 2: once the node is up on management, apply the complete config
+talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
+```
+
+`--mode=try` rolls back on its own if the node becomes unreachable, which is the
+only safety net there is for a network change that can strand a machine with no
+IPMI. Use it for every network-affecting apply, then make it permanent once
+proven.
+
+### Bootstrap etcd, then collect access
+
+Bootstrap **exactly one** node:
+
+```sh
+talosctl -n k8s-1.srv.renner.uno bootstrap
+talosctl -n k8s-1.srv.renner.uno kubeconfig .
+```
+
+Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
+supported way to get the kubeconfig into place afterwards.
 
 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)
 
@@ -149,8 +248,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).
 
 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods
 
 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +257,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
 
-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```
 
 Verify the store recovers before moving on:
@@ -170,7 +273,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```
 
 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.
 
 ## Step 5 — apps: let the rest reconcile
 
@@ -225,9 +330,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
   in-cluster commits made **after** Flux/Gitea broke are not on the mirror
   (Gitea couldn't push them if Gitea itself was down). This is the intended
   recovery point, but be aware of it when reconciling drift.
-- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
-  key must not live in CI. See issue #187 for the open question of whether to
-  wire a semi-automated OpenBao restore.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
+- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
+  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
+  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
+  It has also never had an `openbao` phase — Step 4 is manual because the
+  unseal key must not live in CI; see issue #187 on whether to wire a
+  semi-automated OpenBao restore.
+- **OpenBao seals on every restart and each replica must be unsealed
+  individually.** Three raft replicas mean the service survives losing one, but
+  a sealed server is neither a voter nor a standby — unsealing only one leaves
+  the cluster short of quorum. Step 4 covers this; it is repeated here because
+  it is the step most likely to be half-done under pressure.
+- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
+  IPAM, not host networking. Nothing in the rebuild needs special handling for
+  it — the nodes hold no address on that segment — but a rebuilt cluster will
+  not have the whereabouts allocations from before, which is harmless since
+  they are assigned fresh.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
   Ceph/infra bring-up; always read the phase output rather than trusting the
   exit code.

kubernetes/apps/authentik/barman-creds-externalsecret.yaml

@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: authentik
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/authentik/barman-creds

kubernetes/apps/authentik/hetzner-creds-externalsecret.yaml

@@ -15,4 +15,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/authentik/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/authentik/kustomization.yaml

@@ -27,5 +27,4 @@ resources:
   - cluster.yaml
   - objectstore.yaml
   - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
   - hetzner-creds-externalsecret.yaml

kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml

@@ -1,21 +1,8 @@
 ---
 # Barman S3 credentials for ha-pg.
 #
-# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
-# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
-# problem and the same note. The values are the same today, so this is correct
-# but coupled: rotating paperless's Hetzner credentials would silently change
-# the backup credentials for Home Assistant and Woodpecker too.
-#
-# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
-# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
-#
-#   bao kv put homelab/seed/home-assistant/hetzner-creds \
-#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
-#
-# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
-# 403 on `bao kv list seed`, which was a wrong-path error rather than a
-# permissions problem.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +22,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/home-assistant/kustomization.yaml

@@ -37,6 +37,13 @@ resources:
   - esphome-service.yaml
   - esphome-gate-service.yaml
   - esphome-ingress.yaml
+  # mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
+  # unicast, which is the only direction the CNI will move them. Without it HA
+  # never learns that a device's DHCP lease moved. See the header comment in
+  # mdns-relay-deployment.yaml.
+  - mdns-relay-network-attachment.yaml
+  - mdns-relay-service.yaml
+  - mdns-relay-deployment.yaml
   # Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
   # runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
   # `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml

kubernetes/apps/home-assistant/mdns-relay-deployment.yaml

@@ -0,0 +1,190 @@
+---
+# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
+# without putting a macvlan on the HA pod itself.
+#
+# THE PROBLEM
+#
+# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
+# ff02::fb — and never hears anything on either.
+# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
+# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
+# it does not replicate multicast between pods either. Both halves of the path
+# are dead, which is why an avahi-style reflector — receive on one interface,
+# re-emit as multicast on the other — cannot work here: the re-emitted packets
+# would be dropped by the CNI.
+#
+# Consequence for HA: the ESPHome integration dials devices at the address
+# cached in its config entry. When a lease rotates, every attempt fails with
+# `[Errno 113] Connect call failed` and the device stays unreachable, because
+# the mDNS announcement that would carry the new address never arrives. New
+# devices are never discovered at all.
+#
+# THE MECHANISM
+#
+# This relay holds the multicast membership on the IoT VLAN through its own
+# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
+# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
+# the CNI will carry.
+#
+# It works because python-zeroconf does not filter on how a packet arrived:
+# its datagram handler validates size, deduplicates, and parses, but performs
+# no check on destination address, source address, or source port. A unicast
+# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
+# unicast responses are part of RFC 6762 — the QU bit — so this is within the
+# protocol, not a trick.)
+#
+# WHAT THIS DOES AND DOES NOT RESTORE
+#
+# Restored: passive reception. RFC 6762 requires a responder to re-announce
+# when its address changes, so a device that moves to a new lease announces
+# the move, HA sees it, and the ESPHome integration updates the config entry.
+# That is the failure this exists to fix. New devices announcing themselves
+# are likewise seen.
+#
+# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
+# multicast and die there, so HA cannot actively solicit a response. Anything
+# depending on a query rather than an announcement stays broken. Making that
+# work needs a querier on the VLAN side, which this relay deliberately is not.
+#
+# EXPOSURE
+#
+# The receive sockets carry so-bindtodevice=net1, so only datagrams that
+# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
+# address instead is not possible — it is DHCP-leased and unknown at exec time
+# — but binding to the device achieves the same confinement without needing to
+# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
+# unprivileged uid here.
+#
+# Without it socat would bind the wildcard, and since the namespace CNP grants
+# every pod in home-assistant unrestricted access to every other pod on every
+# port, any sibling pod could reach :5353 over the cluster network and have
+# records forwarded into HA's zeroconf cache.
+#
+# What remains, and cannot be closed here: any host on the IoT VLAN can send
+# this port a datagram. That is the same trust already extended to genuine
+# mDNS from that segment, which any device there can equally forge — the relay
+# widens who can reach HA's listener, not what can be claimed. macvlan traffic
+# bypasses Cilium policy entirely, so the namespace CNP does not constrain
+# this interface either way.
+#
+# Note that the relay is not what makes HA's zeroconf reachable from inside
+# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
+# and the namespace CNP already permits any sibling pod to reach it directly.
+# Confining the relay removes a second route to that listener; it does not
+# remove the first.
+#
+# Nothing else is exposed: the containers run socat and only socat, with no
+# second listener to reach.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mdns-relay
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: mdns-relay
+spec:
+  # One replica. Extra replicas would deliver a duplicate of every announcement
+  # — harmless, since zeroconf deduplicates, but they buy nothing: this is a
+  # cache-refresh path, and a missed window is corrected by the next
+  # announcement.
+  replicas: 1
+  # Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
+  # on v30iot for one workload.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mdns-relay
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mdns-relay
+      annotations:
+        k8s.v1.cni.cncf.io/networks: mdns-relay-iot
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      # One container per address family. mDNS is two protocols — 224.0.0.251
+      # and ff02::fb — and a socket joined to one hears nothing of the other.
+      # HA listens on both (its zeroconf holds both groups on eth0), so a
+      # v4-only relay would silently drop every IPv6 announcement, and devices
+      # that announce only over v6 would stay invisible exactly as before.
+      #
+      # Two processes rather than one: socat carries a single address pair per
+      # invocation, and the families need different receive and send addresses.
+      # They are independent — one crashing does not blind the other — and each
+      # is restarted by the kubelet on its own.
+      containers:
+        # -u is load-bearing on both: unidirectional, left to right. Without it
+        # socat would also carry HA's replies back out, and this must stay a
+        # one-way tap.
+        #
+        # UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
+        # forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
+        # a fork per packet would be a process storm.
+        #
+        # The group is joined on net1 specifically, so the relay subscribes on
+        # the IoT VLAN rather than on the pod network.
+        #
+        # Both send targets are the same ClusterIP name, resolved once at start;
+        # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
+        # mdns-relay-service.yaml for why that indirection is required and why
+        # the Service is dual-stack.
+        - name: socat-ipv4
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
+            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          # No probes. socat is a single process with no health surface: if it
+          # exits, the container exits and the kubelet restarts it, which is the
+          # entire liveness story. A probe here could only re-assert that PID 1
+          # is running.
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+        # Joining ff02::fb needs no address configuration on net1: the group is
+        # link-local and the kernel's automatic link-local address on the
+        # macvlan is sufficient to receive on it. Whether v30iot hands out
+        # global v6 addresses does not affect reception here.
+        #
+        # ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
+        # pod's network namespace, and a dual-stack v6 socket would overlap the
+        # v4 container's port. Measured behaviour without it was no contention —
+        # the v4 receiver kept taking packets while the v6 one ran — but that
+        # depends on the kernel's bindv6only default rather than on anything
+        # stated here, so the roles are made explicit instead of inherited.
+        - name: socat-ipv6
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
+            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL

kubernetes/apps/home-assistant/mdns-relay-network-attachment.yaml

@@ -0,0 +1,52 @@
+---
+# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
+# presence on the IoT VLAN and can join both mDNS multicast groups
+# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
+# configuration of its own: it is link-local, and the kernel's automatic
+# link-local address on the macvlan is enough to receive on it, independently
+# of whether v30iot hands out global v6 addresses.
+# Multicast is link-local — it does not cross the L3
+# boundary between the cluster pod CIDR and v30iot — so that membership has to
+# be held by a pod sitting physically on that segment.
+#
+# The address is leased from the UniFi DHCP server on v30iot. Nothing
+# references it: the relay only receives on this interface and sends onward
+# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
+#
+# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
+# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
+# via the Image Factory schematic), identical across nodes and independent of
+# driver probe order. This is the same master shelly-manager-iot uses.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: mdns-relay-iot
+  namespace: home-assistant
+spec:
+  # Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
+  # v30iot address by DHCP, then `sbr` confines net1's routes to a
+  # source-based routing table. Without sbr the DHCP-supplied default gateway
+  # (10.2.30.1, option 3) installs a second default route at the same metric
+  # as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
+  # dropped. The relay's forward path to Home Assistant runs over eth0, so
+  # that route has to stay intact.
+  #
+  # Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "mdns-relay-iot",
+      "plugins": [
+        {
+          "type": "macvlan",
+          "master": "ens19",
+          "mode": "bridge",
+          "ipam": {
+            "type": "dhcp"
+          }
+        },
+        {
+          "type": "sbr"
+        }
+      ]
+    }

kubernetes/apps/home-assistant/mdns-relay-service.yaml

@@ -0,0 +1,42 @@
+---
+# Stable delivery target for the relay's forwarded mDNS traffic.
+#
+# The relay resolves its send target once, at process start (socat has no
+# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
+# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
+# Cilium re-DNATs to whatever pod currently backs it.
+#
+# Deliberately a separate Service from `home-assistant`, which targets the
+# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
+# This one bypasses the sidecar and addresses the home-assistant container's
+# own port directly — containers in a pod share a network namespace, so
+# targetPort 5353 lands on python-zeroconf's listener regardless of which
+# container opened it. Keeping the two apart means the UI path and the
+# discovery path cannot be changed by accident through one another.
+#
+# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
+# permits intra-namespace traffic on all ports, and the relay is in this
+# namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: home-assistant-mdns
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: home-assistant
+spec:
+  # Dual-stack, because mDNS is two protocols and HA listens on both: its
+  # zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
+  # relay forwards each family over its own transport rather than folding v6
+  # announcements into a v4 hop, so the Service has to answer on both.
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: home-assistant
+  ports:
+    - name: mdns
+      port: 5353
+      protocol: UDP
+      targetPort: 5353

kubernetes/apps/immich/barman-creds-externalsecret.yaml

@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: immich
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/immich/barman-creds

kubernetes/apps/immich/hetzner-creds-externalsecret.yaml

@@ -15,4 +15,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/immich/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/immich/kustomization.yaml

@@ -13,6 +13,5 @@ resources:
   - cluster.yaml
   - objectstore.yaml
   - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
   - hetzner-creds-externalsecret.yaml
   - pvcs.yaml

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -6,12 +6,14 @@ kind: Kustomization
 resources:
   - prometheusrule-ceph.yaml
   - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
   - prometheusrule-mariadb-backup.yaml
   - prometheusrule-netbird-availability.yaml
   - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
   # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -23,4 +25,5 @@ resources:
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
   - probes-blackbox.yaml

kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml

@@ -0,0 +1,92 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # Wear percentages are self-reported: 100 means the rated endurance is
+        # spent, not that the drive stops. Planning thresholds, not predictions.
+        - alert: NvmeDriveWearHigh
+          # Long `for` because wear is monotonic and cannot flap.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              {{ $value }}% of rated write endurance used. Nothing is failing
+              yet. Urgency depends on what the drive carries — the Disk Health
+              dashboard shows the role. Losing an OSD costs a rebalance;
+              EPHEMERAL is /var and rebuilds, apart from the node's etcd member
+              data, which has to be removed and re-added.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # Wear is a budget; spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # On the increase, not the total: pre-existing errors would otherwise
+          # alert forever. Standing counts are visible on the dashboard only.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # The check that the checks are running: no series fires no alert.
+        # Counted against node-exporter's inventory, not a fixed fleet size.
+        - alert: SmartctlExporterIncomplete
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.

kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml

@@ -40,17 +40,31 @@ spec:
             description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
 
         - alert: VeleroBackupStale
-          # Daily schedule with no successful backup in over 30h: covers
-          # the case where multiple consecutive runs failed (Garage flapping
-          # over the backup window) without the per-event alerts being
-          # acted on.
+          # Daily schedule with no successful backup in over 60h: covers the
+          # case where consecutive runs failed without the per-event alerts
+          # being acted on.
+          #
+          # 60h, not 30h. At 30h a single missed cycle pages, and a single
+          # missed cycle is the normal cost of touching the nodes: Velero moves
+          # data through pods, so draining a node kills whatever upload is in
+          # flight and the schedule is simply lost for that day — it is not
+          # retried. Rolling node work between 03:15 and 08:45 UTC therefore
+          # produced a guaranteed critical alert for every schedule it
+          # overlapped, which is noise about planned work rather than a signal
+          # about backups.
+          #
+          # 60h still catches what this is for. Two consecutive failures is no
+          # longer explicable by one maintenance window, and the per-event
+          # VeleroBackupFailed alert fires on the first failure regardless — so
+          # nothing is hidden, only the escalation to critical is deferred
+          # until a second cycle confirms it.
           expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
             and
             velero_backup_last_successful_timestamp{schedule!=""} > 0
           for: 10m
           labels:
             severity: critical
           annotations:
-            summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
-            description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
+            summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
+            description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."

kubernetes/apps/monitoring-rules/smartctl-exporter-servicemonitor.yaml

@@ -0,0 +1,25 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl polls the drives every 60s; scraping faster gains nothing.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this a series carries only the exporter pod's IP, which does
+        # not identify the machine the drive is in.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node

kubernetes/apps/monitoring/dashboards/disk-health.json

@@ -0,0 +1,475 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
+      ],
+      "title": "OSDs down",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory (identity and role)",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
+      "id": 14,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
+      "title": "Worst NVMe wear",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 15,
+      "options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
+          "legendFormat": "{{node}} {{model_name}}",
+          "refId": "A"
+        }
+      ],
+      "title": "NVMe wear",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": [
+          {"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
+          {"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
+          {"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
+          {"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 16,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
+      ],
+      "title": "SMART health",
+      "transformations": [
+        {"id": "merge", "options": {}},
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
+            "indexByName": {},
+            "renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
   - name: dashboard-gitea-overview
     files:
       - gitea-overview.json

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -275,8 +275,11 @@ spec:
           - alertmanager-smtp
       # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
       # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
       # monitoring ns is not blocked by any CNP today.
       config:
         global:

kubernetes/apps/monitoring/kustomization.yaml

@@ -15,6 +15,7 @@ resources:
   - prometheus-gate-service.yaml
   - prometheus-gate-ingress.yaml
   - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
   - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the

kubernetes/apps/monitoring/smartctl-exporter.yaml

@@ -0,0 +1,89 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# Runs privileged: host device access is gated by the container device cgroup,
+# and Kubernetes offers no per-device allowlist. Narrower settings do not work;
+# the alternatives that were tried are in the commit history.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device: the exporter scans and rescans on its own.
+          # Device names are not stable across these machines.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: true
+            # The API rejects privileged together with
+            # allowPrivilegeEscalation: false.
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # Not read-only: smartmontools opens NVMe devices read-write.
+            - name: dev
+              mountPath: /dev
+            # Scratch space, because the root filesystem is read-only.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP

kubernetes/apps/music-assistant/deployment.yaml

@@ -55,7 +55,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -65,7 +65,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/apps/paperless/barman-creds-externalsecret.yaml

@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: paperless
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/paperless/barman-creds

kubernetes/apps/paperless/hetzner-creds-externalsecret.yaml

@@ -1,3 +1,9 @@
+# One Hetzner S3 key, one path, shared by every barman ObjectStore. Separate
+# per-app paths would not make rotation independent — that needs one Hetzner
+# key per consumer (#397).
+#
+# velero reads the same key from infrastructure/velero-hetzner, which holds it
+# as a credentials file rather than these three fields.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
@@ -15,4 +21,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/paperless/kustomization.yaml

@@ -20,7 +20,6 @@ resources:
   - cluster.yaml
   - objectstore.yaml
   - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
   - hetzner-creds-externalsecret.yaml
   - pvcs.yaml
 configMapGenerator:

kubernetes/apps/trmnl/hetzner-creds-externalsecret.yaml

@@ -12,4 +12,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/trmnl/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,8 @@
 ---
 # Barman S3 credentials for wp-pg.
 #
-# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
-# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
-# that path needs OpenBao write access (`bao kv list seed` → 403 for the
-# identities available here), so it stays open. The values are identical, so it
-# is a one-line swap of the `key:` below once the path exists.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/wordpress/externalsecret.yaml

@@ -36,11 +36,9 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
-    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
-    # written, and repoint this here (the value is the same, so it's a drop-in
-    # swap).
+    # One shared path for one shared credential. There is a single Hetzner S3
+    # key behind all of these — see the note in
+    # ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
+    # the wrong shape and what would have to change for them to be right.
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml

@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -33,22 +33,14 @@ spec:
     remediation:
       retries: 3
   values:
-    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
-    # datapath options once, at agent start, and the chart puts no config
-    # checksum on the pod template — so a changed ConfigMap leaves every agent
-    # running the previous datapath until something restarts it. Nothing did.
+    # Rolls the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once at agent start and the chart puts no config
+    # checksum on the pod template, so without this a changed ConfigMap leaves
+    # every agent on the previous datapath indefinitely (#592).
     #
-    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
-    # v30iot macvlan workload stayed broken until the agents were deleted by
-    # hand, one node at a time (issue #592). policyMapMax has the same
-    # exposure, and its comment below says "takes effect on agent restart" —
-    # documentation standing in for automation.
-    #
-    # The trade: any cilium-config change now rolls the CNI cluster-wide,
-    # node by node. That is disruptive-ish, but it is what a datapath change
-    # already required — the difference is that it now happens on its own
-    # instead of being forgotten. Reloader is already deployed for exactly
-    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    # The trade is that any cilium-config change now rolls the CNI node by
+    # node — which is what a datapath change required anyway, only now it is
+    # not forgotten.
     annotations:
       configmap.reloader.stakater.com/reload: cilium-config
     # Which pod labels feed security-identity computation. Patterns are
@@ -69,57 +61,42 @@ spec:
     socketLB:
       enabled: true
     prometheus:
-      # The agent exported no metrics at all, so BPF map utilisation was
-      # invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
-      # of space the agent only says "no space left on device" in its log and
-      # pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
-      # that observable — and alertable — before it stalls a rollout.
+      # Without agent metrics, BPF map utilisation is invisible: a full policy
+      # map only shows up as "no space left on device" in the log while pod
+      # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
+      # alertable beforehand.
       enabled: true
       serviceMonitor:
         enabled: true
     bpf:
-      # Size of each endpoint's own cilium_policy_v3_<id> map, holding one
-      # entry per (identity, direction, protocol, port) the endpoint is
-      # allowed. When it fills, endpoint regeneration fails with "no space
-      # left on device", the CNI ADD call never returns, and the pod is stuck
-      # in sandbox creation — so this must stay comfortably above the busiest
-      # endpoint's entry count, tracked by cilium_bpf_map_pressure.
-      # Takes effect on agent restart; the map is sized at creation.
-      # Bootstrap installs Cilium from the inline manifest in
-      # kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
-      # must move with this value or a cold rebuild starts with the old size.
-      # kubernetes/talos/cilium.yaml carries a third copy of the same key.
-      # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
+      # Per-endpoint policy map size: one entry per (identity, direction,
+      # protocol, port) allowed. When it fills, endpoint regeneration fails
+      # with "no space left on device" and the pod hangs in sandbox creation.
+      # Keep it comfortably above the busiest endpoint, watched by
+      # cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
+      # restart.
+      #
+      # THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
+      # and talos/cilium.yaml. The latter two are what a cold rebuild installs
+      # from. Nothing enforces agreement; grep bpf-policy-map-max before
+      # changing it.
       policyMapMax: 65536
       # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
       # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
       # filter" in cilium_drop_count_total.
       #
-      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
-      # otbr and music-assistant attachments. On bare metal it is a VLAN
-      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
-      # an untagged virtio NIC, which is why this was not needed before.
+      # Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
+      # carry an address, so it sees mgmt0's frames already untagged by the
+      # 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
+      # address-less, so its frames only meet the filter on the shared parent,
+      # still tagged. Adding 40 would change nothing and would touch the
+      # datapath carrying etcd and the API.
       #
-      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
-      # interesting part. Cilium attaches to devices that carry an address —
-      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
-      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
-      # a device and sees frames the 8021q layer has already untagged; nothing
-      # for a VLAN filter to match. ens19 is deliberately address-less (see
-      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
-      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
-      # meet the filter on the shared parent, still tagged.
+      # The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
+      # reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
+      # "no buffer space available" (#592).
       #
-      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
-      # would touch the datapath carrying etcd and the API for no reason.
-      #
-      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
-      # symptom is the CNI dhcp plugin reporting "no buffer space available"
-      # rather than a DHCP timeout. See issue #592.
-      #
-      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
-      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
-      # must move with this list. Takes effect on agent restart.
+      # Same three-copy problem as policyMapMax above. Takes an agent restart.
       vlanBypass: [30]
       # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
       # redirects the socket directly to the backend pod instead of sending

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -36,25 +36,16 @@ spec:
   values:
     container_runtime: containerd
 
-    # --- mTLS between agents and the LAPI: the durable fix for the recurring
-    # KubeDaemonSetRolloutStuck auth loop (#340) ---
+    # mTLS between agents and the LAPI. Do not switch back to password auth:
+    # the LAPI runs without persistence, so its machine-registration DB is
+    # wiped on every restart, and password-registered agents then get 401
+    # "machine not found" and crashloop until re-registered by hand — which
+    # wedges the whole `infrastructure` Kustomization behind a HelmRelease
+    # that never goes Ready (#340).
     #
-    # The LAPI runs WITHOUT persistence (see persistentVolume note below), so
-    # its sqlite machine-registration DB is ephemeral and every LAPI pod restart
-    # wiped it. Under the old password flow the agents registered exactly ONCE
-    # (init container running `cscli lapi register --token`) and cached a
-    # password the wiped LAPI no longer recognised -> 401 "machine not found"
-    # -> CrashLoopBackOff on all three agents -> DaemonSet never Ready -> the
-    # crowdsec HelmRelease upgrade `--wait` times out forever -> the whole
-    # `infrastructure` Kustomization (and every app depending on it) stays
-    # unhealthy. That is the #340 outage.
-    #
-    # With client-cert auth the agent init container no longer registers at all
-    # (verified via `helm template`: the `cscli lapi register` step is gone).
-    # The agent presents a cert whose OU (agent-ou) the LAPI trusts, and a
-    # freshly restarted LAPI with an EMPTY DB auto-recreates each machine record
-    # from the presented cert on connect. So the system self-heals across LAPI
-    # restarts with no PVC, no Ceph, and no manual re-registration.
+    # With client certs the agent never registers: a restarted LAPI with an
+    # empty DB recreates each machine record from the presented cert, so this
+    # self-heals with no PVC and no intervention.
     #
     # cert-manager (already reconciled earlier in this same `infrastructure`
     # stage) provisions a fully self-contained self-signed CA (selfSigned
@@ -87,13 +78,10 @@ spec:
 
     lapi:
       replicas: 1
-      # The chart defaults request 500m. Measured steady-state draw is 2m, and
-      # the request is what the scheduler reserves, so the default parks half a
-      # core of unusable capacity per LAPI. That is affordable on three nodes
-      # and is not on two: with one node out for maintenance the remaining pair
-      # sit at 100% of requested CPU while actually using ~22%, and nothing new
-      # can be placed. The limit stays at the chart default so a log flood can
-      # still burst; only the reservation shrinks.
+      # Far below the chart default, which reserves half a core per LAPI for a
+      # workload that draws single-digit millicores — enough to leave the
+      # cluster unschedulable with one node out. Only the reservation shrinks;
+      # the limit stays at the chart default so a log flood can still burst.
       resources:
         requests:
           cpu: 50m

kubernetes/infrastructure/multus/kustomization.yaml

@@ -4,3 +4,12 @@ resources:
   - cni-plugins-installer.yaml
   - cni-dhcp-daemon.yaml
   - multus-daemonset.yaml
+  # Whereabouts — cluster-coordinated IPAM, for attachments on segments with
+  # no DHCP server. CRDs first: the DaemonSet's control loop reads them at
+  # startup. See the header in whereabouts-daemonset.yaml for why neither
+  # dhcp nor host-local covers the Ceph SAN.
+  - whereabouts-crd-ippools.yaml
+  - whereabouts-crd-overlappingrangeipreservations.yaml
+  - whereabouts-crd-nodeslicepools.yaml
+  - whereabouts-daemonset.yaml
+  - whereabouts-reconciler.yaml

kubernetes/infrastructure/multus/whereabouts-crd-ippools.yaml

@@ -0,0 +1,71 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: ippools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: IPPool is the Schema for the ippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec defines the desired state of IPPool
+            properties:
+              allocations:
+                additionalProperties:
+                  description: IPAllocation represents metadata about the pod/container
+                    owner of a specific IP
+                  properties:
+                    id:
+                      type: string
+                    ifname:
+                      type: string
+                    podref:
+                      type: string
+                  required:
+                  - id
+                  - podref
+                  type: object
+                description: |-
+                  Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
+                  IP with the same index/offset for the pool's range.
+                type: object
+              range:
+                description: Range is a RFC 4632/4291-style string that represents
+                  an IP address and prefix length in CIDR notation
+                type: string
+            required:
+            - allocations
+            - range
+            type: object
+        type: object
+    served: true
+    storage: true

kubernetes/infrastructure/multus/whereabouts-crd-nodeslicepools.yaml

@@ -0,0 +1,79 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: nodeslicepools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: NodeSlicePool
+    listKind: NodeSlicePoolList
+    plural: nodeslicepools
+    singular: nodeslicepool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NodeSlicePool is the Schema for the nodesliceippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
+            properties:
+              range:
+                description: |-
+                  Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
+                  this refers to the entire range where the node is allocated a subset
+                type: string
+              sliceSize:
+                description: SliceSize is the size of subnets or slices of the range
+                  that each node will be assigned
+                type: string
+            required:
+            - range
+            - sliceSize
+            type: object
+          status:
+            description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
+            properties:
+              allocations:
+                description: Allocations holds the allocations of nodes to slices
+                items:
+                  properties:
+                    nodeName:
+                      description: NodeName is the name of the node assigned to this
+                        slice, empty node name is an available slice for assignment
+                      type: string
+                    sliceRange:
+                      description: SliceRange is the subnet of this slice
+                      type: string
+                  required:
+                  - nodeName
+                  - sliceRange
+                  type: object
+                type: array
+            required:
+            - allocations
+            type: object
+        type: object
+    served: true
+    storage: true

kubernetes/infrastructure/multus/whereabouts-crd-overlappingrangeipreservations.yaml

@@ -0,0 +1,57 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: overlappingrangeipreservations.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: OverlappingRangeIPReservation
+    listKind: OverlappingRangeIPReservationList
+    plural: overlappingrangeipreservations
+    singular: overlappingrangeipreservation
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: OverlappingRangeIPReservationSpec defines the desired state
+              of OverlappingRangeIPReservation
+            properties:
+              containerid:
+                type: string
+              ifname:
+                type: string
+              podref:
+                type: string
+            required:
+            - podref
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true

kubernetes/infrastructure/multus/whereabouts-daemonset.yaml

@@ -0,0 +1,196 @@
+---
+# Whereabouts: cluster-wide IPAM for Multus attachments.
+#
+# WHY THIS EXISTS
+#
+# The attachments in this cluster lease their addresses by DHCP
+# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
+# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
+# Ceph-internal traffic and nothing else, by design: there is no DHCP server
+# there and there must not be one.
+#
+# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
+# in. It keeps its allocation state per node with no cluster-wide
+# coordination, so on a shared L2 segment two nodes hand out the same address.
+# Whereabouts coordinates through the Kubernetes API instead — allocations
+# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
+# the address assignment touches the segment being addressed. That property is
+# what makes it usable on an isolated SAN.
+#
+# HOST ACCESS
+#
+# The DaemonSet runs with hostNetwork and privileged. That is how upstream
+# ships it, and it is what the three CNI components already here do —
+# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
+# all carry both. The reason is the same in each case: the pod writes a CNI
+# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
+# which is node plumbing rather than a workload, and it must work before the
+# attachment it provides addresses for exists.
+#
+# The reconciler is deliberately NOT in here. It needs neither, and runs as an
+# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
+#
+# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
+# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
+# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
+# upstream diff stays readable.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: whereabouts
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: whereabouts
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: whereabouts-cni
+subjects:
+  - kind: ServiceAccount
+    name: whereabouts
+    namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: whereabouts-cni
+rules:
+  - apiGroups:
+      - whereabouts.cni.cncf.io
+    resources:
+      - ippools
+      - overlappingrangeipreservations
+      - nodeslicepools
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - coordination.k8s.io
+    resources:
+      - leases
+    verbs:
+      - '*'
+  - apiGroups: [""]
+    resources:
+      - pods
+    verbs:
+      - list
+      - watch
+      - get
+  - apiGroups: [""]
+    resources:
+      - nodes
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - network-attachment-definitions
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+      - get
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: whereabouts-config
+  namespace: kube-system
+  annotations:
+    kubernetes.io/description: |
+      Configmap containing user customizable cronjob schedule
+data:
+  # Read by the reconciler (whereabouts-reconciler.yaml), not by this
+  # DaemonSet. Governs how often stale allocations are swept — the case that
+  # matters here is a node lost hard enough that the CNI DEL never ran, which
+  # otherwise leaves an address held by a pod that no longer exists.
+  cron-expression: "30 4 * * *"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: whereabouts
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts
+spec:
+  selector:
+    matchLabels:
+      name: whereabouts
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts
+        name: whereabouts
+    spec:
+      hostNetwork: true
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node, so the NoSchedule toleration
+      # is not optional — without it this schedules nowhere.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts
+          command: ["/bin/sh"]
+          args:
+            - -c
+            - |
+              SLEEP=false source /install-cni.sh
+              /token-watcher.sh &
+              /ip-control-loop -log-level debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: NODENAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: spec.nodeName
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "100Mi"
+            limits:
+              cpu: "100m"
+              memory: "200Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
+            - name: cni-net-dir
+              mountPath: /host/etc/cni/net.d
+      volumes:
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: cni-net-dir
+          hostPath:
+            path: /etc/cni/net.d

kubernetes/infrastructure/multus/whereabouts-reconciler.yaml

@@ -0,0 +1,75 @@
+---
+# Sweeps whereabouts allocations that no live pod holds any more.
+#
+# The normal release path is the CNI DEL when a pod goes away. It does not run
+# when a node is lost hard — power cut, kernel panic, a reboot that never
+# comes back — and the allocation then stays claimed by a pod that no longer
+# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
+# of leaked addresses is the difference between an OSD getting an address and
+# not.
+#
+# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
+# this needs no host access at all — it only talks to the Kubernetes API,
+# where the allocations live.
+#
+# replicas stays at 1: there is no leader election, so a second replica would
+# reconcile the same records concurrently.
+#
+# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
+# image tag pinned; upstream ships `:latest`. The schedule comes from the
+# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: whereabouts-reconciler
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts-reconciler
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: whereabouts-reconciler
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts-reconciler
+        name: whereabouts-reconciler
+    spec:
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts-reconciler
+          command: ["/ip-reconciler"]
+          args:
+            - -log-level
+            - debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          volumeMounts:
+            - name: cron-scheduler-configmap
+              mountPath: /cron-schedule
+      volumes:
+        - name: cron-scheduler-configmap
+          configMap:
+            name: whereabouts-config
+            defaultMode: 0744
+            items:
+              - key: "cron-expression"
+                path: "config"

kubernetes/infrastructure/netbird-ingress/daemonset.yaml

@@ -6,19 +6,12 @@
 # fails the route over across all per-node peers (no single point of
 # failure if a node/pod drops).
 #
-# Uses the ROOTLESS image (userspace wireguard-go). The netbird process
-# terminates the WireGuard tunnel in userspace and reaches cluster
-# services by opening its OWN sockets to them — NOT by L3-forwarding
-# packets. That distinction is the whole point:
-#   - Cilium socket-LB DNATs the netbird process's own connections, so
-#     forwarding to ClusterIP / LB VIPs actually works. A kernel L3
-#     routing peer fails here: socket-LB never sees forwarded packets,
-#     so they hit the raw VIP and are blackholed (confirmed via conntrack).
-#   - No NET_ADMIN / no kernel WireGuard → runs fully unprivileged
-#     (drop ALL caps, runAsNonRoot), so it satisfies Pod Security too.
-#
-# Flow: mesh peer (100.96.x.x) --wg--> a per-node pod (userspace netbird)
-#        --own socket (socket-LB DNAT)--> service ClusterIP/LB VIP
+# Must stay on the ROOTLESS image (userspace wireguard-go). The process
+# terminates the tunnel in userspace and reaches services by opening its own
+# sockets, which Cilium's socket-LB can DNAT. A kernel WireGuard peer
+# L3-forwards instead, socket-LB never sees those packets, and they are
+# blackholed at the raw VIP. Rootless also means no NET_ADMIN, so the pod
+# runs fully unprivileged.
 #
 # Egress is locked down by the netbird-ingress-egress CiliumNetworkPolicy
 # (network-policies/ingress-isolation.yaml).

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -41,18 +41,14 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # shelly-manager (:8080) is likewise reached only through its own
-    # Authentik gate outpost. Note this rule and the vscode one are
-    # port-identical: because this CNP's endpointSelector is {}, a
-    # `toPorts: 8080` grant here applies to EVERY pod in the namespace that
-    # listens on 8080, not just the intended one — which is exactly how
-    # shelly-manager silently rode on the old traefik→:8080 allow written
-    # for code-server, and why removing that allow in 760e187 took
-    # shelly-manager's ingress down with it. The `fromEndpoints` selector is
-    # what keeps these two apart: each gate outpost can only reach its own
-    # target, enforced on the egress side by the matching CNPs in the
-    # authentik namespace (vscode-gate-egress.yaml,
-    # shelly-manager-gate-egress.yaml).
+    # shelly-manager, behind its own Authentik gate outpost.
+    #
+    # CAUTION for every :8080 rule here: this policy's endpointSelector is {},
+    # so a toPorts grant on 8080 applies to EVERY pod in the namespace
+    # listening on that port, not just the intended one. Three apps share
+    # :8080. What keeps them apart is the fromEndpoints selector plus the
+    # matching per-gate egress CNPs in the authentik namespace. Changing a
+    # :8080 rule here affects all of them.
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": authentik
@@ -61,11 +57,8 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # esphome (ESPHome Device Builder, :6052) is likewise reached only through
-    # its own Authentik gate outpost. Unlike the :8080 vscode/shelly rules
-    # above, :6052 is unique to esphome, so this namespace-wide (endpointSelector
-    # {}) toPorts grant doesn't silently apply to any other app — the
-    # fromEndpoints selector still scopes it to the esphome-gate outpost.
+    # esphome (:6052), behind its own gate outpost. Port is unique to it, so
+    # the namespace-wide grant does not reach anything else.
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": authentik
@@ -74,13 +67,8 @@ spec:
         - ports:
             - port: "6052"
               protocol: TCP
-    # zigbee2mqtt (Z2M frontend, :8080) is likewise reached only through its own
-    # Authentik gate outpost. Like the vscode/shelly-manager rules above this is
-    # the SHARED :8080 (zigbee2mqtt is the third app on that port), so this
-    # namespace-wide (endpointSelector {}) grant applies to every :8080 pod — the
-    # fromEndpoints selector scopes it to the zigbee2mqtt-gate outpost, and the
-    # matching authentik-side zigbee2mqtt-gate-egress.yaml scopes egress to the
-    # zigbee2mqtt pod specifically.
+    # zigbee2mqtt frontend, behind its own gate outpost. Shared :8080 — see the
+    # caution above.
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": authentik
@@ -89,11 +77,8 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # zone-configurator (Everything Presence mmWave zone configurator, :42069)
-    # is likewise reached only through its own Authentik gate outpost. Like
-    # esphome's :6052, :42069 is unique to it, so this namespace-wide (endpoint
-    # Selector {}) toPorts grant doesn't silently apply to any other app; the
-    # fromEndpoints selector scopes it to the zone-configurator-gate outpost.
+    # zone-configurator (:42069), behind its own gate outpost. Port is unique
+    # to it.
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": authentik
@@ -131,25 +116,16 @@ spec:
     - fromEntities:
         - host
         - remote-node
-    # World ingress (IoT-VLAN devices via the BGP LoadBalancers) is NOT
-    # granted namespace-wide here. A blanket `fromEntities: world` on this
-    # endpointSelector:{} policy exposed EVERY pod in the namespace to the
-    # untrusted IoT VLAN / any BGP-routed source — vscode (:8080),
-    # zone-configurator (:42069), otbr, whisper, etc. — not just the two
-    # LoadBalancers that need it. World is now scoped per-target in the
-    # dedicated policies at the bottom of this file: world -> mosquitto
-    # :1883/:8883 and world -> home-assistant :8123 only. (matter-server on
-    # 10.2.x.226 rides a multus net1 interface that bypasses Cilium policy
-    # entirely, so it is unaffected either way.)
-    # ICMP — all standard ICMPv4 and ICMPv6 types from any cluster pod.
-    # Per Cilium docs, "If any layer 4 policy is specified, then ICMP
-    # will be blocked unless it's related to a connection that is
-    # otherwise allowed by the policy." The CT-RELATED path is not
-    # passing these in practice (Hubble shows ICMPv6 DestinationUnreachable
-    # being POLICY_DENIED on the egress side of established DNS flows),
-    # so allow ICMP explicitly. Same shape as commit 3c517a0 — widened
-    # from the error class to the full standard type set. Cilium's
-    # ICMPField requires an explicit type per entry; no wildcard.
+    # Do NOT add a blanket `fromEntities: world` to this policy. With
+    # endpointSelector {} it exposes every pod in the namespace to the IoT
+    # VLAN and any BGP-routed source, not just the LoadBalancers that need it.
+    # World is scoped per target in the dedicated policies at the bottom of
+    # this file. (matter-server rides a multus interface that bypasses Cilium
+    # policy entirely and is unaffected either way.)
+    # ICMP must be allowed explicitly: once any L4 policy exists Cilium blocks
+    # it unless CT-RELATED passes it, which in practice it does not. Cilium's
+    # ICMPField takes one type per entry — there is no wildcard, so the list
+    # has to be spelled out.
     - fromEndpoints:
         - matchExpressions:
             - key: io.kubernetes.pod.namespace
@@ -186,10 +162,8 @@ spec:
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": apps
-    # In-cluster InfluxDB: HA's HACS InfluxDB integration writes state
-    # changes here (replaces the legacy http://influxdb.srv.renner.uno
-    # target — see kubernetes/apps/influxdb/ha-token-job.yaml for the
-    # write-bucket token wiring).
+    # In-cluster InfluxDB: HA's HACS integration writes state changes here.
+    # Token wiring is in apps/influxdb/ha-token-job.yaml.
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": influxdb
@@ -213,9 +187,8 @@ spec:
     - toEntities:
         - host
         - remote-node
-    # World: IoT-VLAN devices, ESPHome, HA Cloud, garage S3 (initial
-    # tarball seed), legacy InfluxDB on VM 108, third-party integration
-    # endpoints (Hue, Roomba, Philips, Brother IPP, …).
+    # World: IoT-VLAN devices, HA Cloud, object storage, and third-party
+    # integration endpoints on the LAN and internet.
     - toEntities:
         - world
     # ICMP — egress mirror of the ingress ICMP allow above. Lets HA

kubernetes/infrastructure/network-policies/woodpecker-ns.yaml

@@ -1,12 +1,5 @@
 ---
-# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
-# agent arrive in stage 2).
-#
-# Unlike the `apps` baseline this replaces, every allow here lands on a
-# namespace holding one service, so a namespace-scoped grant is not a blanket
-# grant across unrelated workloads. That is the point of the move: in `apps`,
-# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
-# external-dns and ical-proxy to the whole database namespace (issue #433).
+# Baseline ingress policy for the `woodpecker` namespace.
 #
 # CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
 # every rule below applies to EVERY pod in the namespace. A `toPorts` block
@@ -15,11 +8,7 @@
 # belongs in its own policy with its own selector, as
 # `wp-pg-allow-cnpg-operator` below does.
 #
-# No egress section, so egress is unrestricted. Deliberate for now and revisited
-# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
-# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
-# Writing that ruleset once, with the full workload set present, beats guessing
-# it here and silently breaking the WAL archiving this stage exists to verify.
+# There is no egress section, so egress is unrestricted. Still open (#433).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:

kubernetes/infrastructure/openbao/helmrelease.yaml

@@ -42,12 +42,28 @@ spec:
       # the agent-injector mutating webhook is unused surface.
       enabled: false
     server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
       ha:
         enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
         raft:
           enabled: true
           setNodeId: true
@@ -65,8 +81,32 @@ spec:
               cluster_address = "[::]:8201"
             }
 
+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
             storage "raft" {
               path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
             }
 
             service_registration "kubernetes" {}

kubernetes/infrastructure/openbao/rotation/kustomization.yaml

@@ -5,25 +5,15 @@ kind: Kustomization
 # secret. The matching OpenBao policy/role for each lives in ../policies/ and is
 # bootstrapped manually (not GitOps-managed).
 #
-# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
-# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
-# image, so it does not need digest pinning, and a dated tag makes "what actually
-# runs" a reviewable diff.
+# IMAGE PINNING: dated tags (`:YYYY-MM-DD`), never a digest and never
+# `:latest`. Do NOT switch to digests — untagged manifests in this registry are
+# garbage-collected once the tags move past them, so a pinned digest silently
+# stops resolving after a few unrelated rebuilds of build-ci-images.yml. These
+# jobs are quarterly, so the resulting ImagePullBackOff looks like nothing
+# happening rather than like a failure (#458).
 #
-# Digests were tried and actively broke: untagged manifests are garbage-collected
-# once :latest and the dated tags move past them, so a digest pinned today stops
-# resolving after a few rebuilds. That decay is driven by edits to
-# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
-# republishes secret-rotator and orphans the manifest every rotator here was
-# pinned to. 11 of these were pinned to digests that no longer existed and would
-# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
-# are quarterly, that would have looked like nothing happening rather than like a
-# failure.
-#
-# Keep them all on one tag: it is a single binary containing every mode
-# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
-# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
-# the next bump partial.
+# Keep every rotator on the same tag: it is one binary containing all modes, so
+# splitting only makes the next bump partial.
 #
 # ADDING A ROTATOR: copy an existing file, and check you did not inherit a
 # digest pin from an older example — nothing enforces this automatically, so it

kubernetes/infrastructure/openbao/snapshot-cronjob.yaml

@@ -53,7 +53,13 @@ spec:
               command: ["/bin/sh", "-ec"]
               args:
                 - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                   BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                     role=openbao-snapshot \
                     jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

kubernetes/infrastructure/post-crds/cilium-bgp.yaml

@@ -24,12 +24,10 @@ spec:
     - start: "fd99::"
       stop: "fd99::ff"
 ---
-# Cilium BGPv2 control-plane config peering each k8s node with the UDM
-# over its MGMT LAN interface — NOT over v30iot. Control-plane traffic
-# belongs on the management network; the IoT VLAN is the noisy untrusted
-# L2 segment we're explicitly trying to keep cluster infrastructure off
-# of. Each node sources from its eth0 (10.2.40.x / fd40::<host>) via
-# the kernel routing decision; no explicit localAddress override needed.
+# Cilium BGPv2 control-plane config peering each node with the UDM over the
+# management LAN — never over v30iot, which is the untrusted IoT segment.
+# The source address comes from the kernel routing decision, so no explicit
+# localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
 #   - v4 session: peer 10.2.40.1, family IPv4 unicast only

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -1,27 +1,16 @@
 ---
 # Velero Schedules for stateful app namespaces.
 #
-# Scope is restricted to PersistentVolumeClaims via includedResources.
-# Flux owns every other K8s object (Deployment, Service, Ingress,
-# ConfigMap, Secret, …) — backing those up shadows current git state
-# on a rebuild: Velero restores stale manifests from yesterday's
-# snapshot, then the app Kustomization clobbers them with whatever git
-# says — except when the Kustomization is blocked on the restore chain,
-# leaving the cluster temporarily serving the stale spec (e.g. the
-# paperless Ingress missing prod hostnames after the 2026-04-22
-# rebuild). PVC-only is the canonical "back up data, not state" scope
-# (see ~/Notes/wiki/k8s-backup-scope.md and the project memory rule).
+# Scope is PersistentVolumeClaims only. Flux owns every other object, and
+# backing those up shadows git on a rebuild: Velero restores yesterday's
+# manifests and the cluster serves them until the Kustomization catches up.
+# Back up data, not state.
 #
-# snapshotMoveData=true keeps doing the heavy lifting via Kopia +
-# node-agent: PVC contents are uploaded to Garage and restored
+# snapshotMoveData uploads PVC contents through Kopia and the node-agent,
 # independently of the K8s API surface.
 #
-# Naming convention: <app>-daily for the app's Schedule. Restore is a
-# manual, operator-driven procedure — see kubernetes/RUNBOOK-cold-rebuild.md.
-# (There is no automatic bootstrap-restore Job; it was removed as dead code,
-# issue #200.)
-#
-# Retention: 30 days, matching the live CNPG Barman ObjectStore policy.
+# Naming: <app>-daily. Restore is manual — see RUNBOOK-cold-rebuild.md.
+# Retention matches the CNPG Barman policy.
 
 apiVersion: velero.io/v1
 kind: Schedule
@@ -203,22 +192,13 @@ metadata:
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
-  # their manifest (matter-server, otbr, whisper, speech-to-phrase,
-  # zone-configurator all set `namespace: home-assistant` via their
-  # Kustomization, not per-resource): home-assistant-config (.storage
-  # entity registry, recorder DB, secrets.yaml), esphome-data,
-  # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
-  # own path (the HA pod's `seed` initContainer pulling the native HAOS
-  # backup from Garage, see the home-assistant Kustomization comment in
-  # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
-  # PVC restore on top of that flow for the same PVC needs its own review.
-  # Restore of these schedules is manual — see
-  # kubernetes/RUNBOOK-cold-rebuild.md.
+  # Daily at 06:45 local. Sweeps every PVC in the home-assistant namespace —
+  # several apps under apps/ set `namespace: home-assistant` in their
+  # Kustomization, so their PVCs land here too.
+  #
+  # home-assistant-config has its own restore path: the HA pod's `seed`
+  # initContainer pulls the native HAOS backup. Layering a Velero PVC restore
+  # on top of that for the same PVC needs its own review.
   schedule: "45 6 * * *"
   useOwnerReferencesInBackup: false
   template:
@@ -229,6 +209,13 @@ spec:
     storageLocation: default
     ttl: 720h0m0s
     snapshotMoveData: true
+    # Excludes the CNPG cluster's PVC. A crash-consistent snapshot of a
+    # running Postgres data directory is not a restorable backup, and Barman
+    # covers that database anyway — see ha-pg-scheduled-backup.yaml.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -8,6 +8,10 @@ spec:
     image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
+  # Defaults to false. With three OSDs and size 3 there is no spare copy, so
+  # this gate is the only thing keeping a rolling OSD change from taking the
+  # next one down mid-recovery. Paired with storage.osdMaxUpdatesInParallel.
+  upgradeOSDRequiresHealthyPGs: true
   mon:
     count: 3
     allowMultiplePerNode: false
@@ -28,112 +32,40 @@ spec:
       daemon:
         keyRotationPolicy: KeyGeneration
         keyGeneration: 2
-  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
-  # Data Movement provisions a backupPVC per source PVC by cloning the
-  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's
-  # multi-PVC Backup with `rados: "all cloner threads are busy"`. 16
-  # gives headroom for parallel restores during cluster rebuild too
-  # (paperless + audiobookshelf + immich + ... all clone at once).
+  # CephFS cloner pool raised from the default 4. Velero clones a
+  # VolumeSnapshot per PVC, and 4 threads stall a multi-PVC backup with
+  # `rados: "all cloner threads are busy"`.
   cephConfig:
     mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than omitted. Deleting a line here does
+      # NOT restore the default: `ceph config` keeps whatever was last applied,
+      # so the old value would sit in the mon config database indefinitely.
+      # Change values in place; to actually drop one, `ceph config rm` it.
+      mon_data_avail_warn: "30"
+      # Off, against the upstream default. The forced compact_range on every
+      # paxos trim rewrote the whole store hundreds of times a day and was the
+      # single largest source of disk writes on these nodes.
       #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
-      # Stop compacting the whole RocksDB prefix on every paxos trim.
-      #
-      # Upstream default is true — "Compact a certain prefix (including paxos)
-      # when we trim its old states" — which is fine at low paxos rates and
-      # pathological here. Measured 2026-08-19:
-      #
-      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
-      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
-      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
-      #                                        (cgroup io.stat: 2.15 TB/27 d)
-      #   => ~98x write amplification, for a ~106 MB database
-      #
-      # That was ~62% of all OS-disk writes on these nodes and roughly a
-      # quarter of everything the cluster writes — more than every client
-      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
-      # means hundreds of forced range compactions per day, each rewriting the
-      # store.
-      #
-      # The churn itself is 99% cluster log and mgr stats, not real cluster
-      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
-      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
-      # report / osd dump — 23 audit entries per minute per mon) becomes a
-      # cluster-log entry, hence a paxos transaction, replicated to all three
-      # mons. Reducing that polling is the complementary fix and is tracked
-      # separately; this setting removes the amplification regardless of rate.
-      #
-      # This does NOT disable compaction. RocksDB's own levelled compaction
-      # still runs and still reclaims trimmed keys; what stops is the extra
-      # forced compact_range on every trim. Store growth is therefore bounded
-      # by normal compaction, and there are two backstops if that assumption is
-      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
-      # CephHealthWarn in monitoring-rules already alerts on, and
-      # mon_compact_on_start below compacts fully on every mon restart.
-      #
-      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
-      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
-      # not come back down, revisit rather than leaving it.
+      # This does NOT disable compaction — RocksDB's levelled compaction still
+      # reclaims trimmed keys. Backstops if that proves wrong:
+      # mon_data_size_warn raises HEALTH_WARN, and mon_compact_on_start below
+      # compacts fully on every mon restart.
       mon_compact_on_trim: "false"
-      # Compensating control for the above: a full compaction on mon start.
-      # Upstream default is false. Mons restart on upgrades, node reboots and
-      # reschedules, so this gives a periodic full reclaim without paying for
-      # one on every trim. Cheap here — the store is ~106 MB.
+      # Compensating control, against the upstream default: mons restart often
+      # enough for this to give periodic full reclaim without paying for one
+      # on every trim.
       mon_compact_on_start: "true"
     mgr:
       mgr/volumes/max_concurrent_clones: "16"
-    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
-    # earlier comments here said "P2", which is a different drive). These need
-    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
-    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM. Init-time settings — an OSD restart is needed to apply.
     #
-    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
-    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
-    # again without re-reading this.
-    #
-    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
-    # prefer_deferred_size defers every write SMALLER than the threshold: the
-    # write goes first into RocksDB's WAL and is applied to the object store
-    # afterwards. That is a double write, and at 65536 it caught essentially
-    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
-    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
-    # state does not need the random-write batching that helps spinning rust.
-    #
-    # Two reasons it was the wrong trade here:
-    #   - The benchmark that justified it cannot have exercised it. It used
-    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
-    #     was ever deferred. The setting was validated by a workload it does
-    #     not affect.
-    #   - It doubles writes onto QLC, which has the lowest endurance of any
-    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
-    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
-    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
-    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
-    #
-    # This cluster is write-light (~117 GB/day at the RBD client layer), so
-    # throughput was never the binding constraint. Endurance is. If small-write
-    # latency ever becomes the real problem, fix it with more OSDs rather than
-    # by paying a 2x write tax on every 4 KiB page.
+    # bluestore_prefer_deferred_size_ssd and bluestore_deferred_batch_ops_ssd
+    # are DELIBERATELY LEFT AT THE CEPH DEFAULTS and must not be raised again.
+    # Deferral double-writes everything below the threshold, and an HDD-style
+    # value catches every write this cluster makes (4 KiB SQLite, 8 KiB
+    # Postgres pages). Ceph disables it on SSD for that reason. Throughput has
+    # never been the binding constraint here; endurance is. If small-write
+    # latency becomes a real problem, add OSDs instead.
     osd:
       bdev_enable_discard: "true"
       bdev_async_discard: "true"
@@ -147,52 +79,45 @@ spec:
     ssl: false
   monitoring:
     enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
+  # Multus. The attachment is san-network-attachment.yaml.
+  #
+  # Only `cluster` is selected. Do not add `public`, and do not reach for
+  # `addressRanges` — that is honoured only under provider: host, which puts
+  # every daemon including the mons on host networking. A mon's address then
+  # becomes the node's DHCP lease, and when a lease moves the mon crash-loops
+  # on "Cannot assign requested address" with no node free to fail over to,
+  # because all three already hold the mon host ports.
+  #
+  # With `public` unset, client traffic stays on the pod network and VLAN 49
+  # carries Ceph-internal traffic only — which is also why Rook's Multus
+  # prerequisites about host routing do not apply here.
+  #
+  # Changing the provider on a LIVE cluster needs a step this file cannot
+  # express: `ceph config` retains public_network and cluster_network after
+  # they disappear from here, and an OSD that comes up on the pod network with
+  # a stale public_network crash-loops on "Failed to pick public address".
+  # Clear them with `ceph config rm global public_network` (and cluster_network)
+  # before the rollout. A cluster built from scratch starts with an empty
+  # config database and never sees this.
   network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
-  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
-  # taken out of the storage fleet by removing it here. Under useAllNodes the
-  # operator re-provisions an OSD on any node whose device still carries a
-  # BlueStore signature, which makes purging an OSD ahead of pulling its drive
-  # impossible: the OSD is recreated within a reconcile and starts backfilling
-  # onto the drive being replaced.
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
+  # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
+  # re-provisions an OSD on any node whose device still carries a BlueStore
+  # signature, so purging an OSD before pulling its drive is impossible — it
+  # comes back within a reconcile and backfills onto the drive being replaced.
   #
-  # All three nodes are bare metal and see three NVMe drives each, so they name
-  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
-  # Talos enumerates these machines' drives in a different order than Proxmox
-  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
-  # depending on the boot. Rook would probably decline the system disk for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
-  #
-  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
-  # and the Crucial as nvme0n1 on the very next one.
-  #
-  #   k8s-1 (jupiter)
-  #   50026B7382B48B86  Kingston  — Talos system disk
-  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M400426    PM983     — this OSD
-  #
-  #   k8s-2 (saturn)
-  #   50026B7382DA5F6E  Kingston  — Talos system disk
-  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M403161    PM983     — this OSD
-  #
-  #   k8s-3 (neptune)
-  #   50026B7382B48B90  Kingston  — Talos system disk
-  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NF0M501176    PM983     — this OSD
+  # Devices are named by /dev/disk/by-id path. Kernel names are not stable
+  # across boots on these machines — k8s-3 enumerated a different drive as
+  # nvme0n1 on two consecutive boots — and a bare name can land on the system
+  # disk or on EPHEMERAL.
   storage:
+    # One at a time. The default of 20 would restart every OSD in this cluster
+    # simultaneously — every PG inactive until they return. Binds any change
+    # that rewrites the OSD deployment spec, not just version upgrades.
+    osdMaxUpdatesInParallel: 1
     useAllNodes: false
     useAllDevices: false
     nodes:
@@ -207,18 +132,12 @@ spec:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
-  # Storage layer must never be evicted by kubelet under node pressure
-  # and must never be picked first by the Linux OOM killer. system-node-
-  # critical (PriorityValue 2000001000) is the highest built-in class:
-  # kubelet preserves these pods last, and kernel sets oom_score_adj to
-  # -997 so anything else dies before a Ceph daemon does.
+  # Highest built-in class: kubelet evicts these last and the kernel gives
+  # them the lowest oom_score_adj.
   priorityClassNames:
     all: system-node-critical
-  # memory requests == limits => Guaranteed QoS. Combined with the
-  # priority class above, the pod's oom_score_adj is the lowest possible.
-  # CPU limits are deliberately NOT set — Ceph daemons benefit from
-  # bursting under recovery/backfill, and a CPU limit would throttle
-  # them; CPU pressure on this cluster is not a real concern (~15%).
+  # requests == limits gives Guaranteed QoS. CPU limits are deliberately NOT
+  # set — Ceph daemons burst during recovery and a limit would throttle them.
   resources:
     mon:
       requests:

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
   - cephcluster.yaml
   - toolbox.yaml
   - storageclass.yaml

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -0,0 +1,69 @@
+---
+# The Ceph cluster network as a Multus attachment, so the OSDs reach VLAN 49
+# without every Ceph daemon landing on host networking to get there. The
+# alternative and why it was rejected is in cephcluster.yaml.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }

kubernetes/infrastructure/traefik/helmrelease.yaml

@@ -7,11 +7,8 @@ spec:
   interval: 24h
   url: https://traefik.github.io/charts
 ---
-# Traefik runs alongside ingress-nginx during the migration off the
-# archived kubernetes/ingress-nginx project. Shares the `ingress` namespace
-# so the post-migration topology is just "ingress namespace, one controller".
-# Pilots: shelly-manager.home.renner.uno. ingress-nginx remains the default
-# IngressClass; this controller is opt-in via `ingressClassName: traefik`.
+# The cluster's only ingress controller, and the only IngressClass. It
+# replaced ingress-nginx, which was retired after that project was archived.
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
@@ -44,27 +41,18 @@ spec:
   values:
     deployment:
       replicas: 2
-    # Entrypoint-default Middlewares for websecure: baseline security
-    # response headers (HSTS/CSP/X-Frame-Options/etc.) and a generous
-    # abuse/brute-force rate limit, both applied to every router on this
-    # entrypoint without per-Ingress annotations. See
-    # security-headers-middleware.yaml / rate-limit-middleware.yaml for the
-    # actual settings and rationale. This is the same entrypoint-default
-    # wiring mechanism the (currently detached) crowdsec-bouncer Middleware
-    # documents re-enabling via — independent of that CrowdSec enforcement
-    # decision, which stays off pending the client-IP-visibility fix
-    # referenced below.
+    # Entrypoint-default Middlewares for websecure: security response headers
+    # and a rate limit, applied to every router on this entrypoint without
+    # per-Ingress annotations. Settings live in
+    # security-headers-middleware.yaml and rate-limit-middleware.yaml.
     additionalArguments:
       - --entryPoints.websecure.http.middlewares=ingress-security-headers@kubernetescrd,ingress-rate-limit@kubernetescrd
-    # CrowdSec bouncer plugin (Yaegi), loaded but NOT globally attached.
-    # Enforcement was disabled after it caused an IPv6 ingress outage on
-    # first deploy: Traefik sees a single shared/SNAT'd source IP for v6
-    # ingress (no real per-client IP), so one http-admin-interface-probing
-    # hit (the e2e scan) banned that shared IP and 403'd all v6 clients.
-    # Re-attach the bouncer (additionalArguments entrypoint middleware ->
-    # ingress-crowdsec-bouncer@kubernetescrd) only after fixing client-IP
-    # visibility (forwardedHeaders/trustedIPs or the v6 masquerade) and
-    # adding an internal/LAN whitelist. The plugin + crowdsec-middleware.yaml
+    # CrowdSec bouncer plugin (Yaegi), loaded but deliberately NOT attached.
+    # Traefik sees one shared SNAT'd source for IPv6 ingress, so a single
+    # probe banned that address and 403'd every v6 client. Do not re-attach
+    # it until client-IP visibility is fixed (forwardedHeaders/trustedIPs or
+    # the v6 masquerade) and a LAN whitelist exists.
+    # The plugin + crowdsec-middleware.yaml
     # stay in place so re-enabling is a one-line change. Detection (the
     # crowdsec agents parsing Traefik logs) remains on and is unaffected.
     experimental:

kubernetes/talos/patches/cluster.yaml

@@ -5,31 +5,26 @@ machine:
     - k8s-3.srv.renner.uno
     - k8s.renner.uno
     - 10.2.40.60
-  # Install from the Image Factory metal installer, not the stock
-  # ghcr.io/siderolabs/installer that `talosctl gen config` defaults to.
-  # Extensions live in the installer image, so with the stock one the schematic
-  # is only ever a property of the boot media and nothing lands on disk —
-  # `talosctl get extensions` returns empty on a node installed that way.
+  # Image Factory metal installer, NOT the stock ghcr.io/siderolabs/installer
+  # that `talosctl gen config` defaults to. Extensions live in the installer
+  # image; install with the stock one and nothing lands on disk —
+  # `talosctl get extensions` comes back empty.
   #
-  # What the schematic carries, and why each matters here:
+  # The schematic carries intel-ucode, thunderbolt and i915, and sets
+  # net.ifnames=1, which is what the per-node VLAN parents match against.
+  # Changing which extensions are present means a new schematic hash and a
+  # reinstall or upgrade, not an edit here.
   #
-  #   siderolabs/intel-ucode   CPU microcode. On a VM the hypervisor supplied
-  #                            it; on bare metal nothing does except the BIOS.
-  #   siderolabs/thunderbolt   Inert until machine.kernel.modules asks for
-  #                            thunderbolt/thunderbolt_net. Present so enabling
-  #                            the MS-01 mesh later needs neither a new image
-  #                            nor a reinstall — which is exactly the property
-  #                            lost by installing without it.
+  # Read only at install and upgrade time — editing this does nothing to a
+  # running node. The tag must match the version being upgraded to, or the
+  # upgrade is a downgrade. Talos only tests migration between adjacent minor
+  # releases: take the latest patch of each intermediate minor rather than
+  # skipping one.
   #
-  # a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
-  # same net.ifnames=1 as the Proxmox one (the predictable interface names the
-  # per-node VLAN parents match against), minus qemu-guest-agent.
-  #
-  # Only read at install and upgrade time. Changing it here does nothing to a
-  # running node until that node is upgraded. Bump the tag with the Talos
-  # version — it must match the running version, or the upgrade is a downgrade.
+  # The Kubernetes version is a separate decision, moved by
+  # `talosctl upgrade-k8s`.
   install:
-    image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
   features:
     rbac: true
     stableHostname: true
@@ -51,20 +46,13 @@ machine:
     nodeIP:
       validSubnets:
         - 10.2.40.0/24
-    # Lower the image GC thresholds from the kubelet defaults (85/80) to
-    # bound containerd's image cache, which is the largest single consumer
-    # of /var. Spegel P2P pull-through softens the re-pull cost when GC'd
-    # images are needed back.
+    # Lower than the kubelet defaults (85/80) to bound containerd's image
+    # cache, the largest single consumer of /var.
     #
-    # These thresholds do NOT control whether Ceph's mon reports
-    # MON_DISK_LOW, and must not be tuned for that. Measured on k8s-2:
-    # image GC frees only the content store (~8.6 GB), while the bulk of
-    # /var/lib/containerd is ~20 GB of overlayfs snapshots — unpacked
-    # layers shared with running containers, which GC cannot reclaim.
-    # Dropping high to 60 deleted 28 images and moved free space by 0.02
-    # percentage points. The mon's own data is ~106 MB; its 30%-free
-    # warning is measuring a filesystem it merely shares, so the lever for
-    # it is elsewhere (mon volume, /var size, or mon_data_avail_warn).
+    # Do NOT tune these to clear Ceph's MON_DISK_LOW. Image GC reclaims only
+    # the content store; the bulk of /var/lib/containerd is overlayfs
+    # snapshots shared with running containers, which GC cannot touch. The
+    # lever for the mon warning is elsewhere.
     extraConfig:
       imageGCHighThresholdPercent: 70
       imageGCLowThresholdPercent: 60

kubernetes/talos/patches/k8s-1.yaml

@@ -1,32 +1,25 @@
-# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
-# still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and all
-# four physical NICs and all three NVMe drives are visible to Talos at once.
-#
-# Networking uses the v1.12 multi-document config rather than the deprecated
-# machine.network.interfaces. That is deliberate: the docs do not define what
-# happens when the same link is configured by both schemas, so this file keeps
-# every link in one schema instead of straddling them.
+# Networking uses the multi-document config, not the deprecated
+# machine.network.interfaces. Keep every link in one schema: the docs do not
+# define what happens when both configure the same link.
 machine:
   network:
     hostname: k8s-1
   sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
-    # the macvlan parent for the v30iot pod attachments
-    # (matter/otbr/shelly-manager/music-assistant), whose children do their own
-    # DHCP on that L2. Host addresses were never needed and are actively
-    # harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
-    # clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
-    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
-    # This disables v6 entirely on that link, including link-local.
+    # ens19 (VLAN 30, v30iot) is LINK ONLY — it is the macvlan parent for the
+    # v30iot pod attachments and must never get host addressing. A connected
+    # 10.2.30.0/24 route makes node replies bypass the UDM and breaks every
+    # DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # These two sit on the untagged fallback network, where nothing may run.
+    # Left enabled they autoconfigure a ULA and an IPv6 default route from its
+    # RAs, which ties with mgmt0's and costs the cluster IPv6 egress.
+    # Disabling v6 on a VLAN parent does not affect the VLAN child.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
-    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
-    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
-    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
-    # VolumeConfig below). Every one of them is selected by serial. Nothing in
-    # this file may use a broad selector: `disk.transport == 'nvme'` matches all
-    # three, and the failure mode is Talos taking the OSD drive.
+    # Select disks by serial only. A broad selector such as
+    # `disk.transport == 'nvme'` matches every drive in the machine, and the
+    # failure mode is the installer taking the Ceph OSD drive.
     diskSelector:
       serial: 50026B7382B48B86
   certSANs:
@@ -36,15 +29,10 @@ machine:
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk.
-#
-# That drive is at 87% of its rated endurance and was this node's Ceph OSD until
-# the PM983 replaced it. /var is exactly the right role for a disk in that
-# state: containerd's image and snapshot store, pod logs and the Ceph mon store
-# are all reconstructible, so the drive is consumable here in a way it is not as
-# an OSD. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had,
-# which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
-# they absorb another node's pods.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable here in a way it
+# is not as an OSD. Note /var/lib/etcd lives here too, which is the one thing
+# on the volume that is not free to lose.
 #
 # THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; a config applied after EPHEMERAL
@@ -52,8 +40,8 @@ cluster:
 # requires wiping the disk and booting install media again.
 #
 # Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
-# would match the PM983 as well, and losing that coin flip means Talos formats
-# the OSD drive.
+# still matches the OSD drive, and losing that coin flip means Talos formats
+# it.
 apiVersion: v1alpha1
 kind: VolumeConfig
 name: EPHEMERAL
@@ -63,30 +51,25 @@ provisioning:
   minSize: 50GiB
   grow: true
 ---
-# X710 port 0 carries the switch trunk. It takes no addressing itself — the
-# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
-# business being on — so management and v30iot both ride it as tagged VLANs.
+# X710 port 0 carries the switch trunk and takes no addressing itself — its
+# untagged VLAN is a network this node has no business being on. Management
+# and v30iot ride it as tagged VLANs.
 #
-# Links are addressed by predictable kernel name. Those derive from PCI
-# topology, not probe order — though not immune: adding the third NVMe
-# renumbered the buses and moved the unused second I226 from enp89s0 to
-# enp90s0. Neither name used here was affected.
+# Links are named by predictable kernel name, which derives from PCI topology
+# and can still shift when hardware is added.
 #
-# LinkAliasConfig would be the better way to pin these, and is worth
-# reintroducing — but only once it is proven. Aliasing both X710 ports by
-# permanent MAC and using the aliases as VLAN parents produced a node with no
-# address on any interface: the aliases were never created, so every document
-# referencing them did nothing, including the static address on port 1. Verify
-# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
-# back on its own, before making it permanent.
+# LinkAliasConfig would pin these better, but aliasing the X710 ports by MAC
+# and using the aliases as VLAN parents left the node with no address on any
+# interface — the aliases were never created and every document referencing
+# them silently did nothing. Prove any alias-based revision with
+# `talosctl apply-config --mode=try` first; it rolls back on its own.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f0np0
 up: true
 ---
 # Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
-# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
+# network.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -94,6 +77,15 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# DHCP, and the lease is not reserved in this repo — reservations live on the
+# UDM out of band. Addresses do move under a running node.
+#
+# kube-apiserver does not follow a re-IP: it advertises $(POD_IP) fixed at pod
+# start and the lease reconciler keeps renewing the masterlease under the old
+# address, so `default/kubernetes` hands out a dead endpoint until the static
+# pod restarts (#635). After any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide` and reboot the node if it does not.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -105,10 +97,8 @@ kind: Layer2VIPConfig
 name: 10.2.40.60
 link: mgmt0
 ---
-# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
-# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
-# serve both VM and bare-metal nodes for as long as the cluster is mixed.
+# Named ens19 on purpose: the v30iot macvlan NADs hard-code `master: ens19`.
+# VLANConfig.name is free-form, so renaming this link means editing them too.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
@@ -116,14 +106,15 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
+# wire. OSD-to-OSD replication and nothing else.
+#
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
+# Giving the node an address would put it on a segment reserved for
+# Ceph-internal traffic.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29

kubernetes/talos/patches/k8s-2.yaml

@@ -9,11 +9,14 @@ machine:
     # macvlan parent for the v30iot pod attachments; host addresses on that L2
     # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same for the physical ports on the untagged fallback network: left
+    # enabled they take an IPv6 default route from its RAs and cost the
+    # cluster IPv6 egress.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
+    # Select the installer target by serial so it cannot land on the OSD drive
+    # or on EPHEMERAL, whatever order the kernel enumerates them in.
     diskSelector:
       serial: 50026B7382DA5F6E
   certSANs:
@@ -23,17 +26,15 @@ machine:
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 81% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable in this role.
 #
 # ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; applied later it is silently
 # ignored and relocating needs a disk wipe and a boot from install media.
 #
 # Matched by serial deliberately: a broad selector such as
-# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+# `disk.transport == 'nvme' && !system_disk` still matches the OSD drive.
 apiVersion: v1alpha1
 kind: VolumeConfig
 name: EPHEMERAL
@@ -76,9 +77,8 @@ kind: Layer2VIPConfig
 name: 10.2.40.60
 link: mgmt0
 ---
-# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
-# name. VLANConfig.name is free-form, so one set of NADs serves both.
+# Named ens19 on purpose: the v30iot macvlan NADs hard-code `master: ens19`.
+# VLANConfig.name is free-form, so renaming this link means editing them too.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
@@ -86,13 +86,13 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. OSD<->OSD replication only. Not routed off its own
-# segment and deliberately has no default route.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
+# wire. OSD-to-OSD replication only, not routed off its own segment.
+#
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29

kubernetes/talos/patches/k8s-3.yaml

@@ -1,18 +1,18 @@
-# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
-# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials and addresses differ. neptune was the last node converted, so all
-# three control-plane nodes now share this layout and no Proxmox remains.
+# Structurally identical to k8s-1.yaml — see that file for the reasoning
+# behind each choice; only the serials and addresses differ.
 machine:
   network:
     hostname: k8s-3
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same for the physical ports on the untagged fallback network — see
+    # k8s-1.yaml.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
+    # Select the installer target by serial so it cannot land on the OSD drive
+    # or on EPHEMERAL, whatever order the kernel enumerates them in.
     diskSelector:
       serial: 50026B7382B48B90
   certSANs:
@@ -22,10 +22,8 @@ machine:
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 83% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. Available spare is still 100% and it reports no media errors.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable in this role.
 #
 # ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; applied later it is silently
@@ -72,11 +70,10 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network — see k8s-1.yaml.
+# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
+# here on purpose; the link is only a macvlan parent for the OSD pods.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29