refactor(cilium): one seed, one value source, no untracked settings

Cleans up the duplication behind the version drift rather than only the drift. talos/cilium.yaml is deleted. It was a second, un-indented copy of the same ~2000 lines already embedded in the patch, and nothing read it — the only references anywhere were two comments in the HelmRelease. It existed to be readable next to the patch and cost a whole file that could go stale on its own, which is precisely what happened. The seed's header now records how to reproduce it: the exact yq and helm template invocation, and why the two overrides exist (the ServiceMonitor CRD is absent at bootstrap; the chart's default cert method would render private keys into git, #188). None of that was written down, which is why reconstructing the original value set today took a diff-until-the-object-kinds-match exercise rather than reading a line. Three settings move from nowhere into the HelmRelease: bgpControlPlane.enabled and the two native-routing CIDRs. They were live but tracked in no file at all, applied by hand and held by the kubectl-patch field manager on cilium-config. Server-side apply left them alone so nothing ever broke — but a cold rebuild installs only the seed, and would have brought the cluster up with the BGP control plane off and no session to the UDM, while BGP looked configured because the peering CRDs are in git. That is not a free change of ownership. Enabling bgpControlPlane through the chart emits five companion keys the hand-patch never set, so Cilium has been running on its own internal defaults for router-id allocation and friends. The comment says which ones and what to watch. The per-key "THREE COPIES of this key exist" warnings are gone, replaced by one note at the top of the values block: these values feed the seed, and nothing enforces agreement. Two copies now, not three, and the surviving one says how it is made. Deliberately not done: a generator script and a CI check to diff the committed seed against a fresh render. That answers duplication with more machinery and still leaves a 2000-line generated artifact in the tree. If this should stop being hand-maintained the real question is whether Talos should carry the CNI at all, which is a rebuild-path decision, not a scripting one. bootstrap.sh is untouched. Its talos phase remains dead.
docs(talos): the cilium patch does not serve bootstrap
2026-08-28 15:21:19 +02:00 · 2026-08-28 15:11:58 +02:00 · 2026-08-28 15:01:43 +02:00 · 2026-08-28 14:31:45 +02:00 · 2026-08-28 13:24:05 +02:00 · 2026-08-28 12:59:19 +02:00
105 changed files with 2680 additions and 4259 deletions
@@ -32,13 +32,13 @@
 #              mechanism as update-audit.yml):
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name alert-monitor --branch master --schedule '*/15 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=alert-monitor k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> alert-monitor
+#              where <n> is an existing pipeline to promote.
 when:
  - event: cron
    cron: alert-monitor
-  - event: manual
-    evaluate: 'PIPELINE == "alert-monitor"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "alert-monitor"'

 steps:
  - name: fetch-alerts
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -20,12 +31,12 @@ when:
      - "ci/images/**"
      - "secret-rotator/**"
      - ".woodpecker/build-ci-images.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "build-ci-images"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-ci-images"'

 steps:
  - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/ansible
    depends_on: []
    # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -33,7 +44,7 @@ steps:
    # secret-rotator commit needlessly rebuilt all four ~1.3G ci-* images and
    # orphaned ~5G of registry layers per run — a main driver of #336.
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/ansible/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -53,11 +64,11 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-ansible:latest

  - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/k8s
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/k8s/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -77,11 +88,11 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest

  - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/tofu
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/tofu/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -108,11 +119,11 @@ steps:
  # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
  # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
  - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: secret-rotator
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["secret-rotator/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -137,11 +148,11 @@ steps:
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
  - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-jq
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -166,11 +177,11 @@ steps:
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/cni-plugins
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -195,11 +206,11 @@ steps:
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-openssl
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -224,11 +235,11 @@ steps:
  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
  # builds the same way and lands in the same registry so the audit sees it.
  - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/heating-can
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/heating-can/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -54,8 +54,8 @@
 #
 # Triggering:
 #   - cron   : the cron NAMED exactly "ci-claude-refresh" (see above).
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=ci-claude-refresh k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> ci-claude-refresh
+#              where <n> is an existing pipeline to promote.
 #   - push   : a change to the Containerfile (or this file) ON master forces a
 #              rebuild of the current latest version so Containerfile fixes ship
 #              promptly. Restricted to master ON PURPOSE — a feature branch must
@@ -65,8 +65,8 @@
 when:
  - event: cron
    cron: ci-claude-refresh
-  - event: manual
-    evaluate: 'PIPELINE == "ci-claude-refresh"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "ci-claude-refresh"'
  - event: push
    branch: master
    path:
@@ -84,14 +84,24 @@ steps:
      - |
        set -eu
        mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
        NEW=$$(npm view @anthropic-ai/claude-code version)
        case "$$NEW" in
          [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
  #    --build-arg to the resolved version (not "latest") guarantees the tag and
  #    the image content agree even if a new release lands mid-run.
  - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/claude
    depends_on: [decide]
    environment:
@@ -176,13 +186,10 @@ steps:
          echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
          exit 0
        fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
        # Exact-string swap of the current tag for the new one, consumers only.
        for f in $$CONSUMERS; do
          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
        {
          printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
          printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
          printf 'run the image so CI stops running a stale Claude Code.\n\n'
          printf 'Merging is a human decision — review, then merge to roll it out.\n'
        } > /tmp/pr-body.md
@@ -17,6 +17,6 @@ when:

 steps:
  - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    commands:
      - sh scripts/ci/test-model-selection.sh
@@ -6,8 +6,8 @@ when:
      - "ansible/roles/claude-workstation/**"
      - "ansible/plays/install-claude-workstation.yml"
      - ".woodpecker/deploy-claude-workstation.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-claude-workstation"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'

 timeout: 30

@@ -8,8 +8,8 @@ when:
    path:
      - "opentofu/**"
      - ".woodpecker/deploy-infrastructure.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-infrastructure"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-infrastructure"'

 steps:
  - name: validate-secrets
@@ -84,7 +84,7 @@ steps:
    image: gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29
    directory: opentofu
    when:
-      - event: manual
+      - event: deployment
        evaluate: 'TOFU_APPLY == "true"'
    environment:
      TF_IN_AUTOMATION: "true"
@@ -7,8 +7,8 @@ when:
      - "ansible/plays/install-netbird-client.yml"
      - "ansible/inventory/group_vars/claude_workstation/**"
      - ".woodpecker/deploy-netbird-client.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-client"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'

 timeout: 30

@@ -9,8 +9,8 @@ when:
      - "ansible/roles/netbird-proxy-watchdog/**"
      - "ansible/plays/install-netbird-proxy.yml"
      - ".woodpecker/deploy-netbird-proxy.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-proxy"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

 timeout: 30

@@ -9,8 +9,8 @@ when:
      - "ansible/roles/netbird-proxy-watchdog/**"
      - "ansible/plays/install-netbird-server.yml"
      - ".woodpecker/deploy-netbird-server.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-server"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'

 timeout: 30

@@ -1,6 +1,6 @@
 when:
-  - event: manual
-    evaluate: 'PIPELINE == "e2e-test"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e-test"'

 steps:
  - name: e2e
@@ -7,7 +7,8 @@
 #                was harmless at daily/weekly cadence — but alert-monitor.yml
 #                fires every 15 minutes, so piggybacking must be opt-in now.
 #                A new cron that SHOULD also run e2e gets its name added here.
-#   - manual   : `woodpecker pipeline create --branch master --var PIPELINE=e2e k.renner/homelab`
+#   - deploy   : `woodpecker-cli pipeline deploy k.renner/homelab <n> e2e`, where
+#                <n> is an existing pipeline to promote.

 when:
  - event: cron
@@ -16,8 +17,8 @@ when:
    cron: update-audit
  - event: cron
    cron: weekly-update
-  - event: manual
-    evaluate: 'PIPELINE == "e2e"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e"'

 steps:
  - name: model-selection-tests
@@ -25,7 +26,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -1,27 +0,0 @@
-when:
-  - event: manual
-    evaluate: 'PIPELINE == "install-garage-synology"'
-
-steps:
-  - name: run-playbook
-    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
-    environment:
-      ANSIBLE_FORCE_COLOR: "true"
-      ANSIBLE_HOST_KEY_CHECKING: "false"
-      SSH_PRIVATE_KEY:
-        from_secret: ssh_private_key
-      VAULT_PASS:
-        from_secret: vault_pass
-      PROXMOX_TOKEN_ID:
-        from_secret: proxmox_token_id
-      PROXMOX_TOKEN_SECRET:
-        from_secret: proxmox_token_secret
-    commands:
-      - mkdir -p ~/.ssh
-      - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
-      - chmod 600 ~/.ssh/id_ed25519
-      - 'printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config'
-      - cd ansible
-      - echo "$$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
-      - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
-      - ansible-playbook -i inventory/ plays/install-garage-synology.yml -v
@@ -34,13 +34,13 @@
 #   - cron   : a cron NAMED exactly "issue-fix":
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name issue-fix --branch master --schedule '41 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=issue-fix k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> issue-fix
+#              where <n> is an existing pipeline to promote.
 when:
  - event: cron
    cron: issue-fix
-  - event: manual
-    evaluate: 'PIPELINE == "issue-fix"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "issue-fix"'

 steps:
  - name: fetch-diag
@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -53,9 +53,9 @@
 #
 # Triggering: pull_request only — automatic on PR open + push-to-open-PR. To
 # re-review a PR, push a commit (the new head sha clears the dedup marker below
-# and earns a fresh review). There is deliberately no manual PIPELINE trigger:
-# Woodpecker exposes a --var only to YAML variable substitution and `when`
-# filters, NOT to a step's shell environment, so a --var PR=<n> re-review would
+# and earns a fresh review). There is deliberately no on-demand trigger for a
+# specific PR: Woodpecker exposes injected variables to YAML substitution and
+# `when` filters, NOT to a step's shell environment, so a PR=<n> re-review would
 # silently no-op (the step would find no PR number and exit green).
 # (Note: keep literal dollar-brace out of THIS file entirely, even in comments —
 # Woodpecker runs its variable substitution over the whole YAML text, comments
@@ -65,7 +65,7 @@ when:

 steps:
  - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,8 +1,20 @@
+# Trivy scan of the IaC tree and every image referenced from ansible/, with an
+# agent pass that opens a PR for CRITICAL/HIGH findings.
+#
+# Triggers:
+#   - cron       : the named `daily-security-scan` cron, matched by NAME.
+#                  Crons cannot carry variables — `woodpecker-cli repo cron add`
+#                  has no --var — so a variable gate can never match a cron.
+#   - deployment : woodpecker-cli pipeline deploy k.renner/homelab <n> security-scan
+#                  The target rides in pipeline metadata as
+#                  CI_PIPELINE_DEPLOY_TARGET, which the workflow-root filter
+#                  does see, unlike manual-run variables on Woodpecker 3.18
+#                  (#582).
 when:
  - event: cron
-    evaluate: 'PIPELINE == "security-scan"'
-  - event: manual
-    evaluate: 'PIPELINE == "security-scan"'
+    cron: daily-security-scan
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "security-scan"'

 steps:
  - name: iac-scan
@@ -28,12 +40,28 @@ steps:
        cat /tmp/images.txt
        echo "---"
        i=0
+        failed=0
+        failed_names=""
        while IFS= read -r img; do
          i=$$((i+1))
          out="trivy-image-$$i.json"
          echo "=== [$$i] $$img -> $$out ==="
-          trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img" || true
+          # Keep going on a bad image so one unpullable reference cannot cost
+          # the scan of every image after it, but record that it happened.
+          if ! trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img"; then
+            failed=$$((failed+1))
+            failed_names="$$failed_names $$img"
+          fi
        done < /tmp/images.txt
+        echo "--- scanned $$((i - failed)) of $$i images ---"
+        if [ "$$failed" -gt 0 ]; then
+          echo "NOT SCANNED:$$failed_names"
+          exit 1
+        fi
+    # Non-zero above turns this step red without failing the workflow, so the
+    # fix agent still runs on what was scanned. Coverage is reported, not
+    # enforced: an incomplete scan is visible in the step status instead of
+    # being indistinguishable from a clean one.
    failure: ignore

  - name: security-fix-agent
@@ -42,7 +70,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -10,14 +10,14 @@
 # report; the pipeline posts it to a single persistent Gitea issue.
 #
 # Triggering:
-#   - cron   : a cron NAMED exactly "update-audit". Unlike the other pipelines
-#              here (which gate on an injected `PIPELINE` var set at cron-create
-#              time in the UI), this uses Woodpecker's documented `cron:` name
-#              filter so the schedule can be created entirely from the CLI:
+#   - cron   : a cron NAMED exactly "update-audit". Crons carry no variables —
+#              `repo cron add` takes name, branch, schedule and enabled — so the
+#              name filter is the only thing that can select a workflow for a
+#              scheduled run:
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name update-audit --branch master --schedule '@weekly'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=update-audit k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> update-audit
+#              where <n> is an existing pipeline to promote.
 #
 # NOTE: e2e.yml runs on an explicit list of cron names that includes this one,
 # so a weekly e2e run piggybacks on the audit (it was a bare `event: cron`
@@ -25,12 +25,12 @@
 when:
  - event: cron
    cron: update-audit
-  - event: manual
-    evaluate: 'PIPELINE == "update-audit"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "update-audit"'

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,37 +0,0 @@
---
-# PBS Client configuration for nas-1
-pbs_backup_paths:
-  - /volume1/Documents
-  - /volume1/homes
-  - /volume1/Homeassistant Backup
-  - /volume1/Games
-  - /volume1/Media
-  - /volume1/Music
-  - /volume1/video
-  - /volume2/Backup
-  - /volume1/Movies
-
-  # Add more paths as needed:
-  # - /volume1/photos
-  # - /volume1/music
-
-# PBS Server connection
-pbs_server_host: "pbs2.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "pool"
-pbs_backup_id: "nas-1"
-
-# PBS Authentication - credentials in vault.yml
-pbs_username: "root@pam!nas-1"
-
-# Garage S3 (K8s backup target)
-garage_image: "dxflrs/garage:v2.3.0"
-garage_zone: "dsm"
-garage_capacity: "2T"
-
-# SSH connection - use password auth (key not supported by DSM)
-ansible_ssh_pass: "{{ nas_ssh_password }}"
-ansible_become_pass: "{{ nas_ssh_password }}"
-
-# Python 3.9+ from Synology package
-ansible_python_interpreter: /usr/local/bin/python3.9
@@ -1,27 +0,0 @@
-$ANSIBLE_VAULT;1.1;AES256
-34663462636437633437613863366266306334666635643839313737343739643864613136613938
-3138393966306561656366396465383539303137333036300a376237376636356230343834386362
-36636535396330333732383233323263613132333166343739396137636538653435336662353138
-6565643866623235610a316266383933303830323936303735623338623165386339383733373435
-62366461363833616438373237383132366438326364363462333265366533323861376630643365
-39343036343838333639363432643738666630363063323636646661313539643465366532363438
-33303837623234613434633563666562346536346435353461373536323737376233386163326537
-61383963343737656438356136613537623538333163343466343739346135333439616432663763
-36663962633037376561326434303035333938646436663832303432346365353439666435613337
-34363636326530333233313231343566306432376532396666643761633136316333376337666135
-65363338623035663438323338663863366362653131336434323865326530326534623631336263
-36633063383031386161363131333636393364623436613331353733313235383062656131313164
-63393665633361313961363637663237616364643236383835393736643937393564656238376362
-31393465383361326264376133393233616264663636383337363962356532613438663637616536
-33653137656335323763313461323633616635346232303062323062333766383931313965346361
-66656461313239633366636463633465636463663937393932636337643861383937663662306466
-32353166643832303234643264633062616330393963333034623432613730366339306637613239
-34626161663666316339323762373639353335653663353635366664643063393432623039306237
-38363161343930363763393364663364353735343236313665313461626165636135303839363164
-37336638396564326166376233353730346563656266323535666265323062646232323938353732
-32636235333839623166633761376334316434613235333730353439343262393537363461333238
-32653435623362666662336532643832356163326332636436346132643237643030343761663332
-36663530356539303636663438646430363335353135323932613863666264363533326137353164
-31663032346236616263343637396439663334396166656364623637666535386530383535663435
-66303037303265616634373763616437333835333136336563323339666563663931396133633131
-3963653266633535303630653335666166313336653831623362
@@ -8,6 +8,8 @@
 #   pve_cluster (jupiter/neptune/saturn)  run Talos on bare metal; no SSH at all
 #   authentik_db / authentik_app          authentik runs in-cluster (CNPG)
 #   grafana                               runs in-cluster
+#   synology_nas (nas-1)                  powered off; Garage and the PBS client
+#                                         it hosted are gone with it
 # Their hosts no longer resolve, except the three that resolve to Talos nodes.

 [netbird_server]
@@ -17,10 +19,6 @@ netbird.renner.uno ansible_user=root
 proxy-1.renner.uno ansible_user=root
 proxy-2.renner.uno ansible_user=root

-
-[synology_nas]
-nas-1.renner.uno ansible_user=ladmin ansible_python_interpreter=/usr/bin/python3
-
 # Every bare-metal edge thin client. Gets the host-generic roles only:
 # flash-wear handling, the log shipper, the MQTT bridge, the VLAN
 # sub-interface. Nothing here assumes any peripheral is attached.
@@ -1,313 +0,0 @@
---
-# Synology-specific playbook using raw module (Python 3.8 workaround)
-# Deploys Garage (https://garagehq.deuxfleurs.fr) as a single-node S3 target
-# for K8s backups (Velero + CNPG Barman Cloud Plugin).
-#
-# Required secrets in host_vars/nas-1.renner.uno/vault.yml:
-#   garage_rpc_secret:     "<openssl rand -hex 32>"
-#   garage_admin_token:    "<openssl rand -base64 32>"
-#   garage_metrics_token:  "<openssl rand -base64 32>"
-#
-# Post-install bootstrap (run once, record the returned secret keys):
-#   docker exec garaged /garage status
-#   docker exec garaged /garage bucket create k8s-velero
-#   docker exec garaged /garage bucket create k8s-cnpg
-#   docker exec garaged /garage bucket create k8s-secrets
-#   docker exec garaged /garage bucket create k8s-migration
-#   docker exec garaged /garage key create velero-key
-#   docker exec garaged /garage key create cnpg-key
-#   docker exec garaged /garage key create secrets-key
-#   docker exec garaged /garage key create migration-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-velero    --key velero-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-cnpg      --key cnpg-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-secrets   --key secrets-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-migration --key migration-key
- name: Install Garage S3 on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    garage_data_dir: "/volume1/docker/garage"
-
-  tasks:
-    - name: Create Garage directory tree
-      ansible.builtin.raw: |
-        mkdir -p {{ garage_data_dir }}/config \
-                 {{ garage_data_dir }}/meta \
-                 {{ garage_data_dir }}/data
-      changed_when: true
-
-    # Secret writes use the same compare-then-write pattern as garage.toml so
-    # rotation is idempotent AND a real change can drive force-recreate below.
-    # Garage reads these files once at startup; without container restart, a
-    # rotated token leaves the running process holding the previous value.
-    - name: Write rpc_secret
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/rpc_secret.new << 'SECRET_EOF'
-        {{ garage_rpc_secret }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/rpc_secret.new
-        if [ ! -f {{ garage_data_dir }}/config/rpc_secret ] \
-        || ! cmp -s {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret; then
-          mv {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret
-          echo RPC_SECRET_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/rpc_secret.new
-        fi
-      register: rpc_secret_write
-      changed_when: "'RPC_SECRET_CHANGED' in rpc_secret_write.stdout"
-      no_log: true
-
-    - name: Write admin_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/admin_token.new << 'SECRET_EOF'
-        {{ garage_admin_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/admin_token.new
-        if [ ! -f {{ garage_data_dir }}/config/admin_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token; then
-          mv {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token
-          echo ADMIN_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/admin_token.new
-        fi
-      register: admin_token_write
-      changed_when: "'ADMIN_TOKEN_CHANGED' in admin_token_write.stdout"
-      no_log: true
-
-    - name: Write metrics_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/metrics_token.new << 'SECRET_EOF'
-        {{ garage_metrics_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/metrics_token.new
-        if [ ! -f {{ garage_data_dir }}/config/metrics_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token; then
-          mv {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token
-          echo METRICS_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/metrics_token.new
-        fi
-      register: metrics_token_write
-      changed_when: "'METRICS_TOKEN_CHANGED' in metrics_token_write.stdout"
-      no_log: true
-
-    - name: Write garage.toml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/garage.toml.new << 'TOML_EOF'
-        metadata_dir = "/var/lib/garage/meta"
-        data_dir     = "/var/lib/garage/data"
-
-        # SQLite chosen over LMDB: single-node deployment has no peers to
-        # rebuild metadata from on unclean-shutdown corruption. SQLite is
-        # slower but portable and has no LMDB corruption window.
-        db_engine = "sqlite"
-
-        # 6-hour auto-snapshot of the metadata DB as a backstop against
-        # corruption; last two snapshots retained.
-        metadata_auto_snapshot_interval = "6h"
-
-        # Single-zone, single-node: only coherent replication factor.
-        replication_factor = 1
-
-        rpc_bind_addr    = "[::]:3901"
-        rpc_secret_file  = "/etc/garage/rpc_secret"
-
-        compression_level = 2
-
-        # 64 MB blocks (default 1 MB; previously 20 MB). On the 5 × 7 TB
-        # Btrfs RAID 5 volume the per-block fixed cost (SQLite lookup,
-        # dir traversal, hash verify, zstd decompress, nginx buffering)
-        # is the dominant overhead per object; larger blocks amortize
-        # it. block_size is a maximum, so 16 MiB CNPG WAL segments
-        # still upload as one block each. Only applies to NEW uploads;
-        # existing objects keep their original chunking until re-uploaded.
-        block_size = "64M"
-
-        [s3_api]
-        api_bind_addr = "[::]:3900"
-        s3_region     = "garage"
-        root_domain   = ".s3.{{ inventory_hostname }}"
-
-        [s3_web]
-        bind_addr   = "[::]:3902"
-        root_domain = ".web.{{ inventory_hostname }}"
-
-        [admin]
-        api_bind_addr      = "[::]:3903"
-        admin_token_file   = "/etc/garage/admin_token"
-        metrics_token_file = "/etc/garage/metrics_token"
-        TOML_EOF
-        if [ ! -f {{ garage_data_dir }}/config/garage.toml ] \
-        || ! cmp -s {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml; then
-          mv {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml
-          echo GARAGE_TOML_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/garage.toml.new
-        fi
-      register: garage_toml_write
-      changed_when: "'GARAGE_TOML_CHANGED' in garage_toml_write.stdout"
-
-    - name: Write docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          garage:
-            image: {{ garage_image }}
-            container_name: garaged
-            restart: unless-stopped
-            # HTTP services (3900/3902/3903) bind to loopback; DSM
-            # reverse proxy fronts and TLS-terminates them. RPC 3901
-            # is exposed directly — the Noise handshake is raw TCP,
-            # so DSM RP cannot front it. rpc_secret gates auth.
-            ports:
-              - "3901:3901"
-              - "127.0.0.1:3900:3900"
-              - "127.0.0.1:3902:3902"
-              - "127.0.0.1:3903:3903"
-            volumes:
-              - {{ garage_data_dir }}/config/garage.toml:/etc/garage.toml:ro
-              - {{ garage_data_dir }}/config/rpc_secret:/etc/garage/rpc_secret:ro
-              - {{ garage_data_dir }}/config/admin_token:/etc/garage/admin_token:ro
-              - {{ garage_data_dir }}/config/metrics_token:/etc/garage/metrics_token:ro
-              - {{ garage_data_dir }}/meta:/var/lib/garage/meta
-              - {{ garage_data_dir }}/data:/var/lib/garage/data
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Pull image and start Garage container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    # Garage does not reload config on SIGHUP and compose does not notice
-    # when a bind-mounted host file changes. Force-recreate the container
-    # when garage.toml OR any of the three secret files changed so the new
-    # config / rotated tokens are picked up.
-    - name: Force-recreate garaged when garage.toml or secrets changed
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose up -d --force-recreate garage
-      become: true
-      when: garage_toml_write.changed
-            or rpc_secret_write.changed
-            or admin_token_write.changed
-            or metrics_token_write.changed
-      changed_when: true
-
-    - name: Wait for Garage to accept CLI calls
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        for i in $(seq 1 30); do
-          if docker exec garaged /garage status >/dev/null 2>&1; then
-            exit 0
-          fi
-          sleep 2
-        done
-        exit 1
-      become: true
-      changed_when: false
-
-    - name: Check whether cluster layout is already applied
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status
-      changed_when: false
-
-    - name: Apply single-node layout (idempotent — only on first run)
-      when: "'NO ROLE ASSIGNED' in garage_status.stdout"
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        NODE_ID=$(docker exec garaged /garage node id | head -n 1 | cut -d@ -f1)
-        docker exec garaged /garage layout assign "$NODE_ID" \
-          -z {{ garage_zone }} \
-          -c {{ garage_capacity }} \
-          -t {{ inventory_hostname_short }}
-        docker exec garaged /garage layout apply --version 1
-      become: true
-      changed_when: true
-
-    - name: Report Garage status
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status_final
-      changed_when: false
-
-    - name: Show Garage status
-      ansible.builtin.debug:
-        var: garage_status_final.stdout_lines
-
-    # Bootstrap buckets + keys. Idempotent: bucket_info / key_info return
-    # non-zero when missing, so we only create on first run. Access-key
-    # secrets are captured at create time and persisted to chmod-600 files
-    # under {{ garage_data_dir }}/config/keys/<key>.creds on nas-1. These
-    # files are the only post-install record of the secrets — Garage does
-    # not expose them again.
-    - name: Bootstrap Garage buckets, keys, and grants
-      ansible.builtin.shell: |
-        set -e
-        export PATH=$PATH:/usr/local/bin
-        KEYS_DIR={{ garage_data_dir }}/config/keys
-        mkdir -p "$KEYS_DIR"
-        chmod 700 "$KEYS_DIR"
-
-        bootstrap_pair() {
-          local BUCKET="$1"
-          local KEY="$2"
-          local CREDS_FILE="$KEYS_DIR/$KEY.creds"
-
-          if ! docker exec garaged /garage bucket info "$BUCKET" >/dev/null 2>&1; then
-            docker exec garaged /garage bucket create "$BUCKET"
-          fi
-
-          if ! docker exec garaged /garage key info "$KEY" >/dev/null 2>&1; then
-            CREATE_OUT=$(docker exec garaged /garage key create "$KEY")
-            KEY_ID=$(echo "$CREATE_OUT"   | awk '/^Key ID:/     {print $3}')
-            SECRET=$(echo "$CREATE_OUT"   | awk '/^Secret key:/ {print $3}')
-            umask 177
-            printf 'KEY_ID=%s\nSECRET_KEY=%s\n' "$KEY_ID" "$SECRET" > "$CREDS_FILE"
-          fi
-
-          docker exec garaged /garage bucket allow \
-            --read --write --owner \
-            "$BUCKET" --key "$KEY"
-        }
-
-        bootstrap_pair k8s-velero    velero-key
-        bootstrap_pair k8s-cnpg      cnpg-key
-        bootstrap_pair k8s-secrets   secrets-key
-        bootstrap_pair k8s-migration migration-key
-        bootstrap_pair tfstate       tfstate-key
-      become: true
-      changed_when: true
-      no_log: true
-
-    - name: Report bucket + key state (no secrets)
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage bucket list
-        echo "---"
-        docker exec garaged /garage key list
-        echo "---"
-        ls -l {{ garage_data_dir }}/config/keys/ 2>&1
-      become: true
-      register: garage_bootstrap_state
-      changed_when: false
-
-    - name: Show bucket + key state
-      ansible.builtin.debug:
-        var: garage_bootstrap_state.stdout_lines
-
-    - name: Post-install note
-      ansible.builtin.debug:
-        msg:
-          - "Buckets: k8s-velero, k8s-cnpg, k8s-secrets, k8s-migration, tfstate"
-          - "Keys:    velero-key, cnpg-key, secrets-key, migration-key, tfstate-key"
-          - "Secrets are written to {{ garage_data_dir }}/config/keys/<key>.creds on nas-1 (chmod 600, root-owned)."
-          - "SSH to nas-1 and `sudo cat` the file to retrieve KEY_ID + SECRET_KEY for Velero / CNPG / migration / tfstate config."
@@ -1,104 +0,0 @@
---
-# Synology-specific playbook using raw module (Python 3.8 workaround)
- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    pbs_client_data_dir: "/volume1/docker/pbs-client"
-
-  tasks:
-    - name: Create PBS client directory
-      ansible.builtin.raw: |
-        mkdir -p {{ pbs_client_data_dir }}/state
-      changed_when: true
-
-    - name: Create docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          pbs-client:
-            image: proxmox/proxmox-backup-client:latest
-            container_name: pbs-client
-            restart: unless-stopped
-            hostname: {{ pbs_backup_id }}
-            env_file:
-              - .env
-            volumes:
-              - {{ pbs_client_data_dir }}/state:/root/.proxmox-backup
-        {% for path in pbs_backup_paths %}
-              - "{{ path }}:{{ path }}:ro"
-        {% endfor %}
-            mem_limit: 2g
-            command: ["sleep", "infinity"]
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Create .env file
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/.env << 'ENV_EOF'
-        PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-        PBS_PASSWORD={{ pbs_password }}
-        ENV_EOF
-        chmod 600 {{ pbs_client_data_dir }}/.env
-      changed_when: true
-
-    - name: Create backup script
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/backup.sh << 'SCRIPT_EOF'
-        #!/bin/bash
-        set -e
-        source {{ pbs_client_data_dir }}/.env
-
-        BACKUP_SOURCES=""
-        {% for path in pbs_backup_paths %}
-        BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-        {% endfor %}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-        docker exec pbs-client proxmox-backup-client backup \
-            $BACKUP_SOURCES \
-            --backup-id {{ pbs_backup_id }} \
-            --repository "${PBS_REPOSITORY}"
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed, pruning..."
-        docker exec pbs-client proxmox-backup-client prune \
-            --repository "${PBS_REPOSITORY}" \
-            --backup-id {{ pbs_backup_id }} \
-            --keep-last {{ pbs_keep_last | default(3) }} \
-            --keep-daily {{ pbs_keep_daily | default(7) }} \
-            --keep-weekly {{ pbs_keep_weekly | default(4) }} \
-            --keep-monthly {{ pbs_keep_monthly | default(6) }}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Done"
-        SCRIPT_EOF
-        chmod 755 {{ pbs_client_data_dir }}/backup.sh
-      changed_when: true
-
-    - name: Start PBS client container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ pbs_client_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    - name: Setup cron job for daily backup at 2 AM (root crontab)
-      ansible.builtin.shell: |
-        # Remove existing entry if present
-        crontab -l 2>/dev/null | grep -v "pbs-client/backup.sh" | crontab - 2>/dev/null || true
-        # Add new entry
-        (crontab -l 2>/dev/null; echo "0 2 * * * {{ pbs_client_data_dir }}/backup.sh >> /var/log/pbs-backup.log 2>&1") | crontab -
-      become: true
-      changed_when: true
-
-    - name: Verify container is running
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker ps --filter name=pbs-client --format "{{ '{{' }}.Status{{ '}}' }}"
-      become: true
-      register: container_status
-      changed_when: false
-
-    - name: Show container status
-      ansible.builtin.debug:
-        msg: "PBS client container status: {{ container_status.stdout }}"
@@ -1,11 +0,0 @@
---
- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  become: true
-  # Variables are set in:
-  # - Role defaults: ansible/roles/pbs-client/defaults/main.yml
-  # - Host overrides: ansible/inventory/host_vars/<hostname>/vars.yml
-  # - Secrets: ansible/inventory/host_vars/<hostname>/vault.yml
-
-  roles:
-    - pbs-client
@@ -1,32 +0,0 @@
---
-# PBS Client configuration
-pbs_client_version: "latest"
-pbs_client_data_dir: "/opt/pbs-client"
-
-# PBS Server connection (override in host_vars)
-pbs_server_host: "pbs.obg.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "backups"
-
-# Authentication (override in vault!)
-pbs_username: "backup@pbs"
-pbs_password: ""
-# pbs_fingerprint: ""  # Optional - omit for Let's Encrypt certs
-
-# Backup configuration
-pbs_backup_id: "nas-1"
-pbs_backup_paths:
-  - /volume1/docker
-  - /volume1/homes
-
-# Schedule (cron format)
-pbs_backup_schedule: "0 2 * * *"  # 2 AM daily
-
-# Retention policy (prune settings)
-pbs_keep_last: 3
-pbs_keep_daily: 7
-pbs_keep_weekly: 4
-pbs_keep_monthly: 6
-
-# Container resource limits
-pbs_memory_limit: "2g"
@@ -1,6 +0,0 @@
---
- name: Restart PBS client
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: restarted
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"
@@ -1,23 +0,0 @@
---
- name: Create PBS client directories
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: directory
-    mode: '0755'
-  loop:
-    - "{{ pbs_client_data_dir }}"
-    - "{{ pbs_client_data_dir }}/state"
-
- name: Copy PBS client docker-compose file
-  ansible.builtin.template:
-    src: docker-compose.yml.j2
-    dest: "{{ pbs_client_data_dir }}/docker-compose.yml"
-    mode: '0644'
-  notify: Restart PBS client
-
- name: Start PBS client via Docker Compose
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: present
-    pull: always
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"
@@ -1,6 +0,0 @@
-# PBS Client Environment Configuration
-PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-PBS_PASSWORD={{ pbs_password }}
-{% if pbs_fingerprint | default('') != '' %}
-PBS_FINGERPRINT={{ pbs_fingerprint }}
-{% endif %}
@@ -1,36 +0,0 @@
-#!/bin/bash
-# PBS Backup Script for {{ pbs_backup_id }}
-# Managed by Ansible - do not edit manually
-
-set -e
-
-# Load environment
-source {{ pbs_client_data_dir }}/.env
-
-# Build backup command with all paths
-BACKUP_SOURCES=""
-{% for path in pbs_backup_paths %}
-BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-{% endfor %}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-
-# Run backup via Docker
-docker exec pbs-client proxmox-backup-client backup \
-    $BACKUP_SOURCES \
-    --backup-id {{ pbs_backup_id }} \
-    --repository "${PBS_REPOSITORY}"
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed"
-
-# Prune old backups
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Pruning old backups"
-docker exec pbs-client proxmox-backup-client prune \
-    --repository "${PBS_REPOSITORY}" \
-    --backup-id {{ pbs_backup_id }} \
-    --keep-last {{ pbs_keep_last }} \
-    --keep-daily {{ pbs_keep_daily }} \
-    --keep-weekly {{ pbs_keep_weekly }} \
-    --keep-monthly {{ pbs_keep_monthly }}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Prune completed"
@@ -1,30 +0,0 @@
-services:
-  pbs-client:
-    image: aterfax/pbs-client:{{ pbs_client_version }}
-    container_name: pbs-client
-    restart: unless-stopped
-    hostname: {{ pbs_backup_id }}
-    environment:
-      - PBS_ENDPOINT={{ pbs_server_host }}:{{ pbs_server_port }}
-      - PBS_DATASTORE={{ pbs_datastore }}
-      - PBS_API_KEY_NAME={{ pbs_username }}
-      - PBS_API_KEY_SECRET={{ pbs_password }}
-      - PBS_PASSWORD={{ pbs_password }}
-      - PBS_ENCRYPTION_PASSWORD={{ pbs_encryption_password | default(pbs_password) }}
-      - PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-      - PBS_BACKUP_ID={{ pbs_backup_id }}
-      - CRON_SCHEDULE={{ pbs_backup_schedule }}
-      - PBS_BACKUP_CMD_APPEND_EXTRA_OPTS=--crypt-mode=none
-      - PRUNE_KEEP_LAST={{ pbs_keep_last | default(3) }}
-      - PRUNE_KEEP_DAILY={{ pbs_keep_daily | default(7) }}
-      - PRUNE_KEEP_WEEKLY={{ pbs_keep_weekly | default(4) }}
-      - PRUNE_KEEP_MONTHLY={{ pbs_keep_monthly | default(6) }}
-      - CRON_BACKUP_ONLY=1
-    volumes:
-      - {{ pbs_client_data_dir }}/state:/root/.config/proxmox-backup
-{% for path in pbs_backup_paths %}
-      - "{{ path }}:/backups/{{ path | basename | replace(' ', '_') }}:ro"
-{% endfor %}
-    tmpfs:
-      - /tmp
-    mem_limit: {{ pbs_memory_limit }}
@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.
@@ -1,10 +1,25 @@
 # Cold-rebuild runbook — homelab cluster from scratch

 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
+in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
+gone).

 Order: **tofu → talos → flux → openbao → apps.**

+All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
+are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
+Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
+resources, and physically booting three machines from install media.
+
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
+
 ## The circular-bootstrap problem this runbook exists to break

 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -61,12 +76,21 @@ These must exist outside the cluster or the rebuild cannot complete:
      `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
      copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
+      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
+      Proxmox credentials — there is no Proxmox any more.
+- [ ] **Install media** written and to hand: the Image Factory ISO named in
+      Step 1, on a USB stick, for all three machines.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.

-## Step 1 — tofu: provision the VMs + supporting infra
+## Step 1a — tofu: DNS, object storage, UniFi

-Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
-UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
+OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
+the tree but nothing instantiates it, so `tofu apply` here creates only the
+platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
+records, the Hetzner object storage, and the NAT hairpin rules.

 ```sh
 cd opentofu
@@ -74,44 +98,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```

-In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
-which also pre-seeds the Talos image onto every Proxmox node. Wait until the
-node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
-Step 2 — `bootstrap.sh` polls DNS but will not create the records.
+Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
+FQDNs to resolve — the control-plane endpoint is baked into every generated
+config, and the nodes are addressed by name.

-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
+## Step 1b — boot all three machines from install media

 ```
-https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```

-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
+Write it to a USB stick and boot each MS-01 from it. Each comes up in
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.

- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
+The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
+interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
+against do not exist, and the node ends up with no address on any interface.
+Beyond that it carries:
+
+- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
+  today; present so wanting hardware transcoding later is a config change
+  rather than another rolling reinstall
+- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
+  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
+  rule auto-authorises any Thunderbolt device attached once those modules load
+- no `qemu-guest-agent`, which has nothing to talk to on bare metal
+
+Verify the disks before going further. Talos enumerates NVMe in an order that
+is not stable across boots, and the per-node configs select by serial for
+exactly that reason — the failure mode is the installer taking the Ceph OSD
+drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
+NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
+in `talos/patches/k8s-N.yaml`.

 ## Step 2 — talos: form the cluster + get kubeconfig

+**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
+config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
+`$patch: delete` document fails against a base with no `inlineManifests`
+(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
+and 6; only config generation is broken.
+
+The procedure below is what actually works, and is the same one
+`RUNBOOK-baremetal-conversion.md` uses.
+
+### Generate the configs
+
+This is a genuinely new cluster — nothing survives to copy secrets from, so
+generate them fresh. (In a *partial* rebuild, where one node still runs, pull
+secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
+--from-controlplane-config`, or the new node forms its own cluster rather than
+joining.)
+
 ```sh
-kubernetes/bootstrap.sh talos
+talosctl gen secrets -o secrets.yaml
+
+for N in 1 2 3; do
+  talosctl gen config homelab https://k8s.renner.uno:6443 \
+    --with-secrets secrets.yaml --output-types controlplane \
+    --config-patch @talos/patches/cluster.yaml \
+    --config-patch @<cilium additive half> \
+    --config-patch @talos/patches/registry-mirror.yaml \
+    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
+done
 ```

-Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
-writes `~/.talos/config` + `~/.kube/config`, and stages both into
-`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
+Two things the generator gets wrong for this repo, both of which will bite
+silently:
+
+- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
+  second document. See issue #581.
+- **Strip the generated `HostnameConfig`** — it collides with
+  `machine.network.hostname` from the per-node patch.
+
+Validate each before applying:
+
+```sh
+talosctl validate --config install-k8s-$N.yaml --mode metal
+```
+
+### Apply in two stages
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
+and nothing times out. Concretely, the `ens19` VLAN document must be removed
+from the first-stage config — its parent exists, but the install stalls anyway.
+
+```sh
+# Stage 1: minimal config, from the maintenance-mode address off the console
+talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
+# Stage 2: once the node is up on management, apply the complete config
+talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
+```
+
+`--mode=try` rolls back on its own if the node becomes unreachable, which is the
+only safety net there is for a network change that can strand a machine with no
+IPMI. Use it for every network-affecting apply, then make it permanent once
+proven.
+
+### Bootstrap etcd, then collect access
+
+Bootstrap **exactly one** node:
+
+```sh
+talosctl -n k8s-1.srv.renner.uno bootstrap
+talosctl -n k8s-1.srv.renner.uno kubeconfig .
+```
+
+Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
+supported way to get the kubeconfig into place afterwards.

 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)

@@ -231,9 +330,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
  in-cluster commits made **after** Flux/Gitea broke are not on the mirror
  (Gitea couldn't push them if Gitea itself was down). This is the intended
  recovery point, but be aware of it when reconciling drift.
- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
-  key must not live in CI. See issue #187 for the open question of whether to
-  wire a semi-automated OpenBao restore.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
+- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
+  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
+  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
+  It has also never had an `openbao` phase — Step 4 is manual because the
+  unseal key must not live in CI; see issue #187 on whether to wire a
+  semi-automated OpenBao restore.
+- **OpenBao seals on every restart and each replica must be unsealed
+  individually.** Three raft replicas mean the service survives losing one, but
+  a sealed server is neither a voter nor a standby — unsealing only one leaves
+  the cluster short of quorum. Step 4 covers this; it is repeated here because
+  it is the step most likely to be half-done under pressure.
+- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
+  IPAM, not host networking. Nothing in the rebuild needs special handling for
+  it — the nodes hold no address on that segment — but a rebuilt cluster will
+  not have the whereabouts allocations from before, which is harmless since
+  they are assigned fresh.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
  Ceph/infra bring-up; always read the phase output rather than trusting the
  exit code.
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: authentik
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/authentik/barman-creds
@@ -25,20 +25,13 @@ data:
        attrs:
          local: true

-      # Retired OIDC layer. Paperless's URL-level access is gated by the
-      # paperless-gate ProxyProvider (below); user identity is forwarded
-      # from the gate to paperless via the X-authentik-* request headers
-      # and consumed by paperless via PAPERLESS_ENABLE_HTTP_REMOTE_USER
-      # (see kubernetes/apps/paperless/paperless.yaml). One Authentik
-      # Application, one Provider — the dual-OAuth2-flow setup with a
-      # separate paperless OIDC client + django-allauth callback was
-      # structurally fine but added a second consent screen on first
-      # login for no real defense-in-depth gain.
+      # Retired OIDC layer. Access is gated by the paperless-gate
+      # ProxyProvider below, which forwards identity as X-authentik-* headers
+      # consumed via PAPERLESS_ENABLE_HTTP_REMOTE_USER.
      #
-      # Order matters: delete the Application BEFORE the OAuth2Provider
-      # because Application.provider is on_delete=PROTECT, so the
-      # OAuth2Provider can't be removed while the Application still
-      # references it.
+      # ORDER MATTERS: delete the Application BEFORE the OAuth2Provider.
+      # Application.provider is on_delete=PROTECT, so the provider cannot be
+      # removed while an Application still references it.
      - model: authentik_core.application
        identifiers:
          slug: paperless
@@ -49,22 +42,15 @@ data:
          name: Paperless
        state: absent

-      # Also clean up the very-early orphan `paperless` ProxyProvider
-      # left behind from this session's first attempts. It's not bound
-      # to anything any more, just leftover state in the DB.
+      # Orphan ProxyProvider, unbound — leftover DB state.
      - model: authentik_providers_proxy.proxyprovider
        identifiers:
          name: paperless
        state: absent

-      # Dedicated forward-auth gate for paperless. This is a SEPARATE
-      # outpost (its own Deployment + Service `ak-outpost-paperless-
-      # gate`) running in front of paperless so every request is
-      # authenticated and authorized by Authentik before it reaches
-      # paperless's own attack surface. The gate sits on top of, and
-      # separate from, the OIDC OAuth2Provider above (which is what
-      # paperless's django-allauth uses for SSO once a request makes
-      # it through the gate).
+      # Dedicated forward-auth gate for paperless: its own outpost
+      # Deployment + Service, so every request is authenticated before it
+      # reaches paperless.
      - id: paperless-gate-provider
        model: authentik_providers_proxy.proxyprovider
        identifiers:
@@ -73,16 +59,13 @@ data:
        attrs:
          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          # Proxy (not forward_domain) so the outpost itself terminates
-          # the request and forwards to paperless — moving the trust
-          # boundary off ingress-nginx and onto paperless-gate. ingress-
-          # nginx now only routes paperless.renner.uno to the outpost
-          # service; it never talks to paperless directly. The outpost
-          # is what populates X-Authentik-Username on the upstream
-          # request, and the NetworkPolicy in the paperless namespace
-          # only allows traffic to paperless from the gate outpost pod
-          # itself, so ingress-nginx (or any other in-cluster workload)
-          # cannot bypass the gate to inject headers.
+          # Proxy, NOT forward_domain: the outpost terminates the request
+          # itself, so the trust boundary sits here rather than at the ingress
+          # controller, which only routes to the outpost Service and never
+          # reaches paperless directly. The outpost populates
+          # X-Authentik-Username, and the paperless NetworkPolicy admits
+          # traffic only from this outpost — so nothing else in the cluster
+          # can bypass the gate and inject those headers.
          mode: proxy
          internal_host: http://paperless.paperless.svc.cluster.local
          internal_host_ssl_validation: false
@@ -91,19 +74,12 @@ data:
          access_token_validity: hours=24
          refresh_token_validity: days=30
          intercept_header_auth: true
-          # Pass paperless's entire static-asset tree through anonymously.
-          # Two reasons:
-          # 1. <link rel="manifest"> is fetched with `credentials: 'omit'`
-          #    (W3C spec, no crossorigin attr) so the cookie is never sent
-          #    — every page load and every SPA navigation turned into a
-          #    302 → auth.renner.uno round-trip with an orphan session JWT.
-          #    Same root cause as the retired music-assistant-gate (see
-          #    comment further below); paperless can't retire the gate so
-          #    it carves out the offending paths instead.
-          # 2. Every CSS/JS/icon fetch otherwise validates the cookie at
-          #    the outpost — pure overhead on assets that carry no
-          #    authentication value. Django's STATIC tree is public by
-          #    design; MEDIA (under /api/documents/*) stays gated.
+          # The static-asset tree passes anonymously, and must. A
+          # <link rel="manifest"> is fetched with `credentials: 'omit'` per
+          # the W3C spec, so the session cookie is never sent and every page
+          # load turns into a redirect loop through the login origin. Django's
+          # STATIC tree is public by design; MEDIA under /api/documents/*
+          # stays gated.
          skip_path_regex: ^/static/.*

      - model: authentik_core.application
@@ -130,18 +106,14 @@ data:
          negate: false
          failure_result: false

-      # Retired: the music-assistant-gate ProxyProvider, Application,
-      # and dedicated Outpost. The gate fundamentally conflicted with
-      # MA's PWA model — Chrome fetches manifest.webmanifest / sw.js /
-      # icons with `credentials: 'omit'` (per W3C spec, no crossorigin
-      # attribute on MA's <link rel="manifest">), and the cached
-      # service worker bypasses fresh navigations on PWA cold-start so
-      # the Authentik cookie never lands in the WebAPK's cookie jar.
-      # MA has its own auth (delegated to Home Assistant); the gate's
-      # marginal value didn't justify the recurring breakage.
-      # Order matters: Application before Provider before Outpost.
-      # Application.provider is on_delete=PROTECT, so the Provider
-      # can't be removed while the Application still references it.
+      # Retired: the music-assistant gate. A forward-auth gate cannot work in
+      # front of a PWA — the manifest and service worker are fetched with
+      # `credentials: 'omit'`, and a cached service worker bypasses fresh
+      # navigations, so the session cookie never reaches the installed app.
+      # Do not reintroduce one there; MA delegates auth to Home Assistant.
+      #
+      # ORDER MATTERS: Application before Provider before Outpost, because
+      # Application.provider is on_delete=PROTECT.
      - model: authentik_core.application
        identifiers:
          slug: music-assistant-gate
@@ -157,18 +129,13 @@ data:
          name: music-assistant-gate
        state: absent

-      # Dedicated outpost: its ONLY job is to serve the in-cluster
-      # /outpost.goauthentik.io/auth/nginx endpoint that nginx
-      # auth_request hits for every request to paperless. The
-      # browser-facing legs of the OAuth flow (/start, /callback, the
-      # `/if/...` login UI, /application/o/authorize, /api, etc.) all
-      # need to happen on auth.renner.uno because that's the origin
-      # WebAuthn/FIDO/TOTP credentials are bound to. So we attach the
-      # ProxyProvider to BOTH outposts (see embedded outpost below) and
-      # set authentik_host_browser to auth.renner.uno here too — the
-      # nginx auth_request happens server-to-server, but if the
-      # dedicated outpost ever does emit a redirect URL, it points at
-      # the right host.
+      # Dedicated outpost, serving the in-cluster auth endpoint only.
+      #
+      # The browser-facing legs of the OAuth flow must happen on
+      # auth.renner.uno, because that is the origin WebAuthn credentials are
+      # bound to. Hence the ProxyProvider is attached to BOTH this outpost and
+      # the embedded one, and authentik_host_browser points at auth.renner.uno
+      # here too — so any redirect this outpost emits names the right host.
      - model: authentik_outposts.outpost
        identifiers:
          name: paperless-gate
@@ -196,12 +163,7 @@ data:
            kubernetes_ingress_secret_name: "authentik-outpost-tls"
            kubernetes_json_patches: null

-      # Tear down the leftover paperless-proxy Application from an
-      # earlier experiment. The orphan `paperless` ProxyProvider that
-      # used to back paperless.app is left in place — it's PROTECTed
-      # by Application.provider as long as anything references it, and
-      # leftover orphans don't break anything. Delete via the Authentik
-      # UI once everything's confirmed working.
+      # Leftover Application from an earlier experiment.
      - model: authentik_policies.policybinding
        identifiers:
          target: !Find [authentik_core.application, [slug, paperless-proxy]]
@@ -217,15 +179,11 @@ data:
      # ------------------------------------------------------------------
      # vscode config-editor gate
      #
-      # code-server (the HA /config editor sidecar) runs with --auth=none,
-      # as root, mounting the whole HA /config incl. plaintext secrets.yaml
-      # and .storage. This gate moves it behind Authentik with a passkey
-      # step-up, mirroring the paperless-gate proxy-outpost pattern above:
-      # the dedicated ak-outpost-vscode-gate terminates the request,
-      # enforces the session, and forwards to the code-server Service.
-      # Traefik only ever routes vscode.home.renner.uno to the outpost; the
-      # home-assistant CNP is tightened so ONLY the gate outpost may reach
-      # :8080 (see home-assistant-ns.yaml + vscode-gate-egress.yaml).
+      # code-server runs with --auth=none, as root, mounting the whole HA
+      # /config including plaintext secrets.yaml and .storage. This gate is
+      # the only thing in front of it: the outpost terminates the request and
+      # enforces a passkey step-up, Traefik routes only to the outpost, and
+      # the home-assistant CNP admits :8080 from the gate alone.
      # ------------------------------------------------------------------
      - id: vscode-config-editors-group
        model: authentik_core.group
@@ -265,17 +223,14 @@ data:
          access_token_validity: hours=24
          refresh_token_validity: days=30
          intercept_header_auth: true
-          # code-server is a service-worker app: <link rel="manifest"> and
-          # the service worker are fetched with credentials:'omit', which
-          # turned into a /authorize redirect loop and forced the RETIREMENT
-          # of the music-assistant gate (see comment above). Carve its
-          # static/manifest/SW tree out so those cookieless fetches pass
-          # through instead of 302-looping; the editor's actual content
-          # (the /config tree) is served over authenticated websocket/API
-          # paths, which stay gated.
-          # NOTE: the exact skip set is code-server-version-specific and
-          # MUST be confirmed post-deploy against the real redirect-loop
-          # behaviour (see the PR validation checklist).
+          # code-server is a service-worker app, so its manifest and SW are
+          # fetched with credentials:'omit' and 302-loop unless carved out.
+          # The editor's actual content rides authenticated websocket/API
+          # paths and stays gated.
+          #
+          # This skip set is version-specific: re-check it after a
+          # code-server bump, because a renamed asset path reintroduces the
+          # loop silently.
          skip_path_regex: ^/(_static/|manifest\.json|.*\.webmanifest|.*serviceWorker\.js).*

      - model: authentik_core.application
@@ -331,16 +286,11 @@ data:
      # ------------------------------------------------------------------
      # shelly-manager gate
      #
-      # shelly-manager has no authentication of its own: anything that can
-      # reach :8080 can rewrite Shelly device config, push scripts, and read
-      # the stored per-device AdminPassword values. Until now the only
-      # control was reachability of shelly-manager.home.renner.uno.
-      #
-      # Same shape as the vscode-gate above: a dedicated outpost terminates
-      # the request, enforces a passkey step-up, and forwards to the
-      # shelly-manager Service. Traefik only ever routes the hostname to the
-      # outpost; the home-assistant CNP allows ONLY this outpost to reach
-      # :8080 (see home-assistant-ns.yaml + shelly-manager-gate-egress.yaml).
+      # shelly-manager has no authentication of its own: anything that reaches
+      # :8080 can rewrite device config, push scripts and read the stored
+      # per-device AdminPassword values. Same shape as the vscode gate — the
+      # outpost is the only control, Traefik routes only to it, and the CNP
+      # admits :8080 from this outpost alone.
      # ------------------------------------------------------------------
      - id: shelly-managers-group
        model: authentik_core.group
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/authentik/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -27,5 +27,4 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
@@ -1,50 +1,10 @@
 ---
-# The goauthentik.io operator (inside authentik-worker) auto-creates a
-# ServiceMonitor per outpost, but it does not stamp the
-# `release: kube-prometheus-stack` label that this cluster's Prometheus
-# requires on serviceMonitorSelector. Those operator-owned SMs are
-# silently ignored. Declare standalone SMs here with the correct label
-# so the outposts actually get scraped.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-ldap
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: ldap-outpost
-      goauthentik.io/outpost-type: ldap
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
---
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-paperless-gate
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: paperless-gate
-      goauthentik.io/outpost-type: proxy
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
---
+# Only radius needs a hand-written ServiceMonitor. The goauthentik.io operator
+# creates one per outpost and Prometheus discovers those directly, because
+# serviceMonitorSelector here is empty (see monitoring/helmrelease-prometheus.yaml).
+# Standalone copies for ldap and paperless-gate used to live here as well; they
+# duplicated the operator's and were removed.
+#
 # Radius requires extra plumbing. The radius binary listens on :9300 just
 # like ldap/proxy (hardcoded in authentik/lib/default.yml), but authentik's
 # Python operator (`providers/radius/controllers/kubernetes.py`) omits the
@@ -53,6 +13,11 @@ spec:
 # it, radius does not. Result: the operator-managed `ak-outpost-local-radius`
 # Service exposes only UDP 1812. Add a sibling Service here that selects the
 # same pod labels and exposes 9300/TCP, then a ServiceMonitor for it.
+#
+# The operator's own `ak-outpost-local-radius` ServiceMonitor cannot cover
+# this: it selects on `app.kubernetes.io/managed-by: goauthentik.io` together
+# with `goauthentik.io/service-type: metrics`, and no Service carries both —
+# the operator's has the first, the one below has the second.
 apiVersion: v1
 kind: Service
 metadata:
@@ -86,6 +51,8 @@ spec:
    matchNames:
      - authentik
  selector:
+    # Pinned to the Service above. Without managed-by this would also match the
+    # operator's Service and scrape the same endpoint twice.
    matchLabels:
      app.kubernetes.io/managed-by: homelab
      goauthentik.io/outpost-name: local-radius
@@ -1,31 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-    # Backend serves HTTPS with a self-signed cert; re-encrypt + skip
-    # verify via the shared insecure-backend ServersTransport (see
-    # apps/insecure-backend-serverstransport.yaml).
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend@kubernetescrd
-spec:
-  ingressClassName: traefik
-  rules:
-    - host: file.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: file
-                port:
-                  number: 7001
-  tls:
-    - secretName: file-tls
-      hosts:
-        - file.renner.uno
@@ -1,5 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - service.yaml
-  - ingress.yaml
@@ -1,18 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-spec:
-  # Caddy reverse-proxied https://nas.renner.uno:7001. nas.renner.uno
-  # is a CNAME -> nas-1.renner.uno owned by UniFi DNS, k8s-gateway
-  # forwards srv lookups so this resolves cluster-side.
-  type: ExternalName
-  externalName: nas.renner.uno
-  ports:
-    - name: https
-      port: 7001
-      targetPort: 7001
-      protocol: TCP
@@ -18,15 +18,18 @@ metadata:
 #
 # Opaque-label map (maintainer reference only — NOT served publicly):
 #   services group                   infrastructure group
-#     Web Service 1 = Authentik        Infrastructure 1 = PBS 2 (pbs2.renner.uno)
+#     Web Service 1 = Authentik        Infrastructure 1 = retired (was PBS 2)
 #     Web Service 2 = Gitea            Infrastructure 2 = retired (was Proxmox VE)
 #     Web Service 3 = Immich           Infrastructure 3 = UDM DNS (10.2.40.1)
 #     Web Service 4 = Woodpecker CI    Infrastructure 4 = Mosquitto MQTT broker
 #     Web Service 5 = Grafana
 #     Web Service 6 = Home Assistant
-#     Web Service 7 = Paperless
+#     Web Service 7 = retired (was the NAS file UI; never Paperless, which
+#                     this map claimed and which nothing here checks)
 #     Web Service 8 = Audiobookshelf
 #     Web Service 9 = Music Assistant
+#     Web Service 10 = TRMNL
+#     Web Service 11 = Paperless (gate-level only, see its entry)
 data:
  config.yaml: |
    storage:
@@ -121,18 +124,10 @@ data:
        conditions:
          - "[STATUS] == any(200, 301, 302)"

-      - name: Web Service 7
-        group: services
-        url: https://file.renner.uno
-        interval: 60s
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: the NAS file UI (file.renner.uno). Its Ingress and
+      # ExternalName Service were deleted in #696 because they pointed at
+      # nas-1, which is powered off; this check outlived them and had been
+      # reporting failure on the public page since.

      - name: Web Service 8
        group: services
@@ -178,6 +173,29 @@ data:
        conditions:
          - "[STATUS] == any(200, 301, 302)"

+      # Paperless sits entirely behind the Authentik gate — every path,
+      # including /api/, answers 302 to unauthenticated callers, so there is no
+      # endpoint that proves the app rather than the gate. This check is
+      # therefore reachability only: DNS, TLS, Traefik and the outpost. It
+      # stays green if paperless itself is dead behind a live gate.
+      #
+      # Closing that blind spot needs an in-cluster probe of the paperless
+      # ClusterIP, which crosses a namespace boundary and so needs a
+      # CiliumNetworkPolicy first — the monitoring probes get away without one
+      # only because their targets share a namespace with the prober.
+      - name: Web Service 11
+        group: services
+        url: https://paperless.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
      # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
      # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
      # no longer deployed. Re-add when/if they come back.
@@ -191,20 +209,11 @@ data:
      # NoData). Monitor the offsite PBS from the offsite side or via a
      # check that doesn't depend on this cluster's DNS path.

-      - name: Infrastructure 1
-        group: infrastructure
-        url: https://pbs2.renner.uno:8007
-        interval: 60s
-        client:
-          insecure-skip-verify: true
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: PBS 2 (pbs2.renner.uno:8007) — it has no clients
+      # left. Its two were the PVE cluster, retired to bare-metal Talos, and
+      # nas-1, powered off and dropped from the repo. Checking the liveness of
+      # a backup server that nothing backs up to reports on nothing. Re-add
+      # alongside the first host that is declared to back up to it.

      - name: Infrastructure 3
        group: infrastructure
@@ -5,6 +5,12 @@ metadata:
  namespace: apps
  labels:
    app: gatus
+  annotations:
+    # Gatus reads its config once at startup and never re-reads the file, so
+    # without this a ConfigMap change reconciles cleanly and then does nothing:
+    # git and the running checks drift apart silently, with no failed resource
+    # to notice. Adding a check would not take effect either.
+    configmap.reloader.stakater.com/reload: gatus-config
 spec:
  replicas: 1
  # RWO PVC (`gatus-data`, ceph-block) — RollingUpdate's default 25% surge
@@ -22,9 +28,30 @@ spec:
      labels:
        app: gatus
    spec:
+      # The upstream image is FROM scratch with no USER, so without this the
+      # pod runs as root and trips PodSecurity restricted on four counts.
+      #
+      # The uid is numeric because a scratch image carries no /etc/passwd for
+      # the kubelet to resolve a name against. fsGroup is what makes the switch
+      # survivable: the only write path is the sqlite store on the RWO volume
+      # (/data/gatus.db, see the storage stanza in the ConfigMap), and fsGroup
+      # both chowns the existing volume and leaves the directory group-writable
+      # so sqlite can create its -wal and -journal siblings next to the db.
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
      containers:
        - name: gatus
          image: twinproduction/gatus:v5.36.0 # {"$imagepolicy": "flux-system:gatus"}
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
          ports:
            - containerPort: 8080
              name: http
@@ -210,6 +210,28 @@ spec:
            - name: vscode
              containerPort: 8080
              protocol: TCP
+          # /healthz answers 200 whenever the HTTP server is up; its body
+          # reports session heartbeat state, which is not health — an idle
+          # editor reports "expired" and is perfectly fine. So this asserts
+          # that code-server is serving, not that anyone is using it.
+          #
+          # Liveness matters here more than elsewhere in this pod: code-server
+          # shares the Home Assistant config volume, and a wedged editor is
+          # otherwise invisible until someone tries to open it.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: config
              mountPath: /config
@@ -1,21 +1,8 @@
 ---
 # Barman S3 credentials for ha-pg.
 #
-# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
-# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
-# problem and the same note. The values are the same today, so this is correct
-# but coupled: rotating paperless's Hetzner credentials would silently change
-# the backup credentials for Home Assistant and Woodpecker too.
-#
-# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
-# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
-#
-#   bao kv put homelab/seed/home-assistant/hetzner-creds \
-#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
-#
-# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
-# 403 on `bao kv list seed`, which was a wrong-path error rather than a
-# permissions problem.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +22,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,80 +1,22 @@
 ---
-# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
-# without putting a macvlan on the HA pod itself.
+# mDNS relay: holds the multicast membership on the IoT VLAN through its own
+# macvlan and forwards each datagram to Home Assistant as ordinary UNICAST UDP.
 #
-# THE PROBLEM
+# It must stay unicast. An avahi-style reflector cannot work here — Cilium runs
+# in tunnel mode without enable-multicast, so re-emitted multicast is dropped by
+# the CNI. Unicast is carried, and python-zeroconf accepts it: its datagram
+# handler checks size and duplicates but never the destination or source
+# address. RFC 6762's QU bit makes this protocol-legal rather than a trick.
 #
-# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
-# ff02::fb — and never hears anything on either.
-# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
-# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
-# it does not replicate multicast between pods either. Both halves of the path
-# are dead, which is why an avahi-style reflector — receive on one interface,
-# re-emit as multicast on the other — cannot work here: the re-emitted packets
-# would be dropped by the CNI.
+# This restores passive reception only — announcements, including the
+# re-announce a device sends when its address changes. HA-initiated browsing
+# still dies on eth0; that would need a querier on the VLAN side, which this
+# deliberately is not.
 #
-# Consequence for HA: the ESPHome integration dials devices at the address
-# cached in its config entry. When a lease rotates, every attempt fails with
-# `[Errno 113] Connect call failed` and the device stays unreachable, because
-# the mDNS announcement that would carry the new address never arrives. New
-# devices are never discovered at all.
-#
-# THE MECHANISM
-#
-# This relay holds the multicast membership on the IoT VLAN through its own
-# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
-# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
-# the CNI will carry.
-#
-# It works because python-zeroconf does not filter on how a packet arrived:
-# its datagram handler validates size, deduplicates, and parses, but performs
-# no check on destination address, source address, or source port. A unicast
-# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
-# unicast responses are part of RFC 6762 — the QU bit — so this is within the
-# protocol, not a trick.)
-#
-# WHAT THIS DOES AND DOES NOT RESTORE
-#
-# Restored: passive reception. RFC 6762 requires a responder to re-announce
-# when its address changes, so a device that moves to a new lease announces
-# the move, HA sees it, and the ESPHome integration updates the config entry.
-# That is the failure this exists to fix. New devices announcing themselves
-# are likewise seen.
-#
-# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
-# multicast and die there, so HA cannot actively solicit a response. Anything
-# depending on a query rather than an announcement stays broken. Making that
-# work needs a querier on the VLAN side, which this relay deliberately is not.
-#
-# EXPOSURE
-#
-# The receive sockets carry so-bindtodevice=net1, so only datagrams that
-# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
-# address instead is not possible — it is DHCP-leased and unknown at exec time
-# — but binding to the device achieves the same confinement without needing to
-# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
-# unprivileged uid here.
-#
-# Without it socat would bind the wildcard, and since the namespace CNP grants
-# every pod in home-assistant unrestricted access to every other pod on every
-# port, any sibling pod could reach :5353 over the cluster network and have
-# records forwarded into HA's zeroconf cache.
-#
-# What remains, and cannot be closed here: any host on the IoT VLAN can send
-# this port a datagram. That is the same trust already extended to genuine
-# mDNS from that segment, which any device there can equally forge — the relay
-# widens who can reach HA's listener, not what can be claimed. macvlan traffic
-# bypasses Cilium policy entirely, so the namespace CNP does not constrain
-# this interface either way.
-#
-# Note that the relay is not what makes HA's zeroconf reachable from inside
-# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
-# and the namespace CNP already permits any sibling pod to reach it directly.
-# Confining the relay removes a second route to that listener; it does not
-# remove the first.
-#
-# Nothing else is exposed: the containers run socat and only socat, with no
-# second listener to reach.
+# so-bindtodevice=net1 on both receive sockets is load-bearing: without it
+# socat binds the wildcard, and the namespace CNP lets any sibling pod reach
+# :5353 and inject records into HA's zeroconf cache. Binding net1's address
+# instead is impossible — it is DHCP-leased and unknown at exec time.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -6,32 +6,23 @@ metadata:
  labels:
    app.kubernetes.io/name: mosquitto
  annotations:
-    # REQUIRED for credential rotation to take effect at all. The startup shim
-    # rebuilds /mosquitto/auth/passwd from these Secrets on every pod start, so
-    # a rotated Secret is inert until the pod restarts — without this the new
-    # password would sit unused until some unrelated restart, while the old one
-    # stayed valid indefinitely.
+    # REQUIRED for credential rotation to take effect. The startup shim rebuilds
+    # the password file only on pod start, so without this a rotated Secret sits
+    # inert while the old password stays valid indefinitely.
    #
-    # Listed per-Secret rather than using the blanket `reloader.stakater.com/
-    # auto: "true"`, so that enrolling each remaining broker user in rotation is
-    # a visible, reviewable edit to this line rather than something that starts
-    # happening implicitly.
+    # Listed per-Secret rather than with the blanket auto annotation, so that
+    # enrolling another broker user is a visible edit to this line.
    secret.reloader.stakater.com/reload: mosquitto-health-credentials
 spec:
  replicas: 1
  strategy:
-    # No PVC anymore, so nothing to fence: the new broker can come up and pass
-    # its readiness probe before the old one is torn down. maxUnavailable:0 keeps
-    # a broker serving throughout an image bump / reschedule (clients on the old
-    # pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
-    # overlap — two independent stateless brokers behind the LB, fine since there
-    # is no shared state.
+    # No PVC, so nothing to fence — the new broker can serve before the old one
+    # is torn down. Two stateless brokers overlapping behind the LB is fine.
    #
-    # NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
-    # new pod starts with an empty retained store, so a roll is equivalent to a
-    # broker restart for every client that depends on retention. Zigbee2MQTT
-    # does not recover from that on its own; see mosquitto-config.yaml and
-    # gitea issue #381.
+    # Connectivity survives a roll but RETAINED STATE DOES NOT: the new pod
+    # starts with an empty retained store, so every roll is a broker restart
+    # for clients that depend on retention. Zigbee2MQTT does not recover from
+    # that on its own (#381).
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
@@ -44,14 +35,10 @@ spec:
      labels:
        app.kubernetes.io/name: mosquitto
    spec:
-      # Runs on any node (no nodeSelector); the mosquitto-iot LB forwards
-      # IoT-VLAN traffic to the broker wherever it lands.
-      # Availability: home-critical priority (never preempted, reschedules
-      # first) + short fault tolerations so a node failure reschedules the
-      # broker in ~30s instead of the default ~5min. With persistence off there
-      # is no PVC to fence/detach/reattach, so the broker can (re)start anywhere
-      # instantly, no longer depends on Ceph being healthy, and rolls without a
-      # tear-down-first gap (see the RollingUpdate strategy above).
+      # Runs anywhere; the mosquitto-iot LB finds it. Short fault tolerations
+      # reschedule the broker in ~30s rather than the default ~5min, which is
+      # only possible because there is no PVC to fence — so it also does not
+      # depend on Ceph being healthy.
      priorityClassName: home-critical
      terminationGracePeriodSeconds: 10
      tolerations:
@@ -6,12 +6,9 @@ metadata:
  labels:
    app.kubernetes.io/name: zigbee2mqtt
 spec:
-  # CUTOVER DONE (2026-07-23): ZHA disabled, the coordinator-backup Secret is
-  # populated in OpenBao, and Z2M is now the live Zigbee stack — so replicas is
-  # 1. It shipped at 0 as a safety interlock (Z2M contends with ZHA for the
-  # coordinator's single TCP session, and starting without the restored backup
-  # would form a NEW network and drop every device); that interlock is spent
-  # once the migration is committed. To roll back, set 0 and re-enable ZHA.
+  # Z2M is the live Zigbee stack. Never run it alongside ZHA: they contend for
+  # the coordinator's single TCP session, and a Z2M start without the restored
+  # coordinator backup forms a NEW network and drops every paired device.
  replicas: 1
  # RWO PVC → single writer; never run two pods against the same /app/data.
  strategy:
@@ -39,14 +36,10 @@ spec:
        # network_key, the latter extracted from the OpenBao coordinator backup
        # as a natural-byte-order int array.
        - name: seed-data
-          # alpine + jq, built by .woodpecker/build-ci-images.yml from
-          # ci/images/alpine-jq/. Replaces `apk add --no-cache jq` on every pod
-          # start: tools belong in the image, not installed at runtime.
-          # Referenced by tag rather than digest — these are our own images.
-          # DATED tag, not :latest, matching the ci-ansible:2026-04-29 precedent
-          # in kubernetes/infrastructure/netbird-backup/cronjob.yaml. With
-          # :latest, rebuilding this base for some unrelated consumer would
-          # silently change what this init container runs on its next pod
+          # Our own alpine + jq image, built by build-ci-images.yml. DATED tag,
+          # never :latest — rebuilding this base for an unrelated consumer
+          # would otherwise silently change what this init container runs on
+          # its next pod
          # restart, with no diff here to review — and this is the container that
          # renders the Zigbee network key. Bumping the tag is a deliberate,
          # reviewable act; the cost is remembering to bump it.
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: immich
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/immich/barman-creds
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/immich/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -13,6 +13,5 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
@@ -1,20 +0,0 @@
-# Shared Traefik ServersTransport for ingresses that re-encrypt to a
-# backend serving HTTPS with a self-signed / hostname-mismatched cert:
-#
-#   apps/file       — NAS web UI on a self-signed cert
-#
-# Replaces the nginx `backend-protocol: HTTPS` + `proxy-ssl-verify: off`
-# annotation pair. Each Ingress that wants this behaviour sets
-#   traefik.ingress.kubernetes.io/service.serversscheme: https
-#   traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend
-#
-# (bare name — kubernetesIngress provider resolves ServersTransport in
-# the same namespace as the Ingress; the `<ns>-<name>@kubernetescrd`
-# qualified form is for IngressRoute CRDs only).
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: insecure-backend
-  namespace: apps
-spec:
-  insecureSkipVerify: true
@@ -1,10 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - insecure-backend-serverstransport.yaml
  - gatus/
  - external-dns/
-  - file/
  - gitea/
  - ical-proxy/
  # woodpecker/ moved to its own dedicated `woodpecker` Kustomization
@@ -5,9 +5,8 @@ metadata:
  namespace: monitoring
  labels:
    app.kubernetes.io/name: alloy
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  selector:
@@ -1,7 +1,8 @@
 # Scrape the CNPG Postgres instances (port 9187, built-in exporter).
-# CNPG's own enablePodMonitor would generate PodMonitors without the
-# `release: kube-prometheus-stack` label, which this Prometheus requires
-# (podMonitorSelector), so we define them here instead.
+# Defined here rather than via CNPG's own enablePodMonitor so the scrape
+# config lives with the other monitors. The `release` label below is not
+# required for discovery — podMonitorSelector is empty — but kept for
+# consistency.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
@@ -5,9 +5,8 @@ metadata:
  namespace: home-assistant
  labels:
    app.kubernetes.io/name: home-assistant
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered. Same trap the garage PrometheusRule documents.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  selector:
@@ -4,14 +4,18 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-blackbox.yaml
  - prometheusrule-ceph.yaml
+  - prometheusrule-ci-openbao.yaml
  - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
  - prometheusrule-velero.yaml
  - prometheusrule-woodpecker-rotation.yaml
  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -23,4 +27,5 @@ resources:
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
@@ -5,9 +5,8 @@ metadata:
  namespace: monitoring
  labels:
    app.kubernetes.io/name: loki
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  # Matches the `loki` (single-binary) and `loki-canary` services, but
@@ -1,5 +1,9 @@
 # HTTP probes — fan out to blackbox-exporter. Probe targets mirror the
 # legacy Ansible role's blackbox_http_targets list.
+#
+# The `release` label on every Probe here IS load-bearing: probeSelector still
+# matches on it, unlike serviceMonitorSelector and podMonitorSelector, which
+# are empty. Removing it silently stops the probe from being discovered.
 apiVersion: monitoring.coreos.com/v1
 kind: Probe
 metadata:
@@ -45,6 +49,28 @@ spec:
        # ports outside the hairpin's 80/443 — real WAN path.
        - http://proxy-1.renner.uno:9000/health
        - http://proxy-2.renner.uno:9000/health
+        # Woodpecker. Ungated, root returns 200 with no redirect. Every
+        # scheduled job in this repo runs here — the alert-monitor cron, the
+        # nightly scans, the deploy targets — so an outage here is silent by
+        # construction: the thing that would have told you is the thing that
+        # is down.
+        - https://ci.renner.uno
+        # OpenBao. Probes the health API rather than the UI on purpose: a
+        # SEALED instance still serves the UI (307 -> /ui/ -> 200) while every
+        # ExternalSecret in the cluster fails, which is exactly the
+        # green-while-broken shape this file warns about elsewhere.
+        #
+        # standbyok=true because the endpoint answers 200 only on the ACTIVE
+        # node and 429 on a standby — the external route currently lands on a
+        # standby, so the bare path would be permanently red. With the
+        # parameter, 200 means initialized, unsealed and serving, on whichever
+        # node answers; 503 sealed and 501 uninitialized both stay red.
+        - https://openbao.renner.uno/v1/sys/health?standbyok=true
+        # ical-proxy serves an unauthenticated /health that answers 200 with
+        # {"status":"healthy"}. Unlike the gated hostnames this is real app
+        # health, not reachability — the root path returns 404 by design, since
+        # the service only answers on calendar paths.
+        - https://ical-proxy.renner.uno/health
        # APP-HEALTH probes for the gated monitoring UIs, from the correct
        # in-cluster vantage (monitoring-vantage-point-rule): the gate is
        # only on the ingress, so the ClusterIP /-/healthy endpoints are
@@ -57,6 +83,11 @@ spec:
        # external reachability probe above; it is not the health vantage.)
        - http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/-/healthy
        - http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/-/healthy
+        # Loki has no external vantage at all: its ingress requires auth on
+        # every path, so an outside probe can only ever report 401. In-cluster
+        # /ready answers 200 unauthenticated and is in this namespace, so the
+        # same intra-namespace allowance the two above rely on covers it.
+        - http://loki.monitoring.svc.cluster.local:3100/ready
 # WAN-path probes of the public edge BY PROXY IP. The name-based
 # home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
 # and never leaves the LAN — the vantage-point blind spot that let the
@@ -0,0 +1,59 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: blackbox
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: blackbox.rules
+      rules:
+        # Catch-all for every blackbox-http target that has no rule of its own.
+        # Before this existed, a target could be probed, graphed and completely
+        # unalerted — which is how file.renner.uno served 502 for months.
+        #
+        # THE EXCLUSION LIST AND THE DEDICATED RULES ARE ONE INVARIANT: an
+        # instance belongs here only while a rule elsewhere covers it, and
+        # adding a dedicated rule means adding its instance here. Get it wrong
+        # in one direction and the alert fires twice; wrong in the other and it
+        # never fires at all. Currently excluded:
+        #
+        #   fsn1.your-objectstorage.com   HetznerBackupEndpointDown (critical)
+        #   netbird.renner.uno            prometheusrule-netbird-availability
+        #   proxy-1/2:9000/health         same file, relay redundancy
+        #   ci.renner.uno                 WoodpeckerDown
+        #   openbao .../sys/health        OpenBaoUnavailable
+        #
+        # Warning, not critical, on purpose: this rule cannot know what a given
+        # target means. A target whose loss deserves a critical page gets its
+        # own rule — and lands in the list above. That is what keeps the
+        # exclusion list from being arbitrary.
+        - alert: BlackboxProbeFailing
+          expr: probe_success{job="blackbox-http", instance!~"https://fsn1\\.your-objectstorage\\.com|https://netbird\\.renner\\.uno|http://proxy-[12]\\.renner\\.uno:9000/health|https://ci\\.renner\\.uno|https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 10m
+          annotations:
+            summary: "Blackbox probe of {{ $labels.instance }} has been failing for 10 minutes"
+            description: >-
+              The endpoint is not answering as expected. For the Authentik-gated
+              hostnames this is a reachability check covering DNS, TLS, Traefik
+              and the gate outpost — it says nothing about the app behind the
+              gate, which the internal /-/healthy probes cover separately.
+          labels:
+            severity: warning
+        # The check that the checks are running. Every rule above reads
+        # probe_success; if the series stop being produced at all, none of them
+        # can fire and the silence looks exactly like health.
+        - alert: BlackboxProbesAbsent
+          expr: absent(probe_success{job="blackbox-http"})
+          for: 15m
+          annotations:
+            summary: "No blackbox-http probe results are reaching Prometheus"
+            description: >-
+              Either the blackbox exporter is down or the Probe objects stopped
+              being discovered. Every endpoint alert depends on this job, so
+              they are all blind until it returns.
+          labels:
+            severity: critical
@@ -0,0 +1,47 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ci-openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ci-openbao.rules
+      rules:
+        - alert: WoodpeckerDown
+          # Probe target defined in probes-blackbox.yaml. Root, ungated, 200.
+          expr: probe_success{job="blackbox-http", instance="https://ci.renner.uno"} == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Woodpecker is not answering on ci.renner.uno"
+            description: >-
+              Every scheduled job in this repo runs here: the alert-monitor
+              cron that files these issues, the nightly scans, and every deploy
+              target. Nothing else reports this outage, because the component
+              that would have reported it is the one that is down. Alertmanager
+              itself is unaffected, which is why this alert exists.
+        - alert: OpenBaoUnavailable
+          # Probes /v1/sys/health?standbyok=true — 200 on any unsealed node,
+          # 503 sealed, 501 uninitialized. The UI stays up when sealed, so a
+          # root probe would not catch this.
+          #
+          # Matched by prefix rather than the full string: the target carries a
+          # query parameter, and an exact match would break silently if the
+          # instance label ever encodes it differently.
+          expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao is sealed, uninitialized or unreachable"
+            description: >-
+              Every ExternalSecret in the cluster resolves through OpenBao.
+              Existing Secrets keep working, so nothing breaks at once — the
+              damage shows up at the next refresh interval or the next pod that
+              needs a fresh credential. Check whether it is sealed before
+              assuming a network fault.
@@ -11,25 +11,22 @@ spec:
  groups:
    - name: cnpg-backup.rules
      rules:
-        # WHY THIS FILE EXISTS
-        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
-        # stopped. The only related rule was HetznerBackupEndpointDown, which
-        # probes the S3 endpoint — it proves the bucket is reachable, not that
-        # anything was written to it. Same failure shape as the WordPress backup
-        # that failed nightly for three days against a bucket that was declared
-        # in tofu but never applied: the status surface looked fine because
-        # nothing checked the thing that mattered.
+        # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
+        # probes that S3 endpoint, which proves the bucket is reachable and
+        # nothing more — these two rules check that something is actually being
+        # written to it.

        - alert: CNPGWalArchiveBacklog
          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
-          # to be archived. Steady state is 0 across every cluster (verified
-          # 2026-08-19). A sustained backlog means the archiver is failing:
-          # point-in-time recovery is silently broken from that moment, and
-          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          # to be archived. Steady state is 0. A sustained backlog means the
+          # archiver is failing: point-in-time recovery is silently broken from
+          # that moment, and pg_wal keeps growing until it fills the PVC and
+          # takes Postgres down.
          #
-          # This is the load-bearing alert of the two. It depends only on the
-          # instance-level collector, so unlike the backup-age metrics below it
-          # is unaffected by the missing Backup CRs (#522).
+          # This is the load-bearing alert of the two. It reads the
+          # instance-level collector directly, so unlike the backup-age metric
+          # discussed below it does not depend on Cluster status being
+          # populated.
          #
          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
@@ -63,16 +60,17 @@ spec:
        #
        # The natural expression is
        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
-        # and it CANNOT be used today: that metric reads exactly 0 on every
-        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
-        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
-        # successful backup anywhere, because the Backup CRs it derives that
-        # from are being deleted within a minute of completing — see #522.
+        # and it cannot be used, because that metric reads exactly 0 on every
+        # instance. The collector derives it from `Cluster.status
+        # .lastSuccessfulBackup`, and that field is absent on every Cluster
+        # here: all of them back up with `method: plugin` (barman-cloud), and
+        # the plugin path does not write it back.
        #
-        # Backups themselves DO work; that was verified independently by
-        # listing the bucket with barman-cloud-backup-list and restoring a WAL
-        # segment back out of S3. The gap is in the record, not the data.
+        # The backups themselves run and are retained — completed Backup CRs
+        # going back weeks exist for every scheduled cluster. The gap is in the
+        # status field the metric reads, not in the data.
        #
-        # Adding the rule now would fire permanently for all seven clusters and
-        # be silenced within a day, which is worse than not having it. Add it as
-        # part of resolving #522, once the metric reflects reality.
+        # Adding the rule against that metric would fire permanently for every
+        # cluster, which is worse than not having it. Either the plugin starts
+        # populating the field, or the alert has to be built on the Backup CRs
+        # directly, which Prometheus does not see today.
@@ -0,0 +1,125 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # Wear percentages are self-reported: 100 means the rated endurance is
+        # spent, not that the drive stops.
+        #
+        # Wear alone is not a signal. It only ever rises, so a plain threshold
+        # starts firing the day a drive crosses it and never stops — a status
+        # dressed as an alert (#673, #675, #682, #686 were all one drive set
+        # crossing 80%). What matters is wear that is high AND still climbing,
+        # so the alert below is a projection: at the write rate of the last
+        # week, when does this drive reach 95%?
+        #
+        # The rate is self-calibrating. Each drive reports both its lifetime
+        # bytes written and its own wear percentage, so bytes-per-percent comes
+        # from the drive itself and no rated-endurance table has to be kept in
+        # sync with the hardware. percentage_used is a whole-number percent and
+        # advances only every few weeks here, which is far too coarse to
+        # differentiate; bytes_written moves on every scrape.
+        - alert: NvmeDriveWearProjectedExhaustion
+          # The >= 80 gate is numerical, not editorial: bytes-per-percent is
+          # derived by dividing by an integer percentage, so on a lightly worn
+          # drive a half-percent rounding error swings the projection by tens
+          # of percent. Above 80 the same rounding is under one percent.
+          # Idle drives divide by a zero rate and yield +Inf, which is not < the
+          # threshold, so they stay silent on their own.
+          expr: >-
+            (
+              (95 - smartctl_device_percentage_used)
+              /
+              (
+                rate(smartctl_device_bytes_written[7d]) * 86400
+                  * smartctl_device_percentage_used
+                  / smartctl_device_bytes_written
+              )
+            ) < 180
+            and smartctl_device_percentage_used >= 80
+          # Long, because a projection off a 7d rate should not react to a day.
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} projects to 95% wear in {{ $value | printf \"%.0f\" }} days"
+            description: >-
+              The drive is past 80% of its rated endurance and the write rate
+              of the last week would spend the rest within six months. This is
+              a lead-time warning, not a fault: decide whether to replace the
+              drive or move what is writing to it. The Disk Health dashboard
+              shows which drives carry which role.
+        # Kept as a flat threshold on purpose: past rated endurance the drive
+        # runs outside what the manufacturer stands behind regardless of how
+        # fast it got there, and the alert clears by replacing it.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # Wear is a budget; spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # On the increase, not the total: pre-existing errors would otherwise
+          # alert forever. Standing counts are visible on the dashboard only.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # The check that the checks are running: no series fires no alert.
+        # Counted against node-exporter's inventory, not a fixed fleet size.
+        - alert: SmartctlExporterIncomplete
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.
@@ -17,11 +17,12 @@ spec:
          # drains on recovery) and Velero backups fail. Replaces the old
          # GarageDown alert now that backups live on Hetzner, not nas-1.
          #
-          # CNPG has no metric-based archiving alert yet — its pod metrics
-          # are not scraped (enablePodMonitor=false). Enabling CNPG
-          # PodMonitors for a granular "archiving falling behind" alert is
-          # a follow-up. Velero's own VeleroBackupValidationFailure /
-          # VeleroBackupStale rules already cover backup-level failures.
+          # The granular counterpart is CNPGWalArchiveBacklog in
+          # prometheusrule-cnpg-backup.yaml, reading the instance collector
+          # scraped by cnpg-podmonitors.yaml: this alert says the target is
+          # unreachable, that one says archiving is falling behind whatever the
+          # cause. Velero's VeleroBackupValidationFailure / VeleroBackupStale
+          # cover backup-level failures.
          expr: probe_success{instance="https://fsn1.your-objectstorage.com"} == 0
          for: 10m
          labels:
@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.
@@ -40,17 +40,31 @@ spec:
            description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."

        - alert: VeleroBackupStale
-          # Daily schedule with no successful backup in over 30h: covers
-          # the case where multiple consecutive runs failed (Garage flapping
-          # over the backup window) without the per-event alerts being
-          # acted on.
+          # Daily schedule with no successful backup in over 60h: covers the
+          # case where consecutive runs failed without the per-event alerts
+          # being acted on.
+          #
+          # 60h, not 30h. At 30h a single missed cycle pages, and a single
+          # missed cycle is the normal cost of touching the nodes: Velero moves
+          # data through pods, so draining a node kills whatever upload is in
+          # flight and the schedule is simply lost for that day — it is not
+          # retried. Rolling node work between 03:15 and 08:45 UTC therefore
+          # produced a guaranteed critical alert for every schedule it
+          # overlapped, which is noise about planned work rather than a signal
+          # about backups.
+          #
+          # 60h still catches what this is for. Two consecutive failures is no
+          # longer explicable by one maintenance window, and the per-event
+          # VeleroBackupFailed alert fires on the first failure regardless — so
+          # nothing is hidden, only the escalation to critical is deferred
+          # until a second cycle confirms it.
          expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
            and
            velero_backup_last_successful_timestamp{schedule!=""} > 0
          for: 10m
          labels:
            severity: critical
          annotations:
-            summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
-            description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
+            summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
+            description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
@@ -0,0 +1,24 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl polls the drives every 60s; scraping faster gains nothing.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this a series carries only the exporter pod's IP, which does
+        # not identify the machine the drive is in.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
@@ -0,0 +1,475 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
+      ],
+      "title": "OSDs down",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory (identity and role)",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
+      "id": 14,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
+      "title": "Worst NVMe wear",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 15,
+      "options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
+          "legendFormat": "{{node}} {{model_name}}",
+          "refId": "A"
+        }
+      ],
+      "title": "NVMe wear",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": [
+          {"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
+          {"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
+          {"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
+          {"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 16,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
+      ],
+      "title": "SMART health",
+      "transformations": [
+        {"id": "merge", "options": {}},
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
+            "indexByName": {},
+            "renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}
@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
  - name: dashboard-blackbox-overview
    files:
      - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
  - name: dashboard-gitea-overview
    files:
      - gitea-overview.json
@@ -24,8 +24,10 @@ spec:
      enabled: true
      defaults:
        labels:
-          # Required for kube-prometheus-stack's Prometheus to pick up
-          # this ServiceMonitor (its selector matches release).
+          # Not required for discovery — serviceMonitorSelector is empty — but
+          # kept for consistency with the other monitors. The Probes in
+          # monitoring-rules/probes-blackbox.yaml do need it; probeSelector
+          # still matches on release.
          release: kube-prometheus-stack
        interval: 30s
    # Helm DOES deep-merge this map with the chart's default (that is why
@@ -56,26 +56,21 @@ spec:
        #
        # By default kube-prometheus-stack sets these selectors to
        # `matchLabels: {release: kube-prometheus-stack}`. ServiceMonitors
-        # created at RUNTIME by third-party operators carry that operator's own
-        # labels instead and can never be made to match — no static manifest or
-        # Kustomize patch can relabel an object the operator creates itself.
+        # created at RUNTIME by third-party operators — Rook, goauthentik.io —
+        # carry that operator's own labels instead and can never be made to
+        # match: no static manifest or Kustomize patch can relabel an object
+        # the operator creates itself. Left strict, those targets are dropped
+        # silently, with no failed resource and no gap visible anywhere except
+        # in the missing series.
        #
-        # This was silently dropping 12 of 30 ServiceMonitors (2026-07-27):
-        #   rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter  (team: rook)
-        #   authentik/ak-outpost-*  x10                            (goauthentik.io/...)
-        # Consequence: ZERO `ceph_*` series existed in Prometheus for all 79
-        # days of the cluster's life — the entire storage layer had no
-        # telemetry, no history, and any Ceph alerting rule would have had
-        # nothing to fire on. The Rook side is the reason this was found; the
-        # authentik outpost targets come along with it.
+        # Because these two are empty, the `release` label on this repo's own
+        # ServiceMonitors and PodMonitors is decorative. It is still set for
+        # consistency, and the comments there say so.
        #
-        # This is the ServiceMonitor twin of the trap already documented for
-        # rules in monitoring-rules/prometheusrule-velero.yaml ("without this
-        # label the rule is created but never loaded").
-        #
-        # ruleSelector is deliberately NOT loosened: this repo's PrometheusRules
-        # all carry `release: kube-prometheus-stack` on purpose, and keeping
-        # that selector strict stops a stray third-party rule from firing here.
+        # ruleSelector, probeSelector and scrapeConfigSelector are deliberately
+        # NOT loosened: nothing outside this repo creates those, so keeping the
+        # selectors strict stops a stray third-party rule from firing here. For
+        # PrometheusRules and Probes the label IS required.
        serviceMonitorSelectorNilUsesHelmValues: false
        podMonitorSelectorNilUsesHelmValues: false
        # Rewrites the in-cluster URL embedded in alert generatorURLs and
@@ -275,8 +270,11 @@ spec:
          - alertmanager-smtp
      # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
      # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
      # monitoring ns is not blocked by any CNP today.
      config:
        global:
@@ -15,6 +15,7 @@ resources:
  - prometheus-gate-service.yaml
  - prometheus-gate-ingress.yaml
  - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
  - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -0,0 +1,89 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# Runs privileged: host device access is gated by the container device cgroup,
+# and Kubernetes offers no per-device allowlist. Narrower settings do not work;
+# the alternatives that were tried are in the commit history.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device: the exporter scans and rescans on its own.
+          # Device names are not stable across these machines.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: true
+            # The API rejects privileged together with
+            # allowPrivilegeEscalation: false.
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # Not read-only: smartmontools opens NVMe devices read-write.
+            - name: dev
+              mountPath: /dev
+            # Scratch space, because the root filesystem is read-only.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -24,6 +24,22 @@ spec:
            - name: wyoming
              containerPort: 10400
              protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process. Weaker than an HTTP probe, but the previous state was none
+          # at all: a dead server stayed Ready and kept being routed to.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 15
+            periodSeconds: 10
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 100m
@@ -5,25 +5,15 @@ metadata:
  labels:
    app.kubernetes.io/name: otbr
 spec:
-  # Disabled 2026-07-11 (replicas: 0) after otbr broke Matter.
+  # DISABLED (replicas: 0). Running otbr broke Matter: its radio is RF-isolated
+  # enough from the Nest mesh that it forms a rival partition of the same
+  # network, and the two bridge only over Ethernet — fast enough for a ping,
+  # too slow for the fragmented CASE handshake, so devices stay discoverable
+  # while every commissioning attempt times out.
  #
-  # otbr is seeded with the Nest network's dataset so it should join the
-  # existing partition as an additive co-BR. In practice it does not merge
-  # cleanly: its SLZB-06 radio is RF-isolated enough from the Nest mesh that
-  # otbr elects itself leader of a rival partition (or the network re-addresses
-  # under a new OMR prefix, observed fd5f -> fdd5). The two same-network
-  # partitions then bridge only over the Ethernet backbone, which carries a
-  # ping but is too slow for the large fragmented CASE handshake -- every
-  # Matter device stays mDNS-discoverable while CASE times out. Removing otbr
-  # re-merges the mesh onto the Nest hubs and Matter recovers.
-  #
-  # This is NOT a diagnosis of a specific config bug: the DHCP/sbr routing
-  # changes (19a4482, df61407) were investigated and matter-server's IPv6 path
-  # to the mesh works (reaches devices, ping ok); sbr's v6 table 100 is an
-  # inert orphan and otbr's eth0 default is outranked by net1's pref-high RA
-  # default. The instability is at the Thread partition layer. Re-enable only
-  # after the co-BR integration is verified to MERGE (one partition, one OMR,
-  # otbr's router table shows the Nest routers) instead of split.
+  # The instability is at the Thread partition layer, not in the config below.
+  # Re-enable only once co-BR integration is verified to MERGE: one partition,
+  # one OMR prefix, and the Nest routers visible in otbr's router table.
  replicas: 0
  strategy:
    type: Recreate
@@ -46,25 +36,17 @@ spec:
      # (slzb-thread.renner.uno). Dropping the workload-class=pinned
      # toleration lets the edge taint keep it off the tiny edge box.
      containers:
-        # The Thread RCP lives on a SLZB-06 over Ethernet at
-        # slzb-thread.renner.uno:6638 (Serial-over-IP). otbr-agent's
-        # spinel transport supports `spinel+hdlc+forkpty://<exec>?<args>`
-        # which forks an exec whose stdio becomes the spinel-HDLC
-        # transport, but the upstream openthread/otbr image does NOT
-        # ship socat (or any TCP-stream tool), so that URL form ends in
-        # silent ENOENT and otbr-agent dies right after logging the
-        # radio URL.
+        # The Thread RCP is a network-attached radio (Serial-over-IP), so the
+        # spinel transport needs a TCP-to-PTY bridge. Two obvious approaches
+        # do not work and should not be retried:
        #
-        # A sidecar socat won't work either: each container has its own
-        # devpts namespace, so a PTY (e.g. /dev/pts/0) created in a
-        # sibling container is invisible from this container — a shared
-        # emptyDir symlink to /dev/pts/0 is dangling.
+        #   - `spinel+hdlc+forkpty://` needs socat, which the upstream image
+        #     does not ship; the URL fails with a silent ENOENT and otbr-agent
+        #     dies right after logging it.
+        #   - A sidecar cannot provide it either: devpts is per-container, so
+        #     a PTY created next door is invisible here and the symlink dangles.
        #
-        # The bridge has to live in the otbr container's own namespace.
-        # Install socat at startup via apt (Ubuntu 18.04 base, no egress
-        # restrictions on this namespace), run it in the background as
-        # the spinel-HDLC TCP↔PTY bridge, then exec the upstream
-        # entrypoint with RADIO_URL pointing at the PTY symlink.
+        # Hence socat is installed at startup and run inside this container.
        - name: otbr
          image: openthread/otbr:latest
          command:
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: paperless
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/paperless/barman-creds
@@ -70,6 +70,27 @@ spec:
            - name: consume
              mountPath: /home/scanner
              subPath: scanner
+          # FTP has no health verb, so the control port accepting a connection
+          # is what can be asserted. It is enough for the failure that matters:
+          # if vsftpd dies the scanner cannot deposit anything and nothing else
+          # in the chain notices, because paperless only ever sees an empty
+          # consume directory — which is indistinguishable from nobody having
+          # scanned.
+          #
+          # Only the control port is probed. The passive range is negotiated
+          # per transfer and is not listening at rest.
+          livenessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 20m
@@ -1,3 +1,9 @@
+# One Hetzner S3 key, one path, shared by every barman ObjectStore. Separate
+# per-app paths would not make rotation independent — that needs one Hetzner
+# key per consumer (#397).
+#
+# velero reads the same key from infrastructure/velero-hetzner, which holds it
+# as a credentials file rather than these three fields.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
@@ -15,4 +21,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -20,7 +20,6 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
 configMapGenerator:
@@ -155,21 +155,33 @@ spec:
            - name: bridge-cert
              mountPath: /shared
              readOnly: true
-          # TCP probes — HTTP probes against /api/ flooded the
-          # `paperless.auth` logger every 10 s with "No authentication
-          # provided from private IP …". With ENABLE_HTTP_REMOTE_USER
-          # the middleware logs every unauth API request, and there's
-          # no public health endpoint to probe. TCP confirms gunicorn
-          # is bound; that's the same liveness signal we'd get from
-          # /api/ minus the noise.
+          # /accounts/login/ rather than /api/ or a TCP connect.
+          #
+          # /api/ is unusable as a probe target: with ENABLE_HTTP_REMOTE_USER
+          # the remote-user middleware logs "No authentication provided from
+          # private IP …" for every unauthenticated API request, so probing it
+          # floods `paperless.auth`. Measured: four requests to /api/ produce
+          # four log lines, four to /accounts/login/ produce none — the
+          # middleware only covers the API surface.
+          #
+          # A TCP connect avoids the noise but only proves gunicorn accepted a
+          # socket. /accounts/login/ renders a real Django view, so URL
+          # routing, settings and templating all have to work. It does not
+          # prove Postgres is healthy; nothing cheap here does.
+          #
+          # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
+          # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
+          # means giving these probes an explicit Host header.
          livenessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
              port: http
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 10
          readinessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
@@ -34,6 +34,32 @@ spec:
            - name: wyoming
              containerPort: 10300
              protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process — weaker than an HTTP probe, but the previous state was
+          # none at all.
+          #
+          # A startupProbe rather than a long liveness delay: this container
+          # runs with --retrain-on-start, and retraining against the Home
+          # Assistant entity list takes as long as it takes. The startup gate
+          # allows up to 10 minutes (60 x 10s) and only then hands over to
+          # liveness, so a slow retrain cannot be mistaken for a hang, and a
+          # genuine hang after startup is still caught within a minute.
+          startupProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: models
              mountPath: /models
@@ -12,4 +12,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/trmnl/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,20 +1,15 @@
 # Headless renderer: screenshots the Home Assistant e-ink dashboard,
 # dithers it to 16 greys and hands it to Terminus.
 #
-# Upstream ships this as a Home Assistant add-on. Our HA is a plain
-# container with no Supervisor, so it runs standalone here instead —
-# viable because the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env
-# vars over /data/options.json and authenticates with an ordinary
-# long-lived token. SUPERVISOR_TOKEN is deliberately NOT set: it only
-# flips an internal isAddOn flag, and setting it would switch the app
-# to add-on behaviour (fixed port, /data assumptions).
+# Upstream ships this as a Home Assistant add-on; it runs standalone here
+# because the app prefers the env vars over /data/options.json. Do NOT set
+# SUPERVISOR_TOKEN — it flips an internal isAddOn flag and switches the app to
+# add-on behaviour (fixed port, /data assumptions).
 #
-# DIRECTION OF TRAVEL, which is the non-obvious part: with the
-# byos-hanami webhook in `uri` delivery mode the renderer does NOT
-# push image bytes. It POSTs a URL pointing at its own /output/<file>
-# endpoint, and TERMINUS FETCHES THE IMAGE BACK from it. That is why
-# both live in this namespace — the pull is intra-namespace and is
-# already permitted by the default-deny-ingress rule.
+# DIRECTION OF TRAVEL, the non-obvious part: in `uri` delivery mode the
+# renderer does not push image bytes. It POSTs a URL to its own /output/<file>
+# and Terminus fetches the image back. Both therefore live in this namespace,
+# where that pull is already permitted.
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -29,6 +29,27 @@ spec:
            - name: wyoming
              containerPort: 10300
              protocol: TCP
+          # Wyoming is a raw TCP protocol with no HTTP surface and no health
+          # verb, so accepting a connection is the only thing that can be
+          # asserted from outside the process. That is weaker than an HTTP
+          # probe but strictly better than the previous state of none at all,
+          # where a dead server stayed Ready and Home Assistant kept routing
+          # speech to it.
+          #
+          # The liveness delay is generous because the model is loaded at
+          # startup from the data volume.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 90
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            timeoutSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /data
@@ -305,7 +305,7 @@ server → agent → step pod → server end to end:

 ```sh
 curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
-woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+woodpecker-cli pipeline deploy k.renner/homelab <pipeline-number> e2e-test
 ```

 Step pods must still reach the server on `:8000` while still being unable to
@@ -94,6 +94,31 @@ spec:
          envFrom:
            - secretRef:
                name: woodpecker-secret
+          ports:
+            # WOODPECKER_HEALTHCHECK defaults to true and serves :3000, so this
+            # port already existed; declaring it just lets the probe name it.
+            - name: health
+              containerPort: 3000
+              protocol: TCP
+          # /healthz answers 500 ONLY when a running workflow has passed its own
+          # timeout plus a one-hour buffer (agent/state.go, State.Healthy). An
+          # idle agent always answers 200.
+          #
+          # So this catches a wedged build holding a slot, which is otherwise
+          # invisible — and it does NOT catch an agent that has lost its server
+          # connection, which reports healthy while quietly taking no work. Do
+          # not read a green probe here as "CI is fine".
+          #
+          # Liveness only: the agent fronts no Service, so readiness would gate
+          # nothing. The upstream condition is already debounced by that hour,
+          # which is why the thresholds here are short.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            initialDelaySeconds: 20
+            periodSeconds: 30
+            timeoutSeconds: 5
          resources:
            requests:
              cpu: 25m
@@ -1,11 +1,8 @@
 ---
 # Barman S3 credentials for wp-pg.
 #
-# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
-# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
-# that path needs OpenBao write access (`bao kv list seed` → 403 for the
-# identities available here), so it stays open. The values are identical, so it
-# is a one-line swap of the `key:` below once the path exists.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -36,11 +36,9 @@ spec:
  target:
    name: hetzner-s3-creds
  dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
-    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
-    # written, and repoint this here (the value is the same, so it's a drop-in
-    # swap).
+    # One shared path for one shared credential. There is a single Hetzner S3
+    # key behind all of these — see the note in
+    # ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
+    # the wrong shape and what would have to change for them to be right.
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.
@@ -33,22 +33,21 @@ spec:
    remediation:
      retries: 3
  values:
-    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
-    # datapath options once, at agent start, and the chart puts no config
-    # checksum on the pod template — so a changed ConfigMap leaves every agent
-    # running the previous datapath until something restarts it. Nothing did.
+    # THESE VALUES ALSO FEED THE BOOTSTRAP SEED. talos/patches/cilium-cni.yaml
+    # is rendered from this block; the command is in that file's header. Change
+    # something here and the seed is stale until it is re-rendered, and nothing
+    # will tell you — the only place the drift shows is a `talosctl upgrade-k8s
+    # --dry-run`, which is how a 1.19.2 seed survived a month behind a 1.20
+    # chart.
    #
-    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
-    # v30iot macvlan workload stayed broken until the agents were deleted by
-    # hand, one node at a time (issue #592). policyMapMax has the same
-    # exposure, and its comment below says "takes effect on agent restart" —
-    # documentation standing in for automation.
+    # Rolls the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once at agent start and the chart puts no config
+    # checksum on the pod template, so without this a changed ConfigMap leaves
+    # every agent on the previous datapath indefinitely (#592).
    #
-    # The trade: any cilium-config change now rolls the CNI cluster-wide,
-    # node by node. That is disruptive-ish, but it is what a datapath change
-    # already required — the difference is that it now happens on its own
-    # instead of being forgotten. Reloader is already deployed for exactly
-    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    # The trade is that any cilium-config change now rolls the CNI node by
+    # node — which is what a datapath change required anyway, only now it is
+    # not forgotten.
    annotations:
      configmap.reloader.stakater.com/reload: cilium-config
    # Which pod labels feed security-identity computation. Patterns are
@@ -58,8 +57,6 @@ spec:
    # and every identity costs one entry per allowed (direction, proto, port)
    # in every endpoint's policy map, cluster-wide. Nothing selects on these
    # labels in policy, so they buy identity churn and nothing else.
-    # Keep in lockstep with kubernetes/talos/patches/cilium-cni.yaml and
-    # kubernetes/talos/cilium.yaml.
    labels: '!woodpecker-ci\.org/.* !step$'
    ipv6:
      enabled: true
@@ -69,57 +66,38 @@ spec:
    socketLB:
      enabled: true
    prometheus:
-      # The agent exported no metrics at all, so BPF map utilisation was
-      # invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
-      # of space the agent only says "no space left on device" in its log and
-      # pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
-      # that observable — and alertable — before it stalls a rollout.
+      # Without agent metrics, BPF map utilisation is invisible: a full policy
+      # map only shows up as "no space left on device" in the log while pod
+      # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
+      # alertable beforehand.
      enabled: true
      serviceMonitor:
        enabled: true
    bpf:
-      # Size of each endpoint's own cilium_policy_v3_<id> map, holding one
-      # entry per (identity, direction, protocol, port) the endpoint is
-      # allowed. When it fills, endpoint regeneration fails with "no space
-      # left on device", the CNI ADD call never returns, and the pod is stuck
-      # in sandbox creation — so this must stay comfortably above the busiest
-      # endpoint's entry count, tracked by cilium_bpf_map_pressure.
-      # Takes effect on agent restart; the map is sized at creation.
-      # Bootstrap installs Cilium from the inline manifest in
-      # kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
-      # must move with this value or a cold rebuild starts with the old size.
-      # kubernetes/talos/cilium.yaml carries a third copy of the same key.
-      # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
+      # Per-endpoint policy map size: one entry per (identity, direction,
+      # protocol, port) allowed. When it fills, endpoint regeneration fails
+      # with "no space left on device" and the pod hangs in sandbox creation.
+      # Keep it comfortably above the busiest endpoint, watched by
+      # cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
+      # restart.
+      #
      policyMapMax: 65536
      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
      # filter" in cilium_drop_count_total.
      #
-      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
-      # otbr and music-assistant attachments. On bare metal it is a VLAN
-      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
-      # an untagged virtio NIC, which is why this was not needed before.
+      # Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
+      # carry an address, so it sees mgmt0's frames already untagged by the
+      # 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
+      # address-less, so its frames only meet the filter on the shared parent,
+      # still tagged. Adding 40 would change nothing and would touch the
+      # datapath carrying etcd and the API.
      #
-      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
-      # interesting part. Cilium attaches to devices that carry an address —
-      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
-      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
-      # a device and sees frames the 8021q layer has already untagged; nothing
-      # for a VLAN filter to match. ens19 is deliberately address-less (see
-      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
-      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
-      # meet the filter on the shared parent, still tagged.
+      # The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
+      # reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
+      # "no buffer space available" (#592).
      #
-      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
-      # would touch the datapath carrying etcd and the API for no reason.
-      #
-      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
-      # symptom is the CNI dhcp plugin reporting "no buffer space available"
-      # rather than a DHCP timeout. See issue #592.
-      #
-      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
-      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
-      # must move with this list. Takes effect on agent restart.
+      # Same three-copy problem as policyMapMax above. Takes an agent restart.
      vlanBypass: [30]
      # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
      # redirects the socket directly to the backend pod instead of sending
@@ -163,3 +141,26 @@ spec:
      enabled: true
    externalIPs:
      enabled: true
+    # These three were live but tracked nowhere: applied by hand, owned by the
+    # `kubectl-patch` field manager on cilium-config, absent from this file and
+    # from the bootstrap seed. Server-side apply left them alone, so nothing
+    # broke and nothing noticed — but a cold rebuild installs only the seed, and
+    # would have come up with the BGP control plane off and no BGP session to
+    # the UDM at all. Declaring them here puts them under Flux and carries them
+    # into the seed, which is generated from these values.
+    #
+    # The values match what the cluster runs, but this is not purely a change of
+    # ownership: enabling bgpControlPlane through the chart also emits five
+    # companion keys the hand-patch never set — bgp-secrets-namespace,
+    # bgp-router-id-allocation-mode and -ip-pool,
+    # enable-bgp-control-plane-status-report, enable-bgp-legacy-origin-attribute.
+    # Cilium has been running on its own internal defaults for those. They
+    # should agree with the chart's, but router-id allocation is the one worth
+    # watching on apply: if it disagrees, the sessions re-establish.
+    #
+    # The peering itself is CRDs in post-crds/cilium-bgp.yaml — that was always
+    # in git; only the switch that starts the control plane was not.
+    bgpControlPlane:
+      enabled: true
+    ipv4NativeRoutingCIDR: 10.0.0.0/8
+    ipv6NativeRoutingCIDR: fd00::/8
@@ -36,25 +36,16 @@ spec:
  values:
    container_runtime: containerd

-    # --- mTLS between agents and the LAPI: the durable fix for the recurring
-    # KubeDaemonSetRolloutStuck auth loop (#340) ---
+    # mTLS between agents and the LAPI. Do not switch back to password auth:
+    # the LAPI runs without persistence, so its machine-registration DB is
+    # wiped on every restart, and password-registered agents then get 401
+    # "machine not found" and crashloop until re-registered by hand — which
+    # wedges the whole `infrastructure` Kustomization behind a HelmRelease
+    # that never goes Ready (#340).
    #
-    # The LAPI runs WITHOUT persistence (see persistentVolume note below), so
-    # its sqlite machine-registration DB is ephemeral and every LAPI pod restart
-    # wiped it. Under the old password flow the agents registered exactly ONCE
-    # (init container running `cscli lapi register --token`) and cached a
-    # password the wiped LAPI no longer recognised -> 401 "machine not found"
-    # -> CrashLoopBackOff on all three agents -> DaemonSet never Ready -> the
-    # crowdsec HelmRelease upgrade `--wait` times out forever -> the whole
-    # `infrastructure` Kustomization (and every app depending on it) stays
-    # unhealthy. That is the #340 outage.
-    #
-    # With client-cert auth the agent init container no longer registers at all
-    # (verified via `helm template`: the `cscli lapi register` step is gone).
-    # The agent presents a cert whose OU (agent-ou) the LAPI trusts, and a
-    # freshly restarted LAPI with an EMPTY DB auto-recreates each machine record
-    # from the presented cert on connect. So the system self-heals across LAPI
-    # restarts with no PVC, no Ceph, and no manual re-registration.
+    # With client certs the agent never registers: a restarted LAPI with an
+    # empty DB recreates each machine record from the presented cert, so this
+    # self-heals with no PVC and no intervention.
    #
    # cert-manager (already reconciled earlier in this same `infrastructure`
    # stage) provisions a fully self-contained self-signed CA (selfSigned
@@ -87,13 +78,10 @@ spec:

    lapi:
      replicas: 1
-      # The chart defaults request 500m. Measured steady-state draw is 2m, and
-      # the request is what the scheduler reserves, so the default parks half a
-      # core of unusable capacity per LAPI. That is affordable on three nodes
-      # and is not on two: with one node out for maintenance the remaining pair
-      # sit at 100% of requested CPU while actually using ~22%, and nothing new
-      # can be placed. The limit stays at the chart default so a log flood can
-      # still burst; only the reservation shrinks.
+      # Far below the chart default, which reserves half a core per LAPI for a
+      # workload that draws single-digit millicores — enough to leave the
+      # cluster unschedulable with one node out. Only the reservation shrinks;
+      # the limit stays at the chart default so a log flood can still burst.
      resources:
        requests:
          cpu: 50m
@@ -33,3 +33,16 @@ spec:
    crds: CreateReplace
    remediation:
      retries: 3
+  values:
+    # The chart ships these disabled, so the controller ran with no probe of
+    # any kind: a wedged reconciler stayed Ready and every ExternalSecret would
+    # simply stop refreshing, which surfaces only when a consumer needs a
+    # credential that has since rotated.
+    #
+    # Enabling them also starts the health server — the chart derives
+    # --health-probe-bind-address from these values, and without them the
+    # controller exposes nothing but its metrics port.
+    livenessProbe:
+      enabled: true
+    readinessProbe:
+      enabled: true
@@ -6,19 +6,12 @@
 # fails the route over across all per-node peers (no single point of
 # failure if a node/pod drops).
 #
-# Uses the ROOTLESS image (userspace wireguard-go). The netbird process
-# terminates the WireGuard tunnel in userspace and reaches cluster
-# services by opening its OWN sockets to them — NOT by L3-forwarding
-# packets. That distinction is the whole point:
-#   - Cilium socket-LB DNATs the netbird process's own connections, so
-#     forwarding to ClusterIP / LB VIPs actually works. A kernel L3
-#     routing peer fails here: socket-LB never sees forwarded packets,
-#     so they hit the raw VIP and are blackholed (confirmed via conntrack).
-#   - No NET_ADMIN / no kernel WireGuard → runs fully unprivileged
-#     (drop ALL caps, runAsNonRoot), so it satisfies Pod Security too.
-#
-# Flow: mesh peer (100.96.x.x) --wg--> a per-node pod (userspace netbird)
-#        --own socket (socket-LB DNAT)--> service ClusterIP/LB VIP
+# Must stay on the ROOTLESS image (userspace wireguard-go). The process
+# terminates the tunnel in userspace and reaches services by opening its own
+# sockets, which Cilium's socket-LB can DNAT. A kernel WireGuard peer
+# L3-forwards instead, socket-LB never sees those packets, and they are
+# blackholed at the raw VIP. Rootless also means no NET_ADMIN, so the pod
+# runs fully unprivileged.
 #
 # Egress is locked down by the netbird-ingress-egress CiliumNetworkPolicy
 # (network-policies/ingress-isolation.yaml).
@@ -41,18 +41,14 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # shelly-manager (:8080) is likewise reached only through its own
-    # Authentik gate outpost. Note this rule and the vscode one are
-    # port-identical: because this CNP's endpointSelector is {}, a
-    # `toPorts: 8080` grant here applies to EVERY pod in the namespace that
-    # listens on 8080, not just the intended one — which is exactly how
-    # shelly-manager silently rode on the old traefik→:8080 allow written
-    # for code-server, and why removing that allow in 760e187 took
-    # shelly-manager's ingress down with it. The `fromEndpoints` selector is
-    # what keeps these two apart: each gate outpost can only reach its own
-    # target, enforced on the egress side by the matching CNPs in the
-    # authentik namespace (vscode-gate-egress.yaml,
-    # shelly-manager-gate-egress.yaml).
+    # shelly-manager, behind its own Authentik gate outpost.
+    #
+    # CAUTION for every :8080 rule here: this policy's endpointSelector is {},
+    # so a toPorts grant on 8080 applies to EVERY pod in the namespace
+    # listening on that port, not just the intended one. Three apps share
+    # :8080. What keeps them apart is the fromEndpoints selector plus the
+    # matching per-gate egress CNPs in the authentik namespace. Changing a
+    # :8080 rule here affects all of them.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -61,11 +57,8 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # esphome (ESPHome Device Builder, :6052) is likewise reached only through
-    # its own Authentik gate outpost. Unlike the :8080 vscode/shelly rules
-    # above, :6052 is unique to esphome, so this namespace-wide (endpointSelector
-    # {}) toPorts grant doesn't silently apply to any other app — the
-    # fromEndpoints selector still scopes it to the esphome-gate outpost.
+    # esphome (:6052), behind its own gate outpost. Port is unique to it, so
+    # the namespace-wide grant does not reach anything else.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -74,13 +67,8 @@ spec:
        - ports:
            - port: "6052"
              protocol: TCP
-    # zigbee2mqtt (Z2M frontend, :8080) is likewise reached only through its own
-    # Authentik gate outpost. Like the vscode/shelly-manager rules above this is
-    # the SHARED :8080 (zigbee2mqtt is the third app on that port), so this
-    # namespace-wide (endpointSelector {}) grant applies to every :8080 pod — the
-    # fromEndpoints selector scopes it to the zigbee2mqtt-gate outpost, and the
-    # matching authentik-side zigbee2mqtt-gate-egress.yaml scopes egress to the
-    # zigbee2mqtt pod specifically.
+    # zigbee2mqtt frontend, behind its own gate outpost. Shared :8080 — see the
+    # caution above.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -89,11 +77,8 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # zone-configurator (Everything Presence mmWave zone configurator, :42069)
-    # is likewise reached only through its own Authentik gate outpost. Like
-    # esphome's :6052, :42069 is unique to it, so this namespace-wide (endpoint
-    # Selector {}) toPorts grant doesn't silently apply to any other app; the
-    # fromEndpoints selector scopes it to the zone-configurator-gate outpost.
+    # zone-configurator (:42069), behind its own gate outpost. Port is unique
+    # to it.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -131,25 +116,16 @@ spec:
    - fromEntities:
        - host
        - remote-node
-    # World ingress (IoT-VLAN devices via the BGP LoadBalancers) is NOT
-    # granted namespace-wide here. A blanket `fromEntities: world` on this
-    # endpointSelector:{} policy exposed EVERY pod in the namespace to the
-    # untrusted IoT VLAN / any BGP-routed source — vscode (:8080),
-    # zone-configurator (:42069), otbr, whisper, etc. — not just the two
-    # LoadBalancers that need it. World is now scoped per-target in the
-    # dedicated policies at the bottom of this file: world -> mosquitto
-    # :1883/:8883 and world -> home-assistant :8123 only. (matter-server on
-    # 10.2.x.226 rides a multus net1 interface that bypasses Cilium policy
-    # entirely, so it is unaffected either way.)
-    # ICMP — all standard ICMPv4 and ICMPv6 types from any cluster pod.
-    # Per Cilium docs, "If any layer 4 policy is specified, then ICMP
-    # will be blocked unless it's related to a connection that is
-    # otherwise allowed by the policy." The CT-RELATED path is not
-    # passing these in practice (Hubble shows ICMPv6 DestinationUnreachable
-    # being POLICY_DENIED on the egress side of established DNS flows),
-    # so allow ICMP explicitly. Same shape as commit 3c517a0 — widened
-    # from the error class to the full standard type set. Cilium's
-    # ICMPField requires an explicit type per entry; no wildcard.
+    # Do NOT add a blanket `fromEntities: world` to this policy. With
+    # endpointSelector {} it exposes every pod in the namespace to the IoT
+    # VLAN and any BGP-routed source, not just the LoadBalancers that need it.
+    # World is scoped per target in the dedicated policies at the bottom of
+    # this file. (matter-server rides a multus interface that bypasses Cilium
+    # policy entirely and is unaffected either way.)
+    # ICMP must be allowed explicitly: once any L4 policy exists Cilium blocks
+    # it unless CT-RELATED passes it, which in practice it does not. Cilium's
+    # ICMPField takes one type per entry — there is no wildcard, so the list
+    # has to be spelled out.
    - fromEndpoints:
        - matchExpressions:
            - key: io.kubernetes.pod.namespace
@@ -186,10 +162,8 @@ spec:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": apps
-    # In-cluster InfluxDB: HA's HACS InfluxDB integration writes state
-    # changes here (replaces the legacy http://influxdb.srv.renner.uno
-    # target — see kubernetes/apps/influxdb/ha-token-job.yaml for the
-    # write-bucket token wiring).
+    # In-cluster InfluxDB: HA's HACS integration writes state changes here.
+    # Token wiring is in apps/influxdb/ha-token-job.yaml.
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": influxdb
@@ -213,9 +187,8 @@ spec:
    - toEntities:
        - host
        - remote-node
-    # World: IoT-VLAN devices, ESPHome, HA Cloud, garage S3 (initial
-    # tarball seed), legacy InfluxDB on VM 108, third-party integration
-    # endpoints (Hue, Roomba, Philips, Brother IPP, …).
+    # World: IoT-VLAN devices, HA Cloud, object storage, and third-party
+    # integration endpoints on the LAN and internet.
    - toEntities:
        - world
    # ICMP — egress mirror of the ingress ICMP allow above. Lets HA
@@ -1,12 +1,5 @@
 ---
-# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
-# agent arrive in stage 2).
-#
-# Unlike the `apps` baseline this replaces, every allow here lands on a
-# namespace holding one service, so a namespace-scoped grant is not a blanket
-# grant across unrelated workloads. That is the point of the move: in `apps`,
-# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
-# external-dns and ical-proxy to the whole database namespace (issue #433).
+# Baseline ingress policy for the `woodpecker` namespace.
 #
 # CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
 # every rule below applies to EVERY pod in the namespace. A `toPorts` block
@@ -15,11 +8,7 @@
 # belongs in its own policy with its own selector, as
 # `wp-pg-allow-cnpg-operator` below does.
 #
-# No egress section, so egress is unrestricted. Deliberate for now and revisited
-# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
-# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
-# Writing that ruleset once, with the full workload set present, beats guessing
-# it here and silently breaking the WAL archiving this stage exists to verify.
+# There is no egress section, so egress is unrestricted. Still open (#433).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -5,25 +5,15 @@ kind: Kustomization
 # secret. The matching OpenBao policy/role for each lives in ../policies/ and is
 # bootstrapped manually (not GitOps-managed).
 #
-# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
-# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
-# image, so it does not need digest pinning, and a dated tag makes "what actually
-# runs" a reviewable diff.
+# IMAGE PINNING: dated tags (`:YYYY-MM-DD`), never a digest and never
+# `:latest`. Do NOT switch to digests — untagged manifests in this registry are
+# garbage-collected once the tags move past them, so a pinned digest silently
+# stops resolving after a few unrelated rebuilds of build-ci-images.yml. These
+# jobs are quarterly, so the resulting ImagePullBackOff looks like nothing
+# happening rather than like a failure (#458).
 #
-# Digests were tried and actively broke: untagged manifests are garbage-collected
-# once :latest and the dated tags move past them, so a digest pinned today stops
-# resolving after a few rebuilds. That decay is driven by edits to
-# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
-# republishes secret-rotator and orphans the manifest every rotator here was
-# pinned to. 11 of these were pinned to digests that no longer existed and would
-# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
-# are quarterly, that would have looked like nothing happening rather than like a
-# failure.
-#
-# Keep them all on one tag: it is a single binary containing every mode
-# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
-# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
-# the next bump partial.
+# Keep every rotator on the same tag: it is one binary containing all modes, so
+# splitting only makes the next bump partial.
 #
 # ADDING A ROTATOR: copy an existing file, and check you did not inherit a
 # digest pin from an older example — nothing enforces this automatically, so it
@@ -24,12 +24,10 @@ spec:
    - start: "fd99::"
      stop: "fd99::ff"
 ---
-# Cilium BGPv2 control-plane config peering each k8s node with the UDM
-# over its MGMT LAN interface — NOT over v30iot. Control-plane traffic
-# belongs on the management network; the IoT VLAN is the noisy untrusted
-# L2 segment we're explicitly trying to keep cluster infrastructure off
-# of. Each node sources from its eth0 (10.2.40.x / fd40::<host>) via
-# the kernel routing decision; no explicit localAddress override needed.
+# Cilium BGPv2 control-plane config peering each node with the UDM over the
+# management LAN — never over v30iot, which is the untrusted IoT segment.
+# The source address comes from the kernel routing decision, so no explicit
+# localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
 #   - v4 session: peer 10.2.40.1, family IPv4 unicast only
@@ -1,27 +1,16 @@
 ---
 # Velero Schedules for stateful app namespaces.
 #
-# Scope is restricted to PersistentVolumeClaims via includedResources.
-# Flux owns every other K8s object (Deployment, Service, Ingress,
-# ConfigMap, Secret, …) — backing those up shadows current git state
-# on a rebuild: Velero restores stale manifests from yesterday's
-# snapshot, then the app Kustomization clobbers them with whatever git
-# says — except when the Kustomization is blocked on the restore chain,
-# leaving the cluster temporarily serving the stale spec (e.g. the
-# paperless Ingress missing prod hostnames after the 2026-04-22
-# rebuild). PVC-only is the canonical "back up data, not state" scope
-# (see ~/Notes/wiki/k8s-backup-scope.md and the project memory rule).
+# Scope is PersistentVolumeClaims only. Flux owns every other object, and
+# backing those up shadows git on a rebuild: Velero restores yesterday's
+# manifests and the cluster serves them until the Kustomization catches up.
+# Back up data, not state.
 #
-# snapshotMoveData=true keeps doing the heavy lifting via Kopia +
-# node-agent: PVC contents are uploaded to Garage and restored
+# snapshotMoveData uploads PVC contents through Kopia and the node-agent,
 # independently of the K8s API surface.
 #
-# Naming convention: <app>-daily for the app's Schedule. Restore is a
-# manual, operator-driven procedure — see kubernetes/RUNBOOK-cold-rebuild.md.
-# (There is no automatic bootstrap-restore Job; it was removed as dead code,
-# issue #200.)
-#
-# Retention: 30 days, matching the live CNPG Barman ObjectStore policy.
+# Naming: <app>-daily. Restore is manual — see RUNBOOK-cold-rebuild.md.
+# Retention matches the CNPG Barman policy.

 apiVersion: velero.io/v1
 kind: Schedule
@@ -203,22 +192,13 @@ metadata:
  labels:
    app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
-  # their manifest (matter-server, otbr, whisper, speech-to-phrase,
-  # zone-configurator all set `namespace: home-assistant` via their
-  # Kustomization, not per-resource): home-assistant-config (.storage
-  # entity registry, recorder DB, secrets.yaml), esphome-data,
-  # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
-  # own path (the HA pod's `seed` initContainer pulling the native HAOS
-  # backup from Garage, see the home-assistant Kustomization comment in
-  # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
-  # PVC restore on top of that flow for the same PVC needs its own review.
-  # Restore of these schedules is manual — see
-  # kubernetes/RUNBOOK-cold-rebuild.md.
+  # Daily at 06:45 local. Sweeps every PVC in the home-assistant namespace —
+  # several apps under apps/ set `namespace: home-assistant` in their
+  # Kustomization, so their PVCs land here too.
+  #
+  # home-assistant-config has its own restore path: the HA pod's `seed`
+  # initContainer pulls the native HAOS backup. Layering a Velero PVC restore
+  # on top of that for the same PVC needs its own review.
  schedule: "45 6 * * *"
  useOwnerReferencesInBackup: false
  template:
@@ -229,6 +209,13 @@ spec:
    storageLocation: default
    ttl: 720h0m0s
    snapshotMoveData: true
+    # Excludes the CNPG cluster's PVC. A crash-consistent snapshot of a
+    # running Postgres data directory is not a restorable backup, and Barman
+    # covers that database anyway — see ha-pg-scheduled-backup.yaml.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule
@@ -8,16 +8,9 @@ spec:
    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
-  # Do not start restarting OSDs while PGs are still recovering. The default
-  # is false, so Rook would proceed regardless — and paired with the old
-  # osdMaxUpdatesInParallel default that means it can take the next OSD down
-  # while the previous one's data is still being caught up.
-  #
-  # With three OSDs and size 3 there is no spare copy to fall back on: the
-  # gate between one OSD returning and the next going down is the only thing
-  # that keeps a rolling change from becoming an outage. See
-  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
-  # from the other side.
+  # Defaults to false. With three OSDs and size 3 there is no spare copy, so
+  # this gate is the only thing keeping a rolling OSD change from taking the
+  # next one down mid-recovery. Paired with storage.osdMaxUpdatesInParallel.
  upgradeOSDRequiresHealthyPGs: true
  mon:
    count: 3
@@ -39,112 +32,40 @@ spec:
      daemon:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
-  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
-  # Data Movement provisions a backupPVC per source PVC by cloning the
-  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's
-  # multi-PVC Backup with `rados: "all cloner threads are busy"`. 16
-  # gives headroom for parallel restores during cluster rebuild too
-  # (paperless + audiobookshelf + immich + ... all clone at once).
+  # CephFS cloner pool raised from the default 4. Velero clones a
+  # VolumeSnapshot per PVC, and 4 threads stall a multi-PVC backup with
+  # `rados: "all cloner threads are busy"`.
  cephConfig:
    mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than omitted. Deleting a line here does
+      # NOT restore the default: `ceph config` keeps whatever was last applied,
+      # so the old value would sit in the mon config database indefinitely.
+      # Change values in place; to actually drop one, `ceph config rm` it.
+      mon_data_avail_warn: "30"
+      # Off, against the upstream default. The forced compact_range on every
+      # paxos trim rewrote the whole store hundreds of times a day and was the
+      # single largest source of disk writes on these nodes.
      #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
-      # Stop compacting the whole RocksDB prefix on every paxos trim.
-      #
-      # Upstream default is true — "Compact a certain prefix (including paxos)
-      # when we trim its old states" — which is fine at low paxos rates and
-      # pathological here. Measured 2026-08-19:
-      #
-      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
-      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
-      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
-      #                                        (cgroup io.stat: 2.15 TB/27 d)
-      #   => ~98x write amplification, for a ~106 MB database
-      #
-      # That was ~62% of all OS-disk writes on these nodes and roughly a
-      # quarter of everything the cluster writes — more than every client
-      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
-      # means hundreds of forced range compactions per day, each rewriting the
-      # store.
-      #
-      # The churn itself is 99% cluster log and mgr stats, not real cluster
-      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
-      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
-      # report / osd dump — 23 audit entries per minute per mon) becomes a
-      # cluster-log entry, hence a paxos transaction, replicated to all three
-      # mons. Reducing that polling is the complementary fix and is tracked
-      # separately; this setting removes the amplification regardless of rate.
-      #
-      # This does NOT disable compaction. RocksDB's own levelled compaction
-      # still runs and still reclaims trimmed keys; what stops is the extra
-      # forced compact_range on every trim. Store growth is therefore bounded
-      # by normal compaction, and there are two backstops if that assumption is
-      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
-      # CephHealthWarn in monitoring-rules already alerts on, and
-      # mon_compact_on_start below compacts fully on every mon restart.
-      #
-      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
-      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
-      # not come back down, revisit rather than leaving it.
+      # This does NOT disable compaction — RocksDB's levelled compaction still
+      # reclaims trimmed keys. Backstops if that proves wrong:
+      # mon_data_size_warn raises HEALTH_WARN, and mon_compact_on_start below
+      # compacts fully on every mon restart.
      mon_compact_on_trim: "false"
-      # Compensating control for the above: a full compaction on mon start.
-      # Upstream default is false. Mons restart on upgrades, node reboots and
-      # reschedules, so this gives a periodic full reclaim without paying for
-      # one on every trim. Cheap here — the store is ~106 MB.
+      # Compensating control, against the upstream default: mons restart often
+      # enough for this to give periodic full reclaim without paying for one
+      # on every trim.
      mon_compact_on_start: "true"
    mgr:
      mgr/volumes/max_concurrent_clones: "16"
-    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
-    # earlier comments here said "P2", which is a different drive). These need
-    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
-    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM. Init-time settings — an OSD restart is needed to apply.
    #
-    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
-    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
-    # again without re-reading this.
-    #
-    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
-    # prefer_deferred_size defers every write SMALLER than the threshold: the
-    # write goes first into RocksDB's WAL and is applied to the object store
-    # afterwards. That is a double write, and at 65536 it caught essentially
-    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
-    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
-    # state does not need the random-write batching that helps spinning rust.
-    #
-    # Two reasons it was the wrong trade here:
-    #   - The benchmark that justified it cannot have exercised it. It used
-    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
-    #     was ever deferred. The setting was validated by a workload it does
-    #     not affect.
-    #   - It doubles writes onto QLC, which has the lowest endurance of any
-    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
-    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
-    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
-    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
-    #
-    # This cluster is write-light (~117 GB/day at the RBD client layer), so
-    # throughput was never the binding constraint. Endurance is. If small-write
-    # latency ever becomes the real problem, fix it with more OSDs rather than
-    # by paying a 2x write tax on every 4 KiB page.
+    # bluestore_prefer_deferred_size_ssd and bluestore_deferred_batch_ops_ssd
+    # are DELIBERATELY LEFT AT THE CEPH DEFAULTS and must not be raised again.
+    # Deferral double-writes everything below the threshold, and an HDD-style
+    # value catches every write this cluster makes (4 KiB SQLite, 8 KiB
+    # Postgres pages). Ceph disables it on SSD for that reason. Throughput has
+    # never been the binding constraint here; endurance is. If small-write
+    # latency becomes a real problem, add OSDs instead.
    osd:
      bdev_enable_discard: "true"
      bdev_async_discard: "true"
@@ -158,106 +79,44 @@ spec:
    ssl: false
  monitoring:
    enabled: true
-  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
-  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
+  # Multus. The attachment is san-network-attachment.yaml.
  #
-  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  # Only `cluster` is selected. Do not add `public`, and do not reach for
+  # `addressRanges` — that is honoured only under provider: host, which puts
+  # every daemon including the mons on host networking. A mon's address then
+  # becomes the node's DHCP lease, and when a lease moves the mon crash-loops
+  # on "Cannot assign requested address" with no node free to fail over to,
+  # because all three already hold the mon host ports.
  #
-  # Because it reaches it for every daemon, not just the ones that need it.
-  # The cluster network is an OSD concern: per the Ceph documentation it
-  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
-  # operate on the public network". `addressRanges` is only honoured under
-  # provider: host, so using it put the mons on host networking as a side
-  # effect — making a mon's address the node's DHCP lease.
+  # With `public` unset, client traffic stays on the pod network and VLAN 49
+  # carries Ceph-internal traffic only — which is also why Rook's Multus
+  # prerequisites about host routing do not apply here.
  #
-  # That is not theoretical. When a lease moved, the mon could no longer bind
-  # ("Cannot assign requested address") and crash-looped. Rook detected it and
-  # started a failover, but with three mons on three nodes each holding the
-  # mon host ports, the replacement could not be scheduled anywhere — the
-  # canary sat Pending on "didn't have free ports". Two of three nodes hit
-  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
-  #
-  # Only `cluster` is selected here. With `public` left unset Rook puts client
-  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
-  # Rook-managed identities and hold no host ports — which is what removes
-  # that failure mode rather than working around it.
-  #
-  # ISOLATION IS PRESERVED
-  #
-  # Rook's Multus prerequisites about hosts routing to the Multus network
-  # apply to a Multus *public* network. Selecting only `cluster` means they do
-  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
-  # else, no router, no other participants. After this settles, the node
-  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
-  # should come out of the Talos configs — at which point the attachment's
-  # pool can take the whole /29.
-  #
-  # Reached from provider: "" rather than directly from host, which Rook's CRD
-  # rejects. See the phase A commit.
-  #
-  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
-  # reproducible from the manifests alone. Removing addressRanges here does
-  # not clear public_network and cluster_network from Ceph's own config
-  # database — the same trap the bluestore comment above describes: `ceph
-  # config` retains a value that is only deleted from this file. During the
-  # phase A rollout the first OSD came up on the pod network, still had
-  # public_network 10.2.40.0/24 in the config database, found no interface in
-  # it and crash-looped on "Failed to pick public address". Both keys were
-  # removed with `ceph config rm global public_network` and the same for
-  # cluster_network; Rook then wrote cluster_network afresh from the
-  # attachment below. Anyone rebuilding this cluster from scratch will not hit
-  # it — the config database starts empty — but anyone repeating a provider
-  # change on a live cluster will.
-  #
-  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
-  # above: one OSD restarts at a time and only once PGs are clean again.
-  #
-  # The attachment itself is san-network-attachment.yaml.
+  # Changing the provider on a LIVE cluster needs a step this file cannot
+  # express: `ceph config` retains public_network and cluster_network after
+  # they disappear from here, and an OSD that comes up on the pod network with
+  # a stale public_network crash-loops on "Failed to pick public address".
+  # Clear them with `ceph config rm global public_network` (and cluster_network)
+  # before the rollout. A cluster built from scratch starts with an empty
+  # config database and never sees this.
  network:
    provider: multus
    selectors:
      cluster: rook-ceph/ceph-san
-  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
-  # taken out of the storage fleet by removing it here. Under useAllNodes the
-  # operator re-provisions an OSD on any node whose device still carries a
-  # BlueStore signature, which makes purging an OSD ahead of pulling its drive
-  # impossible: the OSD is recreated within a reconcile and starts backfilling
-  # onto the drive being replaced.
+  # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
+  # re-provisions an OSD on any node whose device still carries a BlueStore
+  # signature, so purging an OSD before pulling its drive is impossible — it
+  # comes back within a reconcile and backfills onto the drive being replaced.
  #
-  # All three nodes are bare metal and see three NVMe drives each, so they name
-  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
-  # Talos enumerates these machines' drives in a different order than Proxmox
-  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
-  # depending on the boot. Rook would probably decline the system disk for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
-  #
-  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
-  # and the Crucial as nvme0n1 on the very next one.
-  #
-  #   k8s-1 (jupiter)
-  #   50026B7382B48B86  Kingston  — Talos system disk
-  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M400426    PM983     — this OSD
-  #
-  #   k8s-2 (saturn)
-  #   50026B7382DA5F6E  Kingston  — Talos system disk
-  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M403161    PM983     — this OSD
-  #
-  #   k8s-3 (neptune)
-  #   50026B7382B48B90  Kingston  — Talos system disk
-  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NF0M501176    PM983     — this OSD
+  # Devices are named by /dev/disk/by-id path. Kernel names are not stable
+  # across boots on these machines — k8s-3 enumerated a different drive as
+  # nvme0n1 on two consecutive boots — and a bare name can land on the system
+  # disk or on EPHEMERAL.
  storage:
-    # One OSD restarted at a time. The default is 20, which on a three-OSD
-    # cluster means all of them at once — every PG inactive, every pool
-    # unreachable, for as long as the restart takes. The default suits a
-    # cluster large enough that 20 OSDs is a small fraction of it; here it is
-    # the whole thing.
-    #
-    # This binds any change that rewrites the OSD deployment spec, not just
-    # Ceph version upgrades — a network provider change is one such.
+    # One at a time. The default of 20 would restart every OSD in this cluster
+    # simultaneously — every PG inactive until they return. Binds any change
+    # that rewrites the OSD deployment spec, not just version upgrades.
    osdMaxUpdatesInParallel: 1
    useAllNodes: false
    useAllDevices: false
@@ -273,18 +132,12 @@ spec:
          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
    config:
      osdsPerDevice: "1"
-  # Storage layer must never be evicted by kubelet under node pressure
-  # and must never be picked first by the Linux OOM killer. system-node-
-  # critical (PriorityValue 2000001000) is the highest built-in class:
-  # kubelet preserves these pods last, and kernel sets oom_score_adj to
-  # -997 so anything else dies before a Ceph daemon does.
+  # Highest built-in class: kubelet evicts these last and the kernel gives
+  # them the lowest oom_score_adj.
  priorityClassNames:
    all: system-node-critical
-  # memory requests == limits => Guaranteed QoS. Combined with the
-  # priority class above, the pod's oom_score_adj is the lowest possible.
-  # CPU limits are deliberately NOT set — Ceph daemons benefit from
-  # bursting under recovery/backfill, and a CPU limit would throttle
-  # them; CPU pressure on this cluster is not a real concern (~15%).
+  # requests == limits gives Guaranteed QoS. CPU limits are deliberately NOT
+  # set — Ceph daemons burst during recovery and a limit would throttle them.
  resources:
    mon:
      requests:
@@ -1,22 +1,7 @@
 ---
-# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
-# without every Ceph daemon being put on host networking to get there.
-#
-# WHAT THIS REPLACES, AND WHY
-#
-# `addressRanges` only takes effect under `provider: host` — Rook ignores it
-# for the default CNI provider. So carrying replication over the dedicated
-# 10G link meant setting provider: host, which is not per-daemon: it puts
-# mons on host networking too. A mon's address is then the node's DHCP lease,
-# and when that lease moves the mon cannot bind and fails. Worse, three mons
-# on three nodes each hold the mon host ports, so Rook cannot schedule a
-# replacement anywhere and the failover it starts cannot finish.
-#
-# None of that has anything to do with the goal. Per the Ceph documentation
-# the cluster network carries OSD heartbeat, replication and recovery only —
-# "Ceph Monitors always operate on the public network". Attaching it here
-# gives the OSDs the link and leaves everything else on the pod network,
-# where Rook manages identity itself.
+# The Ceph cluster network as a Multus attachment, so the OSDs reach VLAN 49
+# without every Ceph daemon landing on host networking to get there. The
+# alternative and why it was rejected is in cephcluster.yaml.
 #
 # ISOLATION
 #
@@ -29,20 +14,22 @@
 #   needed — unlike the v30iot attachments, there is no DHCP-supplied default
 #   route to confine, because there is no DHCP and no default route.
 #
-#   The pool stays inside 10.99.0.0/29, on-link with the node addresses, and
-#   this is load-bearing during the migration rather than a detail. OSDs move
-#   one at a time, so the cluster runs mixed for a while: an OSD already on
-#   this attachment has to reach an OSD still using the node's host address.
-#   There is no router on this segment, so anything outside the /29 would be
-#   unreachable from the un-migrated side and the two would mark each other
-#   down — during precisely the window that is supposed to be safe.
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
 #
-#   range_start/range_end reserve .4-.6 and leave .1-.3 to the nodes. That is
-#   exactly one address per OSD with nothing spare, which is sufficient
-#   because osdMaxUpdatesInParallel bounds the migration to one OSD at a
-#   time: allocations grow 1, 2, 3 and never overlap. Once the switch is
-#   confirmed and the node addresses come out of the Talos configs, the pool
-#   can take the whole /29 and the spare returns.
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
 #
 # IPAM
 #
@@ -76,7 +63,7 @@ spec:
      "ipam": {
        "type": "whereabouts",
        "range": "10.99.0.0/29",
-        "range_start": "10.99.0.4",
+        "range_start": "10.99.0.1",
        "range_end": "10.99.0.6"
      }
    }
@@ -7,11 +7,8 @@ spec:
  interval: 24h
  url: https://traefik.github.io/charts
 ---
-# Traefik runs alongside ingress-nginx during the migration off the
-# archived kubernetes/ingress-nginx project. Shares the `ingress` namespace
-# so the post-migration topology is just "ingress namespace, one controller".
-# Pilots: shelly-manager.home.renner.uno. ingress-nginx remains the default
-# IngressClass; this controller is opt-in via `ingressClassName: traefik`.
+# The cluster's only ingress controller, and the only IngressClass. It
+# replaced ingress-nginx, which was retired after that project was archived.
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
@@ -44,27 +41,18 @@ spec:
  values:
    deployment:
      replicas: 2
-    # Entrypoint-default Middlewares for websecure: baseline security
-    # response headers (HSTS/CSP/X-Frame-Options/etc.) and a generous
-    # abuse/brute-force rate limit, both applied to every router on this
-    # entrypoint without per-Ingress annotations. See
-    # security-headers-middleware.yaml / rate-limit-middleware.yaml for the
-    # actual settings and rationale. This is the same entrypoint-default
-    # wiring mechanism the (currently detached) crowdsec-bouncer Middleware
-    # documents re-enabling via — independent of that CrowdSec enforcement
-    # decision, which stays off pending the client-IP-visibility fix
-    # referenced below.
+    # Entrypoint-default Middlewares for websecure: security response headers
+    # and a rate limit, applied to every router on this entrypoint without
+    # per-Ingress annotations. Settings live in
+    # security-headers-middleware.yaml and rate-limit-middleware.yaml.
    additionalArguments:
      - --entryPoints.websecure.http.middlewares=ingress-security-headers@kubernetescrd,ingress-rate-limit@kubernetescrd
-    # CrowdSec bouncer plugin (Yaegi), loaded but NOT globally attached.
-    # Enforcement was disabled after it caused an IPv6 ingress outage on
-    # first deploy: Traefik sees a single shared/SNAT'd source IP for v6
-    # ingress (no real per-client IP), so one http-admin-interface-probing
-    # hit (the e2e scan) banned that shared IP and 403'd all v6 clients.
-    # Re-attach the bouncer (additionalArguments entrypoint middleware ->
-    # ingress-crowdsec-bouncer@kubernetescrd) only after fixing client-IP
-    # visibility (forwardedHeaders/trustedIPs or the v6 masquerade) and
-    # adding an internal/LAN whitelist. The plugin + crowdsec-middleware.yaml
+    # CrowdSec bouncer plugin (Yaegi), loaded but deliberately NOT attached.
+    # Traefik sees one shared SNAT'd source for IPv6 ingress, so a single
+    # probe banned that address and 403'd every v6 client. Do not re-attach
+    # it until client-IP visibility is fixed (forwardedHeaders/trustedIPs or
+    # the v6 masquerade) and a LAN whitelist exists.
+    # The plugin + crowdsec-middleware.yaml
    # stay in place so re-enabling is a one-line change. Detection (the
    # crowdsec agents parsing Traefik logs) remains on and is unaffected.
    experimental:
--- a/Show More
+++ b/Show More