feat(monitoring): alert on any blackbox target without a rule of its own

Nine of the thirteen blackbox-http targets were probed, graphed and completely unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two internal /-/healthy probes that exist specifically to catch a dead app behind a live Authentik gate. That is the shape that let file.renner.uno serve 502 for months. BlackboxProbeFailing covers everything except the instances that already have a dedicated rule. Measured against the running Prometheus before committing: the selector returns exactly those nine and excludes the six with bespoke handling, and the full expression as YAML delivers it evaluates to zero results today. The exclusion list and the dedicated rules are one invariant, stated in the file: an instance belongs in the list only while a rule elsewhere covers it, and a new dedicated rule has to add its instance. Wrong in one direction the alert fires twice, wrong in the other it never fires. Severity is warning because a catch-all cannot know what any given target means — anything deserving critical earns its own rule, which is also what puts it in the list. That is what stops the list from being arbitrary. BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads probe_success; if the job stops producing series none of them can fire, and that silence is indistinguishable from health. Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG pod metrics are not scraped and a granular archiving alert is a follow-up. Both have existed for a while: cnpg-podmonitors.yaml scrapes the instances and CNPGWalArchiveBacklog reads that collector.
Merge pull request 'feat(monitoring): probe and alert on Woodpecker and OpenBao' (#697 ) from probe-ci-openbao into master
2026-08-28 11:10:45 +02:00 · 2026-08-28 08:59:10 +00:00 · 2026-08-28 10:57:07 +02:00 · 2026-08-28 08:47:36 +00:00 · 2026-08-28 08:46:58 +00:00 · 2026-08-28 10:45:11 +02:00
55 changed files with 456 additions and 1186 deletions
@@ -32,13 +32,13 @@
 #              mechanism as update-audit.yml):
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name alert-monitor --branch master --schedule '*/15 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=alert-monitor k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> alert-monitor
+#              where <n> is an existing pipeline to promote.
 when:
  - event: cron
    cron: alert-monitor
-  - event: manual
-    evaluate: 'PIPELINE == "alert-monitor"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "alert-monitor"'

 steps:
  - name: fetch-alerts
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -31,8 +31,8 @@ when:
      - "ci/images/**"
      - "secret-rotator/**"
      - ".woodpecker/build-ci-images.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "build-ci-images"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-ci-images"'

 steps:
  - name: build-ansible
@@ -44,7 +44,7 @@ steps:
    # secret-rotator commit needlessly rebuilt all four ~1.3G ci-* images and
    # orphaned ~5G of registry layers per run — a main driver of #336.
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/ansible/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -68,7 +68,7 @@ steps:
    directory: ci/images/k8s
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/k8s/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -92,7 +92,7 @@ steps:
    directory: ci/images/tofu
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/tofu/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -123,7 +123,7 @@ steps:
    directory: secret-rotator
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["secret-rotator/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -152,7 +152,7 @@ steps:
    directory: ci/images/alpine-jq
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -181,7 +181,7 @@ steps:
    directory: ci/images/cni-plugins
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -210,7 +210,7 @@ steps:
    directory: ci/images/alpine-openssl
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -239,7 +239,7 @@ steps:
    directory: ci/images/heating-can
    depends_on: []
    when:
-      - event: manual
+      - event: deployment
      - event: push
        path: ["ci/images/heating-can/**", ".woodpecker/build-ci-images.yml"]
    environment:
@@ -54,8 +54,8 @@
 #
 # Triggering:
 #   - cron   : the cron NAMED exactly "ci-claude-refresh" (see above).
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=ci-claude-refresh k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> ci-claude-refresh
+#              where <n> is an existing pipeline to promote.
 #   - push   : a change to the Containerfile (or this file) ON master forces a
 #              rebuild of the current latest version so Containerfile fixes ship
 #              promptly. Restricted to master ON PURPOSE — a feature branch must
@@ -65,8 +65,8 @@
 when:
  - event: cron
    cron: ci-claude-refresh
-  - event: manual
-    evaluate: 'PIPELINE == "ci-claude-refresh"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "ci-claude-refresh"'
  - event: push
    branch: master
    path:
@@ -17,6 +17,6 @@ when:

 steps:
  - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    commands:
      - sh scripts/ci/test-model-selection.sh
@@ -6,8 +6,8 @@ when:
      - "ansible/roles/claude-workstation/**"
      - "ansible/plays/install-claude-workstation.yml"
      - ".woodpecker/deploy-claude-workstation.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-claude-workstation"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'

 timeout: 30

@@ -8,8 +8,8 @@ when:
    path:
      - "opentofu/**"
      - ".woodpecker/deploy-infrastructure.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-infrastructure"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-infrastructure"'

 steps:
  - name: validate-secrets
@@ -84,7 +84,7 @@ steps:
    image: gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29
    directory: opentofu
    when:
-      - event: manual
+      - event: deployment
        evaluate: 'TOFU_APPLY == "true"'
    environment:
      TF_IN_AUTOMATION: "true"
@@ -7,8 +7,8 @@ when:
      - "ansible/plays/install-netbird-client.yml"
      - "ansible/inventory/group_vars/claude_workstation/**"
      - ".woodpecker/deploy-netbird-client.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-client"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'

 timeout: 30

@@ -9,8 +9,8 @@ when:
      - "ansible/roles/netbird-proxy-watchdog/**"
      - "ansible/plays/install-netbird-proxy.yml"
      - ".woodpecker/deploy-netbird-proxy.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-proxy"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

 timeout: 30

@@ -9,8 +9,8 @@ when:
      - "ansible/roles/netbird-proxy-watchdog/**"
      - "ansible/plays/install-netbird-server.yml"
      - ".woodpecker/deploy-netbird-server.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-server"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'

 timeout: 30

@@ -1,6 +1,6 @@
 when:
-  - event: manual
-    evaluate: 'PIPELINE == "e2e-test"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e-test"'

 steps:
  - name: e2e
@@ -7,7 +7,8 @@
 #                was harmless at daily/weekly cadence — but alert-monitor.yml
 #                fires every 15 minutes, so piggybacking must be opt-in now.
 #                A new cron that SHOULD also run e2e gets its name added here.
-#   - manual   : `woodpecker pipeline create --branch master --var PIPELINE=e2e k.renner/homelab`
+#   - deploy   : `woodpecker-cli pipeline deploy k.renner/homelab <n> e2e`, where
+#                <n> is an existing pipeline to promote.

 when:
  - event: cron
@@ -16,8 +17,8 @@ when:
    cron: update-audit
  - event: cron
    cron: weekly-update
-  - event: manual
-    evaluate: 'PIPELINE == "e2e"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e"'

 steps:
  - name: model-selection-tests
@@ -25,7 +26,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -1,27 +0,0 @@
-when:
-  - event: manual
-    evaluate: 'PIPELINE == "install-garage-synology"'
-
-steps:
-  - name: run-playbook
-    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
-    environment:
-      ANSIBLE_FORCE_COLOR: "true"
-      ANSIBLE_HOST_KEY_CHECKING: "false"
-      SSH_PRIVATE_KEY:
-        from_secret: ssh_private_key
-      VAULT_PASS:
-        from_secret: vault_pass
-      PROXMOX_TOKEN_ID:
-        from_secret: proxmox_token_id
-      PROXMOX_TOKEN_SECRET:
-        from_secret: proxmox_token_secret
-    commands:
-      - mkdir -p ~/.ssh
-      - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
-      - chmod 600 ~/.ssh/id_ed25519
-      - 'printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config'
-      - cd ansible
-      - echo "$$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
-      - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
-      - ansible-playbook -i inventory/ plays/install-garage-synology.yml -v
@@ -34,13 +34,13 @@
 #   - cron   : a cron NAMED exactly "issue-fix":
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name issue-fix --branch master --schedule '41 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=issue-fix k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> issue-fix
+#              where <n> is an existing pipeline to promote.
 when:
  - event: cron
    cron: issue-fix
-  - event: manual
-    evaluate: 'PIPELINE == "issue-fix"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "issue-fix"'

 steps:
  - name: fetch-diag
@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -53,9 +53,9 @@
 #
 # Triggering: pull_request only — automatic on PR open + push-to-open-PR. To
 # re-review a PR, push a commit (the new head sha clears the dedup marker below
-# and earns a fresh review). There is deliberately no manual PIPELINE trigger:
-# Woodpecker exposes a --var only to YAML variable substitution and `when`
-# filters, NOT to a step's shell environment, so a --var PR=<n> re-review would
+# and earns a fresh review). There is deliberately no on-demand trigger for a
+# specific PR: Woodpecker exposes injected variables to YAML substitution and
+# `when` filters, NOT to a step's shell environment, so a PR=<n> re-review would
 # silently no-op (the step would find no PR number and exit green).
 # (Note: keep literal dollar-brace out of THIS file entirely, even in comments —
 # Woodpecker runs its variable substitution over the whole YAML text, comments
@@ -65,7 +65,7 @@ when:

 steps:
  - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,8 +1,20 @@
+# Trivy scan of the IaC tree and every image referenced from ansible/, with an
+# agent pass that opens a PR for CRITICAL/HIGH findings.
+#
+# Triggers:
+#   - cron       : the named `daily-security-scan` cron, matched by NAME.
+#                  Crons cannot carry variables — `woodpecker-cli repo cron add`
+#                  has no --var — so a variable gate can never match a cron.
+#   - deployment : woodpecker-cli pipeline deploy k.renner/homelab <n> security-scan
+#                  The target rides in pipeline metadata as
+#                  CI_PIPELINE_DEPLOY_TARGET, which the workflow-root filter
+#                  does see, unlike manual-run variables on Woodpecker 3.18
+#                  (#582).
 when:
  - event: cron
-    evaluate: 'PIPELINE == "security-scan"'
-  - event: manual
-    evaluate: 'PIPELINE == "security-scan"'
+    cron: daily-security-scan
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "security-scan"'

 steps:
  - name: iac-scan
@@ -28,12 +40,28 @@ steps:
        cat /tmp/images.txt
        echo "---"
        i=0
+        failed=0
+        failed_names=""
        while IFS= read -r img; do
          i=$$((i+1))
          out="trivy-image-$$i.json"
          echo "=== [$$i] $$img -> $$out ==="
-          trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img" || true
+          # Keep going on a bad image so one unpullable reference cannot cost
+          # the scan of every image after it, but record that it happened.
+          if ! trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img"; then
+            failed=$$((failed+1))
+            failed_names="$$failed_names $$img"
+          fi
        done < /tmp/images.txt
+        echo "--- scanned $$((i - failed)) of $$i images ---"
+        if [ "$$failed" -gt 0 ]; then
+          echo "NOT SCANNED:$$failed_names"
+          exit 1
+        fi
+    # Non-zero above turns this step red without failing the workflow, so the
+    # fix agent still runs on what was scanned. Coverage is reported, not
+    # enforced: an incomplete scan is visible in the step status instead of
+    # being indistinguishable from a clean one.
    failure: ignore

  - name: security-fix-agent
@@ -42,7 +70,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -10,14 +10,14 @@
 # report; the pipeline posts it to a single persistent Gitea issue.
 #
 # Triggering:
-#   - cron   : a cron NAMED exactly "update-audit". Unlike the other pipelines
-#              here (which gate on an injected `PIPELINE` var set at cron-create
-#              time in the UI), this uses Woodpecker's documented `cron:` name
-#              filter so the schedule can be created entirely from the CLI:
+#   - cron   : a cron NAMED exactly "update-audit". Crons carry no variables —
+#              `repo cron add` takes name, branch, schedule and enabled — so the
+#              name filter is the only thing that can select a workflow for a
+#              scheduled run:
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name update-audit --branch master --schedule '@weekly'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=update-audit k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> update-audit
+#              where <n> is an existing pipeline to promote.
 #
 # NOTE: e2e.yml runs on an explicit list of cron names that includes this one,
 # so a weekly e2e run piggybacks on the audit (it was a bare `event: cron`
@@ -25,12 +25,12 @@
 when:
  - event: cron
    cron: update-audit
-  - event: manual
-    evaluate: 'PIPELINE == "update-audit"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "update-audit"'

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,37 +0,0 @@
---
-# PBS Client configuration for nas-1
-pbs_backup_paths:
-  - /volume1/Documents
-  - /volume1/homes
-  - /volume1/Homeassistant Backup
-  - /volume1/Games
-  - /volume1/Media
-  - /volume1/Music
-  - /volume1/video
-  - /volume2/Backup
-  - /volume1/Movies
-
-  # Add more paths as needed:
-  # - /volume1/photos
-  # - /volume1/music
-
-# PBS Server connection
-pbs_server_host: "pbs2.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "pool"
-pbs_backup_id: "nas-1"
-
-# PBS Authentication - credentials in vault.yml
-pbs_username: "root@pam!nas-1"
-
-# Garage S3 (K8s backup target)
-garage_image: "dxflrs/garage:v2.3.0"
-garage_zone: "dsm"
-garage_capacity: "2T"
-
-# SSH connection - use password auth (key not supported by DSM)
-ansible_ssh_pass: "{{ nas_ssh_password }}"
-ansible_become_pass: "{{ nas_ssh_password }}"
-
-# Python 3.9+ from Synology package
-ansible_python_interpreter: /usr/local/bin/python3.9
@@ -1,27 +0,0 @@
-$ANSIBLE_VAULT;1.1;AES256
-34663462636437633437613863366266306334666635643839313737343739643864613136613938
-3138393966306561656366396465383539303137333036300a376237376636356230343834386362
-36636535396330333732383233323263613132333166343739396137636538653435336662353138
-6565643866623235610a316266383933303830323936303735623338623165386339383733373435
-62366461363833616438373237383132366438326364363462333265366533323861376630643365
-39343036343838333639363432643738666630363063323636646661313539643465366532363438
-33303837623234613434633563666562346536346435353461373536323737376233386163326537
-61383963343737656438356136613537623538333163343466343739346135333439616432663763
-36663962633037376561326434303035333938646436663832303432346365353439666435613337
-34363636326530333233313231343566306432376532396666643761633136316333376337666135
-65363338623035663438323338663863366362653131336434323865326530326534623631336263
-36633063383031386161363131333636393364623436613331353733313235383062656131313164
-63393665633361313961363637663237616364643236383835393736643937393564656238376362
-31393465383361326264376133393233616264663636383337363962356532613438663637616536
-33653137656335323763313461323633616635346232303062323062333766383931313965346361
-66656461313239633366636463633465636463663937393932636337643861383937663662306466
-32353166643832303234643264633062616330393963333034623432613730366339306637613239
-34626161663666316339323762373639353335653663353635366664643063393432623039306237
-38363161343930363763393364663364353735343236313665313461626165636135303839363164
-37336638396564326166376233353730346563656266323535666265323062646232323938353732
-32636235333839623166633761376334316434613235333730353439343262393537363461333238
-32653435623362666662336532643832356163326332636436346132643237643030343761663332
-36663530356539303636663438646430363335353135323932613863666264363533326137353164
-31663032346236616263343637396439663334396166656364623637666535386530383535663435
-66303037303265616634373763616437333835333136336563323339666563663931396133633131
-3963653266633535303630653335666166313336653831623362
@@ -8,6 +8,8 @@
 #   pve_cluster (jupiter/neptune/saturn)  run Talos on bare metal; no SSH at all
 #   authentik_db / authentik_app          authentik runs in-cluster (CNPG)
 #   grafana                               runs in-cluster
+#   synology_nas (nas-1)                  powered off; Garage and the PBS client
+#                                         it hosted are gone with it
 # Their hosts no longer resolve, except the three that resolve to Talos nodes.

 [netbird_server]
@@ -17,10 +19,6 @@ netbird.renner.uno ansible_user=root
 proxy-1.renner.uno ansible_user=root
 proxy-2.renner.uno ansible_user=root

-
-[synology_nas]
-nas-1.renner.uno ansible_user=ladmin ansible_python_interpreter=/usr/bin/python3
-
 # Every bare-metal edge thin client. Gets the host-generic roles only:
 # flash-wear handling, the log shipper, the MQTT bridge, the VLAN
 # sub-interface. Nothing here assumes any peripheral is attached.
@@ -1,313 +0,0 @@
---
-# Synology-specific playbook using raw module (Python 3.8 workaround)
-# Deploys Garage (https://garagehq.deuxfleurs.fr) as a single-node S3 target
-# for K8s backups (Velero + CNPG Barman Cloud Plugin).
-#
-# Required secrets in host_vars/nas-1.renner.uno/vault.yml:
-#   garage_rpc_secret:     "<openssl rand -hex 32>"
-#   garage_admin_token:    "<openssl rand -base64 32>"
-#   garage_metrics_token:  "<openssl rand -base64 32>"
-#
-# Post-install bootstrap (run once, record the returned secret keys):
-#   docker exec garaged /garage status
-#   docker exec garaged /garage bucket create k8s-velero
-#   docker exec garaged /garage bucket create k8s-cnpg
-#   docker exec garaged /garage bucket create k8s-secrets
-#   docker exec garaged /garage bucket create k8s-migration
-#   docker exec garaged /garage key create velero-key
-#   docker exec garaged /garage key create cnpg-key
-#   docker exec garaged /garage key create secrets-key
-#   docker exec garaged /garage key create migration-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-velero    --key velero-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-cnpg      --key cnpg-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-secrets   --key secrets-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-migration --key migration-key
- name: Install Garage S3 on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    garage_data_dir: "/volume1/docker/garage"
-
-  tasks:
-    - name: Create Garage directory tree
-      ansible.builtin.raw: |
-        mkdir -p {{ garage_data_dir }}/config \
-                 {{ garage_data_dir }}/meta \
-                 {{ garage_data_dir }}/data
-      changed_when: true
-
-    # Secret writes use the same compare-then-write pattern as garage.toml so
-    # rotation is idempotent AND a real change can drive force-recreate below.
-    # Garage reads these files once at startup; without container restart, a
-    # rotated token leaves the running process holding the previous value.
-    - name: Write rpc_secret
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/rpc_secret.new << 'SECRET_EOF'
-        {{ garage_rpc_secret }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/rpc_secret.new
-        if [ ! -f {{ garage_data_dir }}/config/rpc_secret ] \
-        || ! cmp -s {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret; then
-          mv {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret
-          echo RPC_SECRET_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/rpc_secret.new
-        fi
-      register: rpc_secret_write
-      changed_when: "'RPC_SECRET_CHANGED' in rpc_secret_write.stdout"
-      no_log: true
-
-    - name: Write admin_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/admin_token.new << 'SECRET_EOF'
-        {{ garage_admin_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/admin_token.new
-        if [ ! -f {{ garage_data_dir }}/config/admin_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token; then
-          mv {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token
-          echo ADMIN_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/admin_token.new
-        fi
-      register: admin_token_write
-      changed_when: "'ADMIN_TOKEN_CHANGED' in admin_token_write.stdout"
-      no_log: true
-
-    - name: Write metrics_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/metrics_token.new << 'SECRET_EOF'
-        {{ garage_metrics_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/metrics_token.new
-        if [ ! -f {{ garage_data_dir }}/config/metrics_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token; then
-          mv {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token
-          echo METRICS_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/metrics_token.new
-        fi
-      register: metrics_token_write
-      changed_when: "'METRICS_TOKEN_CHANGED' in metrics_token_write.stdout"
-      no_log: true
-
-    - name: Write garage.toml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/garage.toml.new << 'TOML_EOF'
-        metadata_dir = "/var/lib/garage/meta"
-        data_dir     = "/var/lib/garage/data"
-
-        # SQLite chosen over LMDB: single-node deployment has no peers to
-        # rebuild metadata from on unclean-shutdown corruption. SQLite is
-        # slower but portable and has no LMDB corruption window.
-        db_engine = "sqlite"
-
-        # 6-hour auto-snapshot of the metadata DB as a backstop against
-        # corruption; last two snapshots retained.
-        metadata_auto_snapshot_interval = "6h"
-
-        # Single-zone, single-node: only coherent replication factor.
-        replication_factor = 1
-
-        rpc_bind_addr    = "[::]:3901"
-        rpc_secret_file  = "/etc/garage/rpc_secret"
-
-        compression_level = 2
-
-        # 64 MB blocks (default 1 MB; previously 20 MB). On the 5 × 7 TB
-        # Btrfs RAID 5 volume the per-block fixed cost (SQLite lookup,
-        # dir traversal, hash verify, zstd decompress, nginx buffering)
-        # is the dominant overhead per object; larger blocks amortize
-        # it. block_size is a maximum, so 16 MiB CNPG WAL segments
-        # still upload as one block each. Only applies to NEW uploads;
-        # existing objects keep their original chunking until re-uploaded.
-        block_size = "64M"
-
-        [s3_api]
-        api_bind_addr = "[::]:3900"
-        s3_region     = "garage"
-        root_domain   = ".s3.{{ inventory_hostname }}"
-
-        [s3_web]
-        bind_addr   = "[::]:3902"
-        root_domain = ".web.{{ inventory_hostname }}"
-
-        [admin]
-        api_bind_addr      = "[::]:3903"
-        admin_token_file   = "/etc/garage/admin_token"
-        metrics_token_file = "/etc/garage/metrics_token"
-        TOML_EOF
-        if [ ! -f {{ garage_data_dir }}/config/garage.toml ] \
-        || ! cmp -s {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml; then
-          mv {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml
-          echo GARAGE_TOML_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/garage.toml.new
-        fi
-      register: garage_toml_write
-      changed_when: "'GARAGE_TOML_CHANGED' in garage_toml_write.stdout"
-
-    - name: Write docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          garage:
-            image: {{ garage_image }}
-            container_name: garaged
-            restart: unless-stopped
-            # HTTP services (3900/3902/3903) bind to loopback; DSM
-            # reverse proxy fronts and TLS-terminates them. RPC 3901
-            # is exposed directly — the Noise handshake is raw TCP,
-            # so DSM RP cannot front it. rpc_secret gates auth.
-            ports:
-              - "3901:3901"
-              - "127.0.0.1:3900:3900"
-              - "127.0.0.1:3902:3902"
-              - "127.0.0.1:3903:3903"
-            volumes:
-              - {{ garage_data_dir }}/config/garage.toml:/etc/garage.toml:ro
-              - {{ garage_data_dir }}/config/rpc_secret:/etc/garage/rpc_secret:ro
-              - {{ garage_data_dir }}/config/admin_token:/etc/garage/admin_token:ro
-              - {{ garage_data_dir }}/config/metrics_token:/etc/garage/metrics_token:ro
-              - {{ garage_data_dir }}/meta:/var/lib/garage/meta
-              - {{ garage_data_dir }}/data:/var/lib/garage/data
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Pull image and start Garage container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    # Garage does not reload config on SIGHUP and compose does not notice
-    # when a bind-mounted host file changes. Force-recreate the container
-    # when garage.toml OR any of the three secret files changed so the new
-    # config / rotated tokens are picked up.
-    - name: Force-recreate garaged when garage.toml or secrets changed
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose up -d --force-recreate garage
-      become: true
-      when: garage_toml_write.changed
-            or rpc_secret_write.changed
-            or admin_token_write.changed
-            or metrics_token_write.changed
-      changed_when: true
-
-    - name: Wait for Garage to accept CLI calls
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        for i in $(seq 1 30); do
-          if docker exec garaged /garage status >/dev/null 2>&1; then
-            exit 0
-          fi
-          sleep 2
-        done
-        exit 1
-      become: true
-      changed_when: false
-
-    - name: Check whether cluster layout is already applied
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status
-      changed_when: false
-
-    - name: Apply single-node layout (idempotent — only on first run)
-      when: "'NO ROLE ASSIGNED' in garage_status.stdout"
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        NODE_ID=$(docker exec garaged /garage node id | head -n 1 | cut -d@ -f1)
-        docker exec garaged /garage layout assign "$NODE_ID" \
-          -z {{ garage_zone }} \
-          -c {{ garage_capacity }} \
-          -t {{ inventory_hostname_short }}
-        docker exec garaged /garage layout apply --version 1
-      become: true
-      changed_when: true
-
-    - name: Report Garage status
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status_final
-      changed_when: false
-
-    - name: Show Garage status
-      ansible.builtin.debug:
-        var: garage_status_final.stdout_lines
-
-    # Bootstrap buckets + keys. Idempotent: bucket_info / key_info return
-    # non-zero when missing, so we only create on first run. Access-key
-    # secrets are captured at create time and persisted to chmod-600 files
-    # under {{ garage_data_dir }}/config/keys/<key>.creds on nas-1. These
-    # files are the only post-install record of the secrets — Garage does
-    # not expose them again.
-    - name: Bootstrap Garage buckets, keys, and grants
-      ansible.builtin.shell: |
-        set -e
-        export PATH=$PATH:/usr/local/bin
-        KEYS_DIR={{ garage_data_dir }}/config/keys
-        mkdir -p "$KEYS_DIR"
-        chmod 700 "$KEYS_DIR"
-
-        bootstrap_pair() {
-          local BUCKET="$1"
-          local KEY="$2"
-          local CREDS_FILE="$KEYS_DIR/$KEY.creds"
-
-          if ! docker exec garaged /garage bucket info "$BUCKET" >/dev/null 2>&1; then
-            docker exec garaged /garage bucket create "$BUCKET"
-          fi
-
-          if ! docker exec garaged /garage key info "$KEY" >/dev/null 2>&1; then
-            CREATE_OUT=$(docker exec garaged /garage key create "$KEY")
-            KEY_ID=$(echo "$CREATE_OUT"   | awk '/^Key ID:/     {print $3}')
-            SECRET=$(echo "$CREATE_OUT"   | awk '/^Secret key:/ {print $3}')
-            umask 177
-            printf 'KEY_ID=%s\nSECRET_KEY=%s\n' "$KEY_ID" "$SECRET" > "$CREDS_FILE"
-          fi
-
-          docker exec garaged /garage bucket allow \
-            --read --write --owner \
-            "$BUCKET" --key "$KEY"
-        }
-
-        bootstrap_pair k8s-velero    velero-key
-        bootstrap_pair k8s-cnpg      cnpg-key
-        bootstrap_pair k8s-secrets   secrets-key
-        bootstrap_pair k8s-migration migration-key
-        bootstrap_pair tfstate       tfstate-key
-      become: true
-      changed_when: true
-      no_log: true
-
-    - name: Report bucket + key state (no secrets)
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage bucket list
-        echo "---"
-        docker exec garaged /garage key list
-        echo "---"
-        ls -l {{ garage_data_dir }}/config/keys/ 2>&1
-      become: true
-      register: garage_bootstrap_state
-      changed_when: false
-
-    - name: Show bucket + key state
-      ansible.builtin.debug:
-        var: garage_bootstrap_state.stdout_lines
-
-    - name: Post-install note
-      ansible.builtin.debug:
-        msg:
-          - "Buckets: k8s-velero, k8s-cnpg, k8s-secrets, k8s-migration, tfstate"
-          - "Keys:    velero-key, cnpg-key, secrets-key, migration-key, tfstate-key"
-          - "Secrets are written to {{ garage_data_dir }}/config/keys/<key>.creds on nas-1 (chmod 600, root-owned)."
-          - "SSH to nas-1 and `sudo cat` the file to retrieve KEY_ID + SECRET_KEY for Velero / CNPG / migration / tfstate config."
@@ -1,104 +0,0 @@
---
-# Synology-specific playbook using raw module (Python 3.8 workaround)
- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    pbs_client_data_dir: "/volume1/docker/pbs-client"
-
-  tasks:
-    - name: Create PBS client directory
-      ansible.builtin.raw: |
-        mkdir -p {{ pbs_client_data_dir }}/state
-      changed_when: true
-
-    - name: Create docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          pbs-client:
-            image: proxmox/proxmox-backup-client:latest
-            container_name: pbs-client
-            restart: unless-stopped
-            hostname: {{ pbs_backup_id }}
-            env_file:
-              - .env
-            volumes:
-              - {{ pbs_client_data_dir }}/state:/root/.proxmox-backup
-        {% for path in pbs_backup_paths %}
-              - "{{ path }}:{{ path }}:ro"
-        {% endfor %}
-            mem_limit: 2g
-            command: ["sleep", "infinity"]
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Create .env file
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/.env << 'ENV_EOF'
-        PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-        PBS_PASSWORD={{ pbs_password }}
-        ENV_EOF
-        chmod 600 {{ pbs_client_data_dir }}/.env
-      changed_when: true
-
-    - name: Create backup script
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/backup.sh << 'SCRIPT_EOF'
-        #!/bin/bash
-        set -e
-        source {{ pbs_client_data_dir }}/.env
-
-        BACKUP_SOURCES=""
-        {% for path in pbs_backup_paths %}
-        BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-        {% endfor %}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-        docker exec pbs-client proxmox-backup-client backup \
-            $BACKUP_SOURCES \
-            --backup-id {{ pbs_backup_id }} \
-            --repository "${PBS_REPOSITORY}"
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed, pruning..."
-        docker exec pbs-client proxmox-backup-client prune \
-            --repository "${PBS_REPOSITORY}" \
-            --backup-id {{ pbs_backup_id }} \
-            --keep-last {{ pbs_keep_last | default(3) }} \
-            --keep-daily {{ pbs_keep_daily | default(7) }} \
-            --keep-weekly {{ pbs_keep_weekly | default(4) }} \
-            --keep-monthly {{ pbs_keep_monthly | default(6) }}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Done"
-        SCRIPT_EOF
-        chmod 755 {{ pbs_client_data_dir }}/backup.sh
-      changed_when: true
-
-    - name: Start PBS client container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ pbs_client_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    - name: Setup cron job for daily backup at 2 AM (root crontab)
-      ansible.builtin.shell: |
-        # Remove existing entry if present
-        crontab -l 2>/dev/null | grep -v "pbs-client/backup.sh" | crontab - 2>/dev/null || true
-        # Add new entry
-        (crontab -l 2>/dev/null; echo "0 2 * * * {{ pbs_client_data_dir }}/backup.sh >> /var/log/pbs-backup.log 2>&1") | crontab -
-      become: true
-      changed_when: true
-
-    - name: Verify container is running
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker ps --filter name=pbs-client --format "{{ '{{' }}.Status{{ '}}' }}"
-      become: true
-      register: container_status
-      changed_when: false
-
-    - name: Show container status
-      ansible.builtin.debug:
-        msg: "PBS client container status: {{ container_status.stdout }}"
@@ -1,11 +0,0 @@
---
- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  become: true
-  # Variables are set in:
-  # - Role defaults: ansible/roles/pbs-client/defaults/main.yml
-  # - Host overrides: ansible/inventory/host_vars/<hostname>/vars.yml
-  # - Secrets: ansible/inventory/host_vars/<hostname>/vault.yml
-
-  roles:
-    - pbs-client
@@ -1,32 +0,0 @@
---
-# PBS Client configuration
-pbs_client_version: "latest"
-pbs_client_data_dir: "/opt/pbs-client"
-
-# PBS Server connection (override in host_vars)
-pbs_server_host: "pbs.obg.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "backups"
-
-# Authentication (override in vault!)
-pbs_username: "backup@pbs"
-pbs_password: ""
-# pbs_fingerprint: ""  # Optional - omit for Let's Encrypt certs
-
-# Backup configuration
-pbs_backup_id: "nas-1"
-pbs_backup_paths:
-  - /volume1/docker
-  - /volume1/homes
-
-# Schedule (cron format)
-pbs_backup_schedule: "0 2 * * *"  # 2 AM daily
-
-# Retention policy (prune settings)
-pbs_keep_last: 3
-pbs_keep_daily: 7
-pbs_keep_weekly: 4
-pbs_keep_monthly: 6
-
-# Container resource limits
-pbs_memory_limit: "2g"
@@ -1,6 +0,0 @@
---
- name: Restart PBS client
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: restarted
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"
@@ -1,23 +0,0 @@
---
- name: Create PBS client directories
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: directory
-    mode: '0755'
-  loop:
-    - "{{ pbs_client_data_dir }}"
-    - "{{ pbs_client_data_dir }}/state"
-
- name: Copy PBS client docker-compose file
-  ansible.builtin.template:
-    src: docker-compose.yml.j2
-    dest: "{{ pbs_client_data_dir }}/docker-compose.yml"
-    mode: '0644'
-  notify: Restart PBS client
-
- name: Start PBS client via Docker Compose
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: present
-    pull: always
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"
@@ -1,6 +0,0 @@
-# PBS Client Environment Configuration
-PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-PBS_PASSWORD={{ pbs_password }}
-{% if pbs_fingerprint | default('') != '' %}
-PBS_FINGERPRINT={{ pbs_fingerprint }}
-{% endif %}
@@ -1,36 +0,0 @@
-#!/bin/bash
-# PBS Backup Script for {{ pbs_backup_id }}
-# Managed by Ansible - do not edit manually
-
-set -e
-
-# Load environment
-source {{ pbs_client_data_dir }}/.env
-
-# Build backup command with all paths
-BACKUP_SOURCES=""
-{% for path in pbs_backup_paths %}
-BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-{% endfor %}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-
-# Run backup via Docker
-docker exec pbs-client proxmox-backup-client backup \
-    $BACKUP_SOURCES \
-    --backup-id {{ pbs_backup_id }} \
-    --repository "${PBS_REPOSITORY}"
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed"
-
-# Prune old backups
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Pruning old backups"
-docker exec pbs-client proxmox-backup-client prune \
-    --repository "${PBS_REPOSITORY}" \
-    --backup-id {{ pbs_backup_id }} \
-    --keep-last {{ pbs_keep_last }} \
-    --keep-daily {{ pbs_keep_daily }} \
-    --keep-weekly {{ pbs_keep_weekly }} \
-    --keep-monthly {{ pbs_keep_monthly }}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Prune completed"
@@ -1,30 +0,0 @@
-services:
-  pbs-client:
-    image: aterfax/pbs-client:{{ pbs_client_version }}
-    container_name: pbs-client
-    restart: unless-stopped
-    hostname: {{ pbs_backup_id }}
-    environment:
-      - PBS_ENDPOINT={{ pbs_server_host }}:{{ pbs_server_port }}
-      - PBS_DATASTORE={{ pbs_datastore }}
-      - PBS_API_KEY_NAME={{ pbs_username }}
-      - PBS_API_KEY_SECRET={{ pbs_password }}
-      - PBS_PASSWORD={{ pbs_password }}
-      - PBS_ENCRYPTION_PASSWORD={{ pbs_encryption_password | default(pbs_password) }}
-      - PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-      - PBS_BACKUP_ID={{ pbs_backup_id }}
-      - CRON_SCHEDULE={{ pbs_backup_schedule }}
-      - PBS_BACKUP_CMD_APPEND_EXTRA_OPTS=--crypt-mode=none
-      - PRUNE_KEEP_LAST={{ pbs_keep_last | default(3) }}
-      - PRUNE_KEEP_DAILY={{ pbs_keep_daily | default(7) }}
-      - PRUNE_KEEP_WEEKLY={{ pbs_keep_weekly | default(4) }}
-      - PRUNE_KEEP_MONTHLY={{ pbs_keep_monthly | default(6) }}
-      - CRON_BACKUP_ONLY=1
-    volumes:
-      - {{ pbs_client_data_dir }}/state:/root/.config/proxmox-backup
-{% for path in pbs_backup_paths %}
-      - "{{ path }}:/backups/{{ path | basename | replace(' ', '_') }}:ro"
-{% endfor %}
-    tmpfs:
-      - /tmp
-    mem_limit: {{ pbs_memory_limit }}
@@ -25,20 +25,13 @@ data:
        attrs:
          local: true

-      # Retired OIDC layer. Paperless's URL-level access is gated by the
-      # paperless-gate ProxyProvider (below); user identity is forwarded
-      # from the gate to paperless via the X-authentik-* request headers
-      # and consumed by paperless via PAPERLESS_ENABLE_HTTP_REMOTE_USER
-      # (see kubernetes/apps/paperless/paperless.yaml). One Authentik
-      # Application, one Provider — the dual-OAuth2-flow setup with a
-      # separate paperless OIDC client + django-allauth callback was
-      # structurally fine but added a second consent screen on first
-      # login for no real defense-in-depth gain.
+      # Retired OIDC layer. Access is gated by the paperless-gate
+      # ProxyProvider below, which forwards identity as X-authentik-* headers
+      # consumed via PAPERLESS_ENABLE_HTTP_REMOTE_USER.
      #
-      # Order matters: delete the Application BEFORE the OAuth2Provider
-      # because Application.provider is on_delete=PROTECT, so the
-      # OAuth2Provider can't be removed while the Application still
-      # references it.
+      # ORDER MATTERS: delete the Application BEFORE the OAuth2Provider.
+      # Application.provider is on_delete=PROTECT, so the provider cannot be
+      # removed while an Application still references it.
      - model: authentik_core.application
        identifiers:
          slug: paperless
@@ -49,22 +42,15 @@ data:
          name: Paperless
        state: absent

-      # Also clean up the very-early orphan `paperless` ProxyProvider
-      # left behind from this session's first attempts. It's not bound
-      # to anything any more, just leftover state in the DB.
+      # Orphan ProxyProvider, unbound — leftover DB state.
      - model: authentik_providers_proxy.proxyprovider
        identifiers:
          name: paperless
        state: absent

-      # Dedicated forward-auth gate for paperless. This is a SEPARATE
-      # outpost (its own Deployment + Service `ak-outpost-paperless-
-      # gate`) running in front of paperless so every request is
-      # authenticated and authorized by Authentik before it reaches
-      # paperless's own attack surface. The gate sits on top of, and
-      # separate from, the OIDC OAuth2Provider above (which is what
-      # paperless's django-allauth uses for SSO once a request makes
-      # it through the gate).
+      # Dedicated forward-auth gate for paperless: its own outpost
+      # Deployment + Service, so every request is authenticated before it
+      # reaches paperless.
      - id: paperless-gate-provider
        model: authentik_providers_proxy.proxyprovider
        identifiers:
@@ -73,16 +59,13 @@ data:
        attrs:
          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          # Proxy (not forward_domain) so the outpost itself terminates
-          # the request and forwards to paperless — moving the trust
-          # boundary off ingress-nginx and onto paperless-gate. ingress-
-          # nginx now only routes paperless.renner.uno to the outpost
-          # service; it never talks to paperless directly. The outpost
-          # is what populates X-Authentik-Username on the upstream
-          # request, and the NetworkPolicy in the paperless namespace
-          # only allows traffic to paperless from the gate outpost pod
-          # itself, so ingress-nginx (or any other in-cluster workload)
-          # cannot bypass the gate to inject headers.
+          # Proxy, NOT forward_domain: the outpost terminates the request
+          # itself, so the trust boundary sits here rather than at the ingress
+          # controller, which only routes to the outpost Service and never
+          # reaches paperless directly. The outpost populates
+          # X-Authentik-Username, and the paperless NetworkPolicy admits
+          # traffic only from this outpost — so nothing else in the cluster
+          # can bypass the gate and inject those headers.
          mode: proxy
          internal_host: http://paperless.paperless.svc.cluster.local
          internal_host_ssl_validation: false
@@ -91,19 +74,12 @@ data:
          access_token_validity: hours=24
          refresh_token_validity: days=30
          intercept_header_auth: true
-          # Pass paperless's entire static-asset tree through anonymously.
-          # Two reasons:
-          # 1. <link rel="manifest"> is fetched with `credentials: 'omit'`
-          #    (W3C spec, no crossorigin attr) so the cookie is never sent
-          #    — every page load and every SPA navigation turned into a
-          #    302 → auth.renner.uno round-trip with an orphan session JWT.
-          #    Same root cause as the retired music-assistant-gate (see
-          #    comment further below); paperless can't retire the gate so
-          #    it carves out the offending paths instead.
-          # 2. Every CSS/JS/icon fetch otherwise validates the cookie at
-          #    the outpost — pure overhead on assets that carry no
-          #    authentication value. Django's STATIC tree is public by
-          #    design; MEDIA (under /api/documents/*) stays gated.
+          # The static-asset tree passes anonymously, and must. A
+          # <link rel="manifest"> is fetched with `credentials: 'omit'` per
+          # the W3C spec, so the session cookie is never sent and every page
+          # load turns into a redirect loop through the login origin. Django's
+          # STATIC tree is public by design; MEDIA under /api/documents/*
+          # stays gated.
          skip_path_regex: ^/static/.*

      - model: authentik_core.application
@@ -130,18 +106,14 @@ data:
          negate: false
          failure_result: false

-      # Retired: the music-assistant-gate ProxyProvider, Application,
-      # and dedicated Outpost. The gate fundamentally conflicted with
-      # MA's PWA model — Chrome fetches manifest.webmanifest / sw.js /
-      # icons with `credentials: 'omit'` (per W3C spec, no crossorigin
-      # attribute on MA's <link rel="manifest">), and the cached
-      # service worker bypasses fresh navigations on PWA cold-start so
-      # the Authentik cookie never lands in the WebAPK's cookie jar.
-      # MA has its own auth (delegated to Home Assistant); the gate's
-      # marginal value didn't justify the recurring breakage.
-      # Order matters: Application before Provider before Outpost.
-      # Application.provider is on_delete=PROTECT, so the Provider
-      # can't be removed while the Application still references it.
+      # Retired: the music-assistant gate. A forward-auth gate cannot work in
+      # front of a PWA — the manifest and service worker are fetched with
+      # `credentials: 'omit'`, and a cached service worker bypasses fresh
+      # navigations, so the session cookie never reaches the installed app.
+      # Do not reintroduce one there; MA delegates auth to Home Assistant.
+      #
+      # ORDER MATTERS: Application before Provider before Outpost, because
+      # Application.provider is on_delete=PROTECT.
      - model: authentik_core.application
        identifiers:
          slug: music-assistant-gate
@@ -157,18 +129,13 @@ data:
          name: music-assistant-gate
        state: absent

-      # Dedicated outpost: its ONLY job is to serve the in-cluster
-      # /outpost.goauthentik.io/auth/nginx endpoint that nginx
-      # auth_request hits for every request to paperless. The
-      # browser-facing legs of the OAuth flow (/start, /callback, the
-      # `/if/...` login UI, /application/o/authorize, /api, etc.) all
-      # need to happen on auth.renner.uno because that's the origin
-      # WebAuthn/FIDO/TOTP credentials are bound to. So we attach the
-      # ProxyProvider to BOTH outposts (see embedded outpost below) and
-      # set authentik_host_browser to auth.renner.uno here too — the
-      # nginx auth_request happens server-to-server, but if the
-      # dedicated outpost ever does emit a redirect URL, it points at
-      # the right host.
+      # Dedicated outpost, serving the in-cluster auth endpoint only.
+      #
+      # The browser-facing legs of the OAuth flow must happen on
+      # auth.renner.uno, because that is the origin WebAuthn credentials are
+      # bound to. Hence the ProxyProvider is attached to BOTH this outpost and
+      # the embedded one, and authentik_host_browser points at auth.renner.uno
+      # here too — so any redirect this outpost emits names the right host.
      - model: authentik_outposts.outpost
        identifiers:
          name: paperless-gate
@@ -196,12 +163,7 @@ data:
            kubernetes_ingress_secret_name: "authentik-outpost-tls"
            kubernetes_json_patches: null

-      # Tear down the leftover paperless-proxy Application from an
-      # earlier experiment. The orphan `paperless` ProxyProvider that
-      # used to back paperless.app is left in place — it's PROTECTed
-      # by Application.provider as long as anything references it, and
-      # leftover orphans don't break anything. Delete via the Authentik
-      # UI once everything's confirmed working.
+      # Leftover Application from an earlier experiment.
      - model: authentik_policies.policybinding
        identifiers:
          target: !Find [authentik_core.application, [slug, paperless-proxy]]
@@ -217,15 +179,11 @@ data:
      # ------------------------------------------------------------------
      # vscode config-editor gate
      #
-      # code-server (the HA /config editor sidecar) runs with --auth=none,
-      # as root, mounting the whole HA /config incl. plaintext secrets.yaml
-      # and .storage. This gate moves it behind Authentik with a passkey
-      # step-up, mirroring the paperless-gate proxy-outpost pattern above:
-      # the dedicated ak-outpost-vscode-gate terminates the request,
-      # enforces the session, and forwards to the code-server Service.
-      # Traefik only ever routes vscode.home.renner.uno to the outpost; the
-      # home-assistant CNP is tightened so ONLY the gate outpost may reach
-      # :8080 (see home-assistant-ns.yaml + vscode-gate-egress.yaml).
+      # code-server runs with --auth=none, as root, mounting the whole HA
+      # /config including plaintext secrets.yaml and .storage. This gate is
+      # the only thing in front of it: the outpost terminates the request and
+      # enforces a passkey step-up, Traefik routes only to the outpost, and
+      # the home-assistant CNP admits :8080 from the gate alone.
      # ------------------------------------------------------------------
      - id: vscode-config-editors-group
        model: authentik_core.group
@@ -265,17 +223,14 @@ data:
          access_token_validity: hours=24
          refresh_token_validity: days=30
          intercept_header_auth: true
-          # code-server is a service-worker app: <link rel="manifest"> and
-          # the service worker are fetched with credentials:'omit', which
-          # turned into a /authorize redirect loop and forced the RETIREMENT
-          # of the music-assistant gate (see comment above). Carve its
-          # static/manifest/SW tree out so those cookieless fetches pass
-          # through instead of 302-looping; the editor's actual content
-          # (the /config tree) is served over authenticated websocket/API
-          # paths, which stay gated.
-          # NOTE: the exact skip set is code-server-version-specific and
-          # MUST be confirmed post-deploy against the real redirect-loop
-          # behaviour (see the PR validation checklist).
+          # code-server is a service-worker app, so its manifest and SW are
+          # fetched with credentials:'omit' and 302-loop unless carved out.
+          # The editor's actual content rides authenticated websocket/API
+          # paths and stays gated.
+          #
+          # This skip set is version-specific: re-check it after a
+          # code-server bump, because a renamed asset path reintroduces the
+          # loop silently.
          skip_path_regex: ^/(_static/|manifest\.json|.*\.webmanifest|.*serviceWorker\.js).*

      - model: authentik_core.application
@@ -331,16 +286,11 @@ data:
      # ------------------------------------------------------------------
      # shelly-manager gate
      #
-      # shelly-manager has no authentication of its own: anything that can
-      # reach :8080 can rewrite Shelly device config, push scripts, and read
-      # the stored per-device AdminPassword values. Until now the only
-      # control was reachability of shelly-manager.home.renner.uno.
-      #
-      # Same shape as the vscode-gate above: a dedicated outpost terminates
-      # the request, enforces a passkey step-up, and forwards to the
-      # shelly-manager Service. Traefik only ever routes the hostname to the
-      # outpost; the home-assistant CNP allows ONLY this outpost to reach
-      # :8080 (see home-assistant-ns.yaml + shelly-manager-gate-egress.yaml).
+      # shelly-manager has no authentication of its own: anything that reaches
+      # :8080 can rewrite device config, push scripts and read the stored
+      # per-device AdminPassword values. Same shape as the vscode gate — the
+      # outpost is the only control, Traefik routes only to it, and the CNP
+      # admits :8080 from this outpost alone.
      # ------------------------------------------------------------------
      - id: shelly-managers-group
        model: authentik_core.group
@@ -1,50 +1,10 @@
 ---
-# The goauthentik.io operator (inside authentik-worker) auto-creates a
-# ServiceMonitor per outpost, but it does not stamp the
-# `release: kube-prometheus-stack` label that this cluster's Prometheus
-# requires on serviceMonitorSelector. Those operator-owned SMs are
-# silently ignored. Declare standalone SMs here with the correct label
-# so the outposts actually get scraped.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-ldap
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: ldap-outpost
-      goauthentik.io/outpost-type: ldap
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
---
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-paperless-gate
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: paperless-gate
-      goauthentik.io/outpost-type: proxy
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
---
+# Only radius needs a hand-written ServiceMonitor. The goauthentik.io operator
+# creates one per outpost and Prometheus discovers those directly, because
+# serviceMonitorSelector here is empty (see monitoring/helmrelease-prometheus.yaml).
+# Standalone copies for ldap and paperless-gate used to live here as well; they
+# duplicated the operator's and were removed.
+#
 # Radius requires extra plumbing. The radius binary listens on :9300 just
 # like ldap/proxy (hardcoded in authentik/lib/default.yml), but authentik's
 # Python operator (`providers/radius/controllers/kubernetes.py`) omits the
@@ -53,6 +13,11 @@ spec:
 # it, radius does not. Result: the operator-managed `ak-outpost-local-radius`
 # Service exposes only UDP 1812. Add a sibling Service here that selects the
 # same pod labels and exposes 9300/TCP, then a ServiceMonitor for it.
+#
+# The operator's own `ak-outpost-local-radius` ServiceMonitor cannot cover
+# this: it selects on `app.kubernetes.io/managed-by: goauthentik.io` together
+# with `goauthentik.io/service-type: metrics`, and no Service carries both —
+# the operator's has the first, the one below has the second.
 apiVersion: v1
 kind: Service
 metadata:
@@ -86,6 +51,8 @@ spec:
    matchNames:
      - authentik
  selector:
+    # Pinned to the Service above. Without managed-by this would also match the
+    # operator's Service and scrape the same endpoint twice.
    matchLabels:
      app.kubernetes.io/managed-by: homelab
      goauthentik.io/outpost-name: local-radius
@@ -1,31 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-    # Backend serves HTTPS with a self-signed cert; re-encrypt + skip
-    # verify via the shared insecure-backend ServersTransport (see
-    # apps/insecure-backend-serverstransport.yaml).
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend@kubernetescrd
-spec:
-  ingressClassName: traefik
-  rules:
-    - host: file.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: file
-                port:
-                  number: 7001
-  tls:
-    - secretName: file-tls
-      hosts:
-        - file.renner.uno
@@ -1,5 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - service.yaml
-  - ingress.yaml
@@ -1,18 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-spec:
-  # Caddy reverse-proxied https://nas.renner.uno:7001. nas.renner.uno
-  # is a CNAME -> nas-1.renner.uno owned by UniFi DNS, k8s-gateway
-  # forwards srv lookups so this resolves cluster-side.
-  type: ExternalName
-  externalName: nas.renner.uno
-  ports:
-    - name: https
-      port: 7001
-      targetPort: 7001
-      protocol: TCP
@@ -1,80 +1,22 @@
 ---
-# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
-# without putting a macvlan on the HA pod itself.
+# mDNS relay: holds the multicast membership on the IoT VLAN through its own
+# macvlan and forwards each datagram to Home Assistant as ordinary UNICAST UDP.
 #
-# THE PROBLEM
+# It must stay unicast. An avahi-style reflector cannot work here — Cilium runs
+# in tunnel mode without enable-multicast, so re-emitted multicast is dropped by
+# the CNI. Unicast is carried, and python-zeroconf accepts it: its datagram
+# handler checks size and duplicates but never the destination or source
+# address. RFC 6762's QU bit makes this protocol-legal rather than a trick.
 #
-# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
-# ff02::fb — and never hears anything on either.
-# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
-# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
-# it does not replicate multicast between pods either. Both halves of the path
-# are dead, which is why an avahi-style reflector — receive on one interface,
-# re-emit as multicast on the other — cannot work here: the re-emitted packets
-# would be dropped by the CNI.
+# This restores passive reception only — announcements, including the
+# re-announce a device sends when its address changes. HA-initiated browsing
+# still dies on eth0; that would need a querier on the VLAN side, which this
+# deliberately is not.
 #
-# Consequence for HA: the ESPHome integration dials devices at the address
-# cached in its config entry. When a lease rotates, every attempt fails with
-# `[Errno 113] Connect call failed` and the device stays unreachable, because
-# the mDNS announcement that would carry the new address never arrives. New
-# devices are never discovered at all.
-#
-# THE MECHANISM
-#
-# This relay holds the multicast membership on the IoT VLAN through its own
-# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
-# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
-# the CNI will carry.
-#
-# It works because python-zeroconf does not filter on how a packet arrived:
-# its datagram handler validates size, deduplicates, and parses, but performs
-# no check on destination address, source address, or source port. A unicast
-# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
-# unicast responses are part of RFC 6762 — the QU bit — so this is within the
-# protocol, not a trick.)
-#
-# WHAT THIS DOES AND DOES NOT RESTORE
-#
-# Restored: passive reception. RFC 6762 requires a responder to re-announce
-# when its address changes, so a device that moves to a new lease announces
-# the move, HA sees it, and the ESPHome integration updates the config entry.
-# That is the failure this exists to fix. New devices announcing themselves
-# are likewise seen.
-#
-# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
-# multicast and die there, so HA cannot actively solicit a response. Anything
-# depending on a query rather than an announcement stays broken. Making that
-# work needs a querier on the VLAN side, which this relay deliberately is not.
-#
-# EXPOSURE
-#
-# The receive sockets carry so-bindtodevice=net1, so only datagrams that
-# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
-# address instead is not possible — it is DHCP-leased and unknown at exec time
-# — but binding to the device achieves the same confinement without needing to
-# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
-# unprivileged uid here.
-#
-# Without it socat would bind the wildcard, and since the namespace CNP grants
-# every pod in home-assistant unrestricted access to every other pod on every
-# port, any sibling pod could reach :5353 over the cluster network and have
-# records forwarded into HA's zeroconf cache.
-#
-# What remains, and cannot be closed here: any host on the IoT VLAN can send
-# this port a datagram. That is the same trust already extended to genuine
-# mDNS from that segment, which any device there can equally forge — the relay
-# widens who can reach HA's listener, not what can be claimed. macvlan traffic
-# bypasses Cilium policy entirely, so the namespace CNP does not constrain
-# this interface either way.
-#
-# Note that the relay is not what makes HA's zeroconf reachable from inside
-# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
-# and the namespace CNP already permits any sibling pod to reach it directly.
-# Confining the relay removes a second route to that listener; it does not
-# remove the first.
-#
-# Nothing else is exposed: the containers run socat and only socat, with no
-# second listener to reach.
+# so-bindtodevice=net1 on both receive sockets is load-bearing: without it
+# socat binds the wildcard, and the namespace CNP lets any sibling pod reach
+# :5353 and inject records into HA's zeroconf cache. Binding net1's address
+# instead is impossible — it is DHCP-leased and unknown at exec time.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -6,32 +6,23 @@ metadata:
  labels:
    app.kubernetes.io/name: mosquitto
  annotations:
-    # REQUIRED for credential rotation to take effect at all. The startup shim
-    # rebuilds /mosquitto/auth/passwd from these Secrets on every pod start, so
-    # a rotated Secret is inert until the pod restarts — without this the new
-    # password would sit unused until some unrelated restart, while the old one
-    # stayed valid indefinitely.
+    # REQUIRED for credential rotation to take effect. The startup shim rebuilds
+    # the password file only on pod start, so without this a rotated Secret sits
+    # inert while the old password stays valid indefinitely.
    #
-    # Listed per-Secret rather than using the blanket `reloader.stakater.com/
-    # auto: "true"`, so that enrolling each remaining broker user in rotation is
-    # a visible, reviewable edit to this line rather than something that starts
-    # happening implicitly.
+    # Listed per-Secret rather than with the blanket auto annotation, so that
+    # enrolling another broker user is a visible edit to this line.
    secret.reloader.stakater.com/reload: mosquitto-health-credentials
 spec:
  replicas: 1
  strategy:
-    # No PVC anymore, so nothing to fence: the new broker can come up and pass
-    # its readiness probe before the old one is torn down. maxUnavailable:0 keeps
-    # a broker serving throughout an image bump / reschedule (clients on the old
-    # pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
-    # overlap — two independent stateless brokers behind the LB, fine since there
-    # is no shared state.
+    # No PVC, so nothing to fence — the new broker can serve before the old one
+    # is torn down. Two stateless brokers overlapping behind the LB is fine.
    #
-    # NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
-    # new pod starts with an empty retained store, so a roll is equivalent to a
-    # broker restart for every client that depends on retention. Zigbee2MQTT
-    # does not recover from that on its own; see mosquitto-config.yaml and
-    # gitea issue #381.
+    # Connectivity survives a roll but RETAINED STATE DOES NOT: the new pod
+    # starts with an empty retained store, so every roll is a broker restart
+    # for clients that depend on retention. Zigbee2MQTT does not recover from
+    # that on its own (#381).
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
@@ -44,14 +35,10 @@ spec:
      labels:
        app.kubernetes.io/name: mosquitto
    spec:
-      # Runs on any node (no nodeSelector); the mosquitto-iot LB forwards
-      # IoT-VLAN traffic to the broker wherever it lands.
-      # Availability: home-critical priority (never preempted, reschedules
-      # first) + short fault tolerations so a node failure reschedules the
-      # broker in ~30s instead of the default ~5min. With persistence off there
-      # is no PVC to fence/detach/reattach, so the broker can (re)start anywhere
-      # instantly, no longer depends on Ceph being healthy, and rolls without a
-      # tear-down-first gap (see the RollingUpdate strategy above).
+      # Runs anywhere; the mosquitto-iot LB finds it. Short fault tolerations
+      # reschedule the broker in ~30s rather than the default ~5min, which is
+      # only possible because there is no PVC to fence — so it also does not
+      # depend on Ceph being healthy.
      priorityClassName: home-critical
      terminationGracePeriodSeconds: 10
      tolerations:
@@ -6,12 +6,9 @@ metadata:
  labels:
    app.kubernetes.io/name: zigbee2mqtt
 spec:
-  # CUTOVER DONE (2026-07-23): ZHA disabled, the coordinator-backup Secret is
-  # populated in OpenBao, and Z2M is now the live Zigbee stack — so replicas is
-  # 1. It shipped at 0 as a safety interlock (Z2M contends with ZHA for the
-  # coordinator's single TCP session, and starting without the restored backup
-  # would form a NEW network and drop every device); that interlock is spent
-  # once the migration is committed. To roll back, set 0 and re-enable ZHA.
+  # Z2M is the live Zigbee stack. Never run it alongside ZHA: they contend for
+  # the coordinator's single TCP session, and a Z2M start without the restored
+  # coordinator backup forms a NEW network and drops every paired device.
  replicas: 1
  # RWO PVC → single writer; never run two pods against the same /app/data.
  strategy:
@@ -39,14 +36,10 @@ spec:
        # network_key, the latter extracted from the OpenBao coordinator backup
        # as a natural-byte-order int array.
        - name: seed-data
-          # alpine + jq, built by .woodpecker/build-ci-images.yml from
-          # ci/images/alpine-jq/. Replaces `apk add --no-cache jq` on every pod
-          # start: tools belong in the image, not installed at runtime.
-          # Referenced by tag rather than digest — these are our own images.
-          # DATED tag, not :latest, matching the ci-ansible:2026-04-29 precedent
-          # in kubernetes/infrastructure/netbird-backup/cronjob.yaml. With
-          # :latest, rebuilding this base for some unrelated consumer would
-          # silently change what this init container runs on its next pod
+          # Our own alpine + jq image, built by build-ci-images.yml. DATED tag,
+          # never :latest — rebuilding this base for an unrelated consumer
+          # would otherwise silently change what this init container runs on
+          # its next pod
          # restart, with no diff here to review — and this is the container that
          # renders the Zigbee network key. Bumping the tag is a deliberate,
          # reviewable act; the cost is remembering to bump it.
@@ -1,20 +0,0 @@
-# Shared Traefik ServersTransport for ingresses that re-encrypt to a
-# backend serving HTTPS with a self-signed / hostname-mismatched cert:
-#
-#   apps/file       — NAS web UI on a self-signed cert
-#
-# Replaces the nginx `backend-protocol: HTTPS` + `proxy-ssl-verify: off`
-# annotation pair. Each Ingress that wants this behaviour sets
-#   traefik.ingress.kubernetes.io/service.serversscheme: https
-#   traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend
-#
-# (bare name — kubernetesIngress provider resolves ServersTransport in
-# the same namespace as the Ingress; the `<ns>-<name>@kubernetescrd`
-# qualified form is for IngressRoute CRDs only).
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: insecure-backend
-  namespace: apps
-spec:
-  insecureSkipVerify: true
@@ -1,10 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - insecure-backend-serverstransport.yaml
  - gatus/
  - external-dns/
-  - file/
  - gitea/
  - ical-proxy/
  # woodpecker/ moved to its own dedicated `woodpecker` Kustomization
@@ -5,9 +5,8 @@ metadata:
  namespace: monitoring
  labels:
    app.kubernetes.io/name: alloy
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  selector:
@@ -1,7 +1,8 @@
 # Scrape the CNPG Postgres instances (port 9187, built-in exporter).
-# CNPG's own enablePodMonitor would generate PodMonitors without the
-# `release: kube-prometheus-stack` label, which this Prometheus requires
-# (podMonitorSelector), so we define them here instead.
+# Defined here rather than via CNPG's own enablePodMonitor so the scrape
+# config lives with the other monitors. The `release` label below is not
+# required for discovery — podMonitorSelector is empty — but kept for
+# consistency.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
@@ -5,9 +5,8 @@ metadata:
  namespace: home-assistant
  labels:
    app.kubernetes.io/name: home-assistant
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered. Same trap the garage PrometheusRule documents.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  selector:
@@ -4,7 +4,9 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-blackbox.yaml
  - prometheusrule-ceph.yaml
+  - prometheusrule-ci-openbao.yaml
  - prometheusrule-cilium.yaml
  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
@@ -5,9 +5,8 @@ metadata:
  namespace: monitoring
  labels:
    app.kubernetes.io/name: loki
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  # Matches the `loki` (single-binary) and `loki-canary` services, but
@@ -1,5 +1,9 @@
 # HTTP probes — fan out to blackbox-exporter. Probe targets mirror the
 # legacy Ansible role's blackbox_http_targets list.
+#
+# The `release` label on every Probe here IS load-bearing: probeSelector still
+# matches on it, unlike serviceMonitorSelector and podMonitorSelector, which
+# are empty. Removing it silently stops the probe from being discovered.
 apiVersion: monitoring.coreos.com/v1
 kind: Probe
 metadata:
@@ -45,6 +49,23 @@ spec:
        # ports outside the hairpin's 80/443 — real WAN path.
        - http://proxy-1.renner.uno:9000/health
        - http://proxy-2.renner.uno:9000/health
+        # Woodpecker. Ungated, root returns 200 with no redirect. Every
+        # scheduled job in this repo runs here — the alert-monitor cron, the
+        # nightly scans, the deploy targets — so an outage here is silent by
+        # construction: the thing that would have told you is the thing that
+        # is down.
+        - https://ci.renner.uno
+        # OpenBao. Probes the health API rather than the UI on purpose: a
+        # SEALED instance still serves the UI (307 -> /ui/ -> 200) while every
+        # ExternalSecret in the cluster fails, which is exactly the
+        # green-while-broken shape this file warns about elsewhere.
+        #
+        # standbyok=true because the endpoint answers 200 only on the ACTIVE
+        # node and 429 on a standby — the external route currently lands on a
+        # standby, so the bare path would be permanently red. With the
+        # parameter, 200 means initialized, unsealed and serving, on whichever
+        # node answers; 503 sealed and 501 uninitialized both stay red.
+        - https://openbao.renner.uno/v1/sys/health?standbyok=true
        # APP-HEALTH probes for the gated monitoring UIs, from the correct
        # in-cluster vantage (monitoring-vantage-point-rule): the gate is
        # only on the ingress, so the ClusterIP /-/healthy endpoints are
@@ -0,0 +1,59 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: blackbox
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: blackbox.rules
+      rules:
+        # Catch-all for every blackbox-http target that has no rule of its own.
+        # Before this existed, a target could be probed, graphed and completely
+        # unalerted — which is how file.renner.uno served 502 for months.
+        #
+        # THE EXCLUSION LIST AND THE DEDICATED RULES ARE ONE INVARIANT: an
+        # instance belongs here only while a rule elsewhere covers it, and
+        # adding a dedicated rule means adding its instance here. Get it wrong
+        # in one direction and the alert fires twice; wrong in the other and it
+        # never fires at all. Currently excluded:
+        #
+        #   fsn1.your-objectstorage.com   HetznerBackupEndpointDown (critical)
+        #   netbird.renner.uno            prometheusrule-netbird-availability
+        #   proxy-1/2:9000/health         same file, relay redundancy
+        #   ci.renner.uno                 WoodpeckerDown
+        #   openbao .../sys/health        OpenBaoUnavailable
+        #
+        # Warning, not critical, on purpose: this rule cannot know what a given
+        # target means. A target whose loss deserves a critical page gets its
+        # own rule — and lands in the list above. That is what keeps the
+        # exclusion list from being arbitrary.
+        - alert: BlackboxProbeFailing
+          expr: probe_success{job="blackbox-http", instance!~"https://fsn1\\.your-objectstorage\\.com|https://netbird\\.renner\\.uno|http://proxy-[12]\\.renner\\.uno:9000/health|https://ci\\.renner\\.uno|https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 10m
+          annotations:
+            summary: "Blackbox probe of {{ $labels.instance }} has been failing for 10 minutes"
+            description: >-
+              The endpoint is not answering as expected. For the Authentik-gated
+              hostnames this is a reachability check covering DNS, TLS, Traefik
+              and the gate outpost — it says nothing about the app behind the
+              gate, which the internal /-/healthy probes cover separately.
+          labels:
+            severity: warning
+        # The check that the checks are running. Every rule above reads
+        # probe_success; if the series stop being produced at all, none of them
+        # can fire and the silence looks exactly like health.
+        - alert: BlackboxProbesAbsent
+          expr: absent(probe_success{job="blackbox-http"})
+          for: 15m
+          annotations:
+            summary: "No blackbox-http probe results are reaching Prometheus"
+            description: >-
+              Either the blackbox exporter is down or the Probe objects stopped
+              being discovered. Every endpoint alert depends on this job, so
+              they are all blind until it returns.
+          labels:
+            severity: critical
@@ -0,0 +1,47 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ci-openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ci-openbao.rules
+      rules:
+        - alert: WoodpeckerDown
+          # Probe target defined in probes-blackbox.yaml. Root, ungated, 200.
+          expr: probe_success{job="blackbox-http", instance="https://ci.renner.uno"} == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Woodpecker is not answering on ci.renner.uno"
+            description: >-
+              Every scheduled job in this repo runs here: the alert-monitor
+              cron that files these issues, the nightly scans, and every deploy
+              target. Nothing else reports this outage, because the component
+              that would have reported it is the one that is down. Alertmanager
+              itself is unaffected, which is why this alert exists.
+        - alert: OpenBaoUnavailable
+          # Probes /v1/sys/health?standbyok=true — 200 on any unsealed node,
+          # 503 sealed, 501 uninitialized. The UI stays up when sealed, so a
+          # root probe would not catch this.
+          #
+          # Matched by prefix rather than the full string: the target carries a
+          # query parameter, and an exact match would break silently if the
+          # instance label ever encodes it differently.
+          expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao is sealed, uninitialized or unreachable"
+            description: >-
+              Every ExternalSecret in the cluster resolves through OpenBao.
+              Existing Secrets keep working, so nothing breaks at once — the
+              damage shows up at the next refresh interval or the next pod that
+              needs a fresh credential. Check whether it is sealed before
+              assuming a network fault.
@@ -11,25 +11,22 @@ spec:
  groups:
    - name: cnpg-backup.rules
      rules:
-        # WHY THIS FILE EXISTS
-        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
-        # stopped. The only related rule was HetznerBackupEndpointDown, which
-        # probes the S3 endpoint — it proves the bucket is reachable, not that
-        # anything was written to it. Same failure shape as the WordPress backup
-        # that failed nightly for three days against a bucket that was declared
-        # in tofu but never applied: the status surface looked fine because
-        # nothing checked the thing that mattered.
+        # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
+        # probes that S3 endpoint, which proves the bucket is reachable and
+        # nothing more — these two rules check that something is actually being
+        # written to it.

        - alert: CNPGWalArchiveBacklog
          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
-          # to be archived. Steady state is 0 across every cluster (verified
-          # 2026-08-19). A sustained backlog means the archiver is failing:
-          # point-in-time recovery is silently broken from that moment, and
-          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          # to be archived. Steady state is 0. A sustained backlog means the
+          # archiver is failing: point-in-time recovery is silently broken from
+          # that moment, and pg_wal keeps growing until it fills the PVC and
+          # takes Postgres down.
          #
-          # This is the load-bearing alert of the two. It depends only on the
-          # instance-level collector, so unlike the backup-age metrics below it
-          # is unaffected by the missing Backup CRs (#522).
+          # This is the load-bearing alert of the two. It reads the
+          # instance-level collector directly, so unlike the backup-age metric
+          # discussed below it does not depend on Cluster status being
+          # populated.
          #
          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
@@ -63,16 +60,17 @@ spec:
        #
        # The natural expression is
        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
-        # and it CANNOT be used today: that metric reads exactly 0 on every
-        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
-        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
-        # successful backup anywhere, because the Backup CRs it derives that
-        # from are being deleted within a minute of completing — see #522.
+        # and it cannot be used, because that metric reads exactly 0 on every
+        # instance. The collector derives it from `Cluster.status
+        # .lastSuccessfulBackup`, and that field is absent on every Cluster
+        # here: all of them back up with `method: plugin` (barman-cloud), and
+        # the plugin path does not write it back.
        #
-        # Backups themselves DO work; that was verified independently by
-        # listing the bucket with barman-cloud-backup-list and restoring a WAL
-        # segment back out of S3. The gap is in the record, not the data.
+        # The backups themselves run and are retained — completed Backup CRs
+        # going back weeks exist for every scheduled cluster. The gap is in the
+        # status field the metric reads, not in the data.
        #
-        # Adding the rule now would fire permanently for all seven clusters and
-        # be silenced within a day, which is worse than not having it. Add it as
-        # part of resolving #522, once the metric reflects reality.
+        # Adding the rule against that metric would fire permanently for every
+        # cluster, which is worse than not having it. Either the plugin starts
+        # populating the field, or the alert has to be built on the Backup CRs
+        # directly, which Prometheus does not see today.
@@ -12,21 +12,54 @@ spec:
    - name: disk-wear.rules
      rules:
        # Wear percentages are self-reported: 100 means the rated endurance is
-        # spent, not that the drive stops. Planning thresholds, not predictions.
-        - alert: NvmeDriveWearHigh
-          # Long `for` because wear is monotonic and cannot flap.
-          expr: smartctl_device_percentage_used >= 80
-          for: 1h
+        # spent, not that the drive stops.
+        #
+        # Wear alone is not a signal. It only ever rises, so a plain threshold
+        # starts firing the day a drive crosses it and never stops — a status
+        # dressed as an alert (#673, #675, #682, #686 were all one drive set
+        # crossing 80%). What matters is wear that is high AND still climbing,
+        # so the alert below is a projection: at the write rate of the last
+        # week, when does this drive reach 95%?
+        #
+        # The rate is self-calibrating. Each drive reports both its lifetime
+        # bytes written and its own wear percentage, so bytes-per-percent comes
+        # from the drive itself and no rated-endurance table has to be kept in
+        # sync with the hardware. percentage_used is a whole-number percent and
+        # advances only every few weeks here, which is far too coarse to
+        # differentiate; bytes_written moves on every scrape.
+        - alert: NvmeDriveWearProjectedExhaustion
+          # The >= 80 gate is numerical, not editorial: bytes-per-percent is
+          # derived by dividing by an integer percentage, so on a lightly worn
+          # drive a half-percent rounding error swings the projection by tens
+          # of percent. Above 80 the same rounding is under one percent.
+          # Idle drives divide by a zero rate and yield +Inf, which is not < the
+          # threshold, so they stay silent on their own.
+          expr: >-
+            (
+              (95 - smartctl_device_percentage_used)
+              /
+              (
+                rate(smartctl_device_bytes_written[7d]) * 86400
+                  * smartctl_device_percentage_used
+                  / smartctl_device_bytes_written
+              )
+            ) < 180
+            and smartctl_device_percentage_used >= 80
+          # Long, because a projection off a 7d rate should not react to a day.
+          for: 6h
          labels:
            severity: warning
          annotations:
-            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} projects to 95% wear in {{ $value | printf \"%.0f\" }} days"
            description: >-
-              {{ $value }}% of rated write endurance used. Nothing is failing
-              yet. Urgency depends on what the drive carries — the Disk Health
-              dashboard shows the role. Losing an OSD costs a rebalance;
-              EPHEMERAL is /var and rebuilds, apart from the node's etcd member
-              data, which has to be removed and re-added.
+              The drive is past 80% of its rated endurance and the write rate
+              of the last week would spend the rest within six months. This is
+              a lead-time warning, not a fault: decide whether to replace the
+              drive or move what is writing to it. The Disk Health dashboard
+              shows which drives carry which role.
+        # Kept as a flat threshold on purpose: past rated endurance the drive
+        # runs outside what the manufacturer stands behind regardless of how
+        # fast it got there, and the alert clears by replacing it.
        - alert: NvmeDriveWearCritical
          expr: smartctl_device_percentage_used >= 95
          for: 1h
@@ -17,11 +17,12 @@ spec:
          # drains on recovery) and Velero backups fail. Replaces the old
          # GarageDown alert now that backups live on Hetzner, not nas-1.
          #
-          # CNPG has no metric-based archiving alert yet — its pod metrics
-          # are not scraped (enablePodMonitor=false). Enabling CNPG
-          # PodMonitors for a granular "archiving falling behind" alert is
-          # a follow-up. Velero's own VeleroBackupValidationFailure /
-          # VeleroBackupStale rules already cover backup-level failures.
+          # The granular counterpart is CNPGWalArchiveBacklog in
+          # prometheusrule-cnpg-backup.yaml, reading the instance collector
+          # scraped by cnpg-podmonitors.yaml: this alert says the target is
+          # unreachable, that one says archiving is falling behind whatever the
+          # cause. Velero's VeleroBackupValidationFailure / VeleroBackupStale
+          # cover backup-level failures.
          expr: probe_success{instance="https://fsn1.your-objectstorage.com"} == 0
          for: 10m
          labels:
@@ -5,9 +5,8 @@ metadata:
  namespace: monitoring
  labels:
    app.kubernetes.io/name: smartctl-exporter
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
    release: kube-prometheus-stack
 spec:
  selector:
@@ -24,8 +24,10 @@ spec:
      enabled: true
      defaults:
        labels:
-          # Required for kube-prometheus-stack's Prometheus to pick up
-          # this ServiceMonitor (its selector matches release).
+          # Not required for discovery — serviceMonitorSelector is empty — but
+          # kept for consistency with the other monitors. The Probes in
+          # monitoring-rules/probes-blackbox.yaml do need it; probeSelector
+          # still matches on release.
          release: kube-prometheus-stack
        interval: 30s
    # Helm DOES deep-merge this map with the chart's default (that is why
@@ -56,26 +56,21 @@ spec:
        #
        # By default kube-prometheus-stack sets these selectors to
        # `matchLabels: {release: kube-prometheus-stack}`. ServiceMonitors
-        # created at RUNTIME by third-party operators carry that operator's own
-        # labels instead and can never be made to match — no static manifest or
-        # Kustomize patch can relabel an object the operator creates itself.
+        # created at RUNTIME by third-party operators — Rook, goauthentik.io —
+        # carry that operator's own labels instead and can never be made to
+        # match: no static manifest or Kustomize patch can relabel an object
+        # the operator creates itself. Left strict, those targets are dropped
+        # silently, with no failed resource and no gap visible anywhere except
+        # in the missing series.
        #
-        # This was silently dropping 12 of 30 ServiceMonitors (2026-07-27):
-        #   rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter  (team: rook)
-        #   authentik/ak-outpost-*  x10                            (goauthentik.io/...)
-        # Consequence: ZERO `ceph_*` series existed in Prometheus for all 79
-        # days of the cluster's life — the entire storage layer had no
-        # telemetry, no history, and any Ceph alerting rule would have had
-        # nothing to fire on. The Rook side is the reason this was found; the
-        # authentik outpost targets come along with it.
+        # Because these two are empty, the `release` label on this repo's own
+        # ServiceMonitors and PodMonitors is decorative. It is still set for
+        # consistency, and the comments there say so.
        #
-        # This is the ServiceMonitor twin of the trap already documented for
-        # rules in monitoring-rules/prometheusrule-velero.yaml ("without this
-        # label the rule is created but never loaded").
-        #
-        # ruleSelector is deliberately NOT loosened: this repo's PrometheusRules
-        # all carry `release: kube-prometheus-stack` on purpose, and keeping
-        # that selector strict stops a stray third-party rule from firing here.
+        # ruleSelector, probeSelector and scrapeConfigSelector are deliberately
+        # NOT loosened: nothing outside this repo creates those, so keeping the
+        # selectors strict stops a stray third-party rule from firing here. For
+        # PrometheusRules and Probes the label IS required.
        serviceMonitorSelectorNilUsesHelmValues: false
        podMonitorSelectorNilUsesHelmValues: false
        # Rewrites the in-cluster URL embedded in alert generatorURLs and
@@ -5,25 +5,15 @@ metadata:
  labels:
    app.kubernetes.io/name: otbr
 spec:
-  # Disabled 2026-07-11 (replicas: 0) after otbr broke Matter.
+  # DISABLED (replicas: 0). Running otbr broke Matter: its radio is RF-isolated
+  # enough from the Nest mesh that it forms a rival partition of the same
+  # network, and the two bridge only over Ethernet — fast enough for a ping,
+  # too slow for the fragmented CASE handshake, so devices stay discoverable
+  # while every commissioning attempt times out.
  #
-  # otbr is seeded with the Nest network's dataset so it should join the
-  # existing partition as an additive co-BR. In practice it does not merge
-  # cleanly: its SLZB-06 radio is RF-isolated enough from the Nest mesh that
-  # otbr elects itself leader of a rival partition (or the network re-addresses
-  # under a new OMR prefix, observed fd5f -> fdd5). The two same-network
-  # partitions then bridge only over the Ethernet backbone, which carries a
-  # ping but is too slow for the large fragmented CASE handshake -- every
-  # Matter device stays mDNS-discoverable while CASE times out. Removing otbr
-  # re-merges the mesh onto the Nest hubs and Matter recovers.
-  #
-  # This is NOT a diagnosis of a specific config bug: the DHCP/sbr routing
-  # changes (19a4482, df61407) were investigated and matter-server's IPv6 path
-  # to the mesh works (reaches devices, ping ok); sbr's v6 table 100 is an
-  # inert orphan and otbr's eth0 default is outranked by net1's pref-high RA
-  # default. The instability is at the Thread partition layer. Re-enable only
-  # after the co-BR integration is verified to MERGE (one partition, one OMR,
-  # otbr's router table shows the Nest routers) instead of split.
+  # The instability is at the Thread partition layer, not in the config below.
+  # Re-enable only once co-BR integration is verified to MERGE: one partition,
+  # one OMR prefix, and the Nest routers visible in otbr's router table.
  replicas: 0
  strategy:
    type: Recreate
@@ -46,25 +36,17 @@ spec:
      # (slzb-thread.renner.uno). Dropping the workload-class=pinned
      # toleration lets the edge taint keep it off the tiny edge box.
      containers:
-        # The Thread RCP lives on a SLZB-06 over Ethernet at
-        # slzb-thread.renner.uno:6638 (Serial-over-IP). otbr-agent's
-        # spinel transport supports `spinel+hdlc+forkpty://<exec>?<args>`
-        # which forks an exec whose stdio becomes the spinel-HDLC
-        # transport, but the upstream openthread/otbr image does NOT
-        # ship socat (or any TCP-stream tool), so that URL form ends in
-        # silent ENOENT and otbr-agent dies right after logging the
-        # radio URL.
+        # The Thread RCP is a network-attached radio (Serial-over-IP), so the
+        # spinel transport needs a TCP-to-PTY bridge. Two obvious approaches
+        # do not work and should not be retried:
        #
-        # A sidecar socat won't work either: each container has its own
-        # devpts namespace, so a PTY (e.g. /dev/pts/0) created in a
-        # sibling container is invisible from this container — a shared
-        # emptyDir symlink to /dev/pts/0 is dangling.
+        #   - `spinel+hdlc+forkpty://` needs socat, which the upstream image
+        #     does not ship; the URL fails with a silent ENOENT and otbr-agent
+        #     dies right after logging it.
+        #   - A sidecar cannot provide it either: devpts is per-container, so
+        #     a PTY created next door is invisible here and the symlink dangles.
        #
-        # The bridge has to live in the otbr container's own namespace.
-        # Install socat at startup via apt (Ubuntu 18.04 base, no egress
-        # restrictions on this namespace), run it in the background as
-        # the spinel-HDLC TCP↔PTY bridge, then exec the upstream
-        # entrypoint with RADIO_URL pointing at the PTY symlink.
+        # Hence socat is installed at startup and run inside this container.
        - name: otbr
          image: openthread/otbr:latest
          command:
@@ -1,20 +1,15 @@
 # Headless renderer: screenshots the Home Assistant e-ink dashboard,
 # dithers it to 16 greys and hands it to Terminus.
 #
-# Upstream ships this as a Home Assistant add-on. Our HA is a plain
-# container with no Supervisor, so it runs standalone here instead —
-# viable because the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env
-# vars over /data/options.json and authenticates with an ordinary
-# long-lived token. SUPERVISOR_TOKEN is deliberately NOT set: it only
-# flips an internal isAddOn flag, and setting it would switch the app
-# to add-on behaviour (fixed port, /data assumptions).
+# Upstream ships this as a Home Assistant add-on; it runs standalone here
+# because the app prefers the env vars over /data/options.json. Do NOT set
+# SUPERVISOR_TOKEN — it flips an internal isAddOn flag and switches the app to
+# add-on behaviour (fixed port, /data assumptions).
 #
-# DIRECTION OF TRAVEL, which is the non-obvious part: with the
-# byos-hanami webhook in `uri` delivery mode the renderer does NOT
-# push image bytes. It POSTs a URL pointing at its own /output/<file>
-# endpoint, and TERMINUS FETCHES THE IMAGE BACK from it. That is why
-# both live in this namespace — the pull is intra-namespace and is
-# already permitted by the default-deny-ingress rule.
+# DIRECTION OF TRAVEL, the non-obvious part: in `uri` delivery mode the
+# renderer does not push image bytes. It POSTs a URL to its own /output/<file>
+# and Terminus fetches the image back. Both therefore live in this namespace,
+# where that pull is already permitted.
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -305,7 +305,7 @@ server → agent → step pod → server end to end:

 ```sh
 curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
-woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+woodpecker-cli pipeline deploy k.renner/homelab <pipeline-number> e2e-test
 ```

 Step pods must still reach the server on `:8000` while still being unable to