fix(rotation): make the documented guard actually pass

Second attempt, and the first one was wrong for a subtler reason than I fixed. Anchoring on `image:` did not help: this file documents the pattern, so the comment text `image:.*@sha256` matches the regex `image:.*@sha256`. The guard kept reporting its own documentation. Added the `| grep -v kustomization.yaml` filter and this time ran the command exactly as written rather than a paraphrase of it. Verified both directions: no output across this directory as it stands, and it still reports a file carrying a real `image: ...@sha256:` pin. Lesson worth the two commits: a documented check has to be executed in the form it is documented in. I had verified a variant twice and shipped a guard whose only hit was itself — which trains people to ignore it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(rotation): make the documented digest-guard not flag its own docs
2026-07-30 19:33:04 +02:00 · 2026-07-30 19:32:08 +02:00 · 2026-07-30 19:31:30 +02:00 · 2026-07-30 19:26:44 +02:00 · 2026-07-30 16:56:41 +00:00 · 2026-07-30 16:54:02 +00:00
26 changed files with 528 additions and 90 deletions
@@ -188,3 +188,32 @@ steps:
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
+
+  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
+  # needing s_client/x509. Named for what it is; first consumer is the paperless
+  # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
+  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
+  # base, so this one needs an image rather than a busybox substitute.
+  - name: build-alpine-openssl
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    directory: ci/images/alpine-openssl
+    depends_on: []
+    when:
+      - event: manual
+      - event: push
+        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
+          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
@@ -0,0 +1,21 @@
+FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
+
+# Alpine plus the openssl CLI. Nothing application-specific — named for what it
+# is, so anything needing `openssl s_client` / `x509` from a shell script can use
+# it.
+#
+# It exists because installing tools at container start is not acceptable here:
+# `apk add` makes every start depend on network and package-index availability,
+# and the resolved version can drift between restarts of an otherwise pinned
+# image.
+#
+# Note the openssl CLI is genuinely absent from the alpine base — verified, the
+# base carries the libraries but not the command — so unlike the multus CNI
+# installer (where busybox wget/tar already covered the need) this one really
+# does require an image rather than just dropping the apk add.
+#
+# First consumer: the paperless fetch-bridge-cert init container, which captures
+# the Proton Mail Bridge's self-signed IMAP certificate via STARTTLS and pins it
+# for paperless to trust. busybox has no s_client equivalent, so openssl is not
+# substitutable there.
+RUN apk add --no-cache openssl
@@ -0,0 +1,179 @@
+# Proposal: enrol the mosquitto broker credentials in Class-A rotation
+
+**Status:** proposal, nothing implemented.
+**Scope:** the 8 MQTT broker users and the credentials behind them.
+
+## Problem
+
+Every self-generated secret in this cluster rotates automatically except these.
+`kubernetes/infrastructure/openbao/rotation/` holds 13 manifests defining 15
+rotator CronJobs (`home-assistant.yaml` alone defines three). Five are **Class-A**
+— authentik, crowdsec, paperless, shelly-manager and woodpecker — meaning
+"self-generated random secret", which is exactly what a broker password is.
+The mosquitto credentials are the only Class-A-shaped secrets not enrolled.
+
+They are currently minted once by seven near-identical bootstrap Jobs
+(`mosquitto-bootstrap.yaml`, 508 lines — the ServiceAccount, Role and RoleBinding
+are inside that same file) which generate a password, hand-build Secret JSON with
+`jq` and POST it to the Kubernetes API with `curl`. Once created they never
+change again.
+
+Three consequences:
+
+1. **They never rotate.** Not on a schedule, not on compromise, not ever.
+2. **They exist only in the Secret object** — not in OpenBao, so not covered by
+   the OpenBao snapshot CronJob, and unrecoverable if lost.
+3. **The Jobs cannot be modified.** A Job's `spec.template` is immutable, so
+   changing even the container image makes the whole `home-assistant`
+   Kustomization fail to reconcile with `field is immutable`. Any edit requires
+   deleting or renaming the Jobs.
+
+## Why the obvious fixes do not work
+
+Recorded because each was tried and rejected on evidence, and the reasons are
+not obvious from the code.
+
+| Approach | Why it fails |
+| --- | --- |
+| ExternalSecrets reading hand-seeded OpenBao paths | Depends on content copied in by hand. Git cannot recreate it; a fresh OpenBao leaves the paths empty and ESO then overwrites live broker credentials with nothing. |
+| ESO `Password` generator | **Overwrites pre-existing Secrets.** Verified: a Secret holding `KNOWN-PREEXISTING-VALUE` was replaced with a freshly generated password on first sync. Adopting it rotates the 7 Job-minted credentials at once, uncoordinated (`evcc` is already ESO-extract-backed and so unaffected). |
+| Keep the Jobs, fix only the image | Blocked by `spec.template` immutability. Renaming the Jobs works but is a hack that recurs on every future edit. |
+
+The common thread: each treats the credential as something to *place*, when the
+real requirement is a credential that can be *replaced*.
+
+## How rotation actually has to work here
+
+The broker rebuilds its password file from scratch on **every pod start**
+(`mosquitto-deployment.yaml`): `rm -f /mosquitto/auth/passwd` followed by
+`mosquitto_passwd -c -b …` for each of the 8 users, sourced from env vars backed
+by the 8 Secrets.
+
+Two things follow, and they shape the whole design:
+
+- **A Secret change has no effect until the broker restarts.** There is no
+  Reloader annotation on the mosquitto Deployment today, unlike the consumers of
+  the existing Class-A rotators.
+- **There is no make-before-break.** A mosquitto user has exactly one password.
+  The LLAT rotators can mint a second token and revoke the first afterwards; a
+  broker user cannot hold old and new simultaneously. Unless a transitional
+  second user is created, broker and client must cut over together.
+
+### Who holds each credential
+
+This is the part that determines cutover cost, and it is not visible from the
+manifests alone.
+
+| Broker user | Held by | Cutover |
+| --- | --- | --- |
+| `health` | the broker's own liveness/readiness probes | in-cluster, same pod restart |
+| `zigbee2mqtt` | zigbee2mqtt Deployment (`secretKeyRef`) | in-cluster, pod restart |
+| `evcc` | evcc app; already OpenBao-backed via ESO | in-cluster, pod restart |
+| `sml2mqtt` | edge host, from `vault_edge_sml2mqtt_mqtt_password` | re-run the edge-host play |
+| `open3e` | edge host, from `vault_edge_open3e_mqtt_password` | re-run the edge-host play |
+| `homeassistant` | HA MQTT integration — a config entry created in the UI | needs an API/WebSocket update, or a manual re-entry |
+| `shellies` | Shelly device firmware | pushed per-device; `shelly-manager` is deployed and is the natural mechanism |
+| `admin` | a workstation client, held by a human | read the new value from OpenBao |
+
+Note `sml2mqtt` and `open3e` currently keep **the same password in two unlinked
+places** — the Kubernetes Secret for the broker, and `ansible-vault` for the
+client — synchronised by hand. That duplication, not any client limitation, is
+what makes rotation feel impossible today.
+
+## Proposed design
+
+Follow the existing Class-A pattern exactly (`rotation/paperless.yaml` is the
+closest model): *generate random bytes → merge-patch into OpenBao → ESO syncs the
+Secret → restart the consumer*.
+
+1. **One rotator CronJob per broker user**, running the existing `secret-rotator`
+   binary in random mode, writing `apps/home-assistant/mosquitto-<user>-credentials`
+   in OpenBao. Reuses the established policy/role/SA pattern; no new machinery.
+2. **ExternalSecrets** project each OpenBao entry to the existing Secret name and
+   key, so `mosquitto-deployment.yaml` and the in-cluster consumers are unchanged.
+   Key names stay as they are (`homeassistant-password` on one, `password` on the
+   rest) — consumers reference them, so normalising is a separate change.
+3. **Add a Reloader annotation to the mosquitto Deployment** so a Secret change
+   restarts the broker, instead of silently taking effect at some unrelated future
+   restart.
+4. **Move the edge host off `ansible-vault` for these two values**, reading the
+   same OpenBao entry instead, so there is one source of truth per credential.
+5. **Rotate one user at a time.** Each rotation then affects exactly one client,
+   not all eight.
+
+### The cutover window
+
+With no make-before-break, each rotation has a window where broker and client
+disagree. Two options:
+
+- **Accept a brief window** (recommended for most users). Order: rotator writes
+  OpenBao → client picks up the new value → broker restarts. MQTT clients
+  reconnect automatically; the window is seconds. Because rotation is per-user,
+  only that one client is affected.
+
+  One failure mode to expect rather than be alarmed by: the broker rolls with
+  `maxUnavailable: 0` / `maxSurge: 1`, so mid-rollout **two broker pods with
+  different passwd files sit behind the LB**. A client reconnecting during that
+  window can land on the old pod and fail auth until the rollout completes. It is
+  self-healing and lasts seconds to a minute, but it will look alarming in logs
+  during the first `health` / `zigbee2mqtt` rotation, so it is worth knowing in
+  advance.
+- **Transitional dual user** (`<user>` and `<user>-next` in the passwd file and
+  ACL) for any credential where even a brief outage is unacceptable. More moving
+  parts; probably only worth it for `homeassistant`, which carries everything.
+
+## Suggested phasing
+
+Ordered so the risky work happens after the pattern is proven, and so each phase
+is independently valuable.
+
+1. **`health` and `zigbee2mqtt` first.** Both are entirely in-cluster and pick up
+   a new password on pod restart with no external coordination. Proves rotator →
+   OpenBao → ESO → Reloader end to end with the smallest blast radius. `health`
+   is the safest possible start: only the broker's own probes consume it.
+2. **`evcc`.** Already OpenBao-backed via ESO; only the rotator is missing.
+3. **`sml2mqtt` and `open3e`**, together with switching the edge host to read from
+   OpenBao. This is where the duplicate-source-of-truth problem gets fixed.
+4. **`shellies`**, via shelly-manager pushing to devices. Needs investigation:
+   shelly-manager has its own Class-A rotator today, but that rotates
+   shelly-manager's own secret and does not touch device MQTT credentials.
+5. **`homeassistant`** last. Highest blast radius, and the config entry is UI-created,
+   so it needs an API/WebSocket update path — the HA LLAT rotators
+   (`rotation/home-assistant.yaml`) already talk to HA over WebSocket and are the
+   obvious precedent.
+6. **`admin`** whenever convenient; a human reads the new value from OpenBao.
+
+Only once a user is enrolled does its bootstrap Job become dead code. Delete the
+Jobs incrementally as their users migrate, rather than in one step — that also
+avoids the immutability problem entirely, since the Jobs are deleted, never
+patched.
+
+## Open questions
+
+- **Does `secret-rotator` random mode fit unchanged?** The existing rotators write
+  a field into an OpenBao entry; here the entry holds a single password field, so
+  it should, but this needs confirming against the binary rather than assumed.
+- **Can shelly-manager push MQTT credentials to devices**, and does it cover both
+  Gen1 (`/settings/mqtt`) and Gen2+ (`Mqtt.SetConfig`) firmwares? If not, the
+  `shellies` user may need a different mechanism or stay manual longest.
+- **How is the HA MQTT config entry updated programmatically?** Reusing the
+  WebSocket approach from the LLAT rotator is plausible but unverified.
+- **Rotation cadence.** The existing rotators' schedules should be the reference
+  rather than inventing a new one.
+- **ESO refresh latency.** Rotator-writes-OpenBao → Secret-updates →
+  Reloader-restarts is delayed by up to one `refreshInterval`; the `evcc`
+  ExternalSecret uses `1h`. That does not widen the outage window — broker and
+  client still move together — but it does stretch how long a rotation takes to
+  land, which matters for verifying one completed and for the cold-rebuild
+  ordering question above. Worth deciding whether these ExternalSecrets want a
+  shorter interval than 1h, or whether the rotator should trigger the sync
+  directly.
+- **Bootstrap on a cold rebuild.** On a fresh cluster the rotator must create the
+  OpenBao entry before ESO can sync it; check this against
+  `kubernetes/RUNBOOK-cold-rebuild.md` so broker bring-up is not blocked.
+
+## What this does not change
+
+- Secret names, key names and every `secretKeyRef` in `mosquitto-deployment.yaml`.
+- The ACL model and the per-user topic scoping.
+- The `evcc` ExternalSecret, which already follows the target pattern.
@@ -13,6 +13,9 @@ resources:
  - prometheus-token-externalsecret.yaml
  - config-sync-token-externalsecret.yaml
  - mosquitto-bootstrap.yaml
+  # health broker credential: OpenBao-backed + Class-A rotated (first of the
+  # eight users to migrate off mosquitto-bootstrap.yaml).
+  - mosquitto-health-credentials-externalsecret.yaml
  - mosquitto-config.yaml
  - mosquitto-certificate.yaml
  - mosquitto-service.yaml
@@ -370,74 +370,6 @@ spec:
            limits:
              memory: 64Mi
 ---
-# Broker user `health`: a dedicated, least-privilege credential used ONLY by
-# mosquitto's own liveness/readiness probes (mosquitto-deployment.yaml). Its
-# ACL is `write health/#` and nothing else — it cannot read any topic or
-# write outside the throwaway probe subtree, so a leak is worthless (nothing
-# subscribes to health/#). Kept on its own Secret, like every other user, so
-# the probe never has to borrow the admin credential.
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: mosquitto-health-credentials-bootstrap
-  namespace: home-assistant
-spec:
-  backoffLimit: 4
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: mosquitto-bootstrap
-    spec:
-      serviceAccountName: mosquitto-bootstrap
-      restartPolicy: OnFailure
-      containers:
-        - name: bootstrap
-          image: alpine:3.21
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              apk add --no-cache curl jq >/dev/null
-
-              NS=home-assistant
-              NAME=mosquitto-health-credentials
-              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
-              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-              API=https://kubernetes.default.svc
-
-              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
-                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
-                echo "$NAME already exists; nothing to do."
-                exit 0
-              fi
-
-              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
-              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
-
-              body=$(jq -nc \
-                --arg name "$NAME" \
-                --arg ns "$NS" \
-                --arg pw "$PW_B64" \
-                '{apiVersion: "v1",
-                  kind: "Secret",
-                  metadata: {name: $name, namespace: $ns},
-                  type: "Opaque",
-                  data: {password: $pw}}')
-
-              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
-                   -H "Content-Type: application/json" -X POST \
-                   --data "$body" \
-                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
-
-              echo "generated $NAME"
-          resources:
-            requests:
-              cpu: 10m
-              memory: 32Mi
-            limits:
-              memory: 64Mi
---
 # Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
 # to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
 # publish HA MQTT-discovery configs, exactly like the `shellies` user) and
@@ -5,6 +5,18 @@ metadata:
  namespace: home-assistant
  labels:
    app.kubernetes.io/name: mosquitto
+  annotations:
+    # REQUIRED for credential rotation to take effect at all. The startup shim
+    # rebuilds /mosquitto/auth/passwd from these Secrets on every pod start, so
+    # a rotated Secret is inert until the pod restarts — without this the new
+    # password would sit unused until some unrelated restart, while the old one
+    # stayed valid indefinitely.
+    #
+    # Listed per-Secret rather than using the blanket `reloader.stakater.com/
+    # auto: "true"`, so that enrolling each remaining broker user in rotation is
+    # a visible, reviewable edit to this line rather than something that starts
+    # happening implicitly.
+    secret.reloader.stakater.com/reload: mosquitto-health-credentials
 spec:
  replicas: 1
  strategy:
@@ -177,7 +189,9 @@ spec:
                secretKeyRef:
                  name: mosquitto-evcc-credentials
                  key: password
-            # Probe-only credential (mosquitto-health-credentials-bootstrap).
+            # Probe-only credential. Now OpenBao-backed and Class-A rotated (see
+            # mosquitto-health-credentials-externalsecret.yaml); its bootstrap Job
+            # is gone.
            # Present here for the same reason as the others — the startup
            # shim hashes it into the passwd file — and reused by the exec
            # probes below, which run inside this container.
@@ -230,7 +244,7 @@ spec:
          #
          # Uses the dedicated least-privilege `health` user (ACL: readwrite
          # health/# only — see the passwd/acl build above and the
-          # mosquitto-health-credentials-bootstrap Job). Its password is in
+          # mosquitto-health-credentials ExternalSecret). Its password is in
          # this container's env (HEALTH_PASSWORD) because the startup shim
          # hashes every user into the passwd file; the exec probes run
          # inside this container and read it directly. A leak of this
@@ -0,0 +1,36 @@
+# Projects the mosquitto `health` broker credential from OpenBao into the Secret
+# the broker already consumes. FIRST of the eight broker users to move off the
+# bootstrap Jobs and onto Class-A rotation (see
+# PROPOSAL-mosquitto-credential-rotation.md (PR #453 — not yet merged; this
+# file lands with it)); the rotator lives at
+# kubernetes/infrastructure/openbao/rotation/mosquitto-health.yaml.
+#
+# Secret name and key are UNCHANGED (`password`), so mosquitto-deployment.yaml's
+# secretKeyRef and the exec probes that read $HEALTH_PASSWORD need no edit.
+#
+# refreshInterval is 10m rather than the 1h used by mosquitto-evcc-credentials:
+# a rotation only lands once ESO re-reads, so 1h would mean up to an hour between
+# the rotator running and the broker actually using the new password, with the
+# old one still valid in the meantime. 10m keeps that window short without
+# polling OpenBao pointlessly hard.
+#
+# ⚠️ OWNERSHIP: creationPolicy is implicitly Owner, so ESO sets an ownerReference
+# on the Secret. Deleting this ExternalSecret therefore DELETES the live Secret
+# and with it the broker's probe credential — the bootstrap Job it replaces had
+# no such coupling. To detach without destroying, set creationPolicy: Orphan
+# before removing it.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: mosquitto-health-credentials
+  namespace: home-assistant
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: mosquitto-health-credentials
+  dataFrom:
+    - extract:
+        key: apps/home-assistant/mosquitto-health-credentials
@@ -52,7 +52,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.9 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.10 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -62,7 +62,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.9 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.10 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -88,13 +88,17 @@ spec:
      # trust without disabling verification.
      initContainers:
        - name: fetch-bridge-cert
-          image: alpine:3.21
+          # alpine + the openssl CLI, built by .woodpecker/build-ci-images.yml
+          # from ci/images/alpine-openssl/. Replaces `apk add --no-cache openssl`
+          # on every paperless pod start: tools belong in the image, not
+          # installed at runtime. Dated tag so a change is a reviewable diff
+          # here, matching the ci-ansible / alpine-jq convention.
+          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
          command:
            - sh
            - -c
            - |
              set -e
-              apk add --no-cache openssl
              # Wait until the bridge presents a cert via STARTTLS.
              # Re-fetched on every paperless restart so a bridge cert
              # rotation only requires a paperless rollout to pick up.
@@ -36,6 +36,15 @@ spec:
    kubeProxyReplacement: true
    socketLB:
      enabled: true
+    prometheus:
+      # The agent exported no metrics at all, so BPF map utilisation was
+      # invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
+      # of space the agent only says "no space left on device" in its log and
+      # pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
+      # that observable — and alertable — before it stalls a rollout.
+      enabled: true
+      serviceMonitor:
+        enabled: true
    bpf:
      # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
      # redirects the socket directly to the backend pod instead of sending
@@ -0,0 +1,46 @@
+# OpenBao policy: rotator-mosquitto-health
+#
+# Least-privilege policy for the mosquitto `health` broker-credential rotation
+# CronJob (kubernetes auth role `rotator-mosquitto-health`). PER-SECRET
+# isolation, same template as rotator-paperless: one policy + one role per
+# rotated secret, so a compromised rotation Job cannot touch any other secret.
+#
+# Scoped to the `health` credential ALONE, deliberately not to a wildcard over
+# apps/home-assistant/mosquitto-*. The broker has eight users and they are being
+# enrolled one at a time; a wildcard would silently grant the first rotator write
+# access to the other seven before they are ready, which is exactly the blast
+# radius this per-secret pattern exists to avoid.
+#
+# Plan and phasing: kubernetes/apps/home-assistant/
+# PROPOSAL-mosquitto-credential-rotation.md — added by PR #453, which is not yet
+# merged, so that path does not resolve until it lands.
+#
+# NOT applied by Flux — OpenBao policies/roles are not Kubernetes resources
+# (same convention as eso-read.hcl and every other rotator policy). Apply ONCE
+# with an admin token:
+#
+#   bao policy write rotator-mosquitto-health \
+#     kubernetes/infrastructure/openbao/policies/rotator-mosquitto-health.hcl
+#
+#   bao write auth/kubernetes/role/rotator-mosquitto-health \
+#     bound_service_account_names=mosquitto-health-secret-rotator \
+#     bound_service_account_namespaces=openbao \
+#     policies=rotator-mosquitto-health ttl=15m
+#
+# Do NOT set token_no_default_policy=true — the `bao kv patch` kv-v2 preflight
+# needs the default policy (see rotator.hcl). The rotator writes via patch, so
+# hardening the role that way would break rotation rather than tighten it.
+#
+# BOOTSTRAP: `random` mode calls assert_fields_exist BEFORE patching, so it
+# ROTATES an existing field and never creates one. The path must already hold a
+# `password` field or the Job fails safe without writing. Seed it once:
+#
+#   bao kv put homelab/apps/home-assistant/mosquitto-health-credentials \
+#     password="$(kubectl -n home-assistant get secret mosquitto-health-credentials \
+#                 -o jsonpath='{.data.password}' | base64 -d)"
+#
+# (Reads are already covered by eso-read.hcl's `homelab/data/*`, so the
+# ExternalSecret needs no policy change.)
+path "homelab/data/apps/home-assistant/mosquitto-health-credentials" {
+  capabilities = ["read", "patch"]
+}
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -68,7 +68,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: sync-woodpecker
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["woodpecker-secret"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -61,7 +61,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:978fe53cad83a8d55319f6201ecd171775a101b21aaa21b7ec3c7b14304c6a42
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["cloudflare-roll"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -56,7 +56,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:978fe53cad83a8d55319f6201ecd171775a101b21aaa21b7ec3c7b14304c6a42
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["cloudflare-roll"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -84,7 +84,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:b41cc3645f144179dddaa5ab1d32282c57c257432a613c15a6e15634fd0c4b81
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["gitea-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -68,7 +68,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -130,7 +130,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -192,7 +192,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -4,6 +4,46 @@ kind: Kustomization
 # (intra-namespace OpenBao access → no NetworkPolicy needed). One file per rotated
 # secret. The matching OpenBao policy/role for each lives in ../policies/ and is
 # bootstrapped manually (not GitOps-managed).
+#
+# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
+# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
+# image, so it does not need digest pinning, and a dated tag makes "what actually
+# runs" a reviewable diff.
+#
+# Digests were tried and actively broke: untagged manifests are garbage-collected
+# once :latest and the dated tags move past them, so a digest pinned today stops
+# resolving after a few rebuilds. That decay is driven by edits to
+# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
+# republishes secret-rotator and orphans the manifest every rotator here was
+# pinned to. 11 of these were pinned to digests that no longer existed and would
+# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
+# are quarterly, that would have looked like nothing happening rather than like a
+# failure.
+#
+# Keep them all on one tag: it is a single binary containing every mode
+# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
+# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
+# the next bump partial.
+#
+# ADDING A ROTATOR: copy an existing file, and check you did not inherit a
+# digest pin from an older example — nothing enforces this automatically, so it
+# is easy to reintroduce silently and the failure only shows up months later on
+# the first scheduled run:
+#
+#   grep -l 'image:.*@sha256' \
+#     kubernetes/infrastructure/openbao/rotation/*.yaml | grep -v kustomization.yaml
+#
+# (want: no output. The trailing filter is not padding — this file documents the
+# pattern, so it necessarily matches its own grep. Verified both directions:
+# silent across this directory as it stands, and it still reports a file that
+# actually carries an `image: …@sha256:` pin.)
+#
+# BUMPING the tag: verify the new one exists BEFORE committing — a tag that was
+# never built fails exactly like the digests did, and just as quietly:
+#
+#   kubectl run tagcheck --rm -i --restart=Never \
+#     --image=gitea.renner.uno/gitea_admin/secret-rotator:<DATE> \
+#     --command -- /secret-rotator          # prints the usage/mode list if the tag resolves
 resources:
  - paperless.yaml
  - woodpecker.yaml
@@ -18,3 +58,4 @@ resources:
  - home-assistant.yaml
  - review-bot.yaml
  - claude-oauth.yaml
+  - mosquitto-health.yaml
@@ -0,0 +1,124 @@
+---
+# Class-A automated rotation of the mosquitto `health` broker credential — a
+# self-generated random secret.
+#
+# FIRST of the eight mosquitto broker users to be enrolled. The credentials were
+# previously minted once by bootstrap Jobs and then never changed at all: no
+# schedule, no compromise response, and they existed only in the Secret object,
+# outside the OpenBao snapshot. See kubernetes/apps/home-assistant/
+# PROPOSAL-mosquitto-credential-rotation.md (PR #453 — not yet merged; this
+# file lands with it) for the full plan and phasing.
+#
+# `health` deliberately goes first: it is the broker's OWN liveness/readiness
+# probe credential, consumed by exec probes inside the mosquitto container via
+# the same Secret that seeds the passwd file. Nothing outside the pod holds it,
+# so a rotation needs no coordination with any client — the smallest possible
+# blast radius for proving rotator -> OpenBao -> ESO -> Reloader end to end.
+#
+# Runs the secret-rotator binary (random mode): generate 32 random bytes,
+# standard-base64 encode, and merge-patch into OpenBao. assert_fields_exist runs
+# FIRST, so a misconfigured path fails safe (no rotation) rather than adding a
+# stray field — which also means this ROTATES an existing value and never
+# bootstraps one. Seeding is a one-time out-of-band step, documented in
+# policies/rotator-mosquitto-health.hcl.
+#
+# Base64 output is safe for this credential specifically: the probes pass it as
+# `-P "$HEALTH_PASSWORD"` to mosquitto_pub/sub inside the container, so the `+`,
+# `/` and `=` characters are never parsed by a URL or a config file. That is NOT
+# automatically true of the other seven users (sml2mqtt and open3e templates on
+# the edge host, Shelly device firmware, the HA config entry), so the charset
+# must be re-checked per user before enrolling each one — the old bootstrap Jobs
+# generated alphanumeric-only passwords with `tr -dc 'A-Za-z0-9'`.
+#
+# openbao-ns pattern: the Job runs in the openbao namespace, reaching the
+# OpenBao API intra-namespace — no NetworkPolicy needed. It writes OpenBao only;
+# ESO syncs mosquitto-health-credentials (ns home-assistant) and Reloader
+# restarts the broker so the passwd file is rebuilt from the new value.
+#
+# Rotation is not instant: it completes only after the ExternalSecret's
+# refreshInterval elapses and the broker rolls. During that roll two broker pods
+# with different passwd files sit behind the LB (maxUnavailable: 0/maxSurge: 1),
+# so a probe against the outgoing pod can fail auth briefly. Self-healing.
+#
+# ON-DEMAND: kubectl -n openbao create job --from=cronjob/mosquitto-health-secret-rotate mosquitto-health-secret-rotate-now
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: mosquitto-health-secret-rotator
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: mosquitto
+    app.kubernetes.io/component: secret-rotator
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: mosquitto-health-secret-rotate
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: mosquitto
+    app.kubernetes.io/component: secret-rotator
+spec:
+  # Quarterly, offset one month from paperless (which runs Jan/Apr/Jul/Oct) so
+  # unrelated rotations do not land in the same window and complicate diagnosis.
+  schedule: "0 4 1 2,5,8,11 *"
+  timeZone: Europe/Berlin
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 2
+  failedJobsHistoryLimit: 5
+  startingDeadlineSeconds: 3600
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 300
+      ttlSecondsAfterFinished: 259200 # 3 d
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: mosquitto
+            app.kubernetes.io/component: secret-rotator
+        spec:
+          serviceAccountName: mosquitto-health-secret-rotator
+          restartPolicy: Never
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 65534
+            runAsGroup: 65534
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: rotate
+              # Dated tag, per the convention documented in kustomization.yaml —
+              # every rotator here now uses the same one. Verified for this tag:
+              # it pulls, `random` is among its modes, and it carries the
+              # RANDOM_BYTES>=16 floor documented below (RANDOM_BYTES=8 exits
+              # "refusing to generate a weak secret").
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              args: ["random"]
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities:
+                  drop: ["ALL"]
+              env:
+                - name: BAO_ADDR
+                  value: http://openbao.openbao.svc.cluster.local:8200
+                - name: BAO_ROLE
+                  value: rotator-mosquitto-health
+                - name: BAO_PATH
+                  value: homelab/apps/home-assistant/mosquitto-health-credentials
+                # Matches the key the ExternalSecret projects and the key the
+                # Secret already carries, so mosquitto-deployment.yaml's
+                # secretKeyRef is unchanged.
+                - name: BAO_FIELDS
+                  value: "password"
+                # 32 bytes -> 44 base64 chars. The binary refuses anything below
+                # 16 bytes outright.
+                - name: RANDOM_BYTES
+                  value: "32"
+              resources:
+                requests:
+                  cpu: 25m
+                  memory: 16Mi
+                limits:
+                  memory: 48Mi
@@ -59,7 +59,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:49bfddd8eeb5fd871cbeb05bc3b4e2fcaaab7d9e24cc3007caae6084b4634c18
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["netbird-key"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:97c22bcff324a93d2bb7bf3adb2894b405ed1a8035d02ad1ff6baced3813cefe
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["proxmox-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -81,7 +81,7 @@ spec:
              type: RuntimeDefault
          initContainers:
            - name: rotate-mint
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["gitea-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -128,7 +128,7 @@ spec:
                  memory: 48Mi
          containers:
            - name: sync-woodpecker
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["woodpecker-secret"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
+              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false