fix(multus): drop runtime apk add from the CNI plugins installer

Owner directive: tools belong in the image, not installed at container start. This is the highest-blast-radius instance of the pattern left in the repo — a DaemonSet, so it runs on every node. No image was needed. `apk add --no-cache curl tar` turns out to have been unnecessary from the start: verified in alpine:3.21 that `tar` is already busybox (v1.37.0) and handles -xzf, and that busybox `wget` follows GitHub's release redirect to objects.githubusercontent.com over TLS using the base image's /etc/ssl/certs/ca-certificates.crt. Only `curl` is genuinely absent, and wget covers what it was used for. So this replaces the install with `wget -q -O` rather than introducing another layered image. Severity note for the record: the apk add sat inside the else branch, so steady-state nodes never reached it. It ran when plugins were absent — i.e. on a fresh node or after /opt/cni/bin was wiped, which is precisely the moment you least want a dependency on package-index availability. Verified by running the RENDERED init script in alpine:3.21 against the real release URL, both paths: empty dir -> Fetching ...; Installed: host-local, macvlan, static macvlan executable, /opt/cni/bin owner 0:0, exit 0 already there -> "CNI plugins already installed (CNI macvlan plugin v1.5.1)", exit 0, no fetch The second line is worth noting: that version string comes from executing the extracted binary, so the download and extraction are confirmed good, not merely present. The chown 0:0 also still applies, which the existing comment explains is required or Cilium's apply-sysctl-overwrites init container fails with "Permission denied". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 13:47:11 +02:00
49 changed files with 180 additions and 1601 deletions
@@ -159,61 +159,3 @@ steps:
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest
-
-  # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
-  # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
-  # node-side fetch from GitHub that had NO checksum verification and sat on the
-  # critical path of every node rebuild. The download now happens here, once,
-  # where the sha256 is checked — see the Containerfile for the full rationale.
-  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/cni-plugins
-    depends_on: []
-    when:
-      - event: manual
-      - event: push
-        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
-
-  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
-  # needing s_client/x509. Named for what it is; first consumer is the paperless
-  # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
-  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
-  # base, so this one needs an image rather than a busybox substitute.
-  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/alpine-openssl
-    depends_on: []
-    when:
-      - event: manual
-      - event: push
-        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
@@ -1,254 +0,0 @@
-# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
-#
-# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
-# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
-# so a broken manifest reached the cluster and was caught by Flux failing to
-# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
-# have been caught here instead:
-#
-#   * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
-#     which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
-#     it happily — it is valid YAML, just a map instead of a string — so
-#     "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
-#   * a rotator pinned to an image digest that no longer existed (#458).
-#   * manifests whose kustomize wiring was only ever checked by hand.
-#
-# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
-# here is a static check that either passes or fails on the checkout alone, so it
-# can gate PRs without becoming a bottleneck or a flake source.
-when:
-  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
-  # step on one already-cached image, so running it everywhere is free; this one
-  # pulls three images and builds every kustomization in the tree, which is real
-  # cost to put on a docs-only PR. The filter covers everything the steps actually read:
-  # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
-  # so an edit to this file re-runs its own checks).
-  - event: pull_request
-    path:
-      - "kubernetes/**"
-      - ".woodpecker/**"
-
-# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
-# the same shape — `find`, a floor guard against empty discovery, then a
-# file-fed `while read`. Two copies is deliberate: the two traps being commented
-# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
-# where `fail` cannot propagate) are different, and both are worth spelling out
-# where they bite. A third instance is the point to stop copying and factor it
-# into a shared script under `scripts/ci/` — which is already covered by
-# ci-scripts-test.yml, so it would gain unit tests in the process.
-#
-# Every step carries `depends_on: []` so all four run in PARALLEL and every
-# failure shows in one run. Woodpecker's default is sequential, which would stop
-# at the first failure and make a contributor fix-and-repush to discover the
-# next — and these four are independent, cheap, offline checks with no ordering
-# between them. Keep it on every step: `depends_on` is per-step, so dropping it
-# from one silently re-serialises that step behind the others.
-steps:
-  # Every kustomization must build. This is the check that catches a resource
-  # entry pointing at a file that does not exist, a bad patch target, or YAML
-  # that is fine in isolation but broken once overlaid — the class of error that
-  # otherwise surfaces as a Flux reconcile failure after merge.
-  #
-  # Verified green across every kustomization in the tree when this was added,
-  # so this gates new breakage rather than pre-existing debt.
-  #
-  # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml,
-  # alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is
-  # staleness in those four, not a deliberate split here — everything should be on
-  # the newest dated tag. They are not bumped here because none of them are
-  # triggered by this PR, so it would ship untested changes under cover of a green
-  # run that never touched them. Tracked in #461.
-  - name: kustomize-build
-    depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
-    commands:
-      - |
-        set -eu
-        # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
-        # image ships BusyBox find (v1.37.0), which rejects it outright with
-        # "find: unrecognized: -printf". Because the result feeds a command
-        # substitution used as a word list, the failure is SILENT — empty output,
-        # zero loop iterations, fail=0, step exits green having validated
-        # nothing. That is exactly what this pipeline shipped in its first
-        # revision, and CI reported it as passing.
-        dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)
-
-        # Assert the discovery actually found something. Without this, any future
-        # breakage of the find/sed above degrades back into a green no-op rather
-        # than a failure — the specific trap that made the first revision of this
-        # step useless while looking healthy.
-        count=$(printf '%s\n' "$dirs" | grep -c . || true)
-        if [ "$count" -lt 50 ]; then
-          echo "FAIL: found only $count kustomizations; floor is 50."
-          echo "      If the tree legitimately shrank, lower this floor in"
-          echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
-          echo "      Otherwise discovery is broken — refusing to report success."
-          exit 1
-        fi
-        echo "discovered $count kustomizations"
-
-        # `while read` fed from a FILE, not a pipe: a pipe would run the loop in
-        # a subshell where `fail` cannot propagate, so a failing build would exit
-        # 0 — the same silent-pass shape this step already shipped once. Reading
-        # a file keeps the loop in the current shell, and read -r handles paths
-        # containing spaces without IFS games.
-        printf '%s\n' "$dirs" > /tmp/dirs
-        fail=0
-        while IFS= read -r d; do
-          [ -n "$d" ] || continue
-          if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
-            echo "ok    $d"
-          else
-            echo "FAIL  $d"
-            sed 's/^/        /' /tmp/err
-            fail=1
-          fi
-        done < /tmp/dirs
-        exit $fail
-
-  # Lint the pipeline definitions themselves. This is the one that would have
-  # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
-  # fails to compile, and exits 0 on mere warnings — so it fails on real breakage
-  # without blocking on pre-existing noise. Both behaviours were verified against
-  # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
-  #
-  # An earlier revision of this comment also asserted that several deploy-*.yml
-  # files emit "Additional property timeout is not allowed" warnings. That was
-  # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
-  # schema drift in a 7-minor-old CLI rather than real debt in those files, which
-  # is one of the reasons this pin moved to latest stable. It cannot be confirmed
-  # from a green run either, because the loop below sends lint output to /tmp/out
-  # and prints it only on failure, so warnings are invisible when the step passes.
-  # The claim is dropped rather than restated: what this step guarantees is the
-  # exit-code contract above, which is what the gate actually rests on.
-  # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
-  #
-  # It matters here because the CLI compiles the config against its own schema:
-  # one behind the server can pass a config the server then rejects, or reject
-  # one it accepts, and the gate stops predicting what the server will do —
-  # which is its only purpose.
-  #
-  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
-  # shell, and the loop below needs `for`, `sed` and `echo` around the binary.
-  # A step using the plain tag fails to start rather than failing a check.
-  #
-  # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
-  # image-automation; this pin cannot, so the two will drift apart.
-  # Image-automation can only patch YAML that is a Kubernetes manifest —
-  # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
-  # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
-  # pipeline file has neither, so an `$imagepolicy` marker here is silently
-  # ignored. That is a property of the FILE, not of `update.path`: widening the
-  # automation's path to `./` would buy nothing, and would additionally drag the
-  # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
-  # like the fix and is not.
-  #
-  # So this bump is manual, and drift is caught by the update-audit agent
-  # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
-  # whenever it trails the deployed server. If you bump the server, bump here.
-  - name: woodpecker-lint
-    depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
-    commands:
-      - |
-        set -eu
-        fail=0
-        for f in .woodpecker/*.yml; do
-          if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
-            echo "ok    $f"
-          else
-            echo "FAIL  $f"
-            sed 's/^/        /' /tmp/out
-            fail=1
-          fi
-        done
-        exit $fail
-
-  # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
-  # mounted via configMapGenerator). Severity is pinned to `error` on purpose:
-  # style and info findings would fail the build on existing scripts for no
-  # safety gain. All scripts in the tree are clean at this level today.
-  #
-  # NOTE this does NOT cover shell embedded in manifests (init containers, probe
-  # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
-  # where several real bugs lived this year, so it remains a gap worth closing
-  # separately rather than one this step quietly implies is covered.
-  - name: shellcheck
-    depends_on: [] # see note above `steps:`
-    image: koalaman/shellcheck-alpine:v0.11.0
-    commands:
-      - |
-        set -eu
-        files=$(find kubernetes -name '*.sh' | sort)
-        # Same no-op guard as kustomize-build: if discovery ever returns nothing,
-        # fail loudly instead of passing having checked no files.
-        #
-        # This floor is 1 while kustomize-build's is 50, and the asymmetry is
-        # deliberate rather than an oversight. That floor is loose relative to the
-        # tree it guards, so it catches both total failure and gross partial loss
-        # while tolerating normal churn. The shell-script set is small enough that
-        # any floor tight enough to catch PARTIAL loss would sit inside normal
-        # churn — deleting one script would fail CI spuriously, and the fix a
-        # hurried person reaches for is lowering the floor, which is how a guard
-        # quietly becomes decoration. A floor that has to be edited whenever the
-        # tree changes is not a guard, it is a snapshot with an exit code.
-        #
-        # So this catches the failure that actually happened here (discovery
-        # silently returning nothing, step green having checked no files) and not
-        # partial loss. Worth revisiting only if kubernetes/ grows a lot more
-        # shell, at which point a loose proportionate floor becomes possible.
-        count=$(printf '%s\n' "$files" | grep -c . || true)
-        if [ "$count" -lt 1 ]; then
-          echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
-          exit 1
-        fi
-        echo "discovered $count shell scripts"
-
-        # Same file-fed while-read reasoning as kustomize-build above.
-        printf '%s\n' "$files" > /tmp/files
-        fail=0
-        while IFS= read -r f; do
-          [ -n "$f" ] || continue
-          if shellcheck -S error "$f"; then
-            echo "ok    $f"
-          else
-            echo "FAIL  $f"
-            fail=1
-          fi
-        done < /tmp/files
-        exit $fail
-
-  # Guard the convention that cost a day to diagnose: registry-local images are
-  # pinned by DATED TAG, never by digest. Digests here are garbage-collected once
-  # :latest and the dated tags move past them — and that happens on edits to
-  # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
-  # silently orphans the manifest a rotator was pinned to. 11 rotators were dead
-  # this way and would have failed on their first scheduled run months later.
-  #
-  # Scoped to the rotation directory, which is where the convention is documented
-  # and where the failure actually bit.
-  #
-  # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
-  # documents this very rule, so its comment contains the literal string
-  # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
-  # does not prevent that — verified by running the bare grep, which returns
-  # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
-  # this step fails permanently on its own documentation.
-  - name: no-digest-pins
-    depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
-    commands:
-      - |
-        set -eu
-        hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
-                 | grep -v kustomization.yaml || true)
-        if [ -n "$hits" ]; then
-          echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
-          echo "$hits" | sed 's/^/        /'
-          exit 1
-        fi
-        echo "ok    no digest pins under openbao/rotation/"
@@ -1,18 +0,0 @@
---
-# Shared configuration for every NTP server (see plays/install-ntp-ha.yml).
-
-# IPv6 service address -- what ntp.srv.renner.uno resolves to, and the path
-# clients actually use. The IPv4 VIP is legacy.
-#
-# Defined at group level rather than in the play because install-ntp-ha.yml runs
-# two plays over this group -- withdraw the static assignments fleet-wide, then
-# hand the address to keepalived -- and both need the same value. Duplicating a
-# service address across plays is exactly the sort of thing that drifts.
-ntp_vip6: "fd40::123/128"
-
-# Link-local address carried as the FIRST virtual_ipaddress of the IPv6 VRRP
-# instance. RFC 5798 gives an IPv6 virtual router a link-local primary, and
-# keepalived warns "the first IPv6 VIP address should be link local" without
-# one. fe80::123 mirrors the service address and cannot collide with a real
-# NIC's EUI-64 link-local.
-ntp_vip6_linklocal: "fe80::123/64"
@@ -1,19 +1,7 @@
-# Reached by name, like every other host in this file. These three were the last
-# entries still addressed by IPv4 literal; authentik, netbird, proxy, nas-1,
-# grafana and edge-1 have always used their FQDN.
-#
-# An address in an inventory is allocator state standing in for identity, and it
-# pins access to the legacy family -- the names resolve IPv6-first, which is the
-# primary path on this network. DNS is the authoritative source for where a host
-# is; this file should say which host, not where it currently sits.
-#
-# The short inventory_hostname is kept deliberately: plays key on it
-# (ntp_servers[inventory_hostname] in install-ntp-ha.yml, node names in opentofu,
-# the prometheus targets), so only ansible_host changes.
 [pve_cluster]
-jupiter ansible_host=jupiter.renner.uno ansible_user=root
-neptune ansible_host=neptune.renner.uno ansible_user=root
-saturn ansible_host=saturn.renner.uno ansible_user=root
+jupiter ansible_host=10.2.40.201 ansible_user=root
+neptune ansible_host=10.2.40.202 ansible_user=root
+saturn ansible_host=10.2.40.203 ansible_user=root

 [authentik_db]
 authentik-db-1.srv.renner.uno ansible_user=root
@@ -22,54 +22,12 @@
 # so it stays BACKUP while the PVE nodes exist and wins the VIP by default once
 # they are gone -- no config change required at cutover.

-# Phase 1. Every static assignment of the IPv6 service address must be gone
-# fleet-wide before ANY host claims it via VRRP.
-#
-# DAD is enabled on the VRRP address (deliberately -- under single-master VRRP a
-# duplicate is a real fault worth detecting). So a static copy on one host and a
-# VRRP-managed copy on another cannot coexist: the kernel marks the newcomer
-# `dadfailed deprecated`, and its master then believes it is serving while
-# carrying an unusable address. That is not hypothetical -- it is what edge-1
-# did on 2026-08-07 when it took the address while three PVE nodes still held it
-# statically.
-#
-# Deliberately NOT serial: this play must complete everywhere before phase 2
-# starts anywhere. Doing the withdrawal per host inside phase 2 cannot avoid the
-# overlap, because with serial: 1 the first host claims the address while the
-# rest are still static.
-#
-# There is a gap between the phases where no host holds the address. It is
-# bounded by one play's run time and clients simply retry.
- name: Withdraw static assignments of the NTP service address
-  hosts: ntp_cluster
-  become: true
-
-  vars:
-    keepalived_ntp_vip6: "{{ ntp_vip6 }}"
-    keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
-
-  tasks:
-    - name: Withdraw any static assignment of the service address
-      ansible.builtin.include_role:
-        name: keepalived-ntp
-        tasks_from: withdraw-static.yml
-      when: ntp_keepalived_enabled | default(false)
-
-# Phase 2. Configure chrony and hand the address to keepalived.
 - name: Install NTP Server (HA)
  hosts: ntp_cluster
  become: true
-  # One node at a time. The keepalived role withdraws the statically assigned
-  # IPv6 service address and restarts keepalived so the address is re-taken
-  # under VRRP control; doing that across every NTP server at once would leave
-  # the whole fleet with nobody holding fd40::123 until the elections settle.
-  # Serialised, the remaining members keep answering throughout.
-  serial: 1

  vars:
    ntp_vip: "10.2.40.4"
-    # ntp_vip6 / ntp_vip6_linklocal come from group_vars/ntp_cluster.yml, so
-    # both plays in this file see one definition.
    ntp_servers:
      jupiter:
        priority: 100
@@ -91,8 +49,6 @@
      when: ntp_keepalived_enabled | default(false)
      vars:
        keepalived_ntp_vip: "{{ ntp_vip }}"
-        keepalived_ntp_vip6: "{{ ntp_vip6 }}"
-        keepalived_ntp_vip6_linklocal: "{{ ntp_vip6_linklocal }}"
        keepalived_ntp_state: "{{ ntp_servers[inventory_hostname].state }}"
        keepalived_ntp_priority: "{{ ntp_servers[inventory_hostname].priority }}"
        keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
@@ -134,49 +90,6 @@
          {% endif %}
      when: ntp_keepalived_enabled | default(false)

-    # The v6 address is the path clients actually use, so it gets the same
-    # check as the v4 VIP. Without this a broken v6 rollout -- the address
-    # never claimed, or -nodad not behaving as expected on a given kernel --
-    # would not surface here at all, only later by reading
-    # /var/log/keepalived-state.log by hand.
-    #
-    # `ip -6 addr show ... to <addr>` prints the address when present and
-    # nothing when absent, so emptiness is the test.
-    #
-    # Presence cannot itself distinguish a VRRP-held address from a static
-    # leftover -- they look identical. It is only meaningful because the
-    # keepalived-ntp role asserts, on both distros, that no static assignment
-    # survives: the play fails before reaching here if one does.
-    - name: Check if this server holds the IPv6 service address
-      ansible.builtin.command:
-        argv:
-          - ip
-          - -6
-          - addr
-          - show
-          - dev
-          - "{{ ntp_keepalived_interface }}"
-          - to
-          - "{{ ntp_vip6 }}"
-      register: vip6_check
-      failed_when: false
-      changed_when: false
-      when: ntp_keepalived_enabled | default(false)
-
-    - name: Display IPv6 service address status
-      ansible.builtin.debug:
-        msg: >
-          {% if vip6_check.stdout | default('') | length > 0 %}
-          This server ({{ inventory_hostname }}) holds the IPv6 service address
-          {{ ntp_vip6 }} -- it is the v6 MASTER and is serving ntp.srv.renner.uno
-          {% else %}
-          This server ({{ inventory_hostname }}) does not hold {{ ntp_vip6 }}.
-          Expected on a v6 BACKUP. If NO host holds it, check_ntp is failing
-          everywhere -- inspect `chronyc tracking` and
-          /var/log/keepalived-ntp-health.log
-          {% endif %}
-      when: ntp_keepalived_enabled | default(false)
-
    - name: Report hosts serving NTP without participating in the VIP
      ansible.builtin.debug:
        msg: >
@@ -1,25 +1,9 @@
 ---
 # Chrony NTP Server Role Defaults

-# Networks allowed to use this server as NTP source.
-#
-# BOTH families are required. IPv6 is the primary address family here -- the
-# service address ntp.srv.renner.uno resolves to fd40::123 -- so an IPv4-only
-# allow list refuses every client that uses the documented name.
-#
-# The hosts carried `allow fd00::/8` for a long time without it ever being in
-# this file. Applying the role on 2026-08-07 rendered the template over it and
-# dropped the line on all four NTP servers at once, which took internal NTP
-# down over IPv6 until it was added here. Same class of untracked host state as
-# the anycast service address, and the same failure: config that exists only on
-# the box survives right up until something re-renders it.
+# Networks allowed to use this server as NTP source
 chrony_allowed_networks:
  - "10.2.0.0/16"
-  - "fd00::/8"
-  # Link-local. A client on the same segment may source its query from fe80::,
-  # and without this chrony refuses it -- the request arrives and is dropped,
-  # which looks like the server being unreachable rather than a policy refusal.
-  - "fe80::/10"

 # Primary NTP servers (PTB - Physikalisch-Technische Bundesanstalt, Stratum 1)
 chrony_upstream_servers:
@@ -34,26 +18,6 @@ chrony_upstream_pools:
 # Local stratum if disconnected from upstream
 chrony_local_stratum: 10

-# Step (rather than slew) the clock when the offset exceeds the threshold, but
-# only for the first N updates after chronyd starts. Without a makestep
-# directive chronyd only ever slews, so a large offset is never closed in
-# practice.
-#
-# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
-# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
-# that forward to its own built-in epoch (2026-06-26), and it ended up at
-# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
-# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
-# With host time before the cluster certificate's NotBefore, every outbound TLS
-# handshake failed x509 validation, taking out the MQTT bridge (heating
-# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
-# is just where the software flooring landed; the fault is that nothing steps.
-#
-# Limiting to the first few updates keeps steps confined to boot, so a
-# long-running server is never jumped out from under its workloads.
-chrony_makestep_threshold: 1.0
-chrony_makestep_limit: 3
-
 # Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
 # (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
 # error per interface when asked for it. Overridden to false in
@@ -14,11 +14,6 @@ pool {{ pool }}
 # Record the rate at which the system clock gains/losses time
 driftfile /var/lib/chrony/chrony.drift

-# Step the clock instead of slewing it when the offset is large, limited to the
-# first few updates after start so only boot-time corrections jump. Rationale
-# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
-makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
-
 # Allow NTP client access from local networks
 {% for network in chrony_allowed_networks %}
 allow {{ network }}
@@ -118,11 +118,8 @@ After the `kubernetes/apps/claude-workstation/` PR merges and Flux reconciles:
 NS=apps
 TOKEN=$(kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.token}' | base64 -d)
 kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt
-# server = the cluster's stable API endpoint (kubernetes/talos/patches/cluster.yaml).
-# Use the name, not an address: it is what the API server certificate carries as a
-# SAN, so an IP here fails TLS verification even when it is reachable. It must
-# resolve from the workstation over the site-to-site route.
-kubectl config set-cluster homelab --server=https://k8s.renner.uno:6443 \
+# server = the API endpoint reachable over the site-to-site route (a CP node or VIP):
+kubectl config set-cluster homelab --server=https://10.2.40.201:6443 \
  --certificate-authority=/tmp/ca.crt --embed-certs=true --kubeconfig=/tmp/claude-ws.kubeconfig
 kubectl config set-credentials claude-workstation --token="$TOKEN" --kubeconfig=/tmp/claude-ws.kubeconfig
 kubectl config set-context homelab --cluster=homelab --user=claude-workstation --kubeconfig=/tmp/claude-ws.kubeconfig
@@ -4,29 +4,6 @@
 keepalived_ntp_router_id: 4
 keepalived_ntp_vip: "10.2.40.4"
 keepalived_ntp_interface: "vmbr1.40"
-
-# IPv6 service address for NTP (ntp.srv.renner.uno). Empty disables the v6
-# instance entirely. IPv6 is the primary address family on this network, so
-# this is the path that actually matters -- the v4 VIP above is legacy.
-#
-# Held by keepalived rather than assigned statically by networkd so that it is
-# gated on check_ntp: present only while this host is fit to serve time, and
-# withdrawn the moment it is not. A statically configured address is carried
-# unconditionally from boot, which is how edge-1 came to answer NTP queries on
-# 2026-08-04 while its clock was ~16 days in the past.
-#
-# Needs its own virtual_router_id: it is a distinct VRRP instance, and a VRID
-# collision with the IPv4 instance on the same interface would have the two
-# fighting over a single election.
-keepalived_ntp_vip6: ""
-keepalived_ntp_router_id6: 6
-
-# Link-local address for the IPv6 VRRP instance, carried as its FIRST
-# virtual_ipaddress. RFC 5798 gives an IPv6 virtual router a link-local primary,
-# and keepalived warns "the first IPv6 VIP address should be link local" without
-# one. Chosen to mirror the service address rather than derived from the virtual
-# MAC, since no real NIC generates fe80::123 so there is nothing to collide with.
-keepalived_ntp_vip6_linklocal: "fe80::123/64"
 # NOTE: plaintext VRRP auth, unvaulted. Carried over as-is from the original
 # role; VRRP PASS auth is weak regardless (sent in the clear on the segment),
 # so this is a documented weakness rather than a secret worth vaulting.
@@ -41,17 +18,6 @@ keepalived_ntp_check_script: "/usr/local/bin/check-ntp-health.sh"
 keepalived_ntp_check_interval: 2
 keepalived_ntp_check_weight: -20

-# Maximum |system clock offset| in seconds before the host refuses to hold the
-# VIP. chronyd reports this as "System time : <n> seconds slow/fast of NTP
-# time". Anything above this means the host knows its clock is wrong and must
-# not serve time to clients -- see the reasoning in check-ntp-health.sh.j2.
-#
-# A serving NTP host should be within milliseconds of real time, so 1s is
-# already several orders of magnitude beyond normal. It is deliberately not
-# tighter: a brief excursion at boot, before makestep lands, should not flap
-# the VIP (fall 2 at a 2s interval also requires the failure to persist).
-keepalived_ntp_max_offset: 1.0
-
 # sysctl applied for VRRP. Default preserves the PVE hosts' existing behaviour
 # exactly. ip_forward is NOT appropriate everywhere -- on a host with legs in
 # two VLANs it turns the box into a router between them, bypassing the gateway's
@@ -1,39 +1,22 @@
 #!/bin/bash
 # Keepalived State Change Notification Script
-#
-# Two VRRP instances call this: NTP_VIP (IPv4, legacy) passes MASTER/BACKUP/
-# FAULT, and NTP_VIP6 (IPv6, the primary path) passes the same suffixed with
-# '6'. They are logged distinctly because the two can legitimately diverge --
-# a host can hold one family and not the other -- and a log line that does not
-# say which family it means is useless during an incident.

 TYPE=$1
 NAME=$(hostname)
 TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

-log() {
-    echo "$TIMESTAMP: $NAME $1" >> /var/log/keepalived-state.log
-    logger -t keepalived "$NAME $1"
-}
-
 case $TYPE in
    MASTER)
-        log "transitioned to MASTER state (NTP, IPv4)"
+        echo "$TIMESTAMP: $NAME transitioned to MASTER state (NTP)" >> /var/log/keepalived-state.log
+        logger -t keepalived "$NAME is now NTP MASTER"
        ;;
    BACKUP)
-        log "transitioned to BACKUP state (NTP, IPv4)"
+        echo "$TIMESTAMP: $NAME transitioned to BACKUP state (NTP)" >> /var/log/keepalived-state.log
+        logger -t keepalived "$NAME is now NTP BACKUP"
        ;;
    FAULT)
-        log "transitioned to FAULT state (NTP, IPv4)"
-        ;;
-    MASTER6)
-        log "transitioned to MASTER state (NTP, IPv6) - now serving ntp.srv.renner.uno"
-        ;;
-    BACKUP6)
-        log "transitioned to BACKUP state (NTP, IPv6) - released ntp.srv.renner.uno"
-        ;;
-    FAULT6)
-        log "transitioned to FAULT state (NTP, IPv6) - withdrew ntp.srv.renner.uno, unfit to serve time"
+        echo "$TIMESTAMP: $NAME transitioned to FAULT state (NTP)" >> /var/log/keepalived-state.log
+        logger -t keepalived "$NAME is now in NTP FAULT state"
        ;;
    *)
        echo "$TIMESTAMP: $NAME unknown state: $TYPE" >> /var/log/keepalived-state.log
@@ -35,121 +35,6 @@
    state: present
  when: ansible_facts['os_family'] == 'Archlinux'

-# --- Verify nothing else is assigning the IPv6 service address ---------------
-#
-# The removal itself lives in withdraw-static.yml, run as a separate fleet-wide
-# play before any host reaches this role (see install-ntp-ha.yml). It has to
-# complete everywhere first: DAD is enabled on the VRRP address, so a static
-# copy on one host and a VRRP-managed copy on another cannot coexist -- the
-# kernel marks the newcomer `dadfailed` and its master serves nothing.
-#
-# What remains here are the guards. They run on every execution, not just the
-# migration, and fail the play if any static assignment survives -- which would
-# both defeat the health gating (a static copy does not disappear when
-# keepalived decides the host is unfit) and break the VRRP copy via DAD.
-#
-# The config file is what gets checked, not the running address, and that is
-# what makes it safe to run every time: keepalived never writes to these files,
-# so anything found is by definition static, whereas the live address is
-# legitimately present on whichever host is master.
-#
-# Prefix-agnostic and \b-anchored on purpose. Matching the full
-# address/prefixlen would fail open against a hand-written assignment using any
-# other mask.
- name: Detect any remaining static assignment of the service address (ifupdown)
-  ansible.builtin.command:
-    argv:
-      - grep
-      - -rlE
-      - "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
-      - /etc/network/interfaces
-      - /etc/network/interfaces.d/
-  register: keepalived_ntp_static6_ifupdown
-  changed_when: false
-  failed_when: false
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - ansible_facts['os_family'] == 'Debian'
-
- name: Assert ifupdown is not also assigning the service address
-  ansible.builtin.assert:
-    that:
-      - keepalived_ntp_static6_ifupdown.stdout | default('') | length == 0
-    fail_msg: >-
-      {{ keepalived_ntp_vip6 }} is still assigned statically by ifupdown on
-      this host
-      ({{ keepalived_ntp_static6_ifupdown.stdout_lines | default([]) | join(', ') }}).
-      The automatic removal only matches the exact form
-      `up ip -6 addr replace {{ keepalived_ntp_vip6 }} dev <if> nodad`, so a
-      different prefix length or layout is left in place. Handing the address
-      to keepalived while ifupdown also assigns it means it would NOT be
-      withdrawn when check_ntp fails -- the host would keep serving time on an
-      address VRRP believes it has released. Remove the assignment from the
-      file(s) above, then re-run.
-    success_msg: >-
-      No static ifupdown assignment of {{ keepalived_ntp_vip6 }}; safe for
-      keepalived to own it.
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - ansible_facts['os_family'] == 'Debian'
-
-# On Arch (edge-1) the static assignment lives in a systemd-networkd file owned
-# by the edge-host role, so this role must not rewrite it -- but it must not
-# quietly proceed either. install-edge-host.yml is a separate play with no
-# ordering relationship to this one, so nothing guarantees it has been re-run
-# to drop the [Address] block. If it has not, edge-1 would end up holding
-# fd40::123 statically AND running a VRRP instance for the same address:
-# keepalived would believe it had withdrawn the address while networkd kept it
-# assigned, and the host would go on answering NTP with a clock the health
-# check had already rejected.
-#
-# That is the precise failure of 2026-08-04, on the host it happened to, so it
-# fails fast rather than being left to inspection.
-# Matched on the bare address, not the full address/prefixlen, and anchored
-# with \b so it cannot also hit a longer address sharing the prefix (fd40::1234).
-#
-# Searching for the literal "fd40::123/128" would fail open: a hand-written
-# [Address] using any other mask -- /64, say -- would not match, the assert
-# below would pass, and keepalived would be handed an address networkd is
-# still assigning. Since this check is detect-only, with no delete afterwards
-# to catch the mistake, a miss here is silent. The Debian path does not share
-# the problem: `ip -6 addr show ... to <prefix>` matches by range regardless of
-# the assigned prefix length, and its delete is re-verified by assert.
- name: Detect a static assignment of the service address (systemd-networkd)
-  ansible.builtin.command:
-    argv:
-      - grep
-      - -rlE
-      - "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
-      - /etc/systemd/network/
-  register: keepalived_ntp_static6_networkd
-  changed_when: false
-  failed_when: false
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - ansible_facts['os_family'] == 'Archlinux'
-
- name: Assert networkd is not also assigning the service address
-  ansible.builtin.assert:
-    that:
-      - keepalived_ntp_static6_networkd.stdout | default('') | length == 0
-    fail_msg: >-
-      {{ keepalived_ntp_vip6 }} is still assigned statically by
-      systemd-networkd on this host
-      ({{ keepalived_ntp_static6_networkd.stdout_lines | default([]) | join(', ') }}).
-      Handing the address to keepalived while networkd also assigns it means
-      it would NOT be withdrawn when check_ntp fails -- the host would keep
-      serving time on an address VRRP believes it has released.
-      Re-apply the edge-host role first, whose template no longer carries the
-      [Address] block: ansible-playbook -i inventory/ plays/install-edge-host.yml
-      --limit {{ inventory_hostname }} --tags vlan40
-    success_msg: >-
-      No static networkd assignment of {{ keepalived_ntp_vip6 }}; safe for
-      keepalived to own it.
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - ansible_facts['os_family'] == 'Archlinux'
-
 - name: Create keepalived configuration
  ansible.builtin.template:
    src: keepalived.conf.j2
@@ -1,134 +0,0 @@
---
-# Withdraw a static assignment of the IPv6 NTP service address.
-#
-# Included by install-ntp-ha.yml as a FIRST play across the whole ntp_cluster,
-# before any host is given the NTP_VIP6 instance. That ordering is the point of
-# splitting it out.
-#
-# DAD is enabled on the VRRP address (see keepalived.conf.j2), so a static copy
-# on any host and a VRRP-managed copy on another cannot coexist: the kernel sees
-# the duplicate and marks the newcomer `dadfailed`, leaving a master that
-# believes it is serving while carrying an unusable address. Observed on edge-1
-# on 2026-08-07, where keepalived reported MASTER while the address sat
-# `dadfailed deprecated tentative` because three PVE nodes still held it
-# statically.
-#
-# Doing the withdrawal per host inside the main role could not avoid that: with
-# serial: 1 the first host would claim the address via VRRP while the rest were
-# still static. Fleet-wide first, then claim.
-#
-# fd40::123 was assigned by hand on all four servers, in two different ways,
-# none of it in version control:
-#
-#   PVE nodes  an `up ip -6 addr replace ... nodad` hook in
-#              /etc/network/interfaces
-#   edge-1     an [Address] block in /etc/systemd/network/, owned by the
-#              edge-host role and removed by re-applying it -- not touched
-#              here, since this role must not rewrite another role's file
-#
-# There is a service gap between this play and the next, while no host holds
-# the address. It is bounded by the run time of one play and clients simply
-# retry; the alternative is the dadfailed state above, which does not resolve
-# on its own.
-
- name: Remove static assignment of the NTP service address (ifupdown)
-  ansible.builtin.lineinfile:
-    path: /etc/network/interfaces
-    regexp: '^\s*up\s+ip\s+-6\s+addr\s+(replace|add)\s+{{ keepalived_ntp_vip6 | regex_escape }}\b'
-    state: absent
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - ansible_facts['os_family'] == 'Debian'
-  register: keepalived_ntp_static6_removed
-
-# Everything below is gated on the config line having just been removed, so the
-# runtime delete happens once -- on the migration -- and never again.
-#
-# The gate is load-bearing. Once this has landed, the address is legitimately
-# present on whichever host is the elected v6 master, placed there by
-# keepalived, and a delete cannot tell that apart from a static assignment (the
-# address looks identical either way). Ungated, every later run would delete the
-# master's address and restart keepalived, dropping the service address and
-# forcing a re-election each time -- the flapping this whole change exists to
-# prevent.
- name: Check whether the service address is still assigned statically
-  ansible.builtin.command:
-    argv:
-      - ip
-      - -6
-      - addr
-      - show
-      - dev
-      - "{{ keepalived_ntp_interface }}"
-      - to
-      - "{{ keepalived_ntp_vip6 }}"
-  register: keepalived_ntp_vip6_present
-  changed_when: false
-  failed_when: false
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - keepalived_ntp_static6_removed.changed | default(false)
-
-# failed_when: false because the desired end state is "address absent". An
-# `ip addr del` against an address that has already gone is the outcome this
-# task wants, not a reason to abort the play for that host.
- name: Withdraw the statically assigned service address
-  ansible.builtin.command:
-    argv:
-      - ip
-      - -6
-      - addr
-      - del
-      - "{{ keepalived_ntp_vip6 }}"
-      - dev
-      - "{{ keepalived_ntp_interface }}"
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - keepalived_ntp_static6_removed.changed | default(false)
-    - keepalived_ntp_vip6_present.stdout | default('') | length > 0
-  failed_when: false
-
-# The withdrawal gets exactly one attempt -- the gate that makes it idempotent
-# also means it never retries -- and the delete swallows its own errors. So the
-# result is asserted rather than assumed.
-#
-# `ip addr del` matches on address AND prefix length: if a host's static
-# assignment used a different mask or interface than ntp_vip6 /
-# keepalived_ntp_interface, the delete is a silent no-op and the address stays
-# outside VRRP control permanently, with the play reporting success.
- name: Confirm the static service address was actually withdrawn
-  ansible.builtin.command:
-    argv:
-      - ip
-      - -6
-      - addr
-      - show
-      - dev
-      - "{{ keepalived_ntp_interface }}"
-      - to
-      - "{{ keepalived_ntp_vip6 }}"
-  register: keepalived_ntp_vip6_after
-  changed_when: false
-  failed_when: false
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - keepalived_ntp_static6_removed.changed | default(false)
-
- name: Assert the service address is no longer assigned outside keepalived
-  ansible.builtin.assert:
-    that:
-      - keepalived_ntp_vip6_after.stdout | default('') | length == 0
-    fail_msg: >-
-      {{ keepalived_ntp_vip6 }} is still assigned on
-      {{ keepalived_ntp_interface }} after the static assignment was removed.
-      `ip -6 addr del` matches address AND prefix length, so this usually means
-      the on-host assignment differs from ntp_vip6 or uses another interface.
-      Leaving it would both keep the address outside VRRP control and make the
-      VRRP-managed copy fail DAD elsewhere. Inspect with
-      `ip -6 addr show dev {{ keepalived_ntp_interface }}` and
-      `grep -rn {{ keepalived_ntp_vip6 }} /etc/network/ /etc/systemd/network/`.
-    success_msg: >-
-      {{ keepalived_ntp_vip6 }} withdrawn; safe for keepalived to claim it.
-  when:
-    - keepalived_ntp_vip6 | length > 0
-    - keepalived_ntp_static6_removed.changed | default(false)
@@ -6,80 +6,23 @@
 # The chrony unit is named 'chrony' on Debian and 'chronyd' on Arch. This is
 # templated rather than hardcoded: a wrong name here fails every check, which
 # parks the host in permanent FAULT so it never takes the VIP.
-#
-# This check must answer "is it safe for clients to take time from this host",
-# NOT merely "is chronyd alive". Those came apart on 2026-08-04: edge-1's dead
-# CMOS battery left its clock ~16 days in the past, chronyd was running,
-# answering queries and listening on 123, and the `local stratum` directive made
-# it advertise itself to clients as a valid reference. Every liveness check
-# passed for ~18h while the host stood ready to serve a clock from July. The
-# earlier version of this script ran `chronyc tracking` and discarded its
-# output -- the one command that knew the answer.
-#
-# Deliberate distinction, because these must not be conflated:
-#   * "provably wrong"  -> chronyd HAS upstream measurements and they say the
-#                          clock is far off. Fail: drop the VIP so a host with a
-#                          sane clock serves instead.
-#   * "no upstream yet" -> nothing reachable / not yet measured. Also fail, but
-#                          note that when the WAN is down every host fails
-#                          together, they all lose the same weight, relative
-#                          priority is unchanged and the VIP stays put. So the
-#                          local-reference fallback still works; what this
-#                          prevents is one host with a broken clock winning.
-
-LOG=/var/log/keepalived-ntp-health.log
-MAX_OFFSET={{ keepalived_ntp_max_offset }}
-
-fail() {
-    echo "$(date): $1" >> "$LOG"
-    exit 1
-}

 # Check 1: Is chrony service running?
 if ! systemctl is-active --quiet {{ chrony_service_name }}; then
-    fail "{{ chrony_service_name }} service not active"
+    echo "$(date): {{ chrony_service_name }} service not active" >> /var/log/keepalived-ntp-health.log
+    exit 1
 fi

-# Check 2: Is chrony responding, and what does it actually say?
-# Captured, not discarded -- the offset below is the whole point of this script.
-if ! TRACKING=$(chronyc tracking 2>/dev/null); then
-    fail "Chrony not responding to tracking query"
+# Check 2: Is chrony synchronized or at least running?
+if ! chronyc tracking >/dev/null 2>&1; then
+    echo "$(date): Chrony not responding to tracking query" >> /var/log/keepalived-ntp-health.log
+    exit 1
 fi

 # Check 3: Is NTP serving clients (port 123)?
 if ! ss -lun | grep -q ':123 '; then
-    fail "NTP port 123 not listening"
-fi
-
-# Check 4: Is the clock provably wrong?
-#
-# "System time : <n> seconds (slow|fast) of NTP time" is chronyd's own estimate
-# of how far the system clock is from real time, derived from upstream
-# measurements. On edge-1 during the incident this line read
-# "1376235.375000000 seconds slow of NTP time" while every other check passed.
-OFFSET=$(echo "$TRACKING" | awk '/^System time/ {print $4}')
-if [ -n "$OFFSET" ]; then
-    if awk -v o="$OFFSET" -v m="$MAX_OFFSET" 'BEGIN {exit !(o > m)}'; then
-        fail "System clock off by ${OFFSET}s (max ${MAX_OFFSET}s) - refusing to serve time"
-    fi
-fi
-
-# Check 5: Has chronyd ever validated its clock against a real source?
-#
-# With `local stratum N` configured, chronyd advertises itself as a usable
-# reference even when it has never synchronised to anything -- the upstream
-# chrony documentation is explicit that the local reference activates
-# immediately in that case. So a host that has never reached an upstream still
-# looks authoritative to clients. Refuse the VIP until a real source has been
-# selected.
-#
-# `chronyc sources` marks the selected source with '*' (best) or '+' (combined)
-# in the second column. Neither present means nothing upstream is accepted.
-if ! SOURCES=$(chronyc sources 2>/dev/null); then
-    fail "Chrony not responding to sources query"
-fi
-if ! echo "$SOURCES" | grep -qE '^.\*|^.\+'; then
-    fail "No upstream source selected - clock not validated, refusing to serve time"
+    echo "$(date): NTP port 123 not listening" >> /var/log/keepalived-ntp-health.log
+    exit 1
 fi

 # All checks passed
@@ -39,72 +39,3 @@ vrrp_instance NTP_VIP {
    notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP"
    notify_fault "/usr/local/bin/keepalived-notify.sh FAULT"
 }
-{% if keepalived_ntp_vip6 %}
-
-# IPv6 service address for NTP (ntp.srv.renner.uno). IPv6 is the primary
-# address family here, so this instance -- not the IPv4 one above -- is the
-# path that matters.
-#
-# A separate instance is mandatory, not stylistic: keepalived requires that
-# "all addresses in virtual_ipaddress must be of the same family", so the v6
-# address cannot be added to NTP_VIP.
-#
-# No authentication block: VRRPv3 removed authentication (RFC 5798), and
-# keepalived always uses version 3 for IPv6 instances. Configuring auth here
-# is rejected rather than ignored.
-#
-# The first address is link-local, and must be. RFC 5798 gives an IPv6 virtual
-# router a link-local address as its primary, and keepalived warns
-# "the first IPv6 VIP address should be link local" otherwise. It carries on,
-# but the instance is then sourcing VRRPv3 from something the protocol does not
-# expect.
-#
-# DAD is deliberately left ON (no -nodad). The flag was inherited from the
-# previous design, where every NTP server carried fd40::123 simultaneously and
-# duplicate detection had to be suppressed for that to work at all. Under
-# single-master VRRP the opposite is true: exactly one host should hold the
-# address, so a duplicate is a genuine fault and detecting it is the point.
-#
-# This does mean the migration cannot leave a static copy anywhere while a
-# VRRP-managed one exists -- DAD sees the two and marks the newcomer
-# `dadfailed`, leaving a master that believes it is serving while carrying an
-# unusable address. Observed on edge-1 on 2026-08-07. install-ntp-ha.yml
-# therefore withdraws every static assignment fleet-wide in a first play
-# before any host configures this instance.
-#
-# track_script is what makes this health-gated: the address is only held while
-# check_ntp passes, and is withdrawn as soon as it fails. On withdrawal
-# keepalived sends unsolicited Neighbour Advertisements so client neighbour
-# caches are updated immediately -- a statically assigned address cannot do
-# this, and clients would keep sending to the dead holder until their own
-# neighbour unreachability detection expired.
-vrrp_instance NTP_VIP6 {
-    state {{ keepalived_ntp_state }}
-    interface {{ keepalived_ntp_interface }}
-    virtual_router_id {{ keepalived_ntp_router_id6 }}
-    priority {{ keepalived_ntp_priority }}
-    advert_int 1
-
-    virtual_ipaddress {
-        {{ keepalived_ntp_vip6_linklocal }} dev {{ keepalived_ntp_interface }}
-        {{ keepalived_ntp_vip6 }} dev {{ keepalived_ntp_interface }}
-    }
-
-    # weight 0 overrides the script's own weight, and the distinction decides
-    # whether this works at all. A non-zero weight only SUBTRACTS from priority
-    # on failure -- so the last remaining host, having no one to lose the
-    # election to, stays MASTER and keeps serving time with a clock it has
-    # already reported as wrong. Weight 0 instead transitions the instance to
-    # FAULT after <fall> failures, which releases the address unconditionally.
-    #
-    # That is the required behaviour here: a host unfit to serve time must stop
-    # answering, whether or not a replacement exists.
-    track_script {
-        check_ntp weight 0
-    }
-
-    notify_master "/usr/local/bin/keepalived-notify.sh MASTER6"
-    notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP6"
-    notify_fault "/usr/local/bin/keepalived-notify.sh FAULT6"
-}
-{% endif %}
@@ -7,19 +7,19 @@

 # PVE cluster nodes - need direct scraping (no Alloy agent)
 - targets:
-    - 'jupiter.renner.uno:9100'
+    - '10.2.40.201:9100'
  labels:
    job: 'node'
    group: 'pve_cluster'
    hostname: 'jupiter'
 - targets:
-    - 'neptune.renner.uno:9100'
+    - '10.2.40.202:9100'
  labels:
    job: 'node'
    group: 'pve_cluster'
    hostname: 'neptune'
 - targets:
-    - 'saturn.renner.uno:9100'
+    - '10.2.40.203:9100'
  labels:
    job: 'node'
    group: 'pve_cluster'
@@ -51,9 +51,9 @@ scrape_configs:
    scrape_timeout: 55s
    static_configs:
      - targets:
-          - 'jupiter.renner.uno'
-          - 'neptune.renner.uno'
-          - 'saturn.renner.uno'
+          - '10.2.40.201'  # jupiter
+          - '10.2.40.202'  # neptune
+          - '10.2.40.203'  # saturn
    metrics_path: /pve
    params:
      module: [default]
@@ -132,29 +132,28 @@ scrape_configs:
        refresh_interval: 60s

  # Chrony Exporter - NTP metrics from PVE cluster
-  # One static_config per host so each carries its own hostname label. The
-  # previous form listed all three targets together and recovered the hostname
-  # in relabel_configs by regex-matching __address__ against the node's IP --
-  # which silently coupled the labels to the addressing. Changing a target
-  # would have left the regex unmatched and the hostname label simply absent,
-  # with the scrape still succeeding.
  - job_name: 'chrony'
    static_configs:
-      - targets: ['jupiter.renner.uno:9123']
+      - targets:
+          - '10.2.40.201:9123'
+          - '10.2.40.202:9123'
+          - '10.2.40.203:9123'
        labels:
          group: 'ntp'
          type: 'time-server'
-          hostname: 'jupiter'
-      - targets: ['neptune.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'neptune'
-      - targets: ['saturn.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'saturn'
+    relabel_configs:
+      - source_labels: [__address__]
+        regex: '10\.2\.40\.201:.*'
+        target_label: hostname
+        replacement: 'jupiter'
+      - source_labels: [__address__]
+        regex: '10\.2\.40\.202:.*'
+        target_label: hostname
+        replacement: 'neptune'
+      - source_labels: [__address__]
+        regex: '10\.2\.40\.203:.*'
+        target_label: hostname
+        replacement: 'saturn'

  # Immich photo management metrics
  - job_name: 'immich'
@@ -59,22 +59,6 @@ For each, find the **current pin in the repo** (Read/Glob/Grep), then look up th
 5. **Images deliberately excluded from automation** — listed in the
   image-automation `README.md` (e.g. vsftpd, otbr, ical-proxy) and any image with
   no `# {"$imagepolicy": ...}` marker. Check whether a newer usable tag exists.
-   This includes **CI step images in `.woodpecker/*.yml`**, which are all manual:
-   image-automation cannot patch those files at all (they are not Kubernetes
-   manifests — no `apiVersion`/`kind` — so markers there are ignored), and no
-   amount of `update.path` widening changes that. Grep `.woodpecker/` for
-   `image:` and check each pin.
-
-   Give `woodpeckerci/woodpecker-cli` in `.woodpecker/ci-manifests-test.yml`
-   particular attention: the woodpecker **server** IS auto-bumped (`3.x`), the
-   CLI is not, so they silently drift apart. The CLI lints pipeline configs
-   against its own schema, so one behind the server can pass a config the server
-   then rejects — the gate stops predicting server behaviour. Report the CLI as
-   needing an update whenever it trails the server version, not only when it
-   trails the newest CLI release. Its tag carries an `-alpine` suffix (the plain
-   tag is scratch-based with no shell and would break the step) — compare the
-   `vX.Y.Z` part, and ignore the `next-*` / `pull_*` development tags, which are
-   the newest by date on that repo and are not releases.
 6. **Ansible roles & OpenTofu providers** — `ansible/requirements.yml` (Galaxy
   roles/collections) and `opentofu/providers.tf` + `opentofu/.terraform.lock.hcl`.
   Report any with a newer published version.
@@ -1,21 +0,0 @@
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Alpine plus the openssl CLI. Nothing application-specific — named for what it
-# is, so anything needing `openssl s_client` / `x509` from a shell script can use
-# it.
-#
-# It exists because installing tools at container start is not acceptable here:
-# `apk add` makes every start depend on network and package-index availability,
-# and the resolved version can drift between restarts of an otherwise pinned
-# image.
-#
-# Note the openssl CLI is genuinely absent from the alpine base — verified, the
-# base carries the libraries but not the command — so unlike the multus CNI
-# installer (where busybox wget/tar already covered the need) this one really
-# does require an image rather than just dropping the apk add.
-#
-# First consumer: the paperless fetch-bridge-cert init container, which captures
-# the Proton Mail Bridge's self-signed IMAP certificate via STARTTLS and pins it
-# for paperless to trust. busybox has no s_client equivalent, so openssl is not
-# substitutable there.
-RUN apk add --no-cache openssl
@@ -1,68 +0,0 @@
-# Upstream containernetworking/plugins binaries, baked in at build time.
-#
-# WHY THIS IMAGE EXISTS
-# The multus cni-plugins-installer DaemonSet used to fetch this tarball from
-# GitHub on the node itself, at node-provisioning time, and extract it straight
-# into /opt/cni/bin. Two problems with that:
-#
-#   1. Security. The download had NO integrity verification — wget, then tar,
-#      into a directory whose contents are executed as root in the CNI path on
-#      every node. A tampered or MITM'd artifact would have got privileged
-#      execution cluster-wide, with nothing to detect it.
-#   2. Node rebuild. It put a live dependency on github.com (and previously the
-#      Alpine package index too) on the critical path of bringing a node up, so
-#      a GitHub outage or slow transfer stalled or broke CNI installation
-#      exactly when a node was being rebuilt.
-#
-# Baking the binaries in fixes both: the fetch happens once, here, where the
-# checksum IS verified, and the installer becomes a local file copy with no
-# network at all.
-#
-# The version and digest are pinned literals, so bumping them is a reviewable
-# commit rather than a silent change in what runs on every node. The digest was
-# cross-checked against upstream's own published
-# cni-plugins-linux-amd64-<version>.tgz.sha256 asset rather than merely recorded
-# from a local download — a download can be tampered with; the point of the
-# check is to compare against what upstream states.
-#
-# amd64 only, matching the ARCH the installer already hardcoded and the cluster's
-# node architecture. Add a second stage if a different arch is ever needed.
-
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d AS fetch
-
-ARG CNI_PLUGINS_VERSION=v1.5.1
-ARG CNI_PLUGINS_SHA256=77baa2f669980a82255ffa2f2717de823992480271ee778aa51a9c60ae89ff9b
-
-# busybox wget and tar are already in the base; nothing is installed here
-# either. sha256sum -c fails the BUILD if the artifact does not match, which is
-# the whole point — a bad download must never reach a node.
-RUN set -eu; \
-    F="cni-plugins-linux-amd64-${CNI_PLUGINS_VERSION}.tgz"; \
-    wget -q -O "/tmp/${F}" \
-      "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/${F}"; \
-    echo "${CNI_PLUGINS_SHA256}  /tmp/${F}" | sha256sum -c -; \
-    mkdir -p /plugins; \
-    tar -xzf "/tmp/${F}" -C /plugins; \
-    rm -f "/tmp/${F}"; \
-    # Drop the non-executable payload: the tarball ships LICENSE and README.md,
-    # which the old installer extracted into /opt/cni/bin alongside the binaries.
-    rm -f /plugins/LICENSE /plugins/README.md; \
-    # Fail loudly here rather than shipping an image that silently installs
-    # nothing, if upstream ever restructures the archive.
-    #
-    # This list must cover every plugin the repo actually depends on, not just
-    # the ones the installer happens to log. `dhcp` is included because the
-    # sibling cni-dhcp-daemon DaemonSet runs that binary and documents that it
-    # ships from here (see kubernetes/infrastructure/multus/cni-dhcp-daemon.yaml)
-    # — without it in the check, upstream dropping `dhcp` while keeping the
-    # others would build green and only surface later as a broken DHCP daemon on
-    # whichever node rolled first.
-    for p in macvlan static host-local dhcp; do \
-      [ -x "/plugins/$p" ] || { echo "FATAL: /plugins/$p missing or not executable" >&2; exit 1; }; \
-    done
-
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Alpine rather than scratch: the installer needs a shell to copy the binaries
-# onto the host. busybox covers cp/mv/chown/ls, so nothing is added.
-COPY --from=fetch /plugins/ /opt/cni-plugins/
@@ -1,179 +0,0 @@
-# Proposal: enrol the mosquitto broker credentials in Class-A rotation
-
-**Status:** proposal, nothing implemented.
-**Scope:** the 8 MQTT broker users and the credentials behind them.
-
-## Problem
-
-Every self-generated secret in this cluster rotates automatically except these.
-`kubernetes/infrastructure/openbao/rotation/` holds 13 manifests defining 15
-rotator CronJobs (`home-assistant.yaml` alone defines three). Five are **Class-A**
-— authentik, crowdsec, paperless, shelly-manager and woodpecker — meaning
-"self-generated random secret", which is exactly what a broker password is.
-The mosquitto credentials are the only Class-A-shaped secrets not enrolled.
-
-They are currently minted once by seven near-identical bootstrap Jobs
-(`mosquitto-bootstrap.yaml`, 508 lines — the ServiceAccount, Role and RoleBinding
-are inside that same file) which generate a password, hand-build Secret JSON with
-`jq` and POST it to the Kubernetes API with `curl`. Once created they never
-change again.
-
-Three consequences:
-
-1. **They never rotate.** Not on a schedule, not on compromise, not ever.
-2. **They exist only in the Secret object** — not in OpenBao, so not covered by
-   the OpenBao snapshot CronJob, and unrecoverable if lost.
-3. **The Jobs cannot be modified.** A Job's `spec.template` is immutable, so
-   changing even the container image makes the whole `home-assistant`
-   Kustomization fail to reconcile with `field is immutable`. Any edit requires
-   deleting or renaming the Jobs.
-
-## Why the obvious fixes do not work
-
-Recorded because each was tried and rejected on evidence, and the reasons are
-not obvious from the code.
-
-| Approach | Why it fails |
-| --- | --- |
-| ExternalSecrets reading hand-seeded OpenBao paths | Depends on content copied in by hand. Git cannot recreate it; a fresh OpenBao leaves the paths empty and ESO then overwrites live broker credentials with nothing. |
-| ESO `Password` generator | **Overwrites pre-existing Secrets.** Verified: a Secret holding `KNOWN-PREEXISTING-VALUE` was replaced with a freshly generated password on first sync. Adopting it rotates the 7 Job-minted credentials at once, uncoordinated (`evcc` is already ESO-extract-backed and so unaffected). |
-| Keep the Jobs, fix only the image | Blocked by `spec.template` immutability. Renaming the Jobs works but is a hack that recurs on every future edit. |
-
-The common thread: each treats the credential as something to *place*, when the
-real requirement is a credential that can be *replaced*.
-
-## How rotation actually has to work here
-
-The broker rebuilds its password file from scratch on **every pod start**
-(`mosquitto-deployment.yaml`): `rm -f /mosquitto/auth/passwd` followed by
-`mosquitto_passwd -c -b …` for each of the 8 users, sourced from env vars backed
-by the 8 Secrets.
-
-Two things follow, and they shape the whole design:
-
- **A Secret change has no effect until the broker restarts.** There is no
-  Reloader annotation on the mosquitto Deployment today, unlike the consumers of
-  the existing Class-A rotators.
- **There is no make-before-break.** A mosquitto user has exactly one password.
-  The LLAT rotators can mint a second token and revoke the first afterwards; a
-  broker user cannot hold old and new simultaneously. Unless a transitional
-  second user is created, broker and client must cut over together.
-
-### Who holds each credential
-
-This is the part that determines cutover cost, and it is not visible from the
-manifests alone.
-
-| Broker user | Held by | Cutover |
-| --- | --- | --- |
-| `health` | the broker's own liveness/readiness probes | in-cluster, same pod restart |
-| `zigbee2mqtt` | zigbee2mqtt Deployment (`secretKeyRef`) | in-cluster, pod restart |
-| `evcc` | evcc app; already OpenBao-backed via ESO | in-cluster, pod restart |
-| `sml2mqtt` | edge host, from `vault_edge_sml2mqtt_mqtt_password` | re-run the edge-host play |
-| `open3e` | edge host, from `vault_edge_open3e_mqtt_password` | re-run the edge-host play |
-| `homeassistant` | HA MQTT integration — a config entry created in the UI | needs an API/WebSocket update, or a manual re-entry |
-| `shellies` | Shelly device firmware | pushed per-device; `shelly-manager` is deployed and is the natural mechanism |
-| `admin` | a workstation client, held by a human | read the new value from OpenBao |
-
-Note `sml2mqtt` and `open3e` currently keep **the same password in two unlinked
-places** — the Kubernetes Secret for the broker, and `ansible-vault` for the
-client — synchronised by hand. That duplication, not any client limitation, is
-what makes rotation feel impossible today.
-
-## Proposed design
-
-Follow the existing Class-A pattern exactly (`rotation/paperless.yaml` is the
-closest model): *generate random bytes → merge-patch into OpenBao → ESO syncs the
-Secret → restart the consumer*.
-
-1. **One rotator CronJob per broker user**, running the existing `secret-rotator`
-   binary in random mode, writing `apps/home-assistant/mosquitto-<user>-credentials`
-   in OpenBao. Reuses the established policy/role/SA pattern; no new machinery.
-2. **ExternalSecrets** project each OpenBao entry to the existing Secret name and
-   key, so `mosquitto-deployment.yaml` and the in-cluster consumers are unchanged.
-   Key names stay as they are (`homeassistant-password` on one, `password` on the
-   rest) — consumers reference them, so normalising is a separate change.
-3. **Add a Reloader annotation to the mosquitto Deployment** so a Secret change
-   restarts the broker, instead of silently taking effect at some unrelated future
-   restart.
-4. **Move the edge host off `ansible-vault` for these two values**, reading the
-   same OpenBao entry instead, so there is one source of truth per credential.
-5. **Rotate one user at a time.** Each rotation then affects exactly one client,
-   not all eight.
-
-### The cutover window
-
-With no make-before-break, each rotation has a window where broker and client
-disagree. Two options:
-
- **Accept a brief window** (recommended for most users). Order: rotator writes
-  OpenBao → client picks up the new value → broker restarts. MQTT clients
-  reconnect automatically; the window is seconds. Because rotation is per-user,
-  only that one client is affected.
-
-  One failure mode to expect rather than be alarmed by: the broker rolls with
-  `maxUnavailable: 0` / `maxSurge: 1`, so mid-rollout **two broker pods with
-  different passwd files sit behind the LB**. A client reconnecting during that
-  window can land on the old pod and fail auth until the rollout completes. It is
-  self-healing and lasts seconds to a minute, but it will look alarming in logs
-  during the first `health` / `zigbee2mqtt` rotation, so it is worth knowing in
-  advance.
- **Transitional dual user** (`<user>` and `<user>-next` in the passwd file and
-  ACL) for any credential where even a brief outage is unacceptable. More moving
-  parts; probably only worth it for `homeassistant`, which carries everything.
-
-## Suggested phasing
-
-Ordered so the risky work happens after the pattern is proven, and so each phase
-is independently valuable.
-
-1. **`health` and `zigbee2mqtt` first.** Both are entirely in-cluster and pick up
-   a new password on pod restart with no external coordination. Proves rotator →
-   OpenBao → ESO → Reloader end to end with the smallest blast radius. `health`
-   is the safest possible start: only the broker's own probes consume it.
-2. **`evcc`.** Already OpenBao-backed via ESO; only the rotator is missing.
-3. **`sml2mqtt` and `open3e`**, together with switching the edge host to read from
-   OpenBao. This is where the duplicate-source-of-truth problem gets fixed.
-4. **`shellies`**, via shelly-manager pushing to devices. Needs investigation:
-   shelly-manager has its own Class-A rotator today, but that rotates
-   shelly-manager's own secret and does not touch device MQTT credentials.
-5. **`homeassistant`** last. Highest blast radius, and the config entry is UI-created,
-   so it needs an API/WebSocket update path — the HA LLAT rotators
-   (`rotation/home-assistant.yaml`) already talk to HA over WebSocket and are the
-   obvious precedent.
-6. **`admin`** whenever convenient; a human reads the new value from OpenBao.
-
-Only once a user is enrolled does its bootstrap Job become dead code. Delete the
-Jobs incrementally as their users migrate, rather than in one step — that also
-avoids the immutability problem entirely, since the Jobs are deleted, never
-patched.
-
-## Open questions
-
- **Does `secret-rotator` random mode fit unchanged?** The existing rotators write
-  a field into an OpenBao entry; here the entry holds a single password field, so
-  it should, but this needs confirming against the binary rather than assumed.
- **Can shelly-manager push MQTT credentials to devices**, and does it cover both
-  Gen1 (`/settings/mqtt`) and Gen2+ (`Mqtt.SetConfig`) firmwares? If not, the
-  `shellies` user may need a different mechanism or stay manual longest.
- **How is the HA MQTT config entry updated programmatically?** Reusing the
-  WebSocket approach from the LLAT rotator is plausible but unverified.
- **Rotation cadence.** The existing rotators' schedules should be the reference
-  rather than inventing a new one.
- **ESO refresh latency.** Rotator-writes-OpenBao → Secret-updates →
-  Reloader-restarts is delayed by up to one `refreshInterval`; the `evcc`
-  ExternalSecret uses `1h`. That does not widen the outage window — broker and
-  client still move together — but it does stretch how long a rotation takes to
-  land, which matters for verifying one completed and for the cold-rebuild
-  ordering question above. Worth deciding whether these ExternalSecrets want a
-  shorter interval than 1h, or whether the rotator should trigger the sync
-  directly.
- **Bootstrap on a cold rebuild.** On a fresh cluster the rotator must create the
-  OpenBao entry before ESO can sync it; check this against
-  `kubernetes/RUNBOOK-cold-rebuild.md` so broker bring-up is not blocked.
-
-## What this does not change
-
- Secret names, key names and every `secretKeyRef` in `mosquitto-deployment.yaml`.
- The ACL model and the per-user topic scoping.
- The `evcc` ExternalSecret, which already follows the target pattern.
@@ -38,7 +38,7 @@ spec:
      # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.0 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.7.4 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -13,9 +13,6 @@ resources:
  - prometheus-token-externalsecret.yaml
  - config-sync-token-externalsecret.yaml
  - mosquitto-bootstrap.yaml
-  # health broker credential: OpenBao-backed + Class-A rotated (first of the
-  # eight users to migrate off mosquitto-bootstrap.yaml).
-  - mosquitto-health-credentials-externalsecret.yaml
  - mosquitto-config.yaml
  - mosquitto-certificate.yaml
  - mosquitto-service.yaml
@@ -370,6 +370,74 @@ spec:
            limits:
              memory: 64Mi
 ---
+# Broker user `health`: a dedicated, least-privilege credential used ONLY by
+# mosquitto's own liveness/readiness probes (mosquitto-deployment.yaml). Its
+# ACL is `write health/#` and nothing else — it cannot read any topic or
+# write outside the throwaway probe subtree, so a leak is worthless (nothing
+# subscribes to health/#). Kept on its own Secret, like every other user, so
+# the probe never has to borrow the admin credential.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-health-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-health-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
 # Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
 # to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
 # publish HA MQTT-discovery configs, exactly like the `shellies` user) and
@@ -5,18 +5,6 @@ metadata:
  namespace: home-assistant
  labels:
    app.kubernetes.io/name: mosquitto
-  annotations:
-    # REQUIRED for credential rotation to take effect at all. The startup shim
-    # rebuilds /mosquitto/auth/passwd from these Secrets on every pod start, so
-    # a rotated Secret is inert until the pod restarts — without this the new
-    # password would sit unused until some unrelated restart, while the old one
-    # stayed valid indefinitely.
-    #
-    # Listed per-Secret rather than using the blanket `reloader.stakater.com/
-    # auto: "true"`, so that enrolling each remaining broker user in rotation is
-    # a visible, reviewable edit to this line rather than something that starts
-    # happening implicitly.
-    secret.reloader.stakater.com/reload: mosquitto-health-credentials
 spec:
  replicas: 1
  strategy:
@@ -189,9 +177,7 @@ spec:
                secretKeyRef:
                  name: mosquitto-evcc-credentials
                  key: password
-            # Probe-only credential. Now OpenBao-backed and Class-A rotated (see
-            # mosquitto-health-credentials-externalsecret.yaml); its bootstrap Job
-            # is gone.
+            # Probe-only credential (mosquitto-health-credentials-bootstrap).
            # Present here for the same reason as the others — the startup
            # shim hashes it into the passwd file — and reused by the exec
            # probes below, which run inside this container.
@@ -244,7 +230,7 @@ spec:
          #
          # Uses the dedicated least-privilege `health` user (ACL: readwrite
          # health/# only — see the passwd/acl build above and the
-          # mosquitto-health-credentials ExternalSecret). Its password is in
+          # mosquitto-health-credentials-bootstrap Job). Its password is in
          # this container's env (HEALTH_PASSWORD) because the startup shim
          # hashes every user into the passwd file; the exec probes run
          # inside this container and read it directly. A leak of this
@@ -1,36 +0,0 @@
-# Projects the mosquitto `health` broker credential from OpenBao into the Secret
-# the broker already consumes. FIRST of the eight broker users to move off the
-# bootstrap Jobs and onto Class-A rotation (see
-# PROPOSAL-mosquitto-credential-rotation.md (PR #453 — not yet merged; this
-# file lands with it)); the rotator lives at
-# kubernetes/infrastructure/openbao/rotation/mosquitto-health.yaml.
-#
-# Secret name and key are UNCHANGED (`password`), so mosquitto-deployment.yaml's
-# secretKeyRef and the exec probes that read $HEALTH_PASSWORD need no edit.
-#
-# refreshInterval is 10m rather than the 1h used by mosquitto-evcc-credentials:
-# a rotation only lands once ESO re-reads, so 1h would mean up to an hour between
-# the rotator running and the broker actually using the new password, with the
-# old one still valid in the meantime. 10m keeps that window short without
-# polling OpenBao pointlessly hard.
-#
-# ⚠️ OWNERSHIP: creationPolicy is implicitly Owner, so ESO sets an ownerReference
-# on the Secret. Deleting this ExternalSecret therefore DELETES the live Secret
-# and with it the broker's probe credential — the bootstrap Job it replaces had
-# no such coupling. To detach without destroying, set creationPolicy: Orphan
-# before removing it.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-health-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-health-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-health-credentials
@@ -52,7 +52,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.9 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -62,7 +62,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.9 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -88,17 +88,13 @@ spec:
      # trust without disabling verification.
      initContainers:
        - name: fetch-bridge-cert
-          # alpine + the openssl CLI, built by .woodpecker/build-ci-images.yml
-          # from ci/images/alpine-openssl/. Replaces `apk add --no-cache openssl`
-          # on every paperless pod start: tools belong in the image, not
-          # installed at runtime. Dated tag so a change is a reviewable diff
-          # here, matching the ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
+          image: alpine:3.21
          command:
            - sh
            - -c
            - |
              set -e
+              apk add --no-cache openssl
              # Wait until the bridge presents a cert via STARTTLS.
              # Re-fetched on every paperless restart so a bridge cert
              # rotation only requires a paperless rollout to pick up.
@@ -25,7 +25,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: redis
-          image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
+          image: redis:8.8.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
          args:
            - redis-server
            - --save
@@ -34,7 +34,7 @@ spec:
          # Image automation, range 3.x (see image-automation/woodpecker.yaml).
          # MUST stay version-matched with woodpecker-server (gRPC parity) — both
          # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
          env:
            - name: WOODPECKER_SERVER
              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
@@ -40,7 +40,7 @@ spec:
          # minor + patch auto-update within 3.x; only the 4.0 major is held for
          # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
          # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
          env:
            - name: WOODPECKER_HOST
              value: https://ci.renner.uno
@@ -36,15 +36,6 @@ spec:
    kubeProxyReplacement: true
    socketLB:
      enabled: true
-    prometheus:
-      # The agent exported no metrics at all, so BPF map utilisation was
-      # invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
-      # of space the agent only says "no space left on device" in its log and
-      # pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
-      # that observable — and alertable — before it stalls a rollout.
-      enabled: true
-      serviceMonitor:
-        enabled: true
    bpf:
      # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
      # redirects the socket directly to the backend pod instead of sending
@@ -55,31 +55,6 @@ existing `flux-system` GitRepository whose Secret was created by
 | mmwave-configurator | `2.x` | Plain 3-component filter (rejects `addon-`/`-armv7`/`-arm64`/`-amd64` variants) |
 | cnpg-barman-plugin | `0.x` | v-prefix filter; marker lives in upstream-imported manifest (see below) |

-## Not enrolled (structurally impossible)
-
- `woodpeckerci/woodpecker-cli` — the lint image in
-  `.woodpecker/ci-manifests-test.yml`. It publishes a perfectly good
-  version stream (`v3.16.0-alpine`), so unlike the three below this is
-  **not** a tag-format problem. It cannot be enrolled because
-  **image-automation can only patch YAML that is a Kubernetes manifest** —
-  a maintainer states it plainly in fluxcd/flux2#2826: *"Flux can only
-  patch Kubernetes manifests. This means the YAML manifest must have
-  `apiVersion` and `kind` fields."* A Woodpecker pipeline file is plain
-  YAML with neither, so the marker would be silently ignored.
-
-  Note this is **not** fixable by widening `update.path`. The path only
-  controls which directory is walked; objects without `apiVersion`/`kind`
-  are skipped regardless. Widening the path to `./` buys nothing here and
-  additionally pulls the whole `ansible/` and `esphome/` trees into the
-  walk, so don't.
-
-  Consequence: the CLI is a **manual** pin while the server and agent
-  auto-bump, so it *will* drift. That is a known, accepted gap, covered
-  by the update-audit agent (item 5) rather than by automation. When
-  bumping the server major/minor, bump this too — it must stay on the
-  latest stable like everything else, and a CLI behind the server can
-  pass a config the server then rejects.
-
 ## Not enrolled (deliberate)

 These three images publish no usable version stream — there is nothing
@@ -38,57 +38,54 @@ spec:
          effect: NoExecute
      initContainers:
        - name: install
-          # Plugin binaries are baked into this image, checksum-verified at build
-          # time (ci/images/cni-plugins/Containerfile). This replaces fetching the
-          # release tarball from GitHub on the node, which had NO integrity check
-          # at all and put github.com on the critical path of every node rebuild.
-          # Dated tag so a version change is a reviewable diff here, matching the
-          # ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
+          image: alpine:3.21
          command:
            - sh
            - -c
            - |
              set -eu
-              SRC=/opt/cni-plugins
-              DST=/host/opt/cni/bin
-
-              # No network, no download, no "already installed?" guard. The old
-              # guard existed to avoid re-fetching; a local copy is cheap, so
-              # copying unconditionally is better — the node then always matches
-              # the version pinned in this manifest instead of keeping whatever
-              # it happened to install first.
-              for src in "$SRC"/*; do
-                name=${src##*/}
-                # Copy to a temp name in the SAME directory and rename over the
-                # target. A plain `cp` onto a plugin the kubelet is currently
-                # executing can fail with ETXTBSY; rename(2) never does, and is
-                # atomic, so a concurrent CNI invocation sees either the old or
-                # the new binary and never a partial one.
-                cp "$src" "$DST/.$name.tmp"
-                chmod 0755 "$DST/.$name.tmp"
-                mv -f "$DST/.$name.tmp" "$DST/$name"
-              done
-
-              echo "Installed from $(basename "$SRC") image:"
-              ls -la "$DST/macvlan" "$DST/static" "$DST/host-local"
-              echo "Version: $("$DST/macvlan" 2>&1 | head -1 || true)"
-              # Kept, but now as a REPAIR rather than a workaround. The original
-              # cause is gone: extracting the tarball on the node applied the
-              # archive's `./` entry uid/gid (1001/127 — the github-actions runner
-              # that built the release) to the destination directory, and `cp`
-              # into an existing directory does not touch its ownership, so that
-              # can no longer happen.
-              #
-              # It still runs because nodes provisioned by the old installer may
-              # ALREADY carry the wrong ownership, and this is what fixes them.
-              # Why it matters: Cilium's apply-sysctl-overwrites init container
-              # runs as root without DAC_OVERRIDE, so with /opt/cni/bin not owned
-              # by root it fails with "Permission denied" copying cilium-sysctlfix
-              # into /hostbin. The old cluster only survived because the original
-              # Cilium pods initialised before the chown and were never re-rolled;
-              # a rolling daemonset replacement on rebuild is what exposed it.
-              # Safe to drop once every node has been through this version.
+              VERSION=v1.5.1
+              ARCH=amd64
+              URL="https://github.com/containernetworking/plugins/releases/download/${VERSION}/cni-plugins-linux-${ARCH}-${VERSION}.tgz"
+              cd /host/opt/cni/bin
+              if [ -x ./macvlan ] && [ -x ./static ] && [ -x ./host-local ]; then
+                installed=$(./macvlan 2>&1 | head -1 || true)
+                echo "CNI plugins already installed (${installed})"
+              else
+                echo "Fetching ${URL}"
+                # busybox wget + tar, NOT `apk add --no-cache curl tar`.
+                # Installing tools at container start is not acceptable here: it
+                # makes the install path depend on package-index availability at
+                # exactly the moment a node is being (re)built, and this is a
+                # DaemonSet — it runs on every node.
+                #
+                # Nothing needed to be added in the first place. Verified in
+                # alpine:3.21: `tar` is already busybox (v1.37.0) and handles
+                # -xzf, and busybox `wget` follows GitHub's release redirect to
+                # objects.githubusercontent.com over TLS using the base image's
+                # /etc/ssl/certs/ca-certificates.crt — fetched the real 48MB
+                # tarball and extracted an executable macvlan from it.
+                # Only `curl` is genuinely absent from the base, and wget covers
+                # what it was used for here.
+                wget -q -O /tmp/plugins.tgz "${URL}"
+                # Extract everything: the tarball entries are stored with a
+                # `./` prefix, so selective extraction by bare name fails.
+                # All standard plugins are tiny (~5-10MB total) and harmless
+                # to keep alongside Cilium's CNI binaries.
+                tar -xzf /tmp/plugins.tgz -C .
+                echo "Installed: $(ls -la macvlan static host-local)"
+              fi
+              # Always restore root ownership of /opt/cni/bin. The cni-plugins
+              # tarball includes a `./` directory entry whose uid/gid (1001/127
+              # — the github-actions runner that built the release) gets
+              # applied to the destination during extract. After that, Cilium's
+              # apply-sysctl-overwrites init container fails with "Permission
+              # denied" trying to `cp` cilium-sysctlfix into /hostbin, because
+              # the init runs root without DAC_OVERRIDE and the directory is
+              # no longer owned by root. The previous cluster only worked
+              # because the original Cilium pods initialised before the chown
+              # happened and were never re-rolled; on rebuild, the rolling
+              # daemonset replacement bites.
              chown 0:0 /host/opt/cni/bin
          securityContext:
            privileged: true
@@ -1,46 +0,0 @@
-# OpenBao policy: rotator-mosquitto-health
-#
-# Least-privilege policy for the mosquitto `health` broker-credential rotation
-# CronJob (kubernetes auth role `rotator-mosquitto-health`). PER-SECRET
-# isolation, same template as rotator-paperless: one policy + one role per
-# rotated secret, so a compromised rotation Job cannot touch any other secret.
-#
-# Scoped to the `health` credential ALONE, deliberately not to a wildcard over
-# apps/home-assistant/mosquitto-*. The broker has eight users and they are being
-# enrolled one at a time; a wildcard would silently grant the first rotator write
-# access to the other seven before they are ready, which is exactly the blast
-# radius this per-secret pattern exists to avoid.
-#
-# Plan and phasing: kubernetes/apps/home-assistant/
-# PROPOSAL-mosquitto-credential-rotation.md — added by PR #453, which is not yet
-# merged, so that path does not resolve until it lands.
-#
-# NOT applied by Flux — OpenBao policies/roles are not Kubernetes resources
-# (same convention as eso-read.hcl and every other rotator policy). Apply ONCE
-# with an admin token:
-#
-#   bao policy write rotator-mosquitto-health \
-#     kubernetes/infrastructure/openbao/policies/rotator-mosquitto-health.hcl
-#
-#   bao write auth/kubernetes/role/rotator-mosquitto-health \
-#     bound_service_account_names=mosquitto-health-secret-rotator \
-#     bound_service_account_namespaces=openbao \
-#     policies=rotator-mosquitto-health ttl=15m
-#
-# Do NOT set token_no_default_policy=true — the `bao kv patch` kv-v2 preflight
-# needs the default policy (see rotator.hcl). The rotator writes via patch, so
-# hardening the role that way would break rotation rather than tighten it.
-#
-# BOOTSTRAP: `random` mode calls assert_fields_exist BEFORE patching, so it
-# ROTATES an existing field and never creates one. The path must already hold a
-# `password` field or the Job fails safe without writing. Seed it once:
-#
-#   bao kv put homelab/apps/home-assistant/mosquitto-health-credentials \
-#     password="$(kubectl -n home-assistant get secret mosquitto-health-credentials \
-#                 -o jsonpath='{.data.password}' | base64 -d)"
-#
-# (Reads are already covered by eso-read.hcl's `homelab/data/*`, so the
-# ExternalSecret needs no policy change.)
-path "homelab/data/apps/home-assistant/mosquitto-health-credentials" {
-  capabilities = ["read", "patch"]
-}
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -68,7 +68,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: sync-woodpecker
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
              args: ["woodpecker-secret"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -61,7 +61,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:978fe53cad83a8d55319f6201ecd171775a101b21aaa21b7ec3c7b14304c6a42
              args: ["cloudflare-roll"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -56,7 +56,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:978fe53cad83a8d55319f6201ecd171775a101b21aaa21b7ec3c7b14304c6a42
              args: ["cloudflare-roll"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -84,7 +84,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:b41cc3645f144179dddaa5ab1d32282c57c257432a613c15a6e15634fd0c4b81
              args: ["gitea-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -68,7 +68,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -130,7 +130,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -192,7 +192,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["ha-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -4,46 +4,6 @@ kind: Kustomization
 # (intra-namespace OpenBao access → no NetworkPolicy needed). One file per rotated
 # secret. The matching OpenBao policy/role for each lives in ../policies/ and is
 # bootstrapped manually (not GitOps-managed).
-#
-# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
-# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
-# image, so it does not need digest pinning, and a dated tag makes "what actually
-# runs" a reviewable diff.
-#
-# Digests were tried and actively broke: untagged manifests are garbage-collected
-# once :latest and the dated tags move past them, so a digest pinned today stops
-# resolving after a few rebuilds. That decay is driven by edits to
-# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
-# republishes secret-rotator and orphans the manifest every rotator here was
-# pinned to. 11 of these were pinned to digests that no longer existed and would
-# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
-# are quarterly, that would have looked like nothing happening rather than like a
-# failure.
-#
-# Keep them all on one tag: it is a single binary containing every mode
-# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
-# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
-# the next bump partial.
-#
-# ADDING A ROTATOR: copy an existing file, and check you did not inherit a
-# digest pin from an older example — nothing enforces this automatically, so it
-# is easy to reintroduce silently and the failure only shows up months later on
-# the first scheduled run:
-#
-#   grep -l 'image:.*@sha256' \
-#     kubernetes/infrastructure/openbao/rotation/*.yaml | grep -v kustomization.yaml
-#
-# (want: no output. The trailing filter is not padding — this file documents the
-# pattern, so it necessarily matches its own grep. Verified both directions:
-# silent across this directory as it stands, and it still reports a file that
-# actually carries an `image: …@sha256:` pin.)
-#
-# BUMPING the tag: verify the new one exists BEFORE committing — a tag that was
-# never built fails exactly like the digests did, and just as quietly:
-#
-#   kubectl run tagcheck --rm -i --restart=Never \
-#     --image=gitea.renner.uno/gitea_admin/secret-rotator:<DATE> \
-#     --command -- /secret-rotator          # prints the usage/mode list if the tag resolves
 resources:
  - paperless.yaml
  - woodpecker.yaml
@@ -58,4 +18,3 @@ resources:
  - home-assistant.yaml
  - review-bot.yaml
  - claude-oauth.yaml
-  - mosquitto-health.yaml
@@ -1,124 +0,0 @@
---
-# Class-A automated rotation of the mosquitto `health` broker credential — a
-# self-generated random secret.
-#
-# FIRST of the eight mosquitto broker users to be enrolled. The credentials were
-# previously minted once by bootstrap Jobs and then never changed at all: no
-# schedule, no compromise response, and they existed only in the Secret object,
-# outside the OpenBao snapshot. See kubernetes/apps/home-assistant/
-# PROPOSAL-mosquitto-credential-rotation.md (PR #453 — not yet merged; this
-# file lands with it) for the full plan and phasing.
-#
-# `health` deliberately goes first: it is the broker's OWN liveness/readiness
-# probe credential, consumed by exec probes inside the mosquitto container via
-# the same Secret that seeds the passwd file. Nothing outside the pod holds it,
-# so a rotation needs no coordination with any client — the smallest possible
-# blast radius for proving rotator -> OpenBao -> ESO -> Reloader end to end.
-#
-# Runs the secret-rotator binary (random mode): generate 32 random bytes,
-# standard-base64 encode, and merge-patch into OpenBao. assert_fields_exist runs
-# FIRST, so a misconfigured path fails safe (no rotation) rather than adding a
-# stray field — which also means this ROTATES an existing value and never
-# bootstraps one. Seeding is a one-time out-of-band step, documented in
-# policies/rotator-mosquitto-health.hcl.
-#
-# Base64 output is safe for this credential specifically: the probes pass it as
-# `-P "$HEALTH_PASSWORD"` to mosquitto_pub/sub inside the container, so the `+`,
-# `/` and `=` characters are never parsed by a URL or a config file. That is NOT
-# automatically true of the other seven users (sml2mqtt and open3e templates on
-# the edge host, Shelly device firmware, the HA config entry), so the charset
-# must be re-checked per user before enrolling each one — the old bootstrap Jobs
-# generated alphanumeric-only passwords with `tr -dc 'A-Za-z0-9'`.
-#
-# openbao-ns pattern: the Job runs in the openbao namespace, reaching the
-# OpenBao API intra-namespace — no NetworkPolicy needed. It writes OpenBao only;
-# ESO syncs mosquitto-health-credentials (ns home-assistant) and Reloader
-# restarts the broker so the passwd file is rebuilt from the new value.
-#
-# Rotation is not instant: it completes only after the ExternalSecret's
-# refreshInterval elapses and the broker rolls. During that roll two broker pods
-# with different passwd files sit behind the LB (maxUnavailable: 0/maxSurge: 1),
-# so a probe against the outgoing pod can fail auth briefly. Self-healing.
-#
-# ON-DEMAND: kubectl -n openbao create job --from=cronjob/mosquitto-health-secret-rotate mosquitto-health-secret-rotate-now
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: mosquitto-health-secret-rotator
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: mosquitto
-    app.kubernetes.io/component: secret-rotator
---
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: mosquitto-health-secret-rotate
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: mosquitto
-    app.kubernetes.io/component: secret-rotator
-spec:
-  # Quarterly, offset one month from paperless (which runs Jan/Apr/Jul/Oct) so
-  # unrelated rotations do not land in the same window and complicate diagnosis.
-  schedule: "0 4 1 2,5,8,11 *"
-  timeZone: Europe/Berlin
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 2
-  failedJobsHistoryLimit: 5
-  startingDeadlineSeconds: 3600
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 300
-      ttlSecondsAfterFinished: 259200 # 3 d
-      template:
-        metadata:
-          labels:
-            app.kubernetes.io/name: mosquitto
-            app.kubernetes.io/component: secret-rotator
-        spec:
-          serviceAccountName: mosquitto-health-secret-rotator
-          restartPolicy: Never
-          securityContext:
-            runAsNonRoot: true
-            runAsUser: 65534
-            runAsGroup: 65534
-            seccompProfile:
-              type: RuntimeDefault
-          containers:
-            - name: rotate
-              # Dated tag, per the convention documented in kustomization.yaml —
-              # every rotator here now uses the same one. Verified for this tag:
-              # it pulls, `random` is among its modes, and it carries the
-              # RANDOM_BYTES>=16 floor documented below (RANDOM_BYTES=8 exits
-              # "refusing to generate a weak secret").
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
-              args: ["random"]
-              securityContext:
-                allowPrivilegeEscalation: false
-                readOnlyRootFilesystem: true
-                capabilities:
-                  drop: ["ALL"]
-              env:
-                - name: BAO_ADDR
-                  value: http://openbao.openbao.svc.cluster.local:8200
-                - name: BAO_ROLE
-                  value: rotator-mosquitto-health
-                - name: BAO_PATH
-                  value: homelab/apps/home-assistant/mosquitto-health-credentials
-                # Matches the key the ExternalSecret projects and the key the
-                # Secret already carries, so mosquitto-deployment.yaml's
-                # secretKeyRef is unchanged.
-                - name: BAO_FIELDS
-                  value: "password"
-                # 32 bytes -> 44 base64 chars. The binary refuses anything below
-                # 16 bytes outright.
-                - name: RANDOM_BYTES
-                  value: "32"
-              resources:
-                requests:
-                  cpu: 25m
-                  memory: 16Mi
-                limits:
-                  memory: 48Mi
@@ -59,7 +59,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:49bfddd8eeb5fd871cbeb05bc3b4e2fcaaab7d9e24cc3007caae6084b4634c18
              args: ["netbird-key"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:97c22bcff324a93d2bb7bf3adb2894b405ed1a8035d02ad1ff6baced3813cefe
              args: ["proxmox-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -81,7 +81,7 @@ spec:
              type: RuntimeDefault
          initContainers:
            - name: rotate-mint
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
              args: ["gitea-token"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -128,7 +128,7 @@ spec:
                  memory: 48Mi
          containers:
            - name: sync-woodpecker
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:828bf4863286411bfd7ef9fdff617586588050c814743bec601a9222692162e4
              args: ["woodpecker-secret"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false
@@ -63,7 +63,7 @@ spec:
              type: RuntimeDefault
          containers:
            - name: rotate
-              image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
+              image: gitea.renner.uno/gitea_admin/secret-rotator@sha256:159e5888ff4e23f5fe7fb1284711d81591635f840888e61402af311b0a76ba92
              args: ["random"]
              securityContext:
                allowPrivilegeEscalation: false