fix(multus): drop runtime apk add from the CNI plugins installer

Owner directive: tools belong in the image, not installed at container
start. This is the highest-blast-radius instance of the pattern left in the
repo — a DaemonSet, so it runs on every node.

No image was needed. `apk add --no-cache curl tar` turns out to have been
unnecessary from the start: verified in alpine:3.21 that `tar` is already
busybox (v1.37.0) and handles -xzf, and that busybox `wget` follows
GitHub's release redirect to objects.githubusercontent.com over TLS using
the base image's /etc/ssl/certs/ca-certificates.crt. Only `curl` is
genuinely absent, and wget covers what it was used for. So this replaces
the install with `wget -q -O` rather than introducing another layered
image.

Severity note for the record: the apk add sat inside the else branch, so
steady-state nodes never reached it. It ran when plugins were absent — i.e.
on a fresh node or after /opt/cni/bin was wiped, which is precisely the
moment you least want a dependency on package-index availability.

Verified by running the RENDERED init script in alpine:3.21 against the
real release URL, both paths:

  empty dir     -> Fetching ...; Installed: host-local, macvlan, static
                   macvlan executable, /opt/cni/bin owner 0:0, exit 0
  already there -> "CNI plugins already installed (CNI macvlan plugin
                   v1.5.1)", exit 0, no fetch

The second line is worth noting: that version string comes from executing
the extracted binary, so the download and extraction are confirmed good,
not merely present. The chown 0:0 also still applies, which the existing
comment explains is required or Cilium's apply-sysctl-overwrites init
container fails with "Permission denied".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -159,61 +159,3 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
           --destination gitea.renner.uno/gitea_admin/alpine-jq:latest
-
-  # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
-  # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
-  # node-side fetch from GitHub that had NO checksum verification and sat on the
-  # critical path of every node rebuild. The download now happens here, once,
-  # where the sha256 is checked — see the Containerfile for the full rationale.
-  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/cni-plugins
-    depends_on: []
-    when:
-      - event: manual
-      - event: push
-        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
-
-  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
-  # needing s_client/x509. Named for what it is; first consumer is the paperless
-  # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
-  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
-  # base, so this one needs an image rather than a busybox substitute.
-  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/alpine-openssl
-    depends_on: []
-    when:
-      - event: manual
-      - event: push
-        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest

ci/images/alpine-openssl/Containerfile

@@ -1,21 +0,0 @@
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Alpine plus the openssl CLI. Nothing application-specific — named for what it
-# is, so anything needing `openssl s_client` / `x509` from a shell script can use
-# it.
-#
-# It exists because installing tools at container start is not acceptable here:
-# `apk add` makes every start depend on network and package-index availability,
-# and the resolved version can drift between restarts of an otherwise pinned
-# image.
-#
-# Note the openssl CLI is genuinely absent from the alpine base — verified, the
-# base carries the libraries but not the command — so unlike the multus CNI
-# installer (where busybox wget/tar already covered the need) this one really
-# does require an image rather than just dropping the apk add.
-#
-# First consumer: the paperless fetch-bridge-cert init container, which captures
-# the Proton Mail Bridge's self-signed IMAP certificate via STARTTLS and pins it
-# for paperless to trust. busybox has no s_client equivalent, so openssl is not
-# substitutable there.
-RUN apk add --no-cache openssl

ci/images/cni-plugins/Containerfile

@@ -1,68 +0,0 @@
-# Upstream containernetworking/plugins binaries, baked in at build time.
-#
-# WHY THIS IMAGE EXISTS
-# The multus cni-plugins-installer DaemonSet used to fetch this tarball from
-# GitHub on the node itself, at node-provisioning time, and extract it straight
-# into /opt/cni/bin. Two problems with that:
-#
-#   1. Security. The download had NO integrity verification — wget, then tar,
-#      into a directory whose contents are executed as root in the CNI path on
-#      every node. A tampered or MITM'd artifact would have got privileged
-#      execution cluster-wide, with nothing to detect it.
-#   2. Node rebuild. It put a live dependency on github.com (and previously the
-#      Alpine package index too) on the critical path of bringing a node up, so
-#      a GitHub outage or slow transfer stalled or broke CNI installation
-#      exactly when a node was being rebuilt.
-#
-# Baking the binaries in fixes both: the fetch happens once, here, where the
-# checksum IS verified, and the installer becomes a local file copy with no
-# network at all.
-#
-# The version and digest are pinned literals, so bumping them is a reviewable
-# commit rather than a silent change in what runs on every node. The digest was
-# cross-checked against upstream's own published
-# cni-plugins-linux-amd64-<version>.tgz.sha256 asset rather than merely recorded
-# from a local download — a download can be tampered with; the point of the
-# check is to compare against what upstream states.
-#
-# amd64 only, matching the ARCH the installer already hardcoded and the cluster's
-# node architecture. Add a second stage if a different arch is ever needed.
-
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d AS fetch
-
-ARG CNI_PLUGINS_VERSION=v1.5.1
-ARG CNI_PLUGINS_SHA256=77baa2f669980a82255ffa2f2717de823992480271ee778aa51a9c60ae89ff9b
-
-# busybox wget and tar are already in the base; nothing is installed here
-# either. sha256sum -c fails the BUILD if the artifact does not match, which is
-# the whole point — a bad download must never reach a node.
-RUN set -eu; \
-    F="cni-plugins-linux-amd64-${CNI_PLUGINS_VERSION}.tgz"; \
-    wget -q -O "/tmp/${F}" \
-      "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/${F}"; \
-    echo "${CNI_PLUGINS_SHA256}  /tmp/${F}" | sha256sum -c -; \
-    mkdir -p /plugins; \
-    tar -xzf "/tmp/${F}" -C /plugins; \
-    rm -f "/tmp/${F}"; \
-    # Drop the non-executable payload: the tarball ships LICENSE and README.md,
-    # which the old installer extracted into /opt/cni/bin alongside the binaries.
-    rm -f /plugins/LICENSE /plugins/README.md; \
-    # Fail loudly here rather than shipping an image that silently installs
-    # nothing, if upstream ever restructures the archive.
-    #
-    # This list must cover every plugin the repo actually depends on, not just
-    # the ones the installer happens to log. `dhcp` is included because the
-    # sibling cni-dhcp-daemon DaemonSet runs that binary and documents that it
-    # ships from here (see kubernetes/infrastructure/multus/cni-dhcp-daemon.yaml)
-    # — without it in the check, upstream dropping `dhcp` while keeping the
-    # others would build green and only surface later as a broken DHCP daemon on
-    # whichever node rolled first.
-    for p in macvlan static host-local dhcp; do \
-      [ -x "/plugins/$p" ] || { echo "FATAL: /plugins/$p missing or not executable" >&2; exit 1; }; \
-    done
-
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Alpine rather than scratch: the installer needs a shell to copy the binaries
-# onto the host. busybox covers cp/mv/chown/ls, so nothing is added.
-COPY --from=fetch /plugins/ /opt/cni-plugins/

kubernetes/apps/home-assistant/kustomization.yaml

@@ -12,8 +12,7 @@ resources:
   - vscode-ingress.yaml
   - prometheus-token-externalsecret.yaml
   - config-sync-token-externalsecret.yaml
-  # Broker credentials come from OpenBao via ESO (replaced the bootstrap Jobs).
-  - mosquitto-credentials-externalsecrets.yaml
+  - mosquitto-bootstrap.yaml
   - mosquitto-config.yaml
   - mosquitto-certificate.yaml
   - mosquitto-service.yaml
@@ -37,9 +36,8 @@ resources:
   - esphome-ingress.yaml
   # Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
   # runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
-  # `zigbee2mqtt` broker user comes from OpenBao via
-  # mosquitto-credentials-externalsecrets.yaml (above) and wired
-  # into mosquitto-deployment.yaml (passwd + ACL).
+  # `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
+  # (above) and wired into mosquitto-deployment.yaml (passwd + ACL).
   - zigbee2mqtt-coordinator-backup-externalsecret.yaml
   - zigbee2mqtt-config.yaml
   - zigbee2mqtt-pvc.yaml

kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml

@@ -0,0 +1,508 @@
+---
+# Generates mosquitto-credentials on first deploy. The password is created
+# in-cluster, lives only in the Secret object, and is never written to git
+# or seen by a human. The Mosquitto broker hashes it via mosquitto_passwd
+# -U at startup; HA reads the same value from this Secret (configured
+# manually in the HA MQTT integration UI).
+#
+# Idempotent: if the Secret already exists, the Job exits 0 without
+# regenerating, so subsequent Flux reconciles don't rotate the password.
+# To rotate, delete both the Secret and this Job; Flux will re-create
+# the Job and a fresh password.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: mosquitto-bootstrap
+  namespace: home-assistant
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: mosquitto-bootstrap
+  namespace: home-assistant
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: mosquitto-bootstrap
+  namespace: home-assistant
+subjects:
+  - kind: ServiceAccount
+    name: mosquitto-bootstrap
+    namespace: home-assistant
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: mosquitto-bootstrap
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {"homeassistant-password": $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Third broker user: `admin`. Full readwrite (same as HA) but kept on
+# its own credential so a workstation client can hold a password the
+# in-cluster HA config never sees, and rotating one doesn't affect the
+# other. Retrieve with:
+#   kubectl -n home-assistant get secret mosquitto-admin-credentials \
+#     -o jsonpath='{.data.password}' | base64 -d
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-admin-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-admin-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Second broker user: `shellies`. Separate Secret (not a second key on
+# mosquitto-credentials) so each credential's lifecycle is independent —
+# rotating one never touches the other, and existing clusters whose
+# mosquitto-credentials was already generated for `homeassistant` only
+# don't end up with a half-populated Secret.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-shellies-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-shellies-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Fourth broker user: `sml2mqtt` — the power-meter bridge. Publish-only
+# under `sml2mqtt/#` (ACL in mosquitto-deployment.yaml). Own credential so
+# its lifecycle is independent of the others. The sml2mqtt pod reads this
+# password via its init container and renders it into config.yml.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-sml2mqtt-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-sml2mqtt-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Fifth broker user: `open3e` — the Viessmann E3 CAN bridge (Vitocal 250-A
+# + Vitocharge VX3 via open3e). Scoped to `open3e/#` (ACL in
+# mosquitto-deployment.yaml). Own credential so its lifecycle is
+# independent of the others. Retrieve with:
+#   kubectl -n home-assistant get secret mosquitto-open3e-credentials \
+#     -o jsonpath='{.data.password}' | base64 -d
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-open3e-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-open3e-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Broker user `health`: a dedicated, least-privilege credential used ONLY by
+# mosquitto's own liveness/readiness probes (mosquitto-deployment.yaml). Its
+# ACL is `write health/#` and nothing else — it cannot read any topic or
+# write outside the throwaway probe subtree, so a leak is worthless (nothing
+# subscribes to health/#). Kept on its own Secret, like every other user, so
+# the probe never has to borrow the admin credential.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-health-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-health-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+---
+# Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
+# to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
+# publish HA MQTT-discovery configs, exactly like the `shellies` user) and
+# `read homeassistant/status` (Z2M re-publishes discovery when HA restarts —
+# ACL in mosquitto-deployment.yaml). Own credential so its lifecycle is
+# independent of the others. The Z2M init container reads this password and
+# renders it into /app/data/secret.yaml.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: mosquitto-zigbee2mqtt-credentials-bootstrap
+  namespace: home-assistant
+spec:
+  backoffLimit: 4
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mosquitto-bootstrap
+    spec:
+      serviceAccountName: mosquitto-bootstrap
+      restartPolicy: OnFailure
+      containers:
+        - name: bootstrap
+          image: alpine:3.21
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              apk add --no-cache curl jq >/dev/null
+
+              NS=home-assistant
+              NAME=mosquitto-zigbee2mqtt-credentials
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              API=https://kubernetes.default.svc
+
+              if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then
+                echo "$NAME already exists; nothing to do."
+                exit 0
+              fi
+
+              PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)
+              PW_B64=$(printf '%s' "$PW" | base64 | tr -d '\n')
+
+              body=$(jq -nc \
+                --arg name "$NAME" \
+                --arg ns "$NS" \
+                --arg pw "$PW_B64" \
+                '{apiVersion: "v1",
+                  kind: "Secret",
+                  metadata: {name: $name, namespace: $ns},
+                  type: "Opaque",
+                  data: {password: $pw}}')
+
+              curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \
+                   -H "Content-Type: application/json" -X POST \
+                   --data "$body" \
+                   "$API/api/v1/namespaces/$NS/secrets" >/dev/null
+
+              echo "generated $NAME"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi

kubernetes/apps/home-assistant/mosquitto-credentials-externalsecrets.yaml

@@ -1,155 +0,0 @@
-# The seven mosquitto broker credentials that used to be generated by bootstrap
-# Jobs (mosquitto-bootstrap.yaml, now deleted), converted to the same
-# OpenBao-backed pattern mosquitto-evcc-credentials already used.
-#
-# WHY THE JOBS WENT
-# Each was ~70 lines of shell that generated a password, base64'd it, hand-built
-# Secret JSON with jq and POSTed it to the Kubernetes API with curl, needing a
-# ServiceAccount + Role + RoleBinding and an `apk add curl jq` on every run.
-# Together: 518 lines to do what 14 declarative lines do here. They also could not
-# be modified in place at all — a Job's spec.template is immutable, so changing
-# even the container image made the whole Kustomization fail to reconcile.
-#
-# The passwords were also only ever in the Secret object: not in OpenBao, so not
-# backed up, not centrally rotatable, and unrecoverable if the Secret were lost.
-# Now they live in OpenBao like every other credential in this namespace.
-#
-# MIGRATION WAS VALUE-PRESERVING
-# The existing live passwords were copied into OpenBao before this landed, and
-# each round-trip was verified by sha256 — the broker has already hashed these
-# (mosquitto_passwd -U at startup) and every client is using them, so generating
-# fresh ones would have broken Home Assistant, zigbee2mqtt, shellies, sml2mqtt,
-# open3e and the health check at once. ESO therefore rewrites each Secret with
-# byte-identical content and nothing observes a change.
-#
-# ROTATION, which the Jobs made awkward (delete the Secret AND the Job, then wait
-# for Flux), is now: update the value in OpenBao, and within refreshInterval ESO
-# republishes it. The broker still needs a restart to re-hash, as before.
-#
-# ⚠️ NEW COUPLING, absent with the Jobs: creationPolicy is implicitly Owner, so
-# ESO sets an ownerReference from each Secret to its ExternalSecret. Combined with
-# the Kustomization's prune: true, DELETING AN ExternalSecret BLOCK BELOW NOW
-# DELETES THE LIVE SECRET — and with it broker access for that client. The old
-# Jobs created Secrets with no ownerReference, so removing a Job left its Secret
-# untouched; that safety net is gone. To retire a credential deliberately, remove
-# the consumer first. To move one out of ESO's control without losing it, set
-# creationPolicy: Orphan on that block before deleting it.
-#
-# NOTE the key names are not uniform: mosquitto-credentials carries
-# `homeassistant-password` while the rest carry `password`. That asymmetry came
-# from the original Jobs and is preserved deliberately — the consumers reference
-# those key names, so normalising them would be a separate, breaking change.
-# `dataFrom.extract` copies whatever keys the OpenBao entry holds, so each entry
-# was seeded with the matching key name.
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-credentials # unchanged name; HA's MQTT integration reads this
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-admin-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-admin-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-admin-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-health-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-health-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-health-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-open3e-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-open3e-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-open3e-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-shellies-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-shellies-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-shellies-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-sml2mqtt-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-sml2mqtt-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-sml2mqtt-credentials
----
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: mosquitto-zigbee2mqtt-credentials
-  namespace: home-assistant
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: mosquitto-zigbee2mqtt-credentials
-  dataFrom:
-    - extract:
-        key: apps/home-assistant/mosquitto-zigbee2mqtt-credentials

kubernetes/apps/paperless/paperless.yaml

@@ -88,17 +88,13 @@ spec:
       # trust without disabling verification.
       initContainers:
         - name: fetch-bridge-cert
-          # alpine + the openssl CLI, built by .woodpecker/build-ci-images.yml
-          # from ci/images/alpine-openssl/. Replaces `apk add --no-cache openssl`
-          # on every paperless pod start: tools belong in the image, not
-          # installed at runtime. Dated tag so a change is a reviewable diff
-          # here, matching the ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
+          image: alpine:3.21
           command:
             - sh
             - -c
             - |
               set -e
+              apk add --no-cache openssl
               # Wait until the bridge presents a cert via STARTTLS.
               # Re-fetched on every paperless restart so a bridge cert
               # rotation only requires a paperless rollout to pick up.

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -38,57 +38,54 @@ spec:
           effect: NoExecute
       initContainers:
         - name: install
-          # Plugin binaries are baked into this image, checksum-verified at build
-          # time (ci/images/cni-plugins/Containerfile). This replaces fetching the
-          # release tarball from GitHub on the node, which had NO integrity check
-          # at all and put github.com on the critical path of every node rebuild.
-          # Dated tag so a version change is a reviewable diff here, matching the
-          # ci-ansible / alpine-jq convention.
-          image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
+          image: alpine:3.21
           command:
             - sh
             - -c
             - |
               set -eu
-              SRC=/opt/cni-plugins
-              DST=/host/opt/cni/bin
-
-              # No network, no download, no "already installed?" guard. The old
-              # guard existed to avoid re-fetching; a local copy is cheap, so
-              # copying unconditionally is better — the node then always matches
-              # the version pinned in this manifest instead of keeping whatever
-              # it happened to install first.
-              for src in "$SRC"/*; do
-                name=${src##*/}
-                # Copy to a temp name in the SAME directory and rename over the
-                # target. A plain `cp` onto a plugin the kubelet is currently
-                # executing can fail with ETXTBSY; rename(2) never does, and is
-                # atomic, so a concurrent CNI invocation sees either the old or
-                # the new binary and never a partial one.
-                cp "$src" "$DST/.$name.tmp"
-                chmod 0755 "$DST/.$name.tmp"
-                mv -f "$DST/.$name.tmp" "$DST/$name"
-              done
-
-              echo "Installed from $(basename "$SRC") image:"
-              ls -la "$DST/macvlan" "$DST/static" "$DST/host-local"
-              echo "Version: $("$DST/macvlan" 2>&1 | head -1 || true)"
-              # Kept, but now as a REPAIR rather than a workaround. The original
-              # cause is gone: extracting the tarball on the node applied the
-              # archive's `./` entry uid/gid (1001/127 — the github-actions runner
-              # that built the release) to the destination directory, and `cp`
-              # into an existing directory does not touch its ownership, so that
-              # can no longer happen.
-              #
-              # It still runs because nodes provisioned by the old installer may
-              # ALREADY carry the wrong ownership, and this is what fixes them.
-              # Why it matters: Cilium's apply-sysctl-overwrites init container
-              # runs as root without DAC_OVERRIDE, so with /opt/cni/bin not owned
-              # by root it fails with "Permission denied" copying cilium-sysctlfix
-              # into /hostbin. The old cluster only survived because the original
-              # Cilium pods initialised before the chown and were never re-rolled;
-              # a rolling daemonset replacement on rebuild is what exposed it.
-              # Safe to drop once every node has been through this version.
+              VERSION=v1.5.1
+              ARCH=amd64
+              URL="https://github.com/containernetworking/plugins/releases/download/${VERSION}/cni-plugins-linux-${ARCH}-${VERSION}.tgz"
+              cd /host/opt/cni/bin
+              if [ -x ./macvlan ] && [ -x ./static ] && [ -x ./host-local ]; then
+                installed=$(./macvlan 2>&1 | head -1 || true)
+                echo "CNI plugins already installed (${installed})"
+              else
+                echo "Fetching ${URL}"
+                # busybox wget + tar, NOT `apk add --no-cache curl tar`.
+                # Installing tools at container start is not acceptable here: it
+                # makes the install path depend on package-index availability at
+                # exactly the moment a node is being (re)built, and this is a
+                # DaemonSet — it runs on every node.
+                #
+                # Nothing needed to be added in the first place. Verified in
+                # alpine:3.21: `tar` is already busybox (v1.37.0) and handles
+                # -xzf, and busybox `wget` follows GitHub's release redirect to
+                # objects.githubusercontent.com over TLS using the base image's
+                # /etc/ssl/certs/ca-certificates.crt — fetched the real 48MB
+                # tarball and extracted an executable macvlan from it.
+                # Only `curl` is genuinely absent from the base, and wget covers
+                # what it was used for here.
+                wget -q -O /tmp/plugins.tgz "${URL}"
+                # Extract everything: the tarball entries are stored with a
+                # `./` prefix, so selective extraction by bare name fails.
+                # All standard plugins are tiny (~5-10MB total) and harmless
+                # to keep alongside Cilium's CNI binaries.
+                tar -xzf /tmp/plugins.tgz -C .
+                echo "Installed: $(ls -la macvlan static host-local)"
+              fi
+              # Always restore root ownership of /opt/cni/bin. The cni-plugins
+              # tarball includes a `./` directory entry whose uid/gid (1001/127
+              # — the github-actions runner that built the release) gets
+              # applied to the destination during extract. After that, Cilium's
+              # apply-sysctl-overwrites init container fails with "Permission
+              # denied" trying to `cp` cilium-sysctlfix into /hostbin, because
+              # the init runs root without DAC_OVERRIDE and the directory is
+              # no longer owned by root. The previous cluster only worked
+              # because the original Cilium pods initialised before the chown
+              # happened and were never re-rolled; on rebuild, the rolling
+              # daemonset replacement bites.
               chown 0:0 /host/opt/cni/bin
           securityContext:
             privileged: true