fix(multus): drop runtime apk add from the CNI plugins installer

Owner directive: tools belong in the image, not installed at container
start. This is the highest-blast-radius instance of the pattern left in the
repo — a DaemonSet, so it runs on every node.

No image was needed. `apk add --no-cache curl tar` turns out to have been
unnecessary from the start: verified in alpine:3.21 that `tar` is already
busybox (v1.37.0) and handles -xzf, and that busybox `wget` follows
GitHub's release redirect to objects.githubusercontent.com over TLS using
the base image's /etc/ssl/certs/ca-certificates.crt. Only `curl` is
genuinely absent, and wget covers what it was used for. So this replaces
the install with `wget -q -O` rather than introducing another layered
image.

Severity note for the record: the apk add sat inside the else branch, so
steady-state nodes never reached it. It ran when plugins were absent — i.e.
on a fresh node or after /opt/cni/bin was wiped, which is precisely the
moment you least want a dependency on package-index availability.

Verified by running the RENDERED init script in alpine:3.21 against the
real release URL, both paths:

  empty dir     -> Fetching ...; Installed: host-local, macvlan, static
                   macvlan executable, /opt/cni/bin owner 0:0, exit 0
  already there -> "CNI plugins already installed (CNI macvlan plugin
                   v1.5.1)", exit 0, no fetch

The second line is worth noting: that version string comes from executing
the extracted binary, so the download and extraction are confirmed good,
not merely present. The chown 0:0 also still applies, which the existing
comment explains is required or Cilium's apply-sysctl-overwrites init
container fails with "Permission denied".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -130,3 +130,32 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/secret-rotator:$TAG \
           --destination gitea.renner.uno/gitea_admin/secret-rotator:latest
+
+  # Runtime image, not CI tooling: a general-purpose alpine + jq base for
+  # workloads that need to read JSON from a shell script. Named for what it IS,
+  # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
+  # container, but nothing about it is zigbee-specific. Exists so tools come from
+  # the image rather than an `apk add` on every container start.
+  - name: build-alpine-jq
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    directory: ci/images/alpine-jq
+    depends_on: []
+    when:
+      - event: manual
+      - event: push
+        path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest

ci/images/alpine-jq/Containerfile

@@ -0,0 +1,24 @@
+FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
+
+# Alpine plus jq. Nothing application-specific — a general-purpose base for any
+# workload that needs to read JSON from a shell script.
+#
+# It exists because installing tools at container start is not acceptable here:
+# `apk add` makes every start depend on network and package-index availability,
+# and the resolved version can drift between restarts of an otherwise pinned
+# image. Tools belong in the image.
+#
+# Everything else such a script typically needs (sh, sed, cp, cat, tr, awk) comes
+# from busybox in the base, so jq is the only addition.
+#
+# Runs as root like the other ci/images entries. Consumers that can drop
+# privilege should set runAsNonRoot/runAsUser themselves — what the workload
+# needs to read is workload-specific, so it does not belong baked in here.
+#
+# First consumer is the zigbee2mqtt seed-data init container. Worth recording why
+# a real JSON parser is justified there rather than a sed/awk match: "key" occurs
+# 12 times in that document (once per device with an APS link_key) and the FIRST
+# occurrence is a DEVICE key, because "devices" sorts before "network_key". An
+# unanchored extraction therefore yields a device link key, and herdsman then
+# forms a NEW Zigbee network and drops every device.
+RUN apk add --no-cache jq

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -39,16 +39,80 @@ spec:
         # network_key, the latter extracted from the OpenBao coordinator backup
         # as a natural-byte-order int array.
         - name: seed-data
-          image: alpine:3.21
+          # alpine + jq, built by .woodpecker/build-ci-images.yml from
+          # ci/images/alpine-jq/. Replaces `apk add --no-cache jq` on every pod
+          # start: tools belong in the image, not installed at runtime.
+          # Referenced by tag rather than digest — these are our own images.
+          # DATED tag, not :latest, matching the ci-ansible:2026-04-29 precedent
+          # in kubernetes/infrastructure/netbird-backup/cronjob.yaml. With
+          # :latest, rebuilding this base for some unrelated consumer would
+          # silently change what this init container runs on its next pod
+          # restart, with no diff here to review — and this is the container that
+          # renders the Zigbee network key. Bumping the tag is a deliberate,
+          # reviewable act; the cost is remembering to bump it.
+          #
+          # This tag ALREADY EXISTS and was verified pullable before merge — the
+          # build step also fires on branch pushes, so it was published when this
+          # branch was pushed. That matters because Flux reconciles this directory
+          # on a 10m poll with no ordering against the kaniko pipeline, and with
+          # replicas: 1 + strategy: Recreate an unpullable init image would leave
+          # the sole Zigbee coordinator in ImagePullBackOff.
+          #
+          # Caveat when bumping: the build derives TAG from `date` AT RUN TIME, so
+          # never write a date here that a build has not already produced. If a
+          # later master build runs on a different day it publishes that day's tag
+          # and leaves this one untouched (tags are not deleted), so this keeps
+          # resolving — but the image then came from the branch build rather than
+          # master. Re-point it at a master-built tag when convenient.
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
           command:
             - sh
             - -c
             - |
               set -eu
               umask 077
-              apk add --no-cache jq >/dev/null
+              # Read the network key with a real JSON parser. NOT a hand-rolled
+              # sed/awk match: "key" occurs 12 times in the real backup (once per
+              # device with an APS link_key) and the FIRST occurrence is a DEVICE
+              # key, because "devices" sorts before "network_key". An unanchored
+              # extraction yields a device link key, and herdsman then forms a
+              # NEW network and drops every device.
+              # `?` on BOTH selectors, plus the `// ""` fallback, so every
+              # malformed shape reaches the checks below with a readable message
+              # instead of aborting on jq's raw error under `set -e`.
+              #
+              # Measured against jq 1.7.1 in this image. The expression AS WRITTEN
+              # returns an EMPTY string at rc=0 for every one of:
+              #   {}  {"devices":[]}  {"network_key":{}}  {"network_key":null}
+              #   {"network_key":"abc"}  {"network_key":123}   (parent not an object)
+              #   []  "a string"                               (top level not an object)
+              # and the key itself for a well-formed backup.
+              #
+              # The guards are what make that true. Bare `.network_key.key` exits
+              # 5 with `Cannot index ... with string "key"` on the last four
+              # shapes, and `set -e` would abort right here, surfacing jq's error
+              # instead of the FATAL lines below.
+              KEY=$(jq -r '(.network_key?.key? // "")' /backup/coordinator_backup.json)
+              # Fail LOUDLY rather than render a wrong key, for the same reason.
+              # `-z` is the arm that actually fires today: `// ""` maps a null
+              # result to empty, so KEY is never the literal text "null" with the
+              # expression above. The "null" arm is belt-and-braces for a future
+              # edit that drops the fallback, where jq WOULD print `null`. Both
+              # mean "absent" — reporting them as merely "non-hex" would send you
+              # looking at the key instead of at the missing field.
+              if [ "$KEY" = "null" ] || [ -z "$KEY" ]; then
+                echo "FATAL: .network_key.key absent from coordinator_backup.json" >&2
+                exit 1
+              fi
+              case "$KEY" in
+                *[!0-9a-fA-F]*) echo "FATAL: network_key contains non-hex characters" >&2; exit 1 ;;
+              esac
+              if [ "${#KEY}" -ne 32 ]; then
+                echo "FATAL: network_key length ${#KEY}, expected 32 hex chars" >&2
+                exit 1
+              fi
               # hex key "fe0b..2d" -> "0xfe,0x0b,..,0x2d" (natural order)
-              NK=$(jq -r '.network_key.key' /backup/coordinator_backup.json | sed 's/../0x&,/g; s/,$//')
+              NK=$(printf '%s' "$KEY" | sed 's/../0x&,/g; s/,$//')
               cat > /app/data/secret.yaml <<EOF
               mqtt_password: "${MQTT_PASSWORD}"
               network_key: [${NK}]

kubernetes/infrastructure/coredns-config/configmap.yaml

@@ -56,9 +56,17 @@ data:
         forward . /etc/resolv.conf {
            max_concurrent 1000
         }
+        # serve_stale mirrors the renner.uno:53 block (issue #393): when the
+        # upstream resolver browns out, keep answering from expired cache
+        # instead of SERVFAILing every public name cluster-wide. Seen on
+        # 2026-07-30 01:22-01:26Z (~1150 SERVFAILs) — it killed a Velero Kopia
+        # maintenance job, which runs with backoffLimit 0, on the Hetzner S3
+        # endpoint (issue #439). cluster.local is exempt from the cache above,
+        # so in-cluster Service records are never served stale.
         cache 30 {
            disable success cluster.local
            disable denial cluster.local
+           serve_stale 1h immediate
         }
         loop
         reload

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -53,8 +53,21 @@ spec:
                 echo "CNI plugins already installed (${installed})"
               else
                 echo "Fetching ${URL}"
-                apk add --no-cache curl tar
-                curl -fsSL "${URL}" -o /tmp/plugins.tgz
+                # busybox wget + tar, NOT `apk add --no-cache curl tar`.
+                # Installing tools at container start is not acceptable here: it
+                # makes the install path depend on package-index availability at
+                # exactly the moment a node is being (re)built, and this is a
+                # DaemonSet — it runs on every node.
+                #
+                # Nothing needed to be added in the first place. Verified in
+                # alpine:3.21: `tar` is already busybox (v1.37.0) and handles
+                # -xzf, and busybox `wget` follows GitHub's release redirect to
+                # objects.githubusercontent.com over TLS using the base image's
+                # /etc/ssl/certs/ca-certificates.crt — fetched the real 48MB
+                # tarball and extracted an executable macvlan from it.
+                # Only `curl` is genuinely absent from the base, and wget covers
+                # what it was used for here.
+                wget -q -O /tmp/plugins.tgz "${URL}"
                 # Extract everything: the tarball entries are stored with a
                 # `./` prefix, so selective extraction by bare name fails.
                 # All standard plugins are tiny (~5-10MB total) and harmless