ci(kaniko): move the remaining eight builds to the maintained fork

Pipeline 6861 answered the question the staged first step was there to ask.
build-alpine-jq ran on the fork's v1.25.18, unpacked the rootfs, executed the
RUN, and pushed both destinations to the Gitea registry at digest a430c9c6 —
while the seven untouched kaniko steps stayed green in the same run. The
executor behaves the same and the registry auth path is unaffected, so the rest
can follow.

This moves the other seven steps here plus the one in ci-claude-refresh.yml.
Nothing but the image: lines change; no flags, no commands, no credentials.
That leaves zero gcr.io/kaniko-project references in the pipelines.

The per-step note from the first commit is gone. It described a migration state
that no longer exists, and a comment that survives its own situation is how the
velero schedule ended up claiming there was no CNPG cluster in a namespace that
had one. The durable half moved to the file header, where it explains why the
kaniko image is not Google's and why the pin is past the version their repo
ended on.

Also corrects the admission policy's comment, which cited
gcr.io/kaniko-project/executor as the image its exemption was verified against.
Worth being precise about what did not change: the policy matches on the buildx
plugin image only and never on kaniko, so the swap needed no policy edit — which
is exactly why the stale image reference in the prose was easy to miss.

Closes #552

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:
 
 steps:
   - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/ansible
     depends_on: []
     # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
 
   - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/k8s
     depends_on: []
     when:
@@ -77,7 +88,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
     depends_on: []
     when:
@@ -108,7 +119,7 @@ steps:
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
   - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: secret-rotator
     depends_on: []
     when:
@@ -137,7 +148,7 @@ steps:
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
   - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
     depends_on: []
     when:
@@ -166,7 +177,7 @@ steps:
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
   - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
     depends_on: []
     when:
@@ -195,7 +206,7 @@ steps:
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
   - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
     depends_on: []
     when:
@@ -224,7 +235,7 @@ steps:
   # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
   # builds the same way and lands in the same registry so the audit sees it.
   - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/heating-can
     depends_on: []
     when:

.woodpecker/ci-claude-refresh.yml

@@ -130,7 +130,7 @@ steps:
   #    --build-arg to the resolved version (not "latest") guarantees the tag and
   #    the image content agree even if a new release lands mid-run.
   - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/claude
     depends_on: [decide]
     environment:

kubernetes/RUNBOOK-cold-rebuild.md

@@ -11,11 +11,14 @@ are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
 Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
 resources, and physically booting three machines from install media.
 
-**Read this first: the rebuild cannot be done remotely today.** Management is a
-tagged VLAN, and a node in maintenance mode has no VLAN configuration — it can
-only DHCP untagged, so it comes up unreachable. MS-01 has no IPMI. Every node
-therefore needs a keyboard and a monitor until #576 puts VLAN 40 native on the
-I226-V port. Plan for physical access before starting.
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.
 
 ## The circular-bootstrap problem this runbook exists to break
 
@@ -77,9 +80,10 @@ These must exist outside the cluster or the rebuild cannot complete:
       Proxmox credentials — there is no Proxmox any more.
 - [ ] **Install media** written and to hand: the Image Factory ISO named in
       Step 1, on a USB stick, for all three machines.
-- [ ] **Physical access to all three nodes** — keyboard and monitor. Not
-      optional today: see the note at the top. Budget for standing in front of
-      the machines, not for driving this from a desk.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.
 
 ## Step 1a — tofu: DNS, object storage, UniFi
 
@@ -105,9 +109,9 @@ https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced3
 ```
 
 Write it to a USB stick and boot each MS-01 from it. Each comes up in
-maintenance mode with a DHCP address on the **untagged** network — not on
-management, because that is a tagged VLAN a maintenance-mode node does not
-configure. Read the address off the console; that is what Step 2 talks to.
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.
 
 The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
 interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
@@ -326,12 +330,12 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
   in-cluster commits made **after** Flux/Gitea broke are not on the mirror
   (Gitea couldn't push them if Gitea itself was down). This is the intended
   recovery point, but be aware of it when reconciling drift.
-- **The rebuild cannot be driven remotely.** Management is a tagged VLAN and a
-  maintenance-mode node cannot configure VLANs, so a freshly booted machine is
-  unreachable until its config is applied — and MS-01 has no IPMI. Every node
-  needs a keyboard and a monitor. #576 fixes this by putting VLAN 40 native on
-  the I226-V port; until then, physical access is a hard prerequisite rather
-  than a convenience.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
 - **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
   Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
   Steps 3, 5 and 6 and work; do not read this as the whole script being gone.

kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml

@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -203,16 +203,16 @@ metadata:
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
+  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
+  # the home-assistant namespace regardless of which app/ dir owns
   # their manifest (matter-server, otbr, whisper, speech-to-phrase,
   # zone-configurator all set `namespace: home-assistant` via their
   # Kustomization, not per-resource): home-assistant-config (.storage
   # entity registry, recorder DB, secrets.yaml), esphome-data,
   # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
+  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
+  # zwavejs-store, zone-configurator-config. On restore,
+  # home-assistant-config has its
   # own path (the HA pod's `seed` initContainer pulling the native HAOS
   # backup from Garage, see the home-assistant Kustomization comment in
   # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
@@ -229,6 +229,20 @@ spec:
     storageLocation: default
     ttl: 720h0m0s
     snapshotMoveData: true
+    # The ha-pg CNPG cluster now runs in this namespace (it did not when
+    # this schedule was written — the old comment here still claimed
+    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
+    # cnpg.io/cluster label and was being swept in: a crash-consistent
+    # snapshot of a running Postgres data directory is not a restorable
+    # backup, the same trap the wordpress-daily comment below describes.
+    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
+    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
+    # continuous WAL archiving — so this exclusion removes a redundant
+    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule