From 2946dfcd357dfe6c96c3e8b5ae97c3eb0219a0b9 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Fri, 4 Sep 2026 10:01:18 +0200 Subject: [PATCH] feat(bootstrap): let the node fall back to the mirror, not the manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four images are needed while the cluster is still being built: the CNI plugins every pod depends on, paperless's fetch-bridge-cert init container, zigbee2mqtt's seed-data init container and the trmnl renderer. None of them can come from gitea.renner.uno during a rebuild, because Gitea runs inside the cluster being rebuilt. cni-plugins solved that by referencing ghcr.io directly, and the earlier commits in this branch were about to do the same to the other three. That fixes the rebuild and breaks the normal case: every steady-state pull then leaves the network for an image the local registry already holds. The recovery source and the everyday source are different things — the same distinction Flux's git source makes between the in-cluster Gitea and the GitHub break-glass mirror. So the fallback moves into machine.registries.mirrors, where containerd takes an ordered endpoint list: gitea.renner.uno first, ghcr.io second. Normal operation is unchanged and local; ghcr answers only in the window where the first cannot. No manifest names an external registry any more, cni-plugins included. Containerd keeps the repository path when it falls through, so both registries have to agree on it. `gitea_admin/alpine-jq` cannot be served by ghcr, where the namespace is fixed to the account name — hence the konairius/ path on both, one digest pushed to each by the same build. Verified equal on this run: Pushed gitea.renner.uno/konairius/alpine-jq@sha256:87d0bf88... Pushed ghcr.io/konairius/alpine-jq@sha256:87d0bf88... The ghcr namespace stays private — three of the images this will eventually cover are own applications, not alpine+jq — so the node also carries a credential, in machine.registries.config, read from pass as the sixth required secret. An imagePullSecret cannot work here: it would need to exist in five namespaces before any of them does, and it would come from OpenBao, which needs Ceph, which needs these very images. Production still needs the machine config applied before a rebuild relies on it; in steady state the references resolve from Gitea with or without it. --- .../zigbee2mqtt-deployment.yaml | 12 +++--- kubernetes/apps/paperless/paperless.yaml | 15 ++++--- kubernetes/apps/trmnl/renderer.yaml | 10 +++-- kubernetes/bootstrap.sh | 41 ++++++++++++++----- .../multus/cni-plugins-installer.yaml | 37 +++++++++++------ 5 files changed, 77 insertions(+), 38 deletions(-) diff --git a/kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml b/kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml index 5fee1095..e17a0c69 100644 --- a/kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml +++ b/kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml @@ -58,11 +58,13 @@ spec: # resolving — but the image then came from the branch build rather than # master. Re-point it at a master-built tag when convenient. # - # ghcr.io, not gitea.renner.uno: this init container runs during a - # cold rebuild, when Gitea does not exist because it runs inside the - # cluster being rebuilt. The mirror is private; the nodes hold the - # credential in machine.registries, applied by bootstrap.sh from pass. - image: ghcr.io/konairius/alpine-jq:2026-09-04 + # konairius/, not gitea_admin/: this init container runs during a cold + # rebuild, when Gitea does not exist because it runs inside the + # cluster being rebuilt. The node then falls back to a ghcr.io copy + # (machine.registries.mirrors, applied by bootstrap.sh), and containerd + # keeps the repository path when it falls through — so the path has to + # exist on both registries. Steady state still pulls from Gitea. + image: gitea.renner.uno/konairius/alpine-jq:2026-09-04 command: - sh - -c diff --git a/kubernetes/apps/paperless/paperless.yaml b/kubernetes/apps/paperless/paperless.yaml index ed7cfa53..641f4dbb 100644 --- a/kubernetes/apps/paperless/paperless.yaml +++ b/kubernetes/apps/paperless/paperless.yaml @@ -94,14 +94,19 @@ spec: # installed at runtime. Dated tag so a change is a reviewable diff # here, matching the ci-ansible / alpine-jq convention. # - # Pulled from the ghcr.io mirror, not gitea.renner.uno. This is an + # konairius/, not gitea_admin/, and that is not cosmetic. This is an # INIT container, so paperless does not start without it — and during # a cold rebuild gitea.renner.uno does not exist, because Gitea runs # inside the cluster being rebuilt. A rehearsal left this pod in - # Init:ErrImagePull on exactly this image. The mirror is private; the - # nodes carry the credential in machine.registries, applied by - # bootstrap.sh from pass, so no imagePullSecret is needed here. - image: ghcr.io/konairius/alpine-openssl:2026-09-04 + # Init:ErrImagePull on exactly this image. + # + # The node falls back to a ghcr.io copy in that window + # (machine.registries.mirrors, applied by bootstrap.sh). Containerd + # keeps the repository path when it falls through, so the path has to + # exist on both registries — hence konairius/, which matches the ghcr + # namespace. Steady state still pulls from Gitea and never leaves the + # network. + image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04 command: - sh - -c diff --git a/kubernetes/apps/trmnl/renderer.yaml b/kubernetes/apps/trmnl/renderer.yaml index 27e025f7..c90669e4 100644 --- a/kubernetes/apps/trmnl/renderer.yaml +++ b/kubernetes/apps/trmnl/renderer.yaml @@ -177,12 +177,14 @@ spec: # webhook start failing with "expired JWT access token" about # thirty minutes after seeding. - name: render-schedule - # ghcr.io, not gitea.renner.uno: this container starts during a cold + # konairius/, not gitea_admin/: this container starts during a cold # rebuild, when Gitea does not exist because it runs inside the # cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff. - # The mirror is private; the nodes hold the credential in - # machine.registries, applied by bootstrap.sh from pass. - image: ghcr.io/konairius/alpine-jq:2026-09-04 + # The node falls back to a ghcr.io copy then + # (machine.registries.mirrors, applied by bootstrap.sh), and containerd + # keeps the repository path, so it has to exist on both registries. + # Steady state still pulls from Gitea. + image: gitea.renner.uno/konairius/alpine-jq:2026-09-04 command: - sh - -c diff --git a/kubernetes/bootstrap.sh b/kubernetes/bootstrap.sh index d5242232..977872e2 100755 --- a/kubernetes/bootstrap.sh +++ b/kubernetes/bootstrap.sh @@ -528,21 +528,35 @@ if phase_runs talos; then mkdir -p "$(dirname "$TALOSCONFIG_PATH")" cp "$TALOS_GEN_DIR/talosconfig" "$TALOSCONFIG_PATH" - # Node-level pull credential for the private image mirror. + # Lets a node pull the images it needs before the registry that normally + # serves them exists. # # Several apps need an image while the cluster is still being built — # paperless's fetch-bridge-cert init container, zigbee2mqtt's seed-data init - # container, the trmnl renderer. Those images cannot come from - # gitea.renner.uno, which runs inside the cluster being rebuilt, so they are - # mirrored to a private ghcr.io namespace. Pulling from there needs a - # credential, and the usual place for one — an imagePullSecret — does not - # work here twice over: it would have to exist in five namespaces before any - # of them exists, and it would come from OpenBao, which needs Ceph, which - # needs these very images. + # container, the trmnl renderer, and the CNI plugins every pod depends on. + # They are referenced at gitea.renner.uno, which is correct for normal + # operation and useless during a rebuild, because Gitea runs inside the + # cluster being rebuilt. # - # machine.registries.config is containerd's own auth, so it is in place - # before the first pod is scheduled and applies to every pull on the node - # regardless of namespace. Field names follow .docker/config.json + # `mirrors` gives containerd an ordered list to try. Gitea is named first, so + # nothing changes once the cluster is up: pulls stay local and never leave + # the network. ghcr.io is second and only answers when the first cannot, + # which is exactly the rebuild window. The manifests therefore never name an + # external registry — the fallback is a property of the node, not of the + # deployment. + # + # Containerd keeps the repository path when it falls through, which is why + # the images are published under konairius/ on BOTH registries + # (.woodpecker/build-ci-images.yml). Verified equal: the same build pushes + # one digest to both paths. + # + # The ghcr namespace is private, so it needs a credential — and the usual + # place for one, an imagePullSecret, fails here twice over: it would have to + # exist in five namespaces before any of them exists, and it would come from + # OpenBao, which needs Ceph, which needs these very images. + # machine.registries.config is containerd's own auth, in place before the + # first pod is scheduled and applying to every pull on the node regardless + # of namespace. Field names follow .docker/config.json # (pkg/machinery/config/types/v1alpha1, RegistryAuthConfig). # # The patch is written into the in-memory secrets dir, never into the repo @@ -554,6 +568,11 @@ if phase_runs talos; then ( umask 077; cat > "$out" <