feat(bootstrap): let the node fall back to the mirror, not the manifest
Four images are needed while the cluster is still being built: the CNI plugins
every pod depends on, paperless's fetch-bridge-cert init container,
zigbee2mqtt's seed-data init container and the trmnl renderer. None of them can
come from gitea.renner.uno during a rebuild, because Gitea runs inside the
cluster being rebuilt.
cni-plugins solved that by referencing ghcr.io directly, and the earlier commits
in this branch were about to do the same to the other three. That fixes the
rebuild and breaks the normal case: every steady-state pull then leaves the
network for an image the local registry already holds. The recovery source and
the everyday source are different things — the same distinction Flux's git
source makes between the in-cluster Gitea and the GitHub break-glass mirror.
So the fallback moves into machine.registries.mirrors, where containerd takes an
ordered endpoint list: gitea.renner.uno first, ghcr.io second. Normal operation
is unchanged and local; ghcr answers only in the window where the first cannot.
No manifest names an external registry any more, cni-plugins included.
Containerd keeps the repository path when it falls through, so both registries
have to agree on it. `gitea_admin/alpine-jq` cannot be served by ghcr, where the
namespace is fixed to the account name — hence the konairius/ path on both, one
digest pushed to each by the same build. Verified equal on this run:
Pushed gitea.renner.uno/konairius/alpine-jq@sha256:87d0bf88...
Pushed ghcr.io/konairius/alpine-jq@sha256:87d0bf88...
The ghcr namespace stays private — three of the images this will eventually
cover are own applications, not alpine+jq — so the node also carries a
credential, in machine.registries.config, read from pass as the sixth required
secret. An imagePullSecret cannot work here: it would need to exist in five
namespaces before any of them does, and it would come from OpenBao, which needs
Ceph, which needs these very images.
Production still needs the machine config applied before a rebuild relies on
it; in steady state the references resolve from Gitea with or without it.
This commit is contained in:
@@ -58,11 +58,13 @@ spec:
|
||||
# resolving — but the image then came from the branch build rather than
|
||||
# master. Re-point it at a master-built tag when convenient.
|
||||
#
|
||||
# ghcr.io, not gitea.renner.uno: this init container runs during a
|
||||
# cold rebuild, when Gitea does not exist because it runs inside the
|
||||
# cluster being rebuilt. The mirror is private; the nodes hold the
|
||||
# credential in machine.registries, applied by bootstrap.sh from pass.
|
||||
image: ghcr.io/konairius/alpine-jq:2026-09-04
|
||||
# konairius/, not gitea_admin/: this init container runs during a cold
|
||||
# rebuild, when Gitea does not exist because it runs inside the
|
||||
# cluster being rebuilt. The node then falls back to a ghcr.io copy
|
||||
# (machine.registries.mirrors, applied by bootstrap.sh), and containerd
|
||||
# keeps the repository path when it falls through — so the path has to
|
||||
# exist on both registries. Steady state still pulls from Gitea.
|
||||
image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
|
||||
@@ -94,14 +94,19 @@ spec:
|
||||
# installed at runtime. Dated tag so a change is a reviewable diff
|
||||
# here, matching the ci-ansible / alpine-jq convention.
|
||||
#
|
||||
# Pulled from the ghcr.io mirror, not gitea.renner.uno. This is an
|
||||
# konairius/, not gitea_admin/, and that is not cosmetic. This is an
|
||||
# INIT container, so paperless does not start without it — and during
|
||||
# a cold rebuild gitea.renner.uno does not exist, because Gitea runs
|
||||
# inside the cluster being rebuilt. A rehearsal left this pod in
|
||||
# Init:ErrImagePull on exactly this image. The mirror is private; the
|
||||
# nodes carry the credential in machine.registries, applied by
|
||||
# bootstrap.sh from pass, so no imagePullSecret is needed here.
|
||||
image: ghcr.io/konairius/alpine-openssl:2026-09-04
|
||||
# Init:ErrImagePull on exactly this image.
|
||||
#
|
||||
# The node falls back to a ghcr.io copy in that window
|
||||
# (machine.registries.mirrors, applied by bootstrap.sh). Containerd
|
||||
# keeps the repository path when it falls through, so the path has to
|
||||
# exist on both registries — hence konairius/, which matches the ghcr
|
||||
# namespace. Steady state still pulls from Gitea and never leaves the
|
||||
# network.
|
||||
image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
|
||||
@@ -177,12 +177,14 @@ spec:
|
||||
# webhook start failing with "expired JWT access token" about
|
||||
# thirty minutes after seeding.
|
||||
- name: render-schedule
|
||||
# ghcr.io, not gitea.renner.uno: this container starts during a cold
|
||||
# konairius/, not gitea_admin/: this container starts during a cold
|
||||
# rebuild, when Gitea does not exist because it runs inside the
|
||||
# cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff.
|
||||
# The mirror is private; the nodes hold the credential in
|
||||
# machine.registries, applied by bootstrap.sh from pass.
|
||||
image: ghcr.io/konairius/alpine-jq:2026-09-04
|
||||
# The node falls back to a ghcr.io copy then
|
||||
# (machine.registries.mirrors, applied by bootstrap.sh), and containerd
|
||||
# keeps the repository path, so it has to exist on both registries.
|
||||
# Steady state still pulls from Gitea.
|
||||
image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
|
||||
+30
-11
@@ -528,21 +528,35 @@ if phase_runs talos; then
|
||||
mkdir -p "$(dirname "$TALOSCONFIG_PATH")"
|
||||
cp "$TALOS_GEN_DIR/talosconfig" "$TALOSCONFIG_PATH"
|
||||
|
||||
# Node-level pull credential for the private image mirror.
|
||||
# Lets a node pull the images it needs before the registry that normally
|
||||
# serves them exists.
|
||||
#
|
||||
# Several apps need an image while the cluster is still being built —
|
||||
# paperless's fetch-bridge-cert init container, zigbee2mqtt's seed-data init
|
||||
# container, the trmnl renderer. Those images cannot come from
|
||||
# gitea.renner.uno, which runs inside the cluster being rebuilt, so they are
|
||||
# mirrored to a private ghcr.io namespace. Pulling from there needs a
|
||||
# credential, and the usual place for one — an imagePullSecret — does not
|
||||
# work here twice over: it would have to exist in five namespaces before any
|
||||
# of them exists, and it would come from OpenBao, which needs Ceph, which
|
||||
# needs these very images.
|
||||
# container, the trmnl renderer, and the CNI plugins every pod depends on.
|
||||
# They are referenced at gitea.renner.uno, which is correct for normal
|
||||
# operation and useless during a rebuild, because Gitea runs inside the
|
||||
# cluster being rebuilt.
|
||||
#
|
||||
# machine.registries.config is containerd's own auth, so it is in place
|
||||
# before the first pod is scheduled and applies to every pull on the node
|
||||
# regardless of namespace. Field names follow .docker/config.json
|
||||
# `mirrors` gives containerd an ordered list to try. Gitea is named first, so
|
||||
# nothing changes once the cluster is up: pulls stay local and never leave
|
||||
# the network. ghcr.io is second and only answers when the first cannot,
|
||||
# which is exactly the rebuild window. The manifests therefore never name an
|
||||
# external registry — the fallback is a property of the node, not of the
|
||||
# deployment.
|
||||
#
|
||||
# Containerd keeps the repository path when it falls through, which is why
|
||||
# the images are published under konairius/ on BOTH registries
|
||||
# (.woodpecker/build-ci-images.yml). Verified equal: the same build pushes
|
||||
# one digest to both paths.
|
||||
#
|
||||
# The ghcr namespace is private, so it needs a credential — and the usual
|
||||
# place for one, an imagePullSecret, fails here twice over: it would have to
|
||||
# exist in five namespaces before any of them exists, and it would come from
|
||||
# OpenBao, which needs Ceph, which needs these very images.
|
||||
# machine.registries.config is containerd's own auth, in place before the
|
||||
# first pod is scheduled and applying to every pull on the node regardless
|
||||
# of namespace. Field names follow .docker/config.json
|
||||
# (pkg/machinery/config/types/v1alpha1, RegistryAuthConfig).
|
||||
#
|
||||
# The patch is written into the in-memory secrets dir, never into the repo
|
||||
@@ -554,6 +568,11 @@ if phase_runs talos; then
|
||||
( umask 077; cat > "$out" <<EOF
|
||||
machine:
|
||||
registries:
|
||||
mirrors:
|
||||
gitea.renner.uno:
|
||||
endpoints:
|
||||
- https://gitea.renner.uno
|
||||
- https://ghcr.io
|
||||
config:
|
||||
ghcr.io:
|
||||
auth:
|
||||
|
||||
@@ -50,20 +50,31 @@ spec:
|
||||
# Dated tag so a version change is a reviewable diff here, matching the
|
||||
# ci-ansible / alpine-jq convention.
|
||||
#
|
||||
# Pulled from the PUBLIC ghcr.io mirror, not gitea.renner.uno. Gitea
|
||||
# runs inside this cluster, so during a rebuild it does not exist: a
|
||||
# cold-rebuild rehearsal sat here in ImagePullBackOff with "no route to
|
||||
# host", and because the plugins never installed there was no macvlan,
|
||||
# so Multus could not attach ceph-san, so Rook's network canary never
|
||||
# started, so the CephCluster hung at "Configuring Ceph Mons" and
|
||||
# OpenBao waited forever on a PVC that could not bind. The failure
|
||||
# surfaced four layers from its cause. It must stay credential-free —
|
||||
# an imagePullSecret would come from OpenBao, which needs Ceph, which
|
||||
# needs these binaries.
|
||||
# konairius/, not gitea_admin/, and the difference is what makes a
|
||||
# rebuild possible. Gitea runs inside this cluster, so during a rebuild
|
||||
# it does not exist: a rehearsal sat here in ImagePullBackOff with "no
|
||||
# route to host", and because the plugins never installed there was no
|
||||
# macvlan, so Multus could not attach ceph-san, so Rook's network
|
||||
# canary never started, so the CephCluster hung at "Configuring Ceph
|
||||
# Mons" and OpenBao waited forever on a PVC that could not bind. The
|
||||
# failure surfaced four layers from its cause.
|
||||
#
|
||||
# The build pushes the identical image to both registries; this digest
|
||||
# was verified equal on each. Bumping the tag means mirroring first.
|
||||
image: ghcr.io/konairius/cni-plugins:2026-07-30
|
||||
# This used to point straight at ghcr.io, which fixed the rebuild and
|
||||
# broke the normal case: every steady-state pull then left the network
|
||||
# for an image the local registry already holds. The fallback belongs
|
||||
# to the node, not to the manifest — machine.registries.mirrors names
|
||||
# Gitea first and ghcr.io second (bootstrap.sh), so nothing here has to
|
||||
# know about the recovery path.
|
||||
#
|
||||
# Containerd keeps the repository path when it falls through, so the
|
||||
# path must resolve on both registries; the build pushes one digest to
|
||||
# konairius/ on each, verified equal. Bumping the tag means building
|
||||
# first — an unpublished tag is unpullable on both.
|
||||
#
|
||||
# No imagePullSecret: the ghcr namespace is private and its credential
|
||||
# is in machine.registries too, because a Secret would have to come
|
||||
# from OpenBao, which needs Ceph, which needs these binaries.
|
||||
image: gitea.renner.uno/konairius/cni-plugins:2026-09-04
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
|
||||
Reference in New Issue
Block a user