feat(tooling): Hetzner throwaway site and rebuild-harness hardening

None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.

New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.

The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.

bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.

import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.

Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,3 +1,18 @@
+# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
+# isolated from the apps namespace.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    # Image-build steps run a privileged docker/buildx plugin pod (see the
+    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
+    # without this the privileged step pods are rejected at admission. The
+    # namespace is dedicated to ephemeral CI step pods, so scoping the
+    # exemption here keeps it off apps/.
+    pod-security.kubernetes.io/enforce: privileged
+---
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,19 +28,6 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
-    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
-    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
-    # kind is unknown when this path is first applied, and a server-side
-    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
-    #
-    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
-    #   no matches for kind "ObjectStore" in version ...
-    #
-    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
-    # woodpecker — already declares it. This was the only one that did not, and
-    # it took speech-to-phrase, zone-configurator and matter-server down with
-    # it, since all three wait on home-assistant.
-    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,20 +122,8 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
-  # existing cluster could ever hit and no new one could ever escape:
-  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
-  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
-  # defined in this very Kustomization. A cold rebuild stopped here for good,
-  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
-  #
-  # The edge was never needed. infrastructure runs wait: false, so it applies
-  # the rook-ceph namespace and the operator immediately and only its readiness
-  # waits on the healthChecks — everything this path needs exists long before
-  # infrastructure reports Ready. On an empty API server this Kustomization
-  # retries on retryInterval until the operator installs its CRDs, the same
-  # self-healing shape post-crds already relies on; on a live cluster the CRDs
-  # are always present and nothing changes.
+  dependsOn:
+    - name: infrastructure
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -190,31 +190,3 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
----
-# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
-# from `apps`. Distinct from `woodpecker`, which holds the service itself.
-#
-# Here rather than in apps/woodpecker for the reason spelled out above that
-# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
-# ../network-policies/, and infrastructure-crds runs before every app KS. With
-# the namespace declared in the workload KS the order inverted, and a rebuild
-# stopped with
-#
-#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
-#   namespaces "woodpecker-ci" not found
-#
-# which held infrastructure-crds, and with it all twenty app Kustomizations
-# that depend on it. The same reasoning was already written down for
-# `woodpecker` and simply not carried across to its sibling.
-#
-# Image-build steps run a privileged docker/buildx plugin pod (see the server's
-# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
-# label the privileged step pods are rejected at admission. The namespace holds
-# nothing but ephemeral CI step pods, so the exemption stays off apps/.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    pod-security.kubernetes.io/enforce: privileged

kubernetes/talos/patches/hcloud-1.yaml

@@ -0,0 +1,69 @@
+# First machine of the throwaway cluster on Hetzner.
+#
+# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
+# order bootstrap.sh attaches them.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-1
+  nodeLabels:
+    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
+    # rather than MAC: the address follows from the order bootstrap.sh attaches
+    # the networks, while the MAC is assigned by Hetzner at attach time and is
+    # not known when this file is written.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+# In the generated schema rather than machine.network.hostname: naming the host
+# in both is rejected at apply time. `auto: off` switches off the generated
+# `auto: stable`, which is mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-1
+---
+# Every interface is stated, the public one included. Talos stops running its
+# default DHCP operators for the whole machine as soon as one new-style network
+# document exists, so an omitted interface gets no address.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# Ceph cluster network, renamed to the role name by the udev rule in the
+# machine section. The node holds no address here; the link exists for the OSD
+# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    # Two disks: the system disk and the attached volume. Matching on transport
+    # as the qemu patches do selects nothing here — a Hetzner volume arrives
+    # over virtio-scsi, and a selector matching zero disks is skipped silently.
+    match: '!system_disk'
+  # bootstrap.sh reads this size and creates the volume to match.
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -0,0 +1,50 @@
+# Second machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-2
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-2
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -0,0 +1,50 @@
+# Third machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-3
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-3
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/site-hcloud.yaml

@@ -0,0 +1,25 @@
+# Site patch for the throwaway cluster on Hetzner Cloud.
+#
+# This site exists for one thing the other two cannot test: whether a node that
+# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
+# answers, so containerd never reaches the second endpoint in
+# machine.registries.mirrors and the fallback stays unproven.
+#
+# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
+# frames from it do not cross between servers — measured: same-node ARP
+# resolves, cross-node does not. OSD replication therefore does not run here.
+#
+# What belongs in this file is what the cluster is. Interfaces, disks and names
+# are machine properties and live in the node patches.
+machine:
+  kubelet:
+    nodeIP:
+      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
+      # kubelet can pick the public NIC.
+      validSubnets:
+        - 10.90.0.0/16
+cluster:
+  # Three control planes that also carry workloads, which is the production
+  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
+  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
+  allowSchedulingOnControlPlanes: true

kubernetes/vmtest-job.yaml

@@ -0,0 +1,134 @@
+# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
+#
+# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
+# one-shot test, and a Kustomization would recreate it after every completion.
+# It is applied by the vmtest pipeline, or by hand for a single run.
+#
+# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
+# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
+# a single run left under 1G free and two overlapping runs OOM-killed the
+# session. The nodes have ~94G each with about 70G unused.
+#
+# The whole script runs here, flux and OpenBao included, so that what a rebuild
+# rehearses is what a real recovery would execute. The unseal key is not baked
+# in and not stored as a Secret: the operator streams all five secrets into an
+# in-memory emptyDir at the moment the run asks for them, and they die with the
+# Job without ever reaching etcd or a backup of it.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vmtest
+  labels:
+    # The pod is privileged; say so rather than have the admission policy
+    # reject it for reasons that look unrelated.
+    pod-security.kubernetes.io/enforce: privileged
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: vmtest
+  namespace: vmtest
+spec:
+  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
+  ttlSecondsAfterFinished: 86400
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: vmtest
+    spec:
+      restartPolicy: Never
+      # Nothing here should be able to reach the cluster it runs in. The phase
+      # points KUBECONFIG at its own throwaway cluster, and without a mounted
+      # token a stray kubectl cannot fall back to this one.
+      automountServiceAccountToken: false
+      containers:
+        - name: vmtest
+          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
+          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
+          # three builds old, which predated OVMF, and died with "the required
+          # flash image was not found" before the first VM booted. The tag is
+          # carried alongside for readability — containerd resolves the digest.
+          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
+          # Privileged for /dev/kvm and for the bridges and taps the provisioner
+          # builds. It needs no hostNetwork: the provisioner's network lives in
+          # this pod's own namespace, and the script that talks to 10.5.0.x runs
+          # in the same pod.
+          securityContext:
+            privileged: true
+          # A hard ceiling, because the failure mode is known: an unbounded run
+          # is what OOM-killed the workstation. 40G is the VMs (16G control
+          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
+          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
+          # of room for the CNI DaemonSets themselves. The control plane went
+          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
+          # memory and the kubelet evicted kube-apiserver to admit a
+          # node-critical pod — bootstrap.sh carries the full account next to
+          # the sizing flags. The hosts have ~95GiB each, so this fits.
+          resources:
+            requests:
+              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
+              # the guests are never all runnable at once (one cluster booting
+              # in sequence), and there is no CPU limit, so the pod may burst
+              # past the request when the node has headroom. The request only
+              # keeps the scheduler from stacking this next to another big
+              # burst consumer.
+              cpu: "6"
+              memory: 42Gi
+            limits:
+              memory: 48Gi
+          env:
+            # talosctl caches the image and the CNI bundle under $HOME; point it
+            # at the scratch volume rather than the container layer.
+            - name: HOME
+              value: /work
+            # The VM disks. Short on purpose — the provisioner puts a qemu
+            # monitor socket under it and sun_path caps the path at 108 bytes.
+            - name: VM_STATE
+              value: /work/vm
+            # Which branch to test. master unless a run is checking a change.
+            - name: BRANCH
+              value: master
+            # Secrets are streamed in here at the moment the run needs them,
+            # never baked into the image and never stored as a Secret. They live
+            # in this emptyDir for the length of the Job and go with it — so
+            # they do not reach etcd, or any backup of it.
+            - name: OPENBAO_SECRETS_DIR
+              value: /secrets
+          command: ["/bin/bash", "-ec"]
+          args:
+            - |
+              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
+              # that clones from the cluster it is rehearsing the loss of
+              # cannot start when it matters — and a run already died here on
+              # resolving gitea.renner.uno.
+              git clone --depth 1 --branch "$BRANCH" \
+                https://github.com/konairius/Homelab.git /work/homelab
+              exec /work/homelab/kubernetes/bootstrap.sh vmtest
+          volumeMounts:
+            - name: work
+              mountPath: /work
+            - name: secrets
+              mountPath: /secrets
+            - name: dev-kvm
+              mountPath: /dev/kvm
+      volumes:
+        # The VM disks and the provisioner state. emptyDir on the node's disk
+        # rather than a PVC: it is scratch that dies with the Job, and putting it
+        # on Ceph would have the test's own writes compete with the storage it is
+        # meant to leave alone.
+        - name: work
+          emptyDir:
+            # The VM images are sparse, so this bounds what is actually written,
+            # not the 232GiB the geometry declares. 60Gi was already half spent
+            # by four 10GiB systems disks; the node carries this comfortably with
+            # ~810GB free on the filesystem behind it.
+            sizeLimit: 150Gi
+        # In memory, so the material never touches a node's disk.
+        - name: secrets
+          emptyDir:
+            medium: Memory
+            sizeLimit: 1Mi
+        - name: dev-kvm
+          hostPath:
+            path: /dev/kvm
+            type: CharDevice

scripts/import-cluster-access.sh

@@ -13,7 +13,18 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-export AWS_PROFILE="secrets"
+# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
+# The default is unchanged and deliberately so: "secrets" resolves to working
+# credentials and this script has always used it. Both "secrets" and "hetzner"
+# exist in ~/.aws/credentials on the workstation.
+#
+# The defect is only the assignment. Forcing the value defeated the env-based
+# path this file's own header offers, and it also overrode a caller who had
+# already selected a profile — which is how a 403 from HeadObject can appear
+# while the credentials the caller intended were perfectly good. That error
+# reads as a bucket permissions problem and sends you looking in the wrong
+# place.
+export AWS_PROFILE="${AWS_PROFILE:-secrets}"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }