feat(tooling): Hetzner throwaway site and rebuild-harness hardening

None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.

New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.

The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.

bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.

import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.

Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/talos/patches/hcloud-1.yaml

@@ -0,0 +1,69 @@
+# First machine of the throwaway cluster on Hetzner.
+#
+# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
+# order bootstrap.sh attaches them.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-1
+  nodeLabels:
+    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
+    # rather than MAC: the address follows from the order bootstrap.sh attaches
+    # the networks, while the MAC is assigned by Hetzner at attach time and is
+    # not known when this file is written.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+# In the generated schema rather than machine.network.hostname: naming the host
+# in both is rejected at apply time. `auto: off` switches off the generated
+# `auto: stable`, which is mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-1
+---
+# Every interface is stated, the public one included. Talos stops running its
+# default DHCP operators for the whole machine as soon as one new-style network
+# document exists, so an omitted interface gets no address.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# Ceph cluster network, renamed to the role name by the udev rule in the
+# machine section. The node holds no address here; the link exists for the OSD
+# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    # Two disks: the system disk and the attached volume. Matching on transport
+    # as the qemu patches do selects nothing here — a Hetzner volume arrives
+    # over virtio-scsi, and a selector matching zero disks is skipped silently.
+    match: '!system_disk'
+  # bootstrap.sh reads this size and creates the volume to match.
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -0,0 +1,50 @@
+# Second machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-2
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-2
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -0,0 +1,50 @@
+# Third machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-3
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-3
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/site-hcloud.yaml

@@ -0,0 +1,25 @@
+# Site patch for the throwaway cluster on Hetzner Cloud.
+#
+# This site exists for one thing the other two cannot test: whether a node that
+# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
+# answers, so containerd never reaches the second endpoint in
+# machine.registries.mirrors and the fallback stays unproven.
+#
+# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
+# frames from it do not cross between servers — measured: same-node ARP
+# resolves, cross-node does not. OSD replication therefore does not run here.
+#
+# What belongs in this file is what the cluster is. Interfaces, disks and names
+# are machine properties and live in the node patches.
+machine:
+  kubelet:
+    nodeIP:
+      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
+      # kubelet can pick the public NIC.
+      validSubnets:
+        - 10.90.0.0/16
+cluster:
+  # Three control planes that also carry workloads, which is the production
+  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
+  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
+  allowSchedulingOnControlPlanes: true

kubernetes/vmtest-job.yaml

@@ -0,0 +1,134 @@
+# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
+#
+# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
+# one-shot test, and a Kustomization would recreate it after every completion.
+# It is applied by the vmtest pipeline, or by hand for a single run.
+#
+# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
+# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
+# a single run left under 1G free and two overlapping runs OOM-killed the
+# session. The nodes have ~94G each with about 70G unused.
+#
+# The whole script runs here, flux and OpenBao included, so that what a rebuild
+# rehearses is what a real recovery would execute. The unseal key is not baked
+# in and not stored as a Secret: the operator streams all five secrets into an
+# in-memory emptyDir at the moment the run asks for them, and they die with the
+# Job without ever reaching etcd or a backup of it.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vmtest
+  labels:
+    # The pod is privileged; say so rather than have the admission policy
+    # reject it for reasons that look unrelated.
+    pod-security.kubernetes.io/enforce: privileged
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: vmtest
+  namespace: vmtest
+spec:
+  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
+  ttlSecondsAfterFinished: 86400
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: vmtest
+    spec:
+      restartPolicy: Never
+      # Nothing here should be able to reach the cluster it runs in. The phase
+      # points KUBECONFIG at its own throwaway cluster, and without a mounted
+      # token a stray kubectl cannot fall back to this one.
+      automountServiceAccountToken: false
+      containers:
+        - name: vmtest
+          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
+          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
+          # three builds old, which predated OVMF, and died with "the required
+          # flash image was not found" before the first VM booted. The tag is
+          # carried alongside for readability — containerd resolves the digest.
+          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
+          # Privileged for /dev/kvm and for the bridges and taps the provisioner
+          # builds. It needs no hostNetwork: the provisioner's network lives in
+          # this pod's own namespace, and the script that talks to 10.5.0.x runs
+          # in the same pod.
+          securityContext:
+            privileged: true
+          # A hard ceiling, because the failure mode is known: an unbounded run
+          # is what OOM-killed the workstation. 40G is the VMs (16G control
+          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
+          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
+          # of room for the CNI DaemonSets themselves. The control plane went
+          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
+          # memory and the kubelet evicted kube-apiserver to admit a
+          # node-critical pod — bootstrap.sh carries the full account next to
+          # the sizing flags. The hosts have ~95GiB each, so this fits.
+          resources:
+            requests:
+              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
+              # the guests are never all runnable at once (one cluster booting
+              # in sequence), and there is no CPU limit, so the pod may burst
+              # past the request when the node has headroom. The request only
+              # keeps the scheduler from stacking this next to another big
+              # burst consumer.
+              cpu: "6"
+              memory: 42Gi
+            limits:
+              memory: 48Gi
+          env:
+            # talosctl caches the image and the CNI bundle under $HOME; point it
+            # at the scratch volume rather than the container layer.
+            - name: HOME
+              value: /work
+            # The VM disks. Short on purpose — the provisioner puts a qemu
+            # monitor socket under it and sun_path caps the path at 108 bytes.
+            - name: VM_STATE
+              value: /work/vm
+            # Which branch to test. master unless a run is checking a change.
+            - name: BRANCH
+              value: master
+            # Secrets are streamed in here at the moment the run needs them,
+            # never baked into the image and never stored as a Secret. They live
+            # in this emptyDir for the length of the Job and go with it — so
+            # they do not reach etcd, or any backup of it.
+            - name: OPENBAO_SECRETS_DIR
+              value: /secrets
+          command: ["/bin/bash", "-ec"]
+          args:
+            - |
+              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
+              # that clones from the cluster it is rehearsing the loss of
+              # cannot start when it matters — and a run already died here on
+              # resolving gitea.renner.uno.
+              git clone --depth 1 --branch "$BRANCH" \
+                https://github.com/konairius/Homelab.git /work/homelab
+              exec /work/homelab/kubernetes/bootstrap.sh vmtest
+          volumeMounts:
+            - name: work
+              mountPath: /work
+            - name: secrets
+              mountPath: /secrets
+            - name: dev-kvm
+              mountPath: /dev/kvm
+      volumes:
+        # The VM disks and the provisioner state. emptyDir on the node's disk
+        # rather than a PVC: it is scratch that dies with the Job, and putting it
+        # on Ceph would have the test's own writes compete with the storage it is
+        # meant to leave alone.
+        - name: work
+          emptyDir:
+            # The VM images are sparse, so this bounds what is actually written,
+            # not the 232GiB the geometry declares. 60Gi was already half spent
+            # by four 10GiB systems disks; the node carries this comfortably with
+            # ~810GB free on the filesystem behind it.
+            sizeLimit: 150Gi
+        # In memory, so the material never touches a node's disk.
+        - name: secrets
+          emptyDir:
+            medium: Memory
+            sizeLimit: 1Mi
+        - name: dev-kvm
+          hostPath:
+            path: /dev/kvm
+            type: CharDevice

scripts/import-cluster-access.sh

@@ -13,7 +13,18 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-export AWS_PROFILE="secrets"
+# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
+# The default is unchanged and deliberately so: "secrets" resolves to working
+# credentials and this script has always used it. Both "secrets" and "hetzner"
+# exist in ~/.aws/credentials on the workstation.
+#
+# The defect is only the assignment. Forcing the value defeated the env-based
+# path this file's own header offers, and it also overrode a caller who had
+# already selected a profile — which is how a 403 from HeadObject can appear
+# while the credentials the caller intended were perfectly good. That error
+# reads as a bucket permissions problem and sends you looking in the wrong
+# place.
+export AWS_PROFILE="${AWS_PROFILE:-secrets}"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }