feat(tooling): Hetzner throwaway site and rebuild-harness hardening

None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.

New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.

The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.

bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.

import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.

Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/authentik/kustomization.yaml

@@ -16,6 +16,7 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
+  - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -1,46 +0,0 @@
-# Cilium's agent ServiceMonitor, declared here rather than by the chart.
-#
-# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
-# taken field for field off the live object so scraping is unchanged: same name,
-# same namespace, same selector, same 10s interval and node relabeling.
-#
-# It lives here because the chart's template is gated on monitoring.coreos.com/v1
-# existing at install time, and on an empty cluster it does not — those CRDs
-# arrive with kube-prometheus-stack, which reconciles after infrastructure.
-# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
-# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
-#
-# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
-# why; this is the same move for the component where it mattered most.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: cilium-agent
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: cilium-agent
-    app.kubernetes.io/part-of: cilium
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: cilium-agent
-  namespaceSelector:
-    matchNames:
-      - kube-system
-  # The chart's Service keeps rendering (prometheus.enabled), so the target and
-  # its `metrics` port are there regardless of who declares the ServiceMonitor.
-  targetLabels:
-    - k8s-app
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        # Agent metrics are per-node; without this every series lands without a
-        # node label and BPF map pressure cannot be attributed to a machine.
-        - action: replace
-          sourceLabels:
-            - __meta_kubernetes_pod_node_name
-          targetLabel: node
-          replacement: ${1}

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -22,19 +22,8 @@ resources:
   # endpoint, which proves reachability, not that anything was written.
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
-  - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
-  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
-  # kube-prometheus-stack, which reconciles after infrastructure, so these
-  # cannot live beside the Velero HelmRelease.
-  - velero-podmonitor.yaml
-  - velero-servicemonitor.yaml
-  - authentik-outpost-servicemonitors.yaml
-  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
-  # when it cannot create these, which held the CephCluster in Progressing and
-  # deadlocked the whole apps layer. See the file for the cycle.
-  - rook-servicemonitors.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/rook-servicemonitors.yaml

@@ -1,93 +0,0 @@
-# Rook's two ServiceMonitors, declared here rather than by the operator.
-#
-# Reproduced field for field off the live objects in the production cluster, so
-# scraping is unchanged: same names, same namespace, same selectors, same 10s
-# interval and `cluster` relabeling. The only thing dropped is the CephCluster
-# ownerReference, which Rook set and Flux cannot.
-#
-# They live here because CephCluster.spec.monitoring.enabled is now false, and
-# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
-# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
-# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
-# reconcile.go, v1.20.7):
-#
-#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
-#         // exporter reconciled; its errors are logged and reconcile continues
-#         if cephCluster.Spec.Monitoring.Enabled {
-#             if err := EnableCephExporterServiceMonitor(...); err != nil {
-#                 return errors.Wrap(err, "failed to enable service monitor")
-#             }
-#         }
-#     }
-#
-# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
-# fails with "the server could not find the requested resource (post
-# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
-# aborts, and the CephCluster never leaves status.phase Progressing. That is
-# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
-# closes on itself:
-#
-#     infrastructure-ceph-cluster → CephCluster phase Ready
-#                                 → Rook reconcile settles
-#                                 → monitoring.coreos.com CRDs exist
-#                                 → kube-prometheus-stack (monitoring)
-#                                 → grafana-db
-#                                 → infrastructure-ceph-cluster
-#
-# Nothing outside that loop can break it, so a cold rebuild stopped here with
-# every one of the twenty app Kustomizations behind it.
-#
-# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
-# get, and for the same reason: the metrics stay where they are, only the
-# scrape wiring moves to the layer where its CRDs exist.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-exporter
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-exporter
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: ceph-exporter-http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      # Rook stamps the cluster name so series from several Ceph clusters stay
-      # separable. One cluster here, but dropping it would rename every series.
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster
----
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-mgr
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-mgr
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -70,23 +70,9 @@ spec:
       # map only shows up as "no space left on device" in the log while pod
       # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
       # alertable beforehand.
-      #
-      # That reasoning is about the metrics, which this line provides. The
-      # ServiceMonitor is only the scrape wiring and is declared standalone in
-      # apps/monitoring-rules instead — the same treatment velero/helmrelease
-      # already gives its own, and for the same reason: the chart's template is
-      # gated on monitoring.coreos.com/v1 existing at install time.
-      #
-      # Here it is worse than for velero. Cilium's chart does not skip the
-      # template, it aborts rendering — "Service Monitor requires
-      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
-      # healthChecks the infrastructure Kustomization waits on. So on an empty
-      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
-      # blocked monitoring, which is where those CRDs come from. The lowest
-      # layer in the cluster was waiting on the highest.
       enabled: true
       serviceMonitor:
-        enabled: false
+        enabled: true
     bpf:
       # Per-endpoint policy map size: one entry per (identity, direction,
       # protocol, port) allowed. When it fills, endpoint regeneration fails

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -99,7 +99,7 @@ spec:
     # required.
     #
     # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
     # so there is no live impact, but re-enabling it will require pointing the
     # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
     tls:

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -1,5 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - bouncer-key-externalsecret.yaml
+  - edge-bouncer-key-externalsecret.yaml
   - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/external-dns-unifi/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - helmrelease.yaml

kubernetes/infrastructure/netbird-ingress/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - daemonset.yaml

kubernetes/infrastructure/openbao/rotation/flux-git.yaml

@@ -10,8 +10,33 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret that publishes the rotated token to Flux lives in
-# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
+# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
+# own it) so a sync failure leaves the last-good value in place; Flux's
+# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
+# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
+# provisioned by hand (see policies/rotator-gitea-flux.hcl).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: flux-system-git-auth
+  namespace: flux-system
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: flux-system
+    creationPolicy: Merge
+  data:
+    - secretKey: username
+      remoteRef:
+        key: infrastructure/flux-git
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: infrastructure/flux-git
+        property: token
 ---
 apiVersion: v1
 kind: ServiceAccount

kubernetes/infrastructure/post-crds/flux-git-externalsecret.yaml

@@ -1,60 +0,0 @@
-# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
-# WITHOUT owning it, so a sync failure leaves the last-good value in place;
-# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
-# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
-# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
-#
-# See creationPolicy below for why that is Orphan and not Merge — the
-# distinction is what makes a recovery from the break-glass source possible.
-#
-# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
-# rotation CronJob: external-secrets.io CRDs are installed by the ESO
-# HelmRelease in the infrastructure Kustomization, so this manifest cannot
-# pass server-side dry-run there on a fresh cluster.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    # Orphan, not Merge: create the Secret if it is absent, but never own it.
-    #
-    # Merge cannot create — it only writes into a Secret that already exists,
-    # on the assumption that `flux bootstrap` made one. That assumption breaks
-    # in exactly the situation this cluster is rebuilt from. The break-glass
-    # bootstrap pulls from the GitHub mirror and keeps its token in
-    # flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
-    # so this ExternalSecret has nothing to merge into — and reports Ready while
-    # producing no Secret at all. The operator then performs the documented
-    # handover back to Gitea and finds the GitRepository pointing at a Secret
-    # that does not exist.
-    #
-    # Owner would create it but attach an ownerReference, so deleting this
-    # object would take Flux's credential with it. Orphan creates and updates
-    # without ownership, which is what the paragraph at the top of this file
-    # actually wanted: a sync failure, or the loss of this object, leaves the
-    # last-good value in place.
-    #
-    # Safe on a live cluster: the Secret holds exactly the two keys written
-    # below (verified against production), so writing it whole is the same
-    # result as merging into it.
-    creationPolicy: Orphan
-    # Explicit rather than relying on the default, because it is the half that
-    # matters here: if the OpenBao entry ever disappears, keep serving the
-    # credential Flux already has rather than leaving it with none.
-    deletionPolicy: Retain
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -10,25 +10,4 @@ resources:
   # configured (2026-07-13): a store that can't turn Ready would block this
   # KS's wait gate and freeze the dependsOn chain.
   - openbao-clustersecretstore.yaml
-  # ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
-  # the infrastructure Kustomization, so these cannot pass its server-side
-  # dry-run on a fresh cluster — Flux rejects the whole apply set, including
-  # the ESO HelmRelease that would have installed the CRDs. They sat next to
-  # their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
-  # nothing had ever applied this layer against an empty API server before.
-  - crowdsec-bouncer-key-externalsecret.yaml
-  - crowdsec-edge-bouncer-key-externalsecret.yaml
-  - external-dns-unifi-externalsecret.yaml
-  - flux-git-externalsecret.yaml
-  - netbird-ingress-externalsecret.yaml
-  - velero-externalsecret.yaml
-  - velero-hetzner-creds-externalsecret.yaml
-  # traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
-  # infrastructure. Traefik is deliberately absent from that Kustomization's
-  # healthChecks, so this KS may dry-run before the chart lands; it retries on
-  # retryInterval rather than deadlocking, because nothing here installs the
-  # CRDs it is waiting for.
-  - traefik-crowdsec-middleware.yaml
-  - traefik-rate-limit-middleware.yaml
-  - traefik-security-headers-middleware.yaml
   - ../network-policies/

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -78,20 +78,7 @@ spec:
     enabled: true
     ssl: false
   monitoring:
-    # false, and this is not a decision about metrics. `metricsDisabled` is the
-    # switch that turns metrics off; this one only decides whether Rook creates
-    # the ServiceMonitors — and creating them is fatal to the reconcile when it
-    # fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
-    # Rook's node-daemon reconcile aborted, the CephCluster never left
-    # status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
-    # waits on exactly that — with grafana-db → monitoring → those same CRDs
-    # behind it, closing the loop.
-    #
-    # The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
-    # only the scrape wiring moved, to apps/monitoring-rules/
-    # rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
-    # Both objects are reproduced there field for field off the live cluster.
-    enabled: false
+    enabled: true
   # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
   # Multus. The attachment is san-network-attachment.yaml.
   #

kubernetes/infrastructure/traefik/kustomization.yaml

@@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - helmrelease.yaml
+  - crowdsec-middleware.yaml
+  - security-headers-middleware.yaml
+  - rate-limit-middleware.yaml

kubernetes/infrastructure/velero/kustomization.yaml

@@ -1,9 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
   - snapshot-readonly-storageclass.yaml
   - node-agent-config.yaml
   - helmrelease.yaml
+  - servicemonitor.yaml
+  - podmonitor.yaml
   - restore-test-rbac.yaml
   - restore-test-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/

kubernetes/talos/patches/hcloud-1.yaml

@@ -0,0 +1,69 @@
+# First machine of the throwaway cluster on Hetzner.
+#
+# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
+# order bootstrap.sh attaches them.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-1
+  nodeLabels:
+    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
+    # rather than MAC: the address follows from the order bootstrap.sh attaches
+    # the networks, while the MAC is assigned by Hetzner at attach time and is
+    # not known when this file is written.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+# In the generated schema rather than machine.network.hostname: naming the host
+# in both is rejected at apply time. `auto: off` switches off the generated
+# `auto: stable`, which is mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-1
+---
+# Every interface is stated, the public one included. Talos stops running its
+# default DHCP operators for the whole machine as soon as one new-style network
+# document exists, so an omitted interface gets no address.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# Ceph cluster network, renamed to the role name by the udev rule in the
+# machine section. The node holds no address here; the link exists for the OSD
+# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    # Two disks: the system disk and the attached volume. Matching on transport
+    # as the qemu patches do selects nothing here — a Hetzner volume arrives
+    # over virtio-scsi, and a selector matching zero disks is skipped silently.
+    match: '!system_disk'
+  # bootstrap.sh reads this size and creates the volume to match.
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -0,0 +1,50 @@
+# Second machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-2
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-2
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -0,0 +1,50 @@
+# Third machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-3
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-3
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/site-hcloud.yaml

@@ -0,0 +1,25 @@
+# Site patch for the throwaway cluster on Hetzner Cloud.
+#
+# This site exists for one thing the other two cannot test: whether a node that
+# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
+# answers, so containerd never reaches the second endpoint in
+# machine.registries.mirrors and the fallback stays unproven.
+#
+# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
+# frames from it do not cross between servers — measured: same-node ARP
+# resolves, cross-node does not. OSD replication therefore does not run here.
+#
+# What belongs in this file is what the cluster is. Interfaces, disks and names
+# are machine properties and live in the node patches.
+machine:
+  kubelet:
+    nodeIP:
+      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
+      # kubelet can pick the public NIC.
+      validSubnets:
+        - 10.90.0.0/16
+cluster:
+  # Three control planes that also carry workloads, which is the production
+  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
+  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
+  allowSchedulingOnControlPlanes: true

kubernetes/vmtest-job.yaml

@@ -0,0 +1,134 @@
+# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
+#
+# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
+# one-shot test, and a Kustomization would recreate it after every completion.
+# It is applied by the vmtest pipeline, or by hand for a single run.
+#
+# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
+# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
+# a single run left under 1G free and two overlapping runs OOM-killed the
+# session. The nodes have ~94G each with about 70G unused.
+#
+# The whole script runs here, flux and OpenBao included, so that what a rebuild
+# rehearses is what a real recovery would execute. The unseal key is not baked
+# in and not stored as a Secret: the operator streams all five secrets into an
+# in-memory emptyDir at the moment the run asks for them, and they die with the
+# Job without ever reaching etcd or a backup of it.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vmtest
+  labels:
+    # The pod is privileged; say so rather than have the admission policy
+    # reject it for reasons that look unrelated.
+    pod-security.kubernetes.io/enforce: privileged
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: vmtest
+  namespace: vmtest
+spec:
+  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
+  ttlSecondsAfterFinished: 86400
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: vmtest
+    spec:
+      restartPolicy: Never
+      # Nothing here should be able to reach the cluster it runs in. The phase
+      # points KUBECONFIG at its own throwaway cluster, and without a mounted
+      # token a stray kubectl cannot fall back to this one.
+      automountServiceAccountToken: false
+      containers:
+        - name: vmtest
+          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
+          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
+          # three builds old, which predated OVMF, and died with "the required
+          # flash image was not found" before the first VM booted. The tag is
+          # carried alongside for readability — containerd resolves the digest.
+          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
+          # Privileged for /dev/kvm and for the bridges and taps the provisioner
+          # builds. It needs no hostNetwork: the provisioner's network lives in
+          # this pod's own namespace, and the script that talks to 10.5.0.x runs
+          # in the same pod.
+          securityContext:
+            privileged: true
+          # A hard ceiling, because the failure mode is known: an unbounded run
+          # is what OOM-killed the workstation. 40G is the VMs (16G control
+          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
+          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
+          # of room for the CNI DaemonSets themselves. The control plane went
+          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
+          # memory and the kubelet evicted kube-apiserver to admit a
+          # node-critical pod — bootstrap.sh carries the full account next to
+          # the sizing flags. The hosts have ~95GiB each, so this fits.
+          resources:
+            requests:
+              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
+              # the guests are never all runnable at once (one cluster booting
+              # in sequence), and there is no CPU limit, so the pod may burst
+              # past the request when the node has headroom. The request only
+              # keeps the scheduler from stacking this next to another big
+              # burst consumer.
+              cpu: "6"
+              memory: 42Gi
+            limits:
+              memory: 48Gi
+          env:
+            # talosctl caches the image and the CNI bundle under $HOME; point it
+            # at the scratch volume rather than the container layer.
+            - name: HOME
+              value: /work
+            # The VM disks. Short on purpose — the provisioner puts a qemu
+            # monitor socket under it and sun_path caps the path at 108 bytes.
+            - name: VM_STATE
+              value: /work/vm
+            # Which branch to test. master unless a run is checking a change.
+            - name: BRANCH
+              value: master
+            # Secrets are streamed in here at the moment the run needs them,
+            # never baked into the image and never stored as a Secret. They live
+            # in this emptyDir for the length of the Job and go with it — so
+            # they do not reach etcd, or any backup of it.
+            - name: OPENBAO_SECRETS_DIR
+              value: /secrets
+          command: ["/bin/bash", "-ec"]
+          args:
+            - |
+              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
+              # that clones from the cluster it is rehearsing the loss of
+              # cannot start when it matters — and a run already died here on
+              # resolving gitea.renner.uno.
+              git clone --depth 1 --branch "$BRANCH" \
+                https://github.com/konairius/Homelab.git /work/homelab
+              exec /work/homelab/kubernetes/bootstrap.sh vmtest
+          volumeMounts:
+            - name: work
+              mountPath: /work
+            - name: secrets
+              mountPath: /secrets
+            - name: dev-kvm
+              mountPath: /dev/kvm
+      volumes:
+        # The VM disks and the provisioner state. emptyDir on the node's disk
+        # rather than a PVC: it is scratch that dies with the Job, and putting it
+        # on Ceph would have the test's own writes compete with the storage it is
+        # meant to leave alone.
+        - name: work
+          emptyDir:
+            # The VM images are sparse, so this bounds what is actually written,
+            # not the 232GiB the geometry declares. 60Gi was already half spent
+            # by four 10GiB systems disks; the node carries this comfortably with
+            # ~810GB free on the filesystem behind it.
+            sizeLimit: 150Gi
+        # In memory, so the material never touches a node's disk.
+        - name: secrets
+          emptyDir:
+            medium: Memory
+            sizeLimit: 1Mi
+        - name: dev-kvm
+          hostPath:
+            path: /dev/kvm
+            type: CharDevice

scripts/import-cluster-access.sh

@@ -13,7 +13,18 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-export AWS_PROFILE="secrets"
+# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
+# The default is unchanged and deliberately so: "secrets" resolves to working
+# credentials and this script has always used it. Both "secrets" and "hetzner"
+# exist in ~/.aws/credentials on the workstation.
+#
+# The defect is only the assignment. Forcing the value defeated the env-based
+# path this file's own header offers, and it also overrode a caller who had
+# already selected a profile — which is how a 403 from HeadObject can appear
+# while the credentials the caller intended were perfectly good. That error
+# reads as a bucket permissions problem and sends you looking in the wrong
+# place.
+export AWS_PROFILE="${AWS_PROFILE:-secrets}"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }