fix(flux): four cold-rebuild ordering deadlocks

Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.

infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.

home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.

woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,18 +1,3 @@
-# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
-# isolated from the apps namespace.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    # Image-build steps run a privileged docker/buildx plugin pod (see the
-    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
-    # without this the privileged step pods are rejected at admission. The
-    # namespace is dedicated to ephemeral CI step pods, so scoping the
-    # exemption here keeps it off apps/.
-    pod-security.kubernetes.io/enforce: privileged
----
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,6 +28,19 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
+    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
+    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
+    # kind is unknown when this path is first applied, and a server-side
+    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
+    #
+    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
+    #   no matches for kind "ObjectStore" in version ...
+    #
+    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
+    # woodpecker — already declares it. This was the only one that did not, and
+    # it took speech-to-phrase, zone-configurator and matter-server down with
+    # it, since all three wait on home-assistant.
+    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,8 +122,20 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  dependsOn:
-    - name: infrastructure
+  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
+  # existing cluster could ever hit and no new one could ever escape:
+  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
+  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
+  # defined in this very Kustomization. A cold rebuild stopped here for good,
+  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
+  #
+  # The edge was never needed. infrastructure runs wait: false, so it applies
+  # the rook-ceph namespace and the operator immediately and only its readiness
+  # waits on the healthChecks — everything this path needs exists long before
+  # infrastructure reports Ready. On an empty API server this Kustomization
+  # retries on retryInterval until the operator installs its CRDs, the same
+  # self-healing shape post-crds already relies on; on a live cluster the CRDs
+  # are always present and nothing changes.
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -190,3 +190,31 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
+---
+# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
+# from `apps`. Distinct from `woodpecker`, which holds the service itself.
+#
+# Here rather than in apps/woodpecker for the reason spelled out above that
+# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
+# ../network-policies/, and infrastructure-crds runs before every app KS. With
+# the namespace declared in the workload KS the order inverted, and a rebuild
+# stopped with
+#
+#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
+#   namespaces "woodpecker-ci" not found
+#
+# which held infrastructure-crds, and with it all twenty app Kustomizations
+# that depend on it. The same reasoning was already written down for
+# `woodpecker` and simply not carried across to its sibling.
+#
+# Image-build steps run a privileged docker/buildx plugin pod (see the server's
+# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
+# label the privileged step pods are rejected at admission. The namespace holds
+# nothing but ephemeral CI step pods, so the exemption stays off apps/.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    pod-security.kubernetes.io/enforce: privileged

kubernetes/talos/patches/hcloud-1.yaml

@@ -1,69 +0,0 @@
-# First machine of the throwaway cluster on Hetzner.
-#
-# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
-# order bootstrap.sh attaches them.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-1
-  nodeLabels:
-    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
-    # rather than MAC: the address follows from the order bootstrap.sh attaches
-    # the networks, while the MAC is assigned by Hetzner at attach time and is
-    # not known when this file is written.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-# In the generated schema rather than machine.network.hostname: naming the host
-# in both is rejected at apply time. `auto: off` switches off the generated
-# `auto: stable`, which is mutually exclusive with a static name.
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-1
----
-# Every interface is stated, the public one included. Talos stops running its
-# default DHCP operators for the whole machine as soon as one new-style network
-# document exists, so an omitted interface gets no address.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# Ceph cluster network, renamed to the role name by the udev rule in the
-# machine section. The node holds no address here; the link exists for the OSD
-# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: ceph-osd
-provisioning:
-  diskSelector:
-    # Two disks: the system disk and the attached volume. Matching on transport
-    # as the qemu patches do selects nothing here — a Hetzner volume arrives
-    # over virtio-scsi, and a selector matching zero disks is skipped silently.
-    match: '!system_disk'
-  # bootstrap.sh reads this size and creates the volume to match.
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -1,50 +0,0 @@
-# Second machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-2
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-2
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: ceph-osd
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -1,50 +0,0 @@
-# Third machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-3
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-3
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: ceph-osd
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/site-hcloud.yaml

@@ -1,25 +0,0 @@
-# Site patch for the throwaway cluster on Hetzner Cloud.
-#
-# This site exists for one thing the other two cannot test: whether a node that
-# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
-# answers, so containerd never reaches the second endpoint in
-# machine.registries.mirrors and the fallback stays unproven.
-#
-# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
-# frames from it do not cross between servers — measured: same-node ARP
-# resolves, cross-node does not. OSD replication therefore does not run here.
-#
-# What belongs in this file is what the cluster is. Interfaces, disks and names
-# are machine properties and live in the node patches.
-machine:
-  kubelet:
-    nodeIP:
-      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
-      # kubelet can pick the public NIC.
-      validSubnets:
-        - 10.90.0.0/16
-cluster:
-  # Three control planes that also carry workloads, which is the production
-  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
-  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
-  allowSchedulingOnControlPlanes: true

kubernetes/vmtest-job.yaml

@@ -1,134 +0,0 @@
-# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
-#
-# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
-# one-shot test, and a Kustomization would recreate it after every completion.
-# It is applied by the vmtest pipeline, or by hand for a single run.
-#
-# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
-# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
-# a single run left under 1G free and two overlapping runs OOM-killed the
-# session. The nodes have ~94G each with about 70G unused.
-#
-# The whole script runs here, flux and OpenBao included, so that what a rebuild
-# rehearses is what a real recovery would execute. The unseal key is not baked
-# in and not stored as a Secret: the operator streams all five secrets into an
-# in-memory emptyDir at the moment the run asks for them, and they die with the
-# Job without ever reaching etcd or a backup of it.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: vmtest
-  labels:
-    # The pod is privileged; say so rather than have the admission policy
-    # reject it for reasons that look unrelated.
-    pod-security.kubernetes.io/enforce: privileged
----
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: vmtest
-  namespace: vmtest
-spec:
-  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
-  ttlSecondsAfterFinished: 86400
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: vmtest
-    spec:
-      restartPolicy: Never
-      # Nothing here should be able to reach the cluster it runs in. The phase
-      # points KUBECONFIG at its own throwaway cluster, and without a mounted
-      # token a stray kubectl cannot fall back to this one.
-      automountServiceAccountToken: false
-      containers:
-        - name: vmtest
-          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
-          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
-          # three builds old, which predated OVMF, and died with "the required
-          # flash image was not found" before the first VM booted. The tag is
-          # carried alongside for readability — containerd resolves the digest.
-          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
-          # Privileged for /dev/kvm and for the bridges and taps the provisioner
-          # builds. It needs no hostNetwork: the provisioner's network lives in
-          # this pod's own namespace, and the script that talks to 10.5.0.x runs
-          # in the same pod.
-          securityContext:
-            privileged: true
-          # A hard ceiling, because the failure mode is known: an unbounded run
-          # is what OOM-killed the workstation. 40G is the VMs (16G control
-          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
-          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
-          # of room for the CNI DaemonSets themselves. The control plane went
-          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
-          # memory and the kubelet evicted kube-apiserver to admit a
-          # node-critical pod — bootstrap.sh carries the full account next to
-          # the sizing flags. The hosts have ~95GiB each, so this fits.
-          resources:
-            requests:
-              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
-              # the guests are never all runnable at once (one cluster booting
-              # in sequence), and there is no CPU limit, so the pod may burst
-              # past the request when the node has headroom. The request only
-              # keeps the scheduler from stacking this next to another big
-              # burst consumer.
-              cpu: "6"
-              memory: 42Gi
-            limits:
-              memory: 48Gi
-          env:
-            # talosctl caches the image and the CNI bundle under $HOME; point it
-            # at the scratch volume rather than the container layer.
-            - name: HOME
-              value: /work
-            # The VM disks. Short on purpose — the provisioner puts a qemu
-            # monitor socket under it and sun_path caps the path at 108 bytes.
-            - name: VM_STATE
-              value: /work/vm
-            # Which branch to test. master unless a run is checking a change.
-            - name: BRANCH
-              value: master
-            # Secrets are streamed in here at the moment the run needs them,
-            # never baked into the image and never stored as a Secret. They live
-            # in this emptyDir for the length of the Job and go with it — so
-            # they do not reach etcd, or any backup of it.
-            - name: OPENBAO_SECRETS_DIR
-              value: /secrets
-          command: ["/bin/bash", "-ec"]
-          args:
-            - |
-              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
-              # that clones from the cluster it is rehearsing the loss of
-              # cannot start when it matters — and a run already died here on
-              # resolving gitea.renner.uno.
-              git clone --depth 1 --branch "$BRANCH" \
-                https://github.com/konairius/Homelab.git /work/homelab
-              exec /work/homelab/kubernetes/bootstrap.sh vmtest
-          volumeMounts:
-            - name: work
-              mountPath: /work
-            - name: secrets
-              mountPath: /secrets
-            - name: dev-kvm
-              mountPath: /dev/kvm
-      volumes:
-        # The VM disks and the provisioner state. emptyDir on the node's disk
-        # rather than a PVC: it is scratch that dies with the Job, and putting it
-        # on Ceph would have the test's own writes compete with the storage it is
-        # meant to leave alone.
-        - name: work
-          emptyDir:
-            # The VM images are sparse, so this bounds what is actually written,
-            # not the 232GiB the geometry declares. 60Gi was already half spent
-            # by four 10GiB systems disks; the node carries this comfortably with
-            # ~810GB free on the filesystem behind it.
-            sizeLimit: 150Gi
-        # In memory, so the material never touches a node's disk.
-        - name: secrets
-          emptyDir:
-            medium: Memory
-            sizeLimit: 1Mi
-        - name: dev-kvm
-          hostPath:
-            path: /dev/kvm
-            type: CharDevice

scripts/import-cluster-access.sh

@@ -13,18 +13,7 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
-# The default is unchanged and deliberately so: "secrets" resolves to working
-# credentials and this script has always used it. Both "secrets" and "hetzner"
-# exist in ~/.aws/credentials on the workstation.
-#
-# The defect is only the assignment. Forcing the value defeated the env-based
-# path this file's own header offers, and it also overrode a caller who had
-# already selected a profile — which is how a 403 from HeadObject can appear
-# while the credentials the caller intended were perfectly good. That error
-# reads as a bucket permissions problem and sends you looking in the wrong
-# place.
-export AWS_PROFILE="${AWS_PROFILE:-secrets}"
+export AWS_PROFILE="secrets"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }