fix(flux): four cold-rebuild ordering deadlocks

Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.

infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.

home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.

woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/authentik/kustomization.yaml

@@ -16,6 +16,7 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
+  - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -1,46 +0,0 @@
-# Cilium's agent ServiceMonitor, declared here rather than by the chart.
-#
-# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
-# taken field for field off the live object so scraping is unchanged: same name,
-# same namespace, same selector, same 10s interval and node relabeling.
-#
-# It lives here because the chart's template is gated on monitoring.coreos.com/v1
-# existing at install time, and on an empty cluster it does not — those CRDs
-# arrive with kube-prometheus-stack, which reconciles after infrastructure.
-# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
-# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
-#
-# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
-# why; this is the same move for the component where it mattered most.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: cilium-agent
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: cilium-agent
-    app.kubernetes.io/part-of: cilium
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: cilium-agent
-  namespaceSelector:
-    matchNames:
-      - kube-system
-  # The chart's Service keeps rendering (prometheus.enabled), so the target and
-  # its `metrics` port are there regardless of who declares the ServiceMonitor.
-  targetLabels:
-    - k8s-app
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        # Agent metrics are per-node; without this every series lands without a
-        # node label and BPF map pressure cannot be attributed to a machine.
-        - action: replace
-          sourceLabels:
-            - __meta_kubernetes_pod_node_name
-          targetLabel: node
-          replacement: ${1}

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -22,19 +22,8 @@ resources:
   # endpoint, which proves reachability, not that anything was written.
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
-  - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
-  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
-  # kube-prometheus-stack, which reconciles after infrastructure, so these
-  # cannot live beside the Velero HelmRelease.
-  - velero-podmonitor.yaml
-  - velero-servicemonitor.yaml
-  - authentik-outpost-servicemonitors.yaml
-  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
-  # when it cannot create these, which held the CephCluster in Progressing and
-  # deadlocked the whole apps layer. See the file for the cycle.
-  - rook-servicemonitors.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/rook-servicemonitors.yaml

@@ -1,93 +0,0 @@
-# Rook's two ServiceMonitors, declared here rather than by the operator.
-#
-# Reproduced field for field off the live objects in the production cluster, so
-# scraping is unchanged: same names, same namespace, same selectors, same 10s
-# interval and `cluster` relabeling. The only thing dropped is the CephCluster
-# ownerReference, which Rook set and Flux cannot.
-#
-# They live here because CephCluster.spec.monitoring.enabled is now false, and
-# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
-# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
-# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
-# reconcile.go, v1.20.7):
-#
-#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
-#         // exporter reconciled; its errors are logged and reconcile continues
-#         if cephCluster.Spec.Monitoring.Enabled {
-#             if err := EnableCephExporterServiceMonitor(...); err != nil {
-#                 return errors.Wrap(err, "failed to enable service monitor")
-#             }
-#         }
-#     }
-#
-# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
-# fails with "the server could not find the requested resource (post
-# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
-# aborts, and the CephCluster never leaves status.phase Progressing. That is
-# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
-# closes on itself:
-#
-#     infrastructure-ceph-cluster → CephCluster phase Ready
-#                                 → Rook reconcile settles
-#                                 → monitoring.coreos.com CRDs exist
-#                                 → kube-prometheus-stack (monitoring)
-#                                 → grafana-db
-#                                 → infrastructure-ceph-cluster
-#
-# Nothing outside that loop can break it, so a cold rebuild stopped here with
-# every one of the twenty app Kustomizations behind it.
-#
-# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
-# get, and for the same reason: the metrics stay where they are, only the
-# scrape wiring moves to the layer where its CRDs exist.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-exporter
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-exporter
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: ceph-exporter-http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      # Rook stamps the cluster name so series from several Ceph clusters stay
-      # separable. One cluster here, but dropping it would rename every series.
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster
----
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-mgr
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-mgr
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,18 +1,3 @@
-# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
-# isolated from the apps namespace.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    # Image-build steps run a privileged docker/buildx plugin pod (see the
-    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
-    # without this the privileged step pods are rejected at admission. The
-    # namespace is dedicated to ephemeral CI step pods, so scoping the
-    # exemption here keeps it off apps/.
-    pod-security.kubernetes.io/enforce: privileged
----
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,6 +28,19 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
+    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
+    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
+    # kind is unknown when this path is first applied, and a server-side
+    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
+    #
+    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
+    #   no matches for kind "ObjectStore" in version ...
+    #
+    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
+    # woodpecker — already declares it. This was the only one that did not, and
+    # it took speech-to-phrase, zone-configurator and matter-server down with
+    # it, since all three wait on home-assistant.
+    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,8 +122,20 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  dependsOn:
-    - name: infrastructure
+  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
+  # existing cluster could ever hit and no new one could ever escape:
+  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
+  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
+  # defined in this very Kustomization. A cold rebuild stopped here for good,
+  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
+  #
+  # The edge was never needed. infrastructure runs wait: false, so it applies
+  # the rook-ceph namespace and the operator immediately and only its readiness
+  # waits on the healthChecks — everything this path needs exists long before
+  # infrastructure reports Ready. On an empty API server this Kustomization
+  # retries on retryInterval until the operator installs its CRDs, the same
+  # self-healing shape post-crds already relies on; on a live cluster the CRDs
+  # are always present and nothing changes.
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -70,23 +70,9 @@ spec:
       # map only shows up as "no space left on device" in the log while pod
       # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
       # alertable beforehand.
-      #
-      # That reasoning is about the metrics, which this line provides. The
-      # ServiceMonitor is only the scrape wiring and is declared standalone in
-      # apps/monitoring-rules instead — the same treatment velero/helmrelease
-      # already gives its own, and for the same reason: the chart's template is
-      # gated on monitoring.coreos.com/v1 existing at install time.
-      #
-      # Here it is worse than for velero. Cilium's chart does not skip the
-      # template, it aborts rendering — "Service Monitor requires
-      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
-      # healthChecks the infrastructure Kustomization waits on. So on an empty
-      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
-      # blocked monitoring, which is where those CRDs come from. The lowest
-      # layer in the cluster was waiting on the highest.
       enabled: true
       serviceMonitor:
-        enabled: false
+        enabled: true
     bpf:
       # Per-endpoint policy map size: one entry per (identity, direction,
       # protocol, port) allowed. When it fills, endpoint regeneration fails

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -99,7 +99,7 @@ spec:
     # required.
     #
     # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
     # so there is no live impact, but re-enabling it will require pointing the
     # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
     tls:

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -1,5 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - bouncer-key-externalsecret.yaml
+  - edge-bouncer-key-externalsecret.yaml
   - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/external-dns-unifi/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - helmrelease.yaml

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -190,3 +190,31 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
+---
+# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
+# from `apps`. Distinct from `woodpecker`, which holds the service itself.
+#
+# Here rather than in apps/woodpecker for the reason spelled out above that
+# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
+# ../network-policies/, and infrastructure-crds runs before every app KS. With
+# the namespace declared in the workload KS the order inverted, and a rebuild
+# stopped with
+#
+#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
+#   namespaces "woodpecker-ci" not found
+#
+# which held infrastructure-crds, and with it all twenty app Kustomizations
+# that depend on it. The same reasoning was already written down for
+# `woodpecker` and simply not carried across to its sibling.
+#
+# Image-build steps run a privileged docker/buildx plugin pod (see the server's
+# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
+# label the privileged step pods are rejected at admission. The namespace holds
+# nothing but ephemeral CI step pods, so the exemption stays off apps/.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    pod-security.kubernetes.io/enforce: privileged

kubernetes/infrastructure/netbird-ingress/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - daemonset.yaml

kubernetes/infrastructure/openbao/rotation/flux-git.yaml

@@ -10,8 +10,33 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret that publishes the rotated token to Flux lives in
-# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
+# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
+# own it) so a sync failure leaves the last-good value in place; Flux's
+# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
+# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
+# provisioned by hand (see policies/rotator-gitea-flux.hcl).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: flux-system-git-auth
+  namespace: flux-system
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: flux-system
+    creationPolicy: Merge
+  data:
+    - secretKey: username
+      remoteRef:
+        key: infrastructure/flux-git
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: infrastructure/flux-git
+        property: token
 ---
 apiVersion: v1
 kind: ServiceAccount

kubernetes/infrastructure/post-crds/flux-git-externalsecret.yaml

@@ -1,60 +0,0 @@
-# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
-# WITHOUT owning it, so a sync failure leaves the last-good value in place;
-# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
-# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
-# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
-#
-# See creationPolicy below for why that is Orphan and not Merge — the
-# distinction is what makes a recovery from the break-glass source possible.
-#
-# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
-# rotation CronJob: external-secrets.io CRDs are installed by the ESO
-# HelmRelease in the infrastructure Kustomization, so this manifest cannot
-# pass server-side dry-run there on a fresh cluster.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    # Orphan, not Merge: create the Secret if it is absent, but never own it.
-    #
-    # Merge cannot create — it only writes into a Secret that already exists,
-    # on the assumption that `flux bootstrap` made one. That assumption breaks
-    # in exactly the situation this cluster is rebuilt from. The break-glass
-    # bootstrap pulls from the GitHub mirror and keeps its token in
-    # flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
-    # so this ExternalSecret has nothing to merge into — and reports Ready while
-    # producing no Secret at all. The operator then performs the documented
-    # handover back to Gitea and finds the GitRepository pointing at a Secret
-    # that does not exist.
-    #
-    # Owner would create it but attach an ownerReference, so deleting this
-    # object would take Flux's credential with it. Orphan creates and updates
-    # without ownership, which is what the paragraph at the top of this file
-    # actually wanted: a sync failure, or the loss of this object, leaves the
-    # last-good value in place.
-    #
-    # Safe on a live cluster: the Secret holds exactly the two keys written
-    # below (verified against production), so writing it whole is the same
-    # result as merging into it.
-    creationPolicy: Orphan
-    # Explicit rather than relying on the default, because it is the half that
-    # matters here: if the OpenBao entry ever disappears, keep serving the
-    # credential Flux already has rather than leaving it with none.
-    deletionPolicy: Retain
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -10,25 +10,4 @@ resources:
   # configured (2026-07-13): a store that can't turn Ready would block this
   # KS's wait gate and freeze the dependsOn chain.
   - openbao-clustersecretstore.yaml
-  # ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
-  # the infrastructure Kustomization, so these cannot pass its server-side
-  # dry-run on a fresh cluster — Flux rejects the whole apply set, including
-  # the ESO HelmRelease that would have installed the CRDs. They sat next to
-  # their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
-  # nothing had ever applied this layer against an empty API server before.
-  - crowdsec-bouncer-key-externalsecret.yaml
-  - crowdsec-edge-bouncer-key-externalsecret.yaml
-  - external-dns-unifi-externalsecret.yaml
-  - flux-git-externalsecret.yaml
-  - netbird-ingress-externalsecret.yaml
-  - velero-externalsecret.yaml
-  - velero-hetzner-creds-externalsecret.yaml
-  # traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
-  # infrastructure. Traefik is deliberately absent from that Kustomization's
-  # healthChecks, so this KS may dry-run before the chart lands; it retries on
-  # retryInterval rather than deadlocking, because nothing here installs the
-  # CRDs it is waiting for.
-  - traefik-crowdsec-middleware.yaml
-  - traefik-rate-limit-middleware.yaml
-  - traefik-security-headers-middleware.yaml
   - ../network-policies/

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -78,20 +78,7 @@ spec:
     enabled: true
     ssl: false
   monitoring:
-    # false, and this is not a decision about metrics. `metricsDisabled` is the
-    # switch that turns metrics off; this one only decides whether Rook creates
-    # the ServiceMonitors — and creating them is fatal to the reconcile when it
-    # fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
-    # Rook's node-daemon reconcile aborted, the CephCluster never left
-    # status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
-    # waits on exactly that — with grafana-db → monitoring → those same CRDs
-    # behind it, closing the loop.
-    #
-    # The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
-    # only the scrape wiring moved, to apps/monitoring-rules/
-    # rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
-    # Both objects are reproduced there field for field off the live cluster.
-    enabled: false
+    enabled: true
   # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
   # Multus. The attachment is san-network-attachment.yaml.
   #

kubernetes/infrastructure/traefik/kustomization.yaml

@@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - helmrelease.yaml
+  - crowdsec-middleware.yaml
+  - security-headers-middleware.yaml
+  - rate-limit-middleware.yaml

kubernetes/infrastructure/velero/kustomization.yaml

@@ -1,9 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
   - snapshot-readonly-storageclass.yaml
   - node-agent-config.yaml
   - helmrelease.yaml
+  - servicemonitor.yaml
+  - podmonitor.yaml
   - restore-test-rbac.yaml
   - restore-test-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/