refactor(flux): CRD consumers move behind the CRDs they need

One unknown kind fails a server-side dry-run for the WHOLE apply set,
including the very HelmRelease that would have installed the missing
CRDs. On a running cluster every CRD exists, so this never showed; a
cold rebuild deadlocks on it immediately.

Three groups move, content unchanged:

ExternalSecrets and Traefik Middlewares out of infrastructure into
post-crds, which reconciles after the operators. The flux-git
ExternalSecret also changes creationPolicy Merge -> Orphan with
deletionPolicy Retain: Merge never CREATES the target secret, so on a
fresh cluster the Gitea credential handover silently did not happen.

ServiceMonitors/PodMonitors out of their charts and app sets into
apps/monitoring-rules, which applies after kube-prometheus-stack owns
the CRDs. Rook additionally stops creating its own two monitors
(spec.monitoring.enabled: false) — creating them is fatal to its
node-daemon reconcile when the CRD is missing, which held the
CephCluster in Progressing and with it the whole apps layer. Both
objects are reproduced field for field off the live cluster; the
ceph-exporter DaemonSet and the metrics endpoints are untouched.
Cilium is the worst case: its chart aborts rendering outright on the
missing CRD, and cilium is one of the seven infrastructure
healthChecks — the lowest layer in the cluster waiting on the highest.

Steady-state effect: object ownership moves between Kustomizations;
specs are identical. Verified on local vmtest runs and the Hetzner
throwaway cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/authentik/kustomization.yaml

@@ -16,7 +16,6 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
-  - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -0,0 +1,46 @@
+# Cilium's agent ServiceMonitor, declared here rather than by the chart.
+#
+# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
+# taken field for field off the live object so scraping is unchanged: same name,
+# same namespace, same selector, same 10s interval and node relabeling.
+#
+# It lives here because the chart's template is gated on monitoring.coreos.com/v1
+# existing at install time, and on an empty cluster it does not — those CRDs
+# arrive with kube-prometheus-stack, which reconciles after infrastructure.
+# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
+# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
+#
+# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
+# why; this is the same move for the component where it mattered most.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: cilium-agent
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: cilium-agent
+    app.kubernetes.io/part-of: cilium
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cilium-agent
+  namespaceSelector:
+    matchNames:
+      - kube-system
+  # The chart's Service keeps rendering (prometheus.enabled), so the target and
+  # its `metrics` port are there regardless of who declares the ServiceMonitor.
+  targetLabels:
+    - k8s-app
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      relabelings:
+        # Agent metrics are per-node; without this every series lands without a
+        # node label and BPF map pressure cannot be attributed to a machine.
+        - action: replace
+          sourceLabels:
+            - __meta_kubernetes_pod_node_name
+          targetLabel: node
+          replacement: ${1}

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -22,8 +22,19 @@ resources:
   # endpoint, which proves reachability, not that anything was written.
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
+  - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
+  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
+  # kube-prometheus-stack, which reconciles after infrastructure, so these
+  # cannot live beside the Velero HelmRelease.
+  - velero-podmonitor.yaml
+  - velero-servicemonitor.yaml
+  - authentik-outpost-servicemonitors.yaml
+  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
+  # when it cannot create these, which held the CephCluster in Progressing and
+  # deadlocked the whole apps layer. See the file for the cycle.
+  - rook-servicemonitors.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/rook-servicemonitors.yaml

@@ -0,0 +1,93 @@
+# Rook's two ServiceMonitors, declared here rather than by the operator.
+#
+# Reproduced field for field off the live objects in the production cluster, so
+# scraping is unchanged: same names, same namespace, same selectors, same 10s
+# interval and `cluster` relabeling. The only thing dropped is the CephCluster
+# ownerReference, which Rook set and Flux cannot.
+#
+# They live here because CephCluster.spec.monitoring.enabled is now false, and
+# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
+# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
+# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
+# reconcile.go, v1.20.7):
+#
+#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
+#         // exporter reconciled; its errors are logged and reconcile continues
+#         if cephCluster.Spec.Monitoring.Enabled {
+#             if err := EnableCephExporterServiceMonitor(...); err != nil {
+#                 return errors.Wrap(err, "failed to enable service monitor")
+#             }
+#         }
+#     }
+#
+# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
+# fails with "the server could not find the requested resource (post
+# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
+# aborts, and the CephCluster never leaves status.phase Progressing. That is
+# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
+# closes on itself:
+#
+#     infrastructure-ceph-cluster → CephCluster phase Ready
+#                                 → Rook reconcile settles
+#                                 → monitoring.coreos.com CRDs exist
+#                                 → kube-prometheus-stack (monitoring)
+#                                 → grafana-db
+#                                 → infrastructure-ceph-cluster
+#
+# Nothing outside that loop can break it, so a cold rebuild stopped here with
+# every one of the twenty app Kustomizations behind it.
+#
+# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
+# get, and for the same reason: the metrics stay where they are, only the
+# scrape wiring moves to the layer where its CRDs exist.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: rook-ceph-exporter
+  namespace: rook-ceph
+  labels:
+    team: rook
+spec:
+  selector:
+    matchLabels:
+      app: rook-ceph-exporter
+      rook_cluster: rook-ceph
+  namespaceSelector:
+    matchNames:
+      - rook-ceph
+  endpoints:
+    - port: ceph-exporter-http-metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      # Rook stamps the cluster name so series from several Ceph clusters stay
+      # separable. One cluster here, but dropping it would rename every series.
+      relabelings:
+        - action: replace
+          replacement: rook-ceph
+          targetLabel: cluster
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: rook-ceph-mgr
+  namespace: rook-ceph
+  labels:
+    team: rook
+spec:
+  selector:
+    matchLabels:
+      app: rook-ceph-mgr
+      rook_cluster: rook-ceph
+  namespaceSelector:
+    matchNames:
+      - rook-ceph
+  endpoints:
+    - port: http-metrics
+      path: /metrics
+      interval: 10s
+      honorLabels: true
+      relabelings:
+        - action: replace
+          replacement: rook-ceph
+          targetLabel: cluster

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -70,9 +70,23 @@ spec:
       # map only shows up as "no space left on device" in the log while pod
       # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
       # alertable beforehand.
+      #
+      # That reasoning is about the metrics, which this line provides. The
+      # ServiceMonitor is only the scrape wiring and is declared standalone in
+      # apps/monitoring-rules instead — the same treatment velero/helmrelease
+      # already gives its own, and for the same reason: the chart's template is
+      # gated on monitoring.coreos.com/v1 existing at install time.
+      #
+      # Here it is worse than for velero. Cilium's chart does not skip the
+      # template, it aborts rendering — "Service Monitor requires
+      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
+      # healthChecks the infrastructure Kustomization waits on. So on an empty
+      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
+      # blocked monitoring, which is where those CRDs come from. The lowest
+      # layer in the cluster was waiting on the highest.
       enabled: true
       serviceMonitor:
-        enabled: true
+        enabled: false
     bpf:
       # Per-endpoint policy map size: one entry per (identity, direction,
       # protocol, port) allowed. When it fills, endpoint regeneration fails

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -99,7 +99,7 @@ spec:
     # required.
     #
     # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
     # so there is no live impact, but re-enabling it will require pointing the
     # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
     tls:

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -1,7 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - bouncer-key-externalsecret.yaml
-  - edge-bouncer-key-externalsecret.yaml
   - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/external-dns-unifi/kustomization.yaml

@@ -1,5 +1,4 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - externalsecret.yaml
   - helmrelease.yaml

kubernetes/infrastructure/netbird-ingress/kustomization.yaml

@@ -1,5 +1,4 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - externalsecret.yaml
   - daemonset.yaml

kubernetes/infrastructure/openbao/rotation/flux-git.yaml

@@ -10,33 +10,8 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
-# own it) so a sync failure leaves the last-good value in place; Flux's
-# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
-# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
-# provisioned by hand (see policies/rotator-gitea-flux.hcl).
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    creationPolicy: Merge
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token
+# The ExternalSecret that publishes the rotated token to Flux lives in
+# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
 ---
 apiVersion: v1
 kind: ServiceAccount

kubernetes/infrastructure/post-crds/flux-git-externalsecret.yaml

@@ -0,0 +1,60 @@
+# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
+# WITHOUT owning it, so a sync failure leaves the last-good value in place;
+# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
+# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
+# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
+#
+# See creationPolicy below for why that is Orphan and not Merge — the
+# distinction is what makes a recovery from the break-glass source possible.
+#
+# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
+# rotation CronJob: external-secrets.io CRDs are installed by the ESO
+# HelmRelease in the infrastructure Kustomization, so this manifest cannot
+# pass server-side dry-run there on a fresh cluster.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: flux-system-git-auth
+  namespace: flux-system
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: flux-system
+    # Orphan, not Merge: create the Secret if it is absent, but never own it.
+    #
+    # Merge cannot create — it only writes into a Secret that already exists,
+    # on the assumption that `flux bootstrap` made one. That assumption breaks
+    # in exactly the situation this cluster is rebuilt from. The break-glass
+    # bootstrap pulls from the GitHub mirror and keeps its token in
+    # flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
+    # so this ExternalSecret has nothing to merge into — and reports Ready while
+    # producing no Secret at all. The operator then performs the documented
+    # handover back to Gitea and finds the GitRepository pointing at a Secret
+    # that does not exist.
+    #
+    # Owner would create it but attach an ownerReference, so deleting this
+    # object would take Flux's credential with it. Orphan creates and updates
+    # without ownership, which is what the paragraph at the top of this file
+    # actually wanted: a sync failure, or the loss of this object, leaves the
+    # last-good value in place.
+    #
+    # Safe on a live cluster: the Secret holds exactly the two keys written
+    # below (verified against production), so writing it whole is the same
+    # result as merging into it.
+    creationPolicy: Orphan
+    # Explicit rather than relying on the default, because it is the half that
+    # matters here: if the OpenBao entry ever disappears, keep serving the
+    # credential Flux already has rather than leaving it with none.
+    deletionPolicy: Retain
+  data:
+    - secretKey: username
+      remoteRef:
+        key: infrastructure/flux-git
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: infrastructure/flux-git
+        property: token

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -10,4 +10,25 @@ resources:
   # configured (2026-07-13): a store that can't turn Ready would block this
   # KS's wait gate and freeze the dependsOn chain.
   - openbao-clustersecretstore.yaml
+  # ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
+  # the infrastructure Kustomization, so these cannot pass its server-side
+  # dry-run on a fresh cluster — Flux rejects the whole apply set, including
+  # the ESO HelmRelease that would have installed the CRDs. They sat next to
+  # their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
+  # nothing had ever applied this layer against an empty API server before.
+  - crowdsec-bouncer-key-externalsecret.yaml
+  - crowdsec-edge-bouncer-key-externalsecret.yaml
+  - external-dns-unifi-externalsecret.yaml
+  - flux-git-externalsecret.yaml
+  - netbird-ingress-externalsecret.yaml
+  - velero-externalsecret.yaml
+  - velero-hetzner-creds-externalsecret.yaml
+  # traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
+  # infrastructure. Traefik is deliberately absent from that Kustomization's
+  # healthChecks, so this KS may dry-run before the chart lands; it retries on
+  # retryInterval rather than deadlocking, because nothing here installs the
+  # CRDs it is waiting for.
+  - traefik-crowdsec-middleware.yaml
+  - traefik-rate-limit-middleware.yaml
+  - traefik-security-headers-middleware.yaml
   - ../network-policies/

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -78,7 +78,20 @@ spec:
     enabled: true
     ssl: false
   monitoring:
-    enabled: true
+    # false, and this is not a decision about metrics. `metricsDisabled` is the
+    # switch that turns metrics off; this one only decides whether Rook creates
+    # the ServiceMonitors — and creating them is fatal to the reconcile when it
+    # fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
+    # Rook's node-daemon reconcile aborted, the CephCluster never left
+    # status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
+    # waits on exactly that — with grafana-db → monitoring → those same CRDs
+    # behind it, closing the loop.
+    #
+    # The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
+    # only the scrape wiring moved, to apps/monitoring-rules/
+    # rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
+    # Both objects are reproduced there field for field off the live cluster.
+    enabled: false
   # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
   # Multus. The attachment is san-network-attachment.yaml.
   #

kubernetes/infrastructure/traefik/kustomization.yaml

@@ -2,6 +2,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - helmrelease.yaml
-  - crowdsec-middleware.yaml
-  - security-headers-middleware.yaml
-  - rate-limit-middleware.yaml

kubernetes/infrastructure/velero/kustomization.yaml

@@ -1,13 +1,9 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - externalsecret.yaml
-  - hetzner-creds-externalsecret.yaml
   - snapshot-readonly-storageclass.yaml
   - node-agent-config.yaml
   - helmrelease.yaml
-  - servicemonitor.yaml
-  - podmonitor.yaml
   - restore-test-rbac.yaml
   - restore-test-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/