fix(flux): four cold-rebuild ordering deadlocks

Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.

infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.

home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.

woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/paperless-eval/cluster.yaml

@@ -1,116 +0,0 @@
----
-# A read-only view of paperless's backup store, used as a recovery source and
-# nothing else.
-#
-# It points at the production prefix, which is the whole point — the eval
-# instance is a copy of the real archive's metadata — and that makes what is
-# ABSENT here the safety-critical part:
-#
-#   no retentionPolicy   The production store has "30d". Without one here
-#                        there is no policy to enforce, and enforcement is not
-#                        tied to taking a backup: the plugin runs it from the
-#                        instance sidecar on a timer
-#                        (instanceSidecarConfiguration.retentionPolicyIntervalSeconds,
-#                        default 1800). So "this cluster never backs up" would
-#                        not have been enough on its own.
-#
-# A second property makes it belt-and-braces. Retention runs
-# barman-cloud-backup-delete, which takes a server name and works inside
-# <destinationPath>/<serverName> — the store's own status keys its recovery
-# window by "pprl-pg", confirming that layout. This cluster is pprleval-pg, so
-# even an accidental retention policy would address a prefix that holds nothing
-# rather than production's backups.
-#
-# The cluster below completes the guarantee with isWALArchiver: false, so
-# nothing is written back either. It does load the plugin — that is required
-# for the restore to work at all — and read-only comes from the flag rather
-# than from leaving the plugin out.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: pprleval-pg-source
-  namespace: paperless-eval
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
----
-# A disposable clone of the paperless database, for measuring how well the
-# local LLM classifies documents without running the experiment against the
-# live archive.
-#
-# Losing this cluster costs a re-restore and nothing else, which is the intent
-# — it is rebuilt whenever a test needs a clean starting point.
-#
-# One instance, not two. Replication buys availability, and an eval instance
-# that is down is simply an eval that waits.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: pprleval-pg
-  namespace: paperless-eval
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  enableSuperuserAccess: true
-  storage:
-    # Delete-reclaim, unlike the Retain default: this volume is a copy that
-    # gets rebuilt, so it should vanish with the cluster rather than leave a
-    # Released PV and an orphaned RBD image behind (issue #262).
-    storageClass: ceph-block-ephemeral
-    size: 10Gi
-  # Declared with isWALArchiver: false, which is what kubernetes/RECOVERY.md
-  # arrived at after an earlier version of that runbook said to omit it: the
-  # instance manager loads this plugin to replay WAL during the restore, and
-  # without it the instance never starts ("Unknown plugin:
-  # barman-cloud.cloudnative-pg.io").
-  #
-  # externalClusters below only says WHERE the backup is; it does not get the
-  # plugin injected into the pod. So "no plugin" is not the way to keep this
-  # read-only — "plugin, not archiving" is, and it is the combination verified
-  # on im-pg on 2026-07-25.
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      enabled: true
-      isWALArchiver: false
-      parameters:
-        barmanObjectName: pprleval-pg-source
-  bootstrap:
-    recovery:
-      source: pprl-pg
-      database: paperless
-      owner: paperless
-  externalClusters:
-    - name: pprl-pg
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: pprleval-pg-source
-          serverName: pprl-pg
-  postgresql:
-    parameters:
-      shared_buffers: "256MB"
-      max_connections: "100"
-      # The production tuning for write reduction is deliberately not copied.
-      # It exists to spare the SSDs under a database that runs for years; this
-      # one is read-only for hours at a time and then deleted.
-  resources:
-    requests:
-      cpu: 200m
-      memory: 512Mi
-    limits:
-      memory: 1Gi

kubernetes/apps/paperless-eval/kustomization.yaml

@@ -1,13 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: paperless-eval
-# The instance only. The measurement harness lives in scripts/paperless-eval/
-# and is applied by hand — a Job is immutable, re-running one means deleting and
-# recreating it, and Flux would just put the old one back. Nothing here should
-# converge on a finished measurement.
-resources:
-  - secrets.yaml
-  - cluster.yaml
-  - pvcs.yaml
-  - redis.yaml
-  - paperless.yaml

kubernetes/apps/paperless-eval/paperless.yaml

@@ -1,203 +0,0 @@
----
-# Deliberately much smaller than the production ConfigMap. Everything that
-# exists to serve people is gone: no remote-user auth, no CSRF origins, no
-# barcode or mail consumption, no Tika/Gotenberg. Nothing consumes documents
-# here, and nothing signs in.
-#
-# What remains is the database connection and the AI settings, because those
-# are what the evaluation measures.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: paperless-eval-config
-  namespace: paperless-eval
-data:
-  PAPERLESS_URL: "http://paperless.paperless-eval.svc.cluster.local"
-  PAPERLESS_ALLOWED_HOSTS: "*"
-  PAPERLESS_TIME_ZONE: "Europe/Berlin"
-  PAPERLESS_REDIS: "redis://redis:6379"
-  PAPERLESS_DBENGINE: "postgresql"
-  PAPERLESS_DBHOST: "pprleval-pg-rw"
-  PAPERLESS_DBNAME: "paperless"
-  # Polling off, so a pod that finds an empty consume volume does not spend a
-  # thread looking at it every ten seconds for the length of a run.
-  PAPERLESS_CONSUMER_POLLING_INTERVAL: "0"
-
-  # The AI settings under test. The endpoints are placeholders that each eval
-  # job overrides for the model it is measuring — a job runs its own
-  # llama-server beside paperless in the same pod, so the value that matters at
-  # run time is the one in the job, not the one here.
-  #
-  # Same caveat as production, and sharper here: paperless reads these from an
-  # ApplicationConfiguration row and falls back to the environment only when
-  # that row is empty. The row comes across in the restored database, so
-  # anything configured in the production admin UI arrives with it and silently
-  # wins over this ConfigMap — an eval would then measure production's settings
-  # while appearing to measure the job's.
-  #
-  # Check it before trusting any result, and after every fresh restore:
-  #
-  #   select llm_backend, llm_model, llm_embedding_backend, llm_embedding_model
-  #     from paperless_applicationconfiguration;
-  #
-  # Every column must be null for the environment to apply.
-  PAPERLESS_AI_ENABLED: "true"
-  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://127.0.0.1:8080/v1"
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "900"
-  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
-  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
-  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
----
-# A base instance that proves the restored database is usable and that the
-# search index and classifier build against it. The evaluation itself does not
-# run here — each job brings its own paperless container so it can point at its
-# own model — but this is what makes a broken clone visible before a job spends
-# hours on it.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: paperless
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless
-  annotations:
-    configmap.reloader.stakater.com/reload: paperless-eval-config
-    # Both, like production: the secret refreshes on the same 10m cycle, and
-    # without this the pod would keep a rotated key until something else
-    # restarted it — which defeats the point of matching the interval.
-    secret.reloader.stakater.com/reload: paperless-eval-secrets
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paperless
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless
-    spec:
-      enableServiceLinks: false
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: paperless
-          # No $imagepolicy marker, unlike apps/paperless: a measurement is
-          # only comparable against a fixed version, so this tag moves when
-          # somebody decides to re-measure, not when a release lands.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          ports:
-            - name: http
-              containerPort: 8000
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          # 60 minutes, against production's 10, and matching this app's Flux
-          # Kustomization timeout.
-          #
-          # Production's budget was sized for a from-scratch boot on this same
-          # corpus and held, so 10 would probably do — but there the volume is
-          # Retain and the index survives, making that a one-off. Here the
-          # volume is ephemeral and the index is rebuilt on every re-test, so
-          # the expensive path is the normal one. A killed pod starts the work
-          # over rather than resuming, which turns a slow boot into a
-          # crash-loop that stalls Flux behind wait: true.
-          #
-          # The LLM index is not in this budget: it is a celery task, and boot
-          # only runs init-llmindex-migrate, a migration check.
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            timeoutSeconds: 5
-            failureThreshold: 180
-          livenessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 120
-            periodSeconds: 30
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-          # Production's context verbatim, and it has to be: the paperless-ngx
-          # entrypoint starts as root, chowns the mounted data/media/consume
-          # volumes and then drops to uid 1000 via setgid/setuid. Without these
-          # the same image fails that sequence with EPERM.
-          #
-          # It bites harder here than in production, where those volumes were
-          # chowned once years ago and keep their ownership. These are fresh on
-          # every rebuild, so the chown is not a no-op.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-        - name: media
-          persistentVolumeClaim:
-            claimName: paperless-eval-media
-        - name: consume
-          persistentVolumeClaim:
-            claimName: paperless-eval-consume
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: paperless
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: paperless
-  ports:
-    - port: 80
-      targetPort: http

kubernetes/apps/paperless-eval/pvcs.yaml

@@ -1,129 +0,0 @@
----
-# All four use ceph-block-ephemeral rather than the Retain default. Everything
-# here is derived — a restored database, an index built from it, a cache —  so
-# the volumes should go when the namespace does instead of leaving Released PVs
-# and orphaned RBD images to reclaim by hand (issue #262).
-#
-# /data holds the Tantivy search index and the classifier pickle, both rebuilt
-# from the database, and the LLM index — which is the expensive one at roughly
-# three hours for the full corpus.
-#
-# Measured on production for the same ~3900 documents: 36 MiB search index,
-# 125 MiB LLM index. 10Gi is far more than that needs, and deliberately so —
-# an embedding model with wider vectors than bge-m3's 1024 dimensions is
-# exactly what phase two varies, and running out of space mid-run would cost
-# the whole rebuild.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  # Named for its role rather than its mount point: the smoke-test deployment
-  # builds /data here, and every eval job copies it out. Replaces the earlier
-  # RWO paperless-eval-data, whose access mode could not be widened in place —
-  # a bound claim's spec is immutable.
-  name: paperless-eval-golden
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  # RWX, because the eval jobs read it too — each copies it into its own
-  # emptyDir rather than mounting it live. The vector store is SQLite, and
-  # SQLite locking across pods over CephFS is not something to rely on.
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem-ephemeral
-  resources:
-    requests:
-      storage: 10Gi
----
-# Empty, and stays that way. Paperless refuses to start without the path, but
-# the AI path reads title, content and filename out of the database and never
-# opens a file — so the documents themselves are not copied here. That is what
-# keeps this instance cheap and keeps a second copy of the archive off the
-# cluster.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-media
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
----
-# Also empty: nothing is consumed here, but the path has to exist.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-consume
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
----
-# The eval jobs' model cache, RWX so several jobs can read the same weights at
-# once — that is the whole point of measuring two candidates in parallel.
-#
-# Separate from the ai namespace's llama-server-models despite holding some of
-# the same files: PVCs do not cross namespaces. A model therefore downloads
-# once here and is then shared by every job, which is the cost a new candidate
-# carries anyway.
-#
-# NOT ephemeral, unlike everything else in this namespace: re-downloading tens
-# of gigabytes on a slow uplink is a worse outcome than an orphaned volume.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: llama-models
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 120Gi
----
-# Results outlive the jobs that produce them: a Job is deleted and recreated to
-# re-run a configuration, and comparing configurations is the entire exercise.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-results
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 1Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-redis
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi

kubernetes/apps/paperless-eval/redis.yaml

@@ -1,81 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: redis
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 999
-        fsGroup: 999
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: redis
-          # No $imagepolicy marker; see the note on the paperless image.
-          image: redis:8.10.1-alpine
-          # No --save: this holds the celery queue and the suggestion cache,
-          # both of which are regenerated. Persisting them across a restart of
-          # a throwaway instance would only preserve stale suggestions.
-          args:
-            - redis-server
-            - --save
-            - ""
-            - --loglevel
-            - warning
-          ports:
-            - name: redis
-              containerPort: 6379
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          livenessProbe:
-            tcpSocket:
-              port: redis
-            initialDelaySeconds: 10
-          readinessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-redis
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: redis

kubernetes/apps/paperless-eval/secrets.yaml

@@ -1,45 +0,0 @@
----
-# The same Hetzner S3 key every CNPG-using namespace pulls, needed here only so
-# the recovery source can be read. See apps/paperless for why one shared key
-# rather than per-app paths.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: paperless-eval
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: infrastructure/hetzner-s3
----
-# Django's secret key, shared with the production instance rather than given
-# its own vault entry.
-#
-# It signs sessions and CSRF tokens, and this instance has no ingress and no
-# users — nothing signs in, so nothing is at stake in sharing it. A separate
-# entry would mean a vault write before the namespace can start, for a
-# throwaway instance that gets rebuilt.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: paperless-eval-secrets
-  namespace: paperless-eval
-spec:
-  # 10m, matching apps/paperless rather than the 1h default: the point of
-  # sharing the key is that the two hold the same value, and a slower refresh
-  # here would let them diverge for up to an hour after a rotation.
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: paperless-eval-secrets
-  dataFrom:
-    - extract:
-        key: apps/paperless

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,18 +1,3 @@
-# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
-# isolated from the apps namespace.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    # Image-build steps run a privileged docker/buildx plugin pod (see the
-    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
-    # without this the privileged step pods are rejected at admission. The
-    # namespace is dedicated to ephemeral CI step pods, so scoping the
-    # exemption here keeps it off apps/.
-    pod-security.kubernetes.io/enforce: privileged
----
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/bootstrap.sh

@@ -159,8 +159,16 @@ if [[ "$PHASE" == "vmtest" ]]; then
 
     # sun_path caps a unix socket path at 108 bytes and the provisioner puts a
     # qemu monitor socket under the state directory, so this has to stay short.
-    # A path under the repo or a long mktemp dir silently fails to boot.
-    VM_STATE=/tmp/bootstrap-vmtest
+    # A path under the repo or a long mktemp dir silently fails to boot. The
+    # caller may override it to aim the qcow2 images at a sized volume; the
+    # guard turns "too long" from a silent boot failure into an error.
+    VM_STATE="${VM_STATE:-/tmp/bootstrap-vmtest}"
+    [[ ${#VM_STATE} -le 50 ]] \
+        || fail "VM_STATE '$VM_STATE' is ${#VM_STATE} bytes; keep it under 50 so the qemu monitor socket fits in sun_path's 108"
+    # cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`, so refuse anything that is
+    # not a nested absolute path.
+    [[ $VM_STATE == /*/?* ]] \
+        || fail "VM_STATE '$VM_STATE' must be an absolute path at least two levels deep"
     VM_CLUSTER=bootstrap-vmtest
     VM_DIR=$(mktemp -d)
     TALOSCTL_BIN=$(command -v talosctl)

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,6 +28,19 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
+    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
+    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
+    # kind is unknown when this path is first applied, and a server-side
+    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
+    #
+    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
+    #   no matches for kind "ObjectStore" in version ...
+    #
+    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
+    # woodpecker — already declares it. This was the only one that did not, and
+    # it took speech-to-phrase, zone-configurator and matter-server down with
+    # it, since all three wait on home-assistant.
+    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,8 +122,20 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  dependsOn:
-    - name: infrastructure
+  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
+  # existing cluster could ever hit and no new one could ever escape:
+  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
+  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
+  # defined in this very Kustomization. A cold rebuild stopped here for good,
+  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
+  #
+  # The edge was never needed. infrastructure runs wait: false, so it applies
+  # the rook-ceph namespace and the operator immediately and only its readiness
+  # waits on the healthChecks — everything this path needs exists long before
+  # infrastructure reports Ready. On an empty API server this Kustomization
+  # retries on retryInterval until the operator installs its CRDs, the same
+  # self-healing shape post-crds already relies on; on a live cluster the CRDs
+  # are always present and nothing changes.
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/clusters/homelab/paperless-eval.yaml

@@ -1,34 +0,0 @@
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: paperless-eval
-  namespace: flux-system
-spec:
-  # A throwaway copy of paperless for measuring the local LLM against the real
-  # corpus. Its Cluster restores from paperless's backup store and archives
-  # nothing back; see kubernetes/apps/paperless-eval/cluster.yaml.
-  #
-  # prune: true and no prune-disabled annotations anywhere underneath — the
-  # data here is a copy, and being able to delete the whole namespace by
-  # removing this file is the point.
-  interval: 10m
-  retryInterval: 1m
-  # 30m is not enough: the first start restores the database and then builds
-  # the search index over ~3900 documents before the webserver answers.
-  timeout: 60m
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./kubernetes/apps/paperless-eval
-  prune: true
-  wait: true
-  dependsOn:
-    - name: infrastructure
-    - name: infrastructure-crds
-    - name: infrastructure-ceph-cluster
-    - name: infrastructure-cnpg-barman
-  healthCheckExprs:
-    - apiVersion: postgresql.cnpg.io/v1
-      kind: Cluster
-      failed: "has(status.phase) && status.phase.startsWith('Failed')"
-      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -73,11 +73,6 @@ metadata:
 # classifies documents without experimenting on the live archive.
 apiVersion: v1
 kind: Namespace
-metadata:
-  name: paperless-eval
----
-apiVersion: v1
-kind: Namespace
 metadata:
   name: audiobookshelf
 ---
@@ -195,3 +190,31 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
+---
+# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
+# from `apps`. Distinct from `woodpecker`, which holds the service itself.
+#
+# Here rather than in apps/woodpecker for the reason spelled out above that
+# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
+# ../network-policies/, and infrastructure-crds runs before every app KS. With
+# the namespace declared in the workload KS the order inverted, and a rebuild
+# stopped with
+#
+#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
+#   namespaces "woodpecker-ci" not found
+#
+# which held infrastructure-crds, and with it all twenty app Kustomizations
+# that depend on it. The same reasoning was already written down for
+# `woodpecker` and simply not carried across to its sibling.
+#
+# Image-build steps run a privileged docker/buildx plugin pod (see the server's
+# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
+# label the privileged step pods are rejected at admission. The namespace holds
+# nothing but ephemeral CI step pods, so the exemption stays off apps/.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    pod-security.kubernetes.io/enforce: privileged

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,22 +34,6 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # The paperless-eval namespace reaches the embedding endpoint for the same
-    # reason paperless does; its chat model runs inside each eval job's own pod
-    # and needs no grant here.
-    #
-    # Scoped to the namespace rather than to an app label, unlike the grants
-    # around it: that namespace exists only to run evaluations, and its
-    # workload is jobs created per measurement whose labels vary with what is
-    # being measured. A label match would be a trap that presents as an
-    # embedding call silently dropped.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     # Home Assistant's llama.cpp conversation agent calls the chat model only —
     # it has no use for the embedding endpoint, so unlike paperless above this
     # grant does not extend to llama-embed.

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -6,7 +6,6 @@ resources:
   - ingress-isolation.yaml
   - authentik-ns.yaml
   - paperless-ns.yaml
-  - paperless-eval-ns.yaml
   - audiobookshelf-ns.yaml
   - home-assistant-ns.yaml
   - spegel-ns.yaml

kubernetes/infrastructure/network-policies/paperless-eval-ns.yaml

@@ -1,141 +0,0 @@
-# Narrower than paperless-ns.yaml, because this namespace serves nobody.
-#
-# There is no ingress, no gate outpost and no hostname: the only things that
-# talk to it are its own pods. Monitoring is admitted so a run can be watched,
-# and host/remote-node for the kubelet probes.
-#
-# Egress reaches the ai namespace for the embedding endpoint, which is the one
-# dependency an eval job has outside its own pod, and world:443 so the CNPG
-# recovery can read the backup store on Hetzner.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: default-deny-ingress
-  namespace: paperless-eval
-spec:
-  endpointSelector: {}
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": monitoring
-    - fromEntities:
-        - host
-        - remote-node
-    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
-    # home-assistant-ns.yaml for the rationale.
-    - fromEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
-  egress:
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    # The embedding endpoint. The chat model runs inside each job's own pod, so
-    # it needs no grant; the embedder is shared with production because it is
-    # read-only and cheap, and swapping it is a phase-two concern.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-embed
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": rook-ceph
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": kube-system
-            k8s-app: kube-dns
-      toPorts:
-        - ports:
-            - port: "53"
-              protocol: UDP
-    - toEntities:
-        - host
-        - remote-node
-    # world:443 — the CNPG recovery reads the backup store on Hetzner Object
-    # Storage, and image pulls.
-    - toEntities:
-        - world
-      toPorts:
-        - ports:
-            - port: "443"
-              protocol: TCP
-    # ICMP — egress mirror; see ingress block above.
-    - toEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
----
-# The CNPG operator polls each instance's status endpoint over HTTPS
-# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
-# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
-# while Postgres serves normally — issue #433, where the operator's dial timed
-# out because the apps baseline never allowed the database namespace.
-#
-# It matters here even for a throwaway: the Flux Kustomization waits on
-# Cluster status, and a smoke test that cannot report its own health is not one.
-#
-# Scoped to the pprleval-pg endpoints and that one port, like the equivalent
-# rules for wp-pg and ha-pg: the barman and mariadb operators sharing the
-# database namespace have no reason to reach anything here.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: pprleval-pg-allow-cnpg-operator
-  namespace: paperless-eval
-spec:
-  endpointSelector:
-    matchLabels:
-      cnpg.io/cluster: pprleval-pg
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": database
-            app.kubernetes.io/name: cloudnative-pg
-      toPorts:
-        - ports:
-            - port: "8000"
-              protocol: TCP

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -27,7 +27,6 @@ spec:
         - ingress
         - monitoring
         - paperless
-        - paperless-eval # same apps/paperless key plus the shared barman S3 creds
         - trmnl
         - velero
         - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)