feat(tooling): Hetzner throwaway site and rebuild-harness hardening

None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.

New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.

The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.

bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.

import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.

Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/paperless-eval/cluster.yaml

@@ -1,116 +0,0 @@
----
-# A read-only view of paperless's backup store, used as a recovery source and
-# nothing else.
-#
-# It points at the production prefix, which is the whole point — the eval
-# instance is a copy of the real archive's metadata — and that makes what is
-# ABSENT here the safety-critical part:
-#
-#   no retentionPolicy   The production store has "30d". Without one here
-#                        there is no policy to enforce, and enforcement is not
-#                        tied to taking a backup: the plugin runs it from the
-#                        instance sidecar on a timer
-#                        (instanceSidecarConfiguration.retentionPolicyIntervalSeconds,
-#                        default 1800). So "this cluster never backs up" would
-#                        not have been enough on its own.
-#
-# A second property makes it belt-and-braces. Retention runs
-# barman-cloud-backup-delete, which takes a server name and works inside
-# <destinationPath>/<serverName> — the store's own status keys its recovery
-# window by "pprl-pg", confirming that layout. This cluster is pprleval-pg, so
-# even an accidental retention policy would address a prefix that holds nothing
-# rather than production's backups.
-#
-# The cluster below completes the guarantee with isWALArchiver: false, so
-# nothing is written back either. It does load the plugin — that is required
-# for the restore to work at all — and read-only comes from the flag rather
-# than from leaving the plugin out.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: pprleval-pg-source
-  namespace: paperless-eval
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
----
-# A disposable clone of the paperless database, for measuring how well the
-# local LLM classifies documents without running the experiment against the
-# live archive.
-#
-# Losing this cluster costs a re-restore and nothing else, which is the intent
-# — it is rebuilt whenever a test needs a clean starting point.
-#
-# One instance, not two. Replication buys availability, and an eval instance
-# that is down is simply an eval that waits.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: pprleval-pg
-  namespace: paperless-eval
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  enableSuperuserAccess: true
-  storage:
-    # Delete-reclaim, unlike the Retain default: this volume is a copy that
-    # gets rebuilt, so it should vanish with the cluster rather than leave a
-    # Released PV and an orphaned RBD image behind (issue #262).
-    storageClass: ceph-block-ephemeral
-    size: 10Gi
-  # Declared with isWALArchiver: false, which is what kubernetes/RECOVERY.md
-  # arrived at after an earlier version of that runbook said to omit it: the
-  # instance manager loads this plugin to replay WAL during the restore, and
-  # without it the instance never starts ("Unknown plugin:
-  # barman-cloud.cloudnative-pg.io").
-  #
-  # externalClusters below only says WHERE the backup is; it does not get the
-  # plugin injected into the pod. So "no plugin" is not the way to keep this
-  # read-only — "plugin, not archiving" is, and it is the combination verified
-  # on im-pg on 2026-07-25.
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      enabled: true
-      isWALArchiver: false
-      parameters:
-        barmanObjectName: pprleval-pg-source
-  bootstrap:
-    recovery:
-      source: pprl-pg
-      database: paperless
-      owner: paperless
-  externalClusters:
-    - name: pprl-pg
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: pprleval-pg-source
-          serverName: pprl-pg
-  postgresql:
-    parameters:
-      shared_buffers: "256MB"
-      max_connections: "100"
-      # The production tuning for write reduction is deliberately not copied.
-      # It exists to spare the SSDs under a database that runs for years; this
-      # one is read-only for hours at a time and then deleted.
-  resources:
-    requests:
-      cpu: 200m
-      memory: 512Mi
-    limits:
-      memory: 1Gi

kubernetes/apps/paperless-eval/kustomization.yaml

@@ -1,13 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: paperless-eval
-# The instance only. The measurement harness lives in scripts/paperless-eval/
-# and is applied by hand — a Job is immutable, re-running one means deleting and
-# recreating it, and Flux would just put the old one back. Nothing here should
-# converge on a finished measurement.
-resources:
-  - secrets.yaml
-  - cluster.yaml
-  - pvcs.yaml
-  - redis.yaml
-  - paperless.yaml

kubernetes/apps/paperless-eval/paperless.yaml

@@ -1,203 +0,0 @@
----
-# Deliberately much smaller than the production ConfigMap. Everything that
-# exists to serve people is gone: no remote-user auth, no CSRF origins, no
-# barcode or mail consumption, no Tika/Gotenberg. Nothing consumes documents
-# here, and nothing signs in.
-#
-# What remains is the database connection and the AI settings, because those
-# are what the evaluation measures.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: paperless-eval-config
-  namespace: paperless-eval
-data:
-  PAPERLESS_URL: "http://paperless.paperless-eval.svc.cluster.local"
-  PAPERLESS_ALLOWED_HOSTS: "*"
-  PAPERLESS_TIME_ZONE: "Europe/Berlin"
-  PAPERLESS_REDIS: "redis://redis:6379"
-  PAPERLESS_DBENGINE: "postgresql"
-  PAPERLESS_DBHOST: "pprleval-pg-rw"
-  PAPERLESS_DBNAME: "paperless"
-  # Polling off, so a pod that finds an empty consume volume does not spend a
-  # thread looking at it every ten seconds for the length of a run.
-  PAPERLESS_CONSUMER_POLLING_INTERVAL: "0"
-
-  # The AI settings under test. The endpoints are placeholders that each eval
-  # job overrides for the model it is measuring — a job runs its own
-  # llama-server beside paperless in the same pod, so the value that matters at
-  # run time is the one in the job, not the one here.
-  #
-  # Same caveat as production, and sharper here: paperless reads these from an
-  # ApplicationConfiguration row and falls back to the environment only when
-  # that row is empty. The row comes across in the restored database, so
-  # anything configured in the production admin UI arrives with it and silently
-  # wins over this ConfigMap — an eval would then measure production's settings
-  # while appearing to measure the job's.
-  #
-  # Check it before trusting any result, and after every fresh restore:
-  #
-  #   select llm_backend, llm_model, llm_embedding_backend, llm_embedding_model
-  #     from paperless_applicationconfiguration;
-  #
-  # Every column must be null for the environment to apply.
-  PAPERLESS_AI_ENABLED: "true"
-  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://127.0.0.1:8080/v1"
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "900"
-  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
-  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
-  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
----
-# A base instance that proves the restored database is usable and that the
-# search index and classifier build against it. The evaluation itself does not
-# run here — each job brings its own paperless container so it can point at its
-# own model — but this is what makes a broken clone visible before a job spends
-# hours on it.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: paperless
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless
-  annotations:
-    configmap.reloader.stakater.com/reload: paperless-eval-config
-    # Both, like production: the secret refreshes on the same 10m cycle, and
-    # without this the pod would keep a rotated key until something else
-    # restarted it — which defeats the point of matching the interval.
-    secret.reloader.stakater.com/reload: paperless-eval-secrets
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paperless
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless
-    spec:
-      enableServiceLinks: false
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: paperless
-          # No $imagepolicy marker, unlike apps/paperless: a measurement is
-          # only comparable against a fixed version, so this tag moves when
-          # somebody decides to re-measure, not when a release lands.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          ports:
-            - name: http
-              containerPort: 8000
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          # 60 minutes, against production's 10, and matching this app's Flux
-          # Kustomization timeout.
-          #
-          # Production's budget was sized for a from-scratch boot on this same
-          # corpus and held, so 10 would probably do — but there the volume is
-          # Retain and the index survives, making that a one-off. Here the
-          # volume is ephemeral and the index is rebuilt on every re-test, so
-          # the expensive path is the normal one. A killed pod starts the work
-          # over rather than resuming, which turns a slow boot into a
-          # crash-loop that stalls Flux behind wait: true.
-          #
-          # The LLM index is not in this budget: it is a celery task, and boot
-          # only runs init-llmindex-migrate, a migration check.
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            timeoutSeconds: 5
-            failureThreshold: 180
-          livenessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 120
-            periodSeconds: 30
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-          # Production's context verbatim, and it has to be: the paperless-ngx
-          # entrypoint starts as root, chowns the mounted data/media/consume
-          # volumes and then drops to uid 1000 via setgid/setuid. Without these
-          # the same image fails that sequence with EPERM.
-          #
-          # It bites harder here than in production, where those volumes were
-          # chowned once years ago and keep their ownership. These are fresh on
-          # every rebuild, so the chown is not a no-op.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-        - name: media
-          persistentVolumeClaim:
-            claimName: paperless-eval-media
-        - name: consume
-          persistentVolumeClaim:
-            claimName: paperless-eval-consume
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: paperless
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: paperless
-  ports:
-    - port: 80
-      targetPort: http

kubernetes/apps/paperless-eval/pvcs.yaml

@@ -1,129 +0,0 @@
----
-# All four use ceph-block-ephemeral rather than the Retain default. Everything
-# here is derived — a restored database, an index built from it, a cache —  so
-# the volumes should go when the namespace does instead of leaving Released PVs
-# and orphaned RBD images to reclaim by hand (issue #262).
-#
-# /data holds the Tantivy search index and the classifier pickle, both rebuilt
-# from the database, and the LLM index — which is the expensive one at roughly
-# three hours for the full corpus.
-#
-# Measured on production for the same ~3900 documents: 36 MiB search index,
-# 125 MiB LLM index. 10Gi is far more than that needs, and deliberately so —
-# an embedding model with wider vectors than bge-m3's 1024 dimensions is
-# exactly what phase two varies, and running out of space mid-run would cost
-# the whole rebuild.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  # Named for its role rather than its mount point: the smoke-test deployment
-  # builds /data here, and every eval job copies it out. Replaces the earlier
-  # RWO paperless-eval-data, whose access mode could not be widened in place —
-  # a bound claim's spec is immutable.
-  name: paperless-eval-golden
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  # RWX, because the eval jobs read it too — each copies it into its own
-  # emptyDir rather than mounting it live. The vector store is SQLite, and
-  # SQLite locking across pods over CephFS is not something to rely on.
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem-ephemeral
-  resources:
-    requests:
-      storage: 10Gi
----
-# Empty, and stays that way. Paperless refuses to start without the path, but
-# the AI path reads title, content and filename out of the database and never
-# opens a file — so the documents themselves are not copied here. That is what
-# keeps this instance cheap and keeps a second copy of the archive off the
-# cluster.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-media
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
----
-# Also empty: nothing is consumed here, but the path has to exist.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-consume
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
----
-# The eval jobs' model cache, RWX so several jobs can read the same weights at
-# once — that is the whole point of measuring two candidates in parallel.
-#
-# Separate from the ai namespace's llama-server-models despite holding some of
-# the same files: PVCs do not cross namespaces. A model therefore downloads
-# once here and is then shared by every job, which is the cost a new candidate
-# carries anyway.
-#
-# NOT ephemeral, unlike everything else in this namespace: re-downloading tens
-# of gigabytes on a slow uplink is a worse outcome than an orphaned volume.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: llama-models
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 120Gi
----
-# Results outlive the jobs that produce them: a Job is deleted and recreated to
-# re-run a configuration, and comparing configurations is the entire exercise.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-results
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 1Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-redis
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi

kubernetes/apps/paperless-eval/redis.yaml

@@ -1,81 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: redis
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 999
-        fsGroup: 999
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: redis
-          # No $imagepolicy marker; see the note on the paperless image.
-          image: redis:8.10.1-alpine
-          # No --save: this holds the celery queue and the suggestion cache,
-          # both of which are regenerated. Persisting them across a restart of
-          # a throwaway instance would only preserve stale suggestions.
-          args:
-            - redis-server
-            - --save
-            - ""
-            - --loglevel
-            - warning
-          ports:
-            - name: redis
-              containerPort: 6379
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          livenessProbe:
-            tcpSocket:
-              port: redis
-            initialDelaySeconds: 10
-          readinessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-redis
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: redis

kubernetes/apps/paperless-eval/secrets.yaml

@@ -1,45 +0,0 @@
----
-# The same Hetzner S3 key every CNPG-using namespace pulls, needed here only so
-# the recovery source can be read. See apps/paperless for why one shared key
-# rather than per-app paths.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: paperless-eval
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: infrastructure/hetzner-s3
----
-# Django's secret key, shared with the production instance rather than given
-# its own vault entry.
-#
-# It signs sessions and CSRF tokens, and this instance has no ingress and no
-# users — nothing signs in, so nothing is at stake in sharing it. A separate
-# entry would mean a vault write before the namespace can start, for a
-# throwaway instance that gets rebuilt.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: paperless-eval-secrets
-  namespace: paperless-eval
-spec:
-  # 10m, matching apps/paperless rather than the 1h default: the point of
-  # sharing the key is that the two hold the same value, and a slower refresh
-  # here would let them diverge for up to an hour after a rotation.
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: paperless-eval-secrets
-  dataFrom:
-    - extract:
-        key: apps/paperless

kubernetes/clusters/homelab/paperless-eval.yaml

@@ -1,34 +0,0 @@
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: paperless-eval
-  namespace: flux-system
-spec:
-  # A throwaway copy of paperless for measuring the local LLM against the real
-  # corpus. Its Cluster restores from paperless's backup store and archives
-  # nothing back; see kubernetes/apps/paperless-eval/cluster.yaml.
-  #
-  # prune: true and no prune-disabled annotations anywhere underneath — the
-  # data here is a copy, and being able to delete the whole namespace by
-  # removing this file is the point.
-  interval: 10m
-  retryInterval: 1m
-  # 30m is not enough: the first start restores the database and then builds
-  # the search index over ~3900 documents before the webserver answers.
-  timeout: 60m
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./kubernetes/apps/paperless-eval
-  prune: true
-  wait: true
-  dependsOn:
-    - name: infrastructure
-    - name: infrastructure-crds
-    - name: infrastructure-ceph-cluster
-    - name: infrastructure-cnpg-barman
-  healthCheckExprs:
-    - apiVersion: postgresql.cnpg.io/v1
-      kind: Cluster
-      failed: "has(status.phase) && status.phase.startsWith('Failed')"
-      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -73,11 +73,6 @@ metadata:
 # classifies documents without experimenting on the live archive.
 apiVersion: v1
 kind: Namespace
-metadata:
-  name: paperless-eval
----
-apiVersion: v1
-kind: Namespace
 metadata:
   name: audiobookshelf
 ---

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,22 +34,6 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # The paperless-eval namespace reaches the embedding endpoint for the same
-    # reason paperless does; its chat model runs inside each eval job's own pod
-    # and needs no grant here.
-    #
-    # Scoped to the namespace rather than to an app label, unlike the grants
-    # around it: that namespace exists only to run evaluations, and its
-    # workload is jobs created per measurement whose labels vary with what is
-    # being measured. A label match would be a trap that presents as an
-    # embedding call silently dropped.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     # Home Assistant's llama.cpp conversation agent calls the chat model only —
     # it has no use for the embedding endpoint, so unlike paperless above this
     # grant does not extend to llama-embed.

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -6,7 +6,6 @@ resources:
   - ingress-isolation.yaml
   - authentik-ns.yaml
   - paperless-ns.yaml
-  - paperless-eval-ns.yaml
   - audiobookshelf-ns.yaml
   - home-assistant-ns.yaml
   - spegel-ns.yaml

kubernetes/infrastructure/network-policies/paperless-eval-ns.yaml

@@ -1,141 +0,0 @@
-# Narrower than paperless-ns.yaml, because this namespace serves nobody.
-#
-# There is no ingress, no gate outpost and no hostname: the only things that
-# talk to it are its own pods. Monitoring is admitted so a run can be watched,
-# and host/remote-node for the kubelet probes.
-#
-# Egress reaches the ai namespace for the embedding endpoint, which is the one
-# dependency an eval job has outside its own pod, and world:443 so the CNPG
-# recovery can read the backup store on Hetzner.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: default-deny-ingress
-  namespace: paperless-eval
-spec:
-  endpointSelector: {}
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": monitoring
-    - fromEntities:
-        - host
-        - remote-node
-    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
-    # home-assistant-ns.yaml for the rationale.
-    - fromEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
-  egress:
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    # The embedding endpoint. The chat model runs inside each job's own pod, so
-    # it needs no grant; the embedder is shared with production because it is
-    # read-only and cheap, and swapping it is a phase-two concern.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-embed
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": rook-ceph
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": kube-system
-            k8s-app: kube-dns
-      toPorts:
-        - ports:
-            - port: "53"
-              protocol: UDP
-    - toEntities:
-        - host
-        - remote-node
-    # world:443 — the CNPG recovery reads the backup store on Hetzner Object
-    # Storage, and image pulls.
-    - toEntities:
-        - world
-      toPorts:
-        - ports:
-            - port: "443"
-              protocol: TCP
-    # ICMP — egress mirror; see ingress block above.
-    - toEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
----
-# The CNPG operator polls each instance's status endpoint over HTTPS
-# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
-# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
-# while Postgres serves normally — issue #433, where the operator's dial timed
-# out because the apps baseline never allowed the database namespace.
-#
-# It matters here even for a throwaway: the Flux Kustomization waits on
-# Cluster status, and a smoke test that cannot report its own health is not one.
-#
-# Scoped to the pprleval-pg endpoints and that one port, like the equivalent
-# rules for wp-pg and ha-pg: the barman and mariadb operators sharing the
-# database namespace have no reason to reach anything here.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: pprleval-pg-allow-cnpg-operator
-  namespace: paperless-eval
-spec:
-  endpointSelector:
-    matchLabels:
-      cnpg.io/cluster: pprleval-pg
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": database
-            app.kubernetes.io/name: cloudnative-pg
-      toPorts:
-        - ports:
-            - port: "8000"
-              protocol: TCP

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -27,7 +27,6 @@ spec:
         - ingress
         - monitoring
         - paperless
-        - paperless-eval # same apps/paperless key plus the shared barman S3 creds
         - trmnl
         - velero
         - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)

kubernetes/talos/patches/hcloud-1.yaml

@@ -0,0 +1,69 @@
+# First machine of the throwaway cluster on Hetzner.
+#
+# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
+# order bootstrap.sh attaches them.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-1
+  nodeLabels:
+    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
+    # rather than MAC: the address follows from the order bootstrap.sh attaches
+    # the networks, while the MAC is assigned by Hetzner at attach time and is
+    # not known when this file is written.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+# In the generated schema rather than machine.network.hostname: naming the host
+# in both is rejected at apply time. `auto: off` switches off the generated
+# `auto: stable`, which is mutually exclusive with a static name.
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-1
+---
+# Every interface is stated, the public one included. Talos stops running its
+# default DHCP operators for the whole machine as soon as one new-style network
+# document exists, so an omitted interface gets no address.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# Ceph cluster network, renamed to the role name by the udev rule in the
+# machine section. The node holds no address here; the link exists for the OSD
+# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    # Two disks: the system disk and the attached volume. Matching on transport
+    # as the qemu patches do selects nothing here — a Hetzner volume arrives
+    # over virtio-scsi, and a selector matching zero disks is skipped silently.
+    match: '!system_disk'
+  # bootstrap.sh reads this size and creates the volume to match.
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -0,0 +1,50 @@
+# Second machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-2
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-2
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -0,0 +1,50 @@
+# Third machine. See hcloud-1.yaml.
+machine:
+  install:
+    disk: /dev/sda
+  certSANs:
+    - hcloud-3
+  nodeLabels:
+    storage.renner.uno/ceph-osd: "true"
+  udev:
+    # See hcloud-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: off
+hostname: hcloud-3
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp1s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp1s0
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp7s0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp7s0
+---
+# See hcloud-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ceph-san
+up: true
+---
+apiVersion: v1alpha1
+kind: RawVolumeConfig
+name: ceph-osd
+provisioning:
+  diskSelector:
+    match: '!system_disk'
+  minSize: 50GiB
+  grow: true

kubernetes/talos/patches/site-hcloud.yaml

@@ -0,0 +1,25 @@
+# Site patch for the throwaway cluster on Hetzner Cloud.
+#
+# This site exists for one thing the other two cannot test: whether a node that
+# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
+# answers, so containerd never reaches the second endpoint in
+# machine.registries.mirrors and the fallback stays unproven.
+#
+# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
+# frames from it do not cross between servers — measured: same-node ARP
+# resolves, cross-node does not. OSD replication therefore does not run here.
+#
+# What belongs in this file is what the cluster is. Interfaces, disks and names
+# are machine properties and live in the node patches.
+machine:
+  kubelet:
+    nodeIP:
+      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
+      # kubelet can pick the public NIC.
+      validSubnets:
+        - 10.90.0.0/16
+cluster:
+  # Three control planes that also carry workloads, which is the production
+  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
+  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
+  allowSchedulingOnControlPlanes: true

kubernetes/vmtest-job.yaml

@@ -0,0 +1,134 @@
+# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
+#
+# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
+# one-shot test, and a Kustomization would recreate it after every completion.
+# It is applied by the vmtest pipeline, or by hand for a single run.
+#
+# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
+# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
+# a single run left under 1G free and two overlapping runs OOM-killed the
+# session. The nodes have ~94G each with about 70G unused.
+#
+# The whole script runs here, flux and OpenBao included, so that what a rebuild
+# rehearses is what a real recovery would execute. The unseal key is not baked
+# in and not stored as a Secret: the operator streams all five secrets into an
+# in-memory emptyDir at the moment the run asks for them, and they die with the
+# Job without ever reaching etcd or a backup of it.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vmtest
+  labels:
+    # The pod is privileged; say so rather than have the admission policy
+    # reject it for reasons that look unrelated.
+    pod-security.kubernetes.io/enforce: privileged
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: vmtest
+  namespace: vmtest
+spec:
+  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
+  ttlSecondsAfterFinished: 86400
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: vmtest
+    spec:
+      restartPolicy: Never
+      # Nothing here should be able to reach the cluster it runs in. The phase
+      # points KUBECONFIG at its own throwaway cluster, and without a mounted
+      # token a stray kubectl cannot fall back to this one.
+      automountServiceAccountToken: false
+      containers:
+        - name: vmtest
+          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
+          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
+          # three builds old, which predated OVMF, and died with "the required
+          # flash image was not found" before the first VM booted. The tag is
+          # carried alongside for readability — containerd resolves the digest.
+          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
+          # Privileged for /dev/kvm and for the bridges and taps the provisioner
+          # builds. It needs no hostNetwork: the provisioner's network lives in
+          # this pod's own namespace, and the script that talks to 10.5.0.x runs
+          # in the same pod.
+          securityContext:
+            privileged: true
+          # A hard ceiling, because the failure mode is known: an unbounded run
+          # is what OOM-killed the workstation. 40G is the VMs (16G control
+          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
+          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
+          # of room for the CNI DaemonSets themselves. The control plane went
+          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
+          # memory and the kubelet evicted kube-apiserver to admit a
+          # node-critical pod — bootstrap.sh carries the full account next to
+          # the sizing flags. The hosts have ~95GiB each, so this fits.
+          resources:
+            requests:
+              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
+              # the guests are never all runnable at once (one cluster booting
+              # in sequence), and there is no CPU limit, so the pod may burst
+              # past the request when the node has headroom. The request only
+              # keeps the scheduler from stacking this next to another big
+              # burst consumer.
+              cpu: "6"
+              memory: 42Gi
+            limits:
+              memory: 48Gi
+          env:
+            # talosctl caches the image and the CNI bundle under $HOME; point it
+            # at the scratch volume rather than the container layer.
+            - name: HOME
+              value: /work
+            # The VM disks. Short on purpose — the provisioner puts a qemu
+            # monitor socket under it and sun_path caps the path at 108 bytes.
+            - name: VM_STATE
+              value: /work/vm
+            # Which branch to test. master unless a run is checking a change.
+            - name: BRANCH
+              value: master
+            # Secrets are streamed in here at the moment the run needs them,
+            # never baked into the image and never stored as a Secret. They live
+            # in this emptyDir for the length of the Job and go with it — so
+            # they do not reach etcd, or any backup of it.
+            - name: OPENBAO_SECRETS_DIR
+              value: /secrets
+          command: ["/bin/bash", "-ec"]
+          args:
+            - |
+              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
+              # that clones from the cluster it is rehearsing the loss of
+              # cannot start when it matters — and a run already died here on
+              # resolving gitea.renner.uno.
+              git clone --depth 1 --branch "$BRANCH" \
+                https://github.com/konairius/Homelab.git /work/homelab
+              exec /work/homelab/kubernetes/bootstrap.sh vmtest
+          volumeMounts:
+            - name: work
+              mountPath: /work
+            - name: secrets
+              mountPath: /secrets
+            - name: dev-kvm
+              mountPath: /dev/kvm
+      volumes:
+        # The VM disks and the provisioner state. emptyDir on the node's disk
+        # rather than a PVC: it is scratch that dies with the Job, and putting it
+        # on Ceph would have the test's own writes compete with the storage it is
+        # meant to leave alone.
+        - name: work
+          emptyDir:
+            # The VM images are sparse, so this bounds what is actually written,
+            # not the 232GiB the geometry declares. 60Gi was already half spent
+            # by four 10GiB systems disks; the node carries this comfortably with
+            # ~810GB free on the filesystem behind it.
+            sizeLimit: 150Gi
+        # In memory, so the material never touches a node's disk.
+        - name: secrets
+          emptyDir:
+            medium: Memory
+            sizeLimit: 1Mi
+        - name: dev-kvm
+          hostPath:
+            path: /dev/kvm
+            type: CharDevice

scripts/import-cluster-access.sh

@@ -13,7 +13,18 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-export AWS_PROFILE="secrets"
+# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
+# The default is unchanged and deliberately so: "secrets" resolves to working
+# credentials and this script has always used it. Both "secrets" and "hetzner"
+# exist in ~/.aws/credentials on the workstation.
+#
+# The defect is only the assignment. Forcing the value defeated the env-based
+# path this file's own header offers, and it also overrode a caller who had
+# already selected a profile — which is how a 403 from HeadObject can appear
+# while the credentials the caller intended were perfectly good. That error
+# reads as a bucket permissions problem and sends you looking in the wrong
+# place.
+export AWS_PROFILE="${AWS_PROFILE:-secrets}"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }