fix(bootstrap): honour a caller-supplied VM_STATE in vmtest

The vmtest Job passes VM_STATE=/work/vm to aim the qcow2 images at the 150Gi emptyDir it sizes for them. bootstrap.sh overwrote it unconditionally, so the images landed on the container's writable layer instead and the sizeLimit guard never applied to anything. Default to the same path as before when the caller sets nothing, so a plain `bootstrap.sh vmtest` is unchanged. Two guards come with it, because the value is now caller-controlled: * a length cap, since the provisioner puts a qemu monitor socket under this directory and sun_path stops at 108 bytes. Too long used to be a silent failure to boot; it is now an error. Both the old default (21 bytes) and the Job's /work/vm (8) are well inside it. * a shape check, because cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`. That was safe by construction while the path was hardcoded and is not once a caller picks it. This does not by itself reduce drive wear — an emptyDir also lives on /var, so both paths are on the same device. It restores the size guard and makes it possible to point VM state at a different device later. VMTEST_FULL and VMTEST_KEEP, which the Job also sets, are still not read anywhere in this script. Left alone rather than guessed at: the Job is applied out of band and is not in this repo, so the intended semantics are not visible from here. Refs #754 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'chore(paperless-eval): remove the namespace and its whole footprint' (#791 ) from chore/remove-paperless-eval into master
2026-09-06 11:28:06 +02:00 · 2026-09-06 08:46:27 +00:00 · 2026-09-06 10:41:51 +02:00 · 2026-09-06 08:08:43 +00:00 · 2026-09-06 10:03:11 +02:00 · 2026-09-06 07:56:39 +00:00
44 changed files with 1486 additions and 923 deletions
@@ -120,7 +120,14 @@ spec:
    server:
      resources:
        requests:
-          cpu: 200m
+          # 500m, not 200m: the HPA below targets 70% of *this* request, so at
+          # 200m the scale-up threshold sat at 140m — below the pod's own
+          # ordinary peak. Measured over 7d, a warm server pod p99s at ~108m
+          # and peaks at ~190m, and a freshly-started one burns ~300m booting
+          # Django. That made the HPA oscillate 2<->3<->4 all day and pin at
+          # maxReplicas (KubeHpaMaxedOut, issue #761). At 500m the threshold
+          # is 350m/pod, above the boot burst, so only real load scales out.
+          cpu: 500m
          memory: 1Gi
        limits:
          memory: 2Gi
@@ -154,7 +161,11 @@ spec:
    worker:
      resources:
        requests:
-          cpu: 200m
+          # Same reasoning as server above, and the worker is burstier still:
+          # over 7d its per-pod p99 reaches ~336m and peaks hit ~630m while
+          # running blueprint applies and scheduled tasks. 200m understated
+          # that badly enough that routine task batches read as saturation.
+          cpu: 500m
          memory: 1Gi
        limits:
          memory: 2Gi
@@ -95,7 +95,7 @@ spec:
              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.9.0 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -60,7 +60,7 @@ spec:
      # Two processes rather than one: socat carries a single address pair per
      # invocation, and the families need different receive and send addresses.
      # They are independent — one crashing does not blind the other — and each
-      # is restarted by the kubelet on its own.
+      # carries its own supervisor loop.
      containers:
        # -u is load-bearing on both: unidirectional, left to right. Without it
        # socat would also carry HA's replies back out, and this must stay a
@@ -77,17 +77,29 @@ spec:
        # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
        # mdns-relay-service.yaml for why that indirection is required and why
        # the Service is dual-stack.
+        #
+        # Supervisor loop: socat exits 1 on the first datagram it cannot
+        # deliver, and the ClusterIP has no backend while HA restarts. The
+        # trap forwards SIGTERM so the pod still terminates promptly.
        - name: socat-ipv4
          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
          imagePullPolicy: IfNotPresent
-          args:
-            - -u
-            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
-            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
-          # No probes. socat is a single process with no health surface: if it
-          # exits, the container exits and the kubelet restarts it, which is the
-          # entire liveness story. A probe here could only re-assert that PID 1
-          # is running.
+          command:
+            - /bin/sh
+            - -c
+            - |
+              trap 'kill "$pid"; exit 0' TERM INT
+              while :; do
+                socat -u \
+                  "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr" \
+                  "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353" &
+                pid=$!
+                wait "$pid"
+                echo "mdns-relay: socat-ipv4 exited ($?); restarting in 2s" >&2
+                sleep 2
+              done
+          # No probes. The supervisor loop is the liveness story; a probe here
+          # could only re-assert that PID 1 is running.
          resources:
            requests:
              cpu: 10m
@@ -114,10 +126,20 @@ spec:
        - name: socat-ipv6
          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
          imagePullPolicy: IfNotPresent
-          args:
-            - -u
-            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
-            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          command:
+            - /bin/sh
+            - -c
+            - |
+              trap 'kill "$pid"; exit 0' TERM INT
+              while :; do
+                socat -u \
+                  "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr" \
+                  "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353" &
+                pid=$!
+                wait "$pid"
+                echo "mdns-relay: socat-ipv6 exited ($?); restarting in 2s" >&2
+                sleep 2
+              done
          resources:
            requests:
              cpu: 10m
@@ -25,6 +25,13 @@ metadata:
    app.kubernetes.io/name: llama-server
 spec:
  replicas: 1
+  # The startupProbe below sanctions a 40 min cold download, but the default
+  # progressDeadlineSeconds is 600. Left implicit the Deployment declares
+  # ProgressDeadlineExceeded at 10 minutes while the probe is still legitimately
+  # waiting, and KubeDeploymentRolloutStuck fires 15 min after that on a rollout
+  # that is fine. Kept above the startup budget, with room for the image pull
+  # and scheduling that happen before the first probe.
+  progressDeadlineSeconds: 3000
  # A rolling update would load a second copy of the weights onto the same
  # iGPU and into the same node's RAM before releasing the first.
  strategy:
@@ -19,6 +19,11 @@ metadata:
    app.kubernetes.io/name: llama-embed
 spec:
  replicas: 1
+  # Same reasoning as llama-server: the startupProbe budget (600s) must stay
+  # below progressDeadlineSeconds, whose default is also 600 — exactly equal, so
+  # any cold download that uses most of its budget trips
+  # KubeDeploymentRolloutStuck on a rollout that is still progressing normally.
+  progressDeadlineSeconds: 900
  strategy:
    type: Recreate
  selector:
@@ -58,10 +58,30 @@ spec:
          # VeleroBackupFailed alert fires on the first failure regardless — so
          # nothing is hidden, only the escalation to critical is deferred
          # until a second cycle confirms it.
+          #
+          # The `and on(schedule) velero_backup_last_status` gate exists because
+          # velero_backup_last_successful_timestamp is an in-process gauge that
+          # velero does NOT retract when a Schedule is deleted: the series just
+          # stops being updated, ages past 60h and pages critical forever, for a
+          # schedule that no longer exists. That is what decommissioning
+          # netbird-backup-daily did on 2026-09-01.
+          #
+          # velero_backup_last_status is the same shape but IS retracted on
+          # Schedule deletion, so it works as a liveness set for "this schedule
+          # still exists". It hides nothing: velero repopulates it for every
+          # existing Schedule on its first scrape after a restart (verified
+          # across the three pods of the 2026-08-27 rollout), and it is written
+          # on failed runs too (0) as well as successful ones (1) — so a
+          # schedule that exists and is failing, or that exists and has stopped
+          # firing, still carries the series and still alerts.
          expr: |
-            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
-            and
-            velero_backup_last_successful_timestamp{schedule!=""} > 0
+            (
+              time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
+              and
+              velero_backup_last_successful_timestamp{schedule!=""} > 0
+            )
+            and on(schedule)
+            velero_backup_last_status
          for: 10m
          labels:
            severity: critical
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -1,116 +0,0 @@
---
-# A read-only view of paperless's backup store, used as a recovery source and
-# nothing else.
-#
-# It points at the production prefix, which is the whole point — the eval
-# instance is a copy of the real archive's metadata — and that makes what is
-# ABSENT here the safety-critical part:
-#
-#   no retentionPolicy   The production store has "30d". Without one here
-#                        there is no policy to enforce, and enforcement is not
-#                        tied to taking a backup: the plugin runs it from the
-#                        instance sidecar on a timer
-#                        (instanceSidecarConfiguration.retentionPolicyIntervalSeconds,
-#                        default 1800). So "this cluster never backs up" would
-#                        not have been enough on its own.
-#
-# A second property makes it belt-and-braces. Retention runs
-# barman-cloud-backup-delete, which takes a server name and works inside
-# <destinationPath>/<serverName> — the store's own status keys its recovery
-# window by "pprl-pg", confirming that layout. This cluster is pprleval-pg, so
-# even an accidental retention policy would address a prefix that holds nothing
-# rather than production's backups.
-#
-# The cluster below completes the guarantee with isWALArchiver: false, so
-# nothing is written back either. It does load the plugin — that is required
-# for the restore to work at all — and read-only comes from the flag rather
-# than from leaving the plugin out.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: pprleval-pg-source
-  namespace: paperless-eval
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
---
-# A disposable clone of the paperless database, for measuring how well the
-# local LLM classifies documents without running the experiment against the
-# live archive.
-#
-# Losing this cluster costs a re-restore and nothing else, which is the intent
-# — it is rebuilt whenever a test needs a clean starting point.
-#
-# One instance, not two. Replication buys availability, and an eval instance
-# that is down is simply an eval that waits.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: pprleval-pg
-  namespace: paperless-eval
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  enableSuperuserAccess: true
-  storage:
-    # Delete-reclaim, unlike the Retain default: this volume is a copy that
-    # gets rebuilt, so it should vanish with the cluster rather than leave a
-    # Released PV and an orphaned RBD image behind (issue #262).
-    storageClass: ceph-block-ephemeral
-    size: 10Gi
-  # Declared with isWALArchiver: false, which is what kubernetes/RECOVERY.md
-  # arrived at after an earlier version of that runbook said to omit it: the
-  # instance manager loads this plugin to replay WAL during the restore, and
-  # without it the instance never starts ("Unknown plugin:
-  # barman-cloud.cloudnative-pg.io").
-  #
-  # externalClusters below only says WHERE the backup is; it does not get the
-  # plugin injected into the pod. So "no plugin" is not the way to keep this
-  # read-only — "plugin, not archiving" is, and it is the combination verified
-  # on im-pg on 2026-07-25.
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      enabled: true
-      isWALArchiver: false
-      parameters:
-        barmanObjectName: pprleval-pg-source
-  bootstrap:
-    recovery:
-      source: pprl-pg
-      database: paperless
-      owner: paperless
-  externalClusters:
-    - name: pprl-pg
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: pprleval-pg-source
-          serverName: pprl-pg
-  postgresql:
-    parameters:
-      shared_buffers: "256MB"
-      max_connections: "100"
-      # The production tuning for write reduction is deliberately not copied.
-      # It exists to spare the SSDs under a database that runs for years; this
-      # one is read-only for hours at a time and then deleted.
-  resources:
-    requests:
-      cpu: 200m
-      memory: 512Mi
-    limits:
-      memory: 1Gi
@@ -1,13 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: paperless-eval
-# The instance only. The measurement harness lives in scripts/paperless-eval/
-# and is applied by hand — a Job is immutable, re-running one means deleting and
-# recreating it, and Flux would just put the old one back. Nothing here should
-# converge on a finished measurement.
-resources:
-  - secrets.yaml
-  - cluster.yaml
-  - pvcs.yaml
-  - redis.yaml
-  - paperless.yaml
@@ -1,203 +0,0 @@
---
-# Deliberately much smaller than the production ConfigMap. Everything that
-# exists to serve people is gone: no remote-user auth, no CSRF origins, no
-# barcode or mail consumption, no Tika/Gotenberg. Nothing consumes documents
-# here, and nothing signs in.
-#
-# What remains is the database connection and the AI settings, because those
-# are what the evaluation measures.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: paperless-eval-config
-  namespace: paperless-eval
-data:
-  PAPERLESS_URL: "http://paperless.paperless-eval.svc.cluster.local"
-  PAPERLESS_ALLOWED_HOSTS: "*"
-  PAPERLESS_TIME_ZONE: "Europe/Berlin"
-  PAPERLESS_REDIS: "redis://redis:6379"
-  PAPERLESS_DBENGINE: "postgresql"
-  PAPERLESS_DBHOST: "pprleval-pg-rw"
-  PAPERLESS_DBNAME: "paperless"
-  # Polling off, so a pod that finds an empty consume volume does not spend a
-  # thread looking at it every ten seconds for the length of a run.
-  PAPERLESS_CONSUMER_POLLING_INTERVAL: "0"
-
-  # The AI settings under test. The endpoints are placeholders that each eval
-  # job overrides for the model it is measuring — a job runs its own
-  # llama-server beside paperless in the same pod, so the value that matters at
-  # run time is the one in the job, not the one here.
-  #
-  # Same caveat as production, and sharper here: paperless reads these from an
-  # ApplicationConfiguration row and falls back to the environment only when
-  # that row is empty. The row comes across in the restored database, so
-  # anything configured in the production admin UI arrives with it and silently
-  # wins over this ConfigMap — an eval would then measure production's settings
-  # while appearing to measure the job's.
-  #
-  # Check it before trusting any result, and after every fresh restore:
-  #
-  #   select llm_backend, llm_model, llm_embedding_backend, llm_embedding_model
-  #     from paperless_applicationconfiguration;
-  #
-  # Every column must be null for the environment to apply.
-  PAPERLESS_AI_ENABLED: "true"
-  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://127.0.0.1:8080/v1"
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "900"
-  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
-  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
-  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
---
-# A base instance that proves the restored database is usable and that the
-# search index and classifier build against it. The evaluation itself does not
-# run here — each job brings its own paperless container so it can point at its
-# own model — but this is what makes a broken clone visible before a job spends
-# hours on it.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: paperless
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless
-  annotations:
-    configmap.reloader.stakater.com/reload: paperless-eval-config
-    # Both, like production: the secret refreshes on the same 10m cycle, and
-    # without this the pod would keep a rotated key until something else
-    # restarted it — which defeats the point of matching the interval.
-    secret.reloader.stakater.com/reload: paperless-eval-secrets
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paperless
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless
-    spec:
-      enableServiceLinks: false
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: paperless
-          # No $imagepolicy marker, unlike apps/paperless: a measurement is
-          # only comparable against a fixed version, so this tag moves when
-          # somebody decides to re-measure, not when a release lands.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          ports:
-            - name: http
-              containerPort: 8000
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          # 60 minutes, against production's 10, and matching this app's Flux
-          # Kustomization timeout.
-          #
-          # Production's budget was sized for a from-scratch boot on this same
-          # corpus and held, so 10 would probably do — but there the volume is
-          # Retain and the index survives, making that a one-off. Here the
-          # volume is ephemeral and the index is rebuilt on every re-test, so
-          # the expensive path is the normal one. A killed pod starts the work
-          # over rather than resuming, which turns a slow boot into a
-          # crash-loop that stalls Flux behind wait: true.
-          #
-          # The LLM index is not in this budget: it is a celery task, and boot
-          # only runs init-llmindex-migrate, a migration check.
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            timeoutSeconds: 5
-            failureThreshold: 180
-          livenessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 120
-            periodSeconds: 30
-            timeoutSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-          # Production's context verbatim, and it has to be: the paperless-ngx
-          # entrypoint starts as root, chowns the mounted data/media/consume
-          # volumes and then drops to uid 1000 via setgid/setuid. Without these
-          # the same image fails that sequence with EPERM.
-          #
-          # It bites harder here than in production, where those volumes were
-          # chowned once years ago and keep their ownership. These are fresh on
-          # every rebuild, so the chown is not a no-op.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-        - name: media
-          persistentVolumeClaim:
-            claimName: paperless-eval-media
-        - name: consume
-          persistentVolumeClaim:
-            claimName: paperless-eval-consume
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: paperless
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: paperless
-  ports:
-    - port: 80
-      targetPort: http
@@ -1,129 +0,0 @@
---
-# All four use ceph-block-ephemeral rather than the Retain default. Everything
-# here is derived — a restored database, an index built from it, a cache —  so
-# the volumes should go when the namespace does instead of leaving Released PVs
-# and orphaned RBD images to reclaim by hand (issue #262).
-#
-# /data holds the Tantivy search index and the classifier pickle, both rebuilt
-# from the database, and the LLM index — which is the expensive one at roughly
-# three hours for the full corpus.
-#
-# Measured on production for the same ~3900 documents: 36 MiB search index,
-# 125 MiB LLM index. 10Gi is far more than that needs, and deliberately so —
-# an embedding model with wider vectors than bge-m3's 1024 dimensions is
-# exactly what phase two varies, and running out of space mid-run would cost
-# the whole rebuild.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  # Named for its role rather than its mount point: the smoke-test deployment
-  # builds /data here, and every eval job copies it out. Replaces the earlier
-  # RWO paperless-eval-data, whose access mode could not be widened in place —
-  # a bound claim's spec is immutable.
-  name: paperless-eval-golden
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  # RWX, because the eval jobs read it too — each copies it into its own
-  # emptyDir rather than mounting it live. The vector store is SQLite, and
-  # SQLite locking across pods over CephFS is not something to rely on.
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem-ephemeral
-  resources:
-    requests:
-      storage: 10Gi
---
-# Empty, and stays that way. Paperless refuses to start without the path, but
-# the AI path reads title, content and filename out of the database and never
-# opens a file — so the documents themselves are not copied here. That is what
-# keeps this instance cheap and keeps a second copy of the archive off the
-# cluster.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-media
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
---
-# Also empty: nothing is consumed here, but the path has to exist.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-consume
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
---
-# The eval jobs' model cache, RWX so several jobs can read the same weights at
-# once — that is the whole point of measuring two candidates in parallel.
-#
-# Separate from the ai namespace's llama-server-models despite holding some of
-# the same files: PVCs do not cross namespaces. A model therefore downloads
-# once here and is then shared by every job, which is the cost a new candidate
-# carries anyway.
-#
-# NOT ephemeral, unlike everything else in this namespace: re-downloading tens
-# of gigabytes on a slow uplink is a worse outcome than an orphaned volume.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: llama-models
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 120Gi
---
-# Results outlive the jobs that produce them: a Job is deleted and recreated to
-# re-run a configuration, and comparing configurations is the entire exercise.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-results
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 1Gi
---
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-eval-redis
-  namespace: paperless-eval
-  labels:
-    velero.io/exclude-from-backup: "true"
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block-ephemeral
-  resources:
-    requests:
-      storage: 1Gi
@@ -1,81 +0,0 @@
---
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: redis
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 999
-        fsGroup: 999
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: redis
-          # No $imagepolicy marker; see the note on the paperless image.
-          image: redis:8.10.1-alpine
-          # No --save: this holds the celery queue and the suggestion cache,
-          # both of which are regenerated. Persisting them across a restart of
-          # a throwaway instance would only preserve stale suggestions.
-          args:
-            - redis-server
-            - --save
-            - ""
-            - --loglevel
-            - warning
-          ports:
-            - name: redis
-              containerPort: 6379
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          livenessProbe:
-            tcpSocket:
-              port: redis
-            initialDelaySeconds: 10
-          readinessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: paperless-eval-redis
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: paperless-eval
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: redis
@@ -1,45 +0,0 @@
---
-# The same Hetzner S3 key every CNPG-using namespace pulls, needed here only so
-# the recovery source can be read. See apps/paperless for why one shared key
-# rather than per-app paths.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: paperless-eval
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: infrastructure/hetzner-s3
---
-# Django's secret key, shared with the production instance rather than given
-# its own vault entry.
-#
-# It signs sessions and CSRF tokens, and this instance has no ingress and no
-# users — nothing signs in, so nothing is at stake in sharing it. A separate
-# entry would mean a vault write before the namespace can start, for a
-# throwaway instance that gets rebuilt.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: paperless-eval-secrets
-  namespace: paperless-eval
-spec:
-  # 10m, matching apps/paperless rather than the 1h default: the point of
-  # sharing the key is that the two hold the same value, and a slower refresh
-  # here would let them diverge for up to an hour after a rotation.
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: paperless-eval-secrets
-  dataFrom:
-    - extract:
-        key: apps/paperless
@@ -170,7 +170,7 @@ spec:
              drop: ["ALL"]
      containers:
        - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.3 # {"$imagepolicy": "flux-system:paperless-ngx"}
          ports:
            - name: http
              containerPort: 8000
@@ -325,7 +325,6 @@ spec:
    - port: 80
      targetPort: http
 ---
---
 # ExternalName mirror so the paperless-namespace Ingress can backend to
 # the gate outpost in the authentik namespace (cross-namespace Service
 # refs aren't supported in standard Ingress).
@@ -0,0 +1,36 @@
+# trmnl
+
+Terminus (self-hosted TRMNL BYOS server) plus the headless renderer that
+screenshots the Home Assistant entrance dashboard and pushes it in. One
+device: a TRMNL X at the entrance, 1872x1404, 16 greys.
+
+## Device firmware
+
+Terminus mirrors one model-agnostic binary and it is the **ESP32-C3** (OG)
+build. This panel is a TRMNL X, i.e. ESP32-S3, so that file is not the
+artefact for a manual flash here. Use
+`https://trmnl-fw.s3.us-east-2.amazonaws.com/trmnl_x/flash/FW<version>.bin`
+— a merged image at offset `0x0` — and run `esptool image-info` on
+anything before writing it.
+
+The panel runs 1.8.16, flashed over USB on 2026-09-05. It shipped on
+1.8.6, where turning on `firmware_update` froze the screen for as long as
+an update was offered (fixed upstream in 1.8.7, commit `9b7a0f7`).
+
+## Image automation
+
+Both images are enrolled (`infrastructure/image-automation/terminus.yaml`,
+`trmnl-renderer.yaml`) and track the latest `0.x`. A bad bump on either
+surfaces as a **blank or corrupted panel**, not as a failed rollout — no
+readiness probe can tell that a rendered screen became unreadable. Look at
+the panel after a bump lands; revert the tag if it is wrong.
+
+## Device endpoint
+
+`trmnl-device.renner.uno` is the device-facing name, TLS via Traefik
+(`device-ingress.yaml`). `API_URI` must carry the same scheme the device
+polls with, because Terminus bakes it into the absolute `image_url` it
+returns. The LoadBalancer in `device-service.yaml` keeps `10.2.99.89`
+without a hostname as a cleartext fallback that depends on neither
+Traefik nor certificate renewal — the captive portal accepts a bare
+address.
@@ -0,0 +1,52 @@
+# Device-facing endpoint, over TLS.
+#
+# This replaces the reason the cleartext LoadBalancer in device-service.yaml
+# was given a hostname at all. That file claimed the firmware "would reject
+# the internal CA" — but nothing here is signed by an internal CA:
+# letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
+# so trmnl.renner.uno already serves a public Let's Encrypt certificate
+# (issuer CN=YR2) that any client with a normal CA bundle trusts. A
+# LAN-only name gets the same treatment — DNS-01 never needs the host to be
+# publicly reachable — so the device can speak HTTPS like everything else.
+#
+# The hostname moves here from the Service's external-dns annotation rather
+# than being duplicated: two sources claiming trmnl-device.renner.uno would
+# have external-dns flapping the record between the Traefik VIP and the
+# device LB. The LoadBalancer keeps its address and stays reachable at
+# http://10.2.99.89 as a nameless fallback if TLS ever turns out to be the
+# thing that breaks.
+#
+# CONSEQUENCE, and the reason API_URI in terminus.yaml changes with this:
+# Traefik's `web` entrypoint carries a global http -> websecure redirect
+# (helmrelease.yaml: `to: websecure, scheme: https, permanent: true`), so
+# once this name points at Traefik, http://trmnl-device.renner.uno answers
+# 308 rather than serving. Whether the firmware follows that is not
+# documented, so the device is provisioned with the https URL and Terminus
+# hands back https image_urls to match. No mixed-scheme fetch.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: terminus-device
+  namespace: trmnl
+  # No external-dns hostname annotation: external-dns-unifi runs with
+  # --source=ingress, so it registers the hosts in spec.rules on its own.
+  # Same as the terminus Ingress next door.
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - trmnl-device.renner.uno
+      secretName: trmnl-device-tls
+  rules:
+    - host: trmnl-device.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: terminus
+                port:
+                  number: 80
@@ -1,15 +1,24 @@
-# Device-facing endpoint. Cleartext HTTP, deliberately.
+# Cleartext fallback for the device, reachable by address only.
 #
-# The TRMNL firmware initiates every request and cannot be handed the
-# internal CA, so terminating TLS here would simply break polling. Scope of
-# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
-# record), it carries no admin session — the device authenticates with its
-# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
-# UI stays on the TLS route at trmnl.renner.uno.
+# This used to be THE device endpoint, on the premise that the firmware
+# "cannot be handed the internal CA". The premise was wrong: nothing here
+# hands out an internal CA for these names — letsencrypt-cluster solves
+# DNS-01 against the renner.uno Cloudflare zone and issues publicly-trusted
+# certificates for LAN-only hosts, which is why trmnl.renner.uno has always
+# served an issuer CN=YR2 certificate. The device speaks HTTPS via
+# device-ingress.yaml, which now owns the trmnl-device.renner.uno name.
+#
+# Kept, without a hostname, because it is the one path to Terminus that
+# depends on neither Traefik nor certificate renewal: http://10.2.99.89
+# still serves /api/setup and /api/display. If TLS is ever what breaks the
+# panel, the recovery is to type that address into the captive portal.
+#
+# Scope of the exposure is unchanged: LAN-only (no public DNS label, no
+# Cloudflare record), no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key.
 #
 # Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
-# pool rather than a TLS-less Ingress, which would be a new pattern for this
-# repo. The device is provisioned with the HOSTNAME, not the address.
+# pool.
 #
 # The address is requested explicitly, matching every other LB service here.
 # Left to allocate freely, Cilium IPAM handed out the first free address in
@@ -30,8 +39,10 @@ metadata:
  labels:
    app.kubernetes.io/name: terminus
    lb-pool: bgp
+  # No external-dns hostname: device-ingress.yaml claims that name. Two
+  # sources claiming trmnl-device.renner.uno would flap the record between
+  # the Traefik VIP and this address (external-dns runs --policy=sync).
  annotations:
-    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
    lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
 spec:
  type: LoadBalancer
@@ -6,6 +6,9 @@ resources:
  - valkey.yaml
  - terminus.yaml
  - ingress.yaml
+  # TLS device endpoint (owns trmnl-device.renner.uno) plus the nameless
+  # cleartext LoadBalancer it falls back to.
+  - device-ingress.yaml
  - device-service.yaml
  # Headless HA screenshotter. Lives in this namespace because Terminus
  # FETCHES rendered screens back from it (uri delivery mode), so the
@@ -227,16 +227,12 @@ spec:
            capabilities:
              drop: ["ALL"]
      containers:
-        # Pinned, not enrolled in image automation — same reasoning as
-        # Terminus itself. This drives a wall-mounted panel and upstream
-        # is pre-1.0; an unattended bump that changes the dithering
-        # pipeline or the webhook payload shows up as a corrupted
-        # screen, not as a failed rollout.
-        #
-        # Note the image is ARCH-SUFFIXED: upstream publishes no
-        # multi-arch manifest, only trmnl-ha-amd64 and trmnl-ha-aarch64.
+        # Enrolled in image automation (flux-system:trmnl-renderer). A bad
+        # bump shows up as a corrupted screen, not a failed rollout — check
+        # the panel after one lands. ARCH-SUFFIXED image: upstream publishes
+        # no multi-arch manifest.
        - name: renderer
-          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.9.5
+          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.10.3 # {"$imagepolicy": "flux-system:trmnl-renderer"}
          ports:
            - name: http
              containerPort: 10000
@@ -7,11 +7,17 @@ data:
  TZ: "Europe/Berlin"
  HANAMI_PORT: "2345"
  # What the device is told to fetch from. Terminus bakes this into the
-  # absolute `image_url` it hands back on /api/display, so it MUST be the
-  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
-  # HTTP and would reject the internal CA. No trailing slash: the same
-  # string is typed into the captive portal at provisioning time.
-  API_URI: "http://trmnl-device.renner.uno"
+  # absolute `image_url` it hands back on /api/display, so it has to be a
+  # name the DEVICE can resolve and fetch — and it has to carry the same
+  # scheme the device polls with, or every screen fetch is a second,
+  # cleartext connection. No trailing slash: the same string is typed into
+  # the captive portal at provisioning time.
+  #
+  # https since device-ingress.yaml took over trmnl-device.renner.uno. The
+  # earlier value was http on the theory that the firmware would reject an
+  # internal CA; there is no internal CA here — letsencrypt-cluster issues
+  # publicly-trusted certs for LAN-only names via DNS-01.
+  API_URI: "https://trmnl-device.renner.uno"
  KEYVALUE_URL: "redis://terminus-valkey:6379"
  # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
  # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB
@@ -128,7 +134,7 @@ spec:
        # is a device-bricking class of risk. Bumps are a reviewed, manual
        # act — see kubernetes/infrastructure/image-automation/README.md.
        - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.68.0
+          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
          ports:
            - name: http
              containerPort: 2345
@@ -191,7 +197,7 @@ spec:
            capabilities:
              drop: ["ALL"]
        - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.68.0
+          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
          command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
          envFrom:
            - configMapRef:
@@ -159,8 +159,16 @@ if [[ "$PHASE" == "vmtest" ]]; then

    # sun_path caps a unix socket path at 108 bytes and the provisioner puts a
    # qemu monitor socket under the state directory, so this has to stay short.
-    # A path under the repo or a long mktemp dir silently fails to boot.
-    VM_STATE=/tmp/bootstrap-vmtest
+    # A path under the repo or a long mktemp dir silently fails to boot. The
+    # caller may override it to aim the qcow2 images at a sized volume; the
+    # guard turns "too long" from a silent boot failure into an error.
+    VM_STATE="${VM_STATE:-/tmp/bootstrap-vmtest}"
+    [[ ${#VM_STATE} -le 50 ]] \
+        || fail "VM_STATE '$VM_STATE' is ${#VM_STATE} bytes; keep it under 50 so the qemu monitor socket fits in sun_path's 108"
+    # cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`, so refuse anything that is
+    # not a nested absolute path.
+    [[ $VM_STATE == /*/?* ]] \
+        || fail "VM_STATE '$VM_STATE' must be an absolute path at least two levels deep"
    VM_CLUSTER=bootstrap-vmtest
    VM_DIR=$(mktemp -d)
    TALOSCTL_BIN=$(command -v talosctl)
@@ -1,34 +0,0 @@
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: paperless-eval
-  namespace: flux-system
-spec:
-  # A throwaway copy of paperless for measuring the local LLM against the real
-  # corpus. Its Cluster restores from paperless's backup store and archives
-  # nothing back; see kubernetes/apps/paperless-eval/cluster.yaml.
-  #
-  # prune: true and no prune-disabled annotations anywhere underneath — the
-  # data here is a copy, and being able to delete the whole namespace by
-  # removing this file is the point.
-  interval: 10m
-  retryInterval: 1m
-  # 30m is not enough: the first start restores the database and then builds
-  # the search index over ~3900 documents before the webserver answers.
-  timeout: 60m
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./kubernetes/apps/paperless-eval
-  prune: true
-  wait: true
-  dependsOn:
-    - name: infrastructure
-    - name: infrastructure-crds
-    - name: infrastructure-ceph-cluster
-    - name: infrastructure-cnpg-barman
-  healthCheckExprs:
-    - apiVersion: postgresql.cnpg.io/v1
-      kind: Cluster
-      failed: "has(status.phase) && status.phase.startsWith('Failed')"
-      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
@@ -937,7 +937,7 @@ subjects:
 # with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
 stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
  name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
            secretKeyRef:
              key: SIDECAR_IMAGE
              name: plugin-barman-cloud-kg5c2795f7
-        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
+        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
        name: barman-cloud
        ports:
        - containerPort: 9090
@@ -96,28 +96,13 @@ The security-scan agent will continue to surface vuln findings on
 these as "Skipped / Flux image-automation owns this" even without
 a policy.

- `ghcr.io/usetrmnl/terminus` — publishes a usable `0.x.0` tag stream, so
-  unlike the three above this is **not** a tag-format problem. It is left
-  unenrolled because the blast radius is hardware: Terminus is pre-1.0,
-  cuts no GitHub Releases and ships no changelog, and it serves the
-  firmware and `/api/display` contract that the TRMNL X panel polls. An
-  unattended bump that changes either can leave a wall-mounted device
-  dark, with no rollback path that doesn't involve the captive portal.
-  Bumps are a reviewed, manual act: read the commit range, then edit the
-  tag in `kubernetes/apps/trmnl/terminus.yaml` (both containers).
-
-  The Valkey sidecar alongside it *is* enrolled (`valkey.yaml`) — it holds
-  only the Sidekiq queue and touches no device-facing contract.
-
- `ghcr.io/usetrmnl/trmnl-ha-amd64` — the TRMNL screenshot renderer, for
-  the same reason and one more. It owns the dithering pipeline and the
-  webhook payload shape, so a bad bump surfaces as a *corrupted or blank
-  panel*, not as a failed rollout — there is no readiness signal that
-  catches "the image is now unreadable". Separately, upstream publishes
-  **no multi-arch manifest**: only the arch-suffixed `trmnl-ha-amd64` and
-  `trmnl-ha-aarch64` repositories exist, so an ImagePolicy here would also
-  be pinning us to one architecture in a way the tag does not make
-  obvious. Bump manually in `kubernetes/apps/trmnl/renderer.yaml`.
+Only a tag-format problem belongs on that list. The policy is to run the
+latest version and let image-automation hold us there wherever the tags
+allow it, so an image with a usable semver stream gets enrolled even when
+a bump carries risk. Where a bad bump is hard to detect, the answer is a
+note about what to check afterwards — see `apps/trmnl/README.md` for the
+panel, whose failure mode is a blank screen rather than a failed rollout
+— not an exclusion.

 ## Re-import hazard: cnpg-barman-plugin

@@ -18,7 +18,9 @@ resources:
  - mosquitto.yaml
  - redis.yaml
  - shelly-manager.yaml
+  - terminus.yaml
  - tika.yaml
+  - trmnl-renderer.yaml
  - valkey.yaml
  - velero-plugin-for-aws.yaml
  - vsftpd.yaml
@@ -0,0 +1,30 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: terminus
+  namespace: flux-system
+spec:
+  image: ghcr.io/usetrmnl/terminus
+  interval: 1h
+---
+# Pre-1.0, so the range is 0.x rather than a major pin: upstream's minors are
+# its release stream, and holding at a single minor is how this drifted four
+# releases behind while it was excluded from automation.
+#
+# A bad bump here surfaces as a blank or corrupted panel rather than a failed
+# rollout — no readiness probe can tell that a rendered screen became
+# unreadable. Check the panel after one lands; revert the tag if it is wrong.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: terminus
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: terminus
+  filterTags:
+    pattern: '^(?P<ver>\d+\.\d+\.\d+)$'
+    extract: '$ver'
+  policy:
+    semver:
+      range: "0.x"
@@ -0,0 +1,29 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: trmnl-renderer
+  namespace: flux-system
+spec:
+  image: ghcr.io/usetrmnl/trmnl-ha-amd64
+  interval: 1h
+---
+# Upstream ships no multi-arch manifest, only the arch-suffixed repositories.
+# That does not obstruct a policy: the image reference already names the
+# architecture, so this pins nothing the tag did not.
+#
+# The floating `0.9` / `0.10` series tags are filtered out; only fully
+# qualified patch versions are candidates. Same panel caveat as terminus.yaml.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: trmnl-renderer
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: trmnl-renderer
+  filterTags:
+    pattern: '^(?P<ver>\d+\.\d+\.\d+)$'
+    extract: '$ver'
+  policy:
+    semver:
+      range: "0.x"
@@ -73,11 +73,6 @@ metadata:
 # classifies documents without experimenting on the live archive.
 apiVersion: v1
 kind: Namespace
-metadata:
-  name: paperless-eval
---
-apiVersion: v1
-kind: Namespace
 metadata:
  name: audiobookshelf
 ---
@@ -34,22 +34,6 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # The paperless-eval namespace reaches the embedding endpoint for the same
-    # reason paperless does; its chat model runs inside each eval job's own pod
-    # and needs no grant here.
-    #
-    # Scoped to the namespace rather than to an app label, unlike the grants
-    # around it: that namespace exists only to run evaluations, and its
-    # workload is jobs created per measurement whose labels vary with what is
-    # being measured. A label match would be a trap that presents as an
-    # embedding call silently dropped.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
    # Home Assistant's llama.cpp conversation agent calls the chat model only —
    # it has no use for the embedding endpoint, so unlike paperless above this
    # grant does not extend to llama-embed.
@@ -6,7 +6,6 @@ resources:
  - ingress-isolation.yaml
  - authentik-ns.yaml
  - paperless-ns.yaml
-  - paperless-eval-ns.yaml
  - audiobookshelf-ns.yaml
  - home-assistant-ns.yaml
  - spegel-ns.yaml
@@ -1,141 +0,0 @@
-# Narrower than paperless-ns.yaml, because this namespace serves nobody.
-#
-# There is no ingress, no gate outpost and no hostname: the only things that
-# talk to it are its own pods. Monitoring is admitted so a run can be watched,
-# and host/remote-node for the kubelet probes.
-#
-# Egress reaches the ai namespace for the embedding endpoint, which is the one
-# dependency an eval job has outside its own pod, and world:443 so the CNPG
-# recovery can read the backup store on Hetzner.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: default-deny-ingress
-  namespace: paperless-eval
-spec:
-  endpointSelector: {}
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": monitoring
-    - fromEntities:
-        - host
-        - remote-node
-    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
-    # home-assistant-ns.yaml for the rationale.
-    - fromEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
-  egress:
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless-eval
-    # The embedding endpoint. The chat model runs inside each job's own pod, so
-    # it needs no grant; the embedder is shared with production because it is
-    # read-only and cheap, and swapping it is a phase-two concern.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-embed
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": rook-ceph
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": kube-system
-            k8s-app: kube-dns
-      toPorts:
-        - ports:
-            - port: "53"
-              protocol: UDP
-    - toEntities:
-        - host
-        - remote-node
-    # world:443 — the CNPG recovery reads the backup store on Hetzner Object
-    # Storage, and image pulls.
-    - toEntities:
-        - world
-      toPorts:
-        - ports:
-            - port: "443"
-              protocol: TCP
-    # ICMP — egress mirror; see ingress block above.
-    - toEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - { type: 0,   family: IPv4 }
-            - { type: 3,   family: IPv4 }
-            - { type: 8,   family: IPv4 }
-            - { type: 11,  family: IPv4 }
-            - { type: 12,  family: IPv4 }
-            - { type: 1,   family: IPv6 }
-            - { type: 2,   family: IPv6 }
-            - { type: 3,   family: IPv6 }
-            - { type: 4,   family: IPv6 }
-            - { type: 128, family: IPv6 }
-            - { type: 129, family: IPv6 }
-            - { type: 133, family: IPv6 }
-            - { type: 134, family: IPv6 }
-            - { type: 135, family: IPv6 }
-            - { type: 136, family: IPv6 }
-            - { type: 143, family: IPv6 }
---
-# The CNPG operator polls each instance's status endpoint over HTTPS
-# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
-# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
-# while Postgres serves normally — issue #433, where the operator's dial timed
-# out because the apps baseline never allowed the database namespace.
-#
-# It matters here even for a throwaway: the Flux Kustomization waits on
-# Cluster status, and a smoke test that cannot report its own health is not one.
-#
-# Scoped to the pprleval-pg endpoints and that one port, like the equivalent
-# rules for wp-pg and ha-pg: the barman and mariadb operators sharing the
-# database namespace have no reason to reach anything here.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: pprleval-pg-allow-cnpg-operator
-  namespace: paperless-eval
-spec:
-  endpointSelector:
-    matchLabels:
-      cnpg.io/cluster: pprleval-pg
-  ingress:
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": database
-            app.kubernetes.io/name: cloudnative-pg
-      toPorts:
-        - ports:
-            - port: "8000"
-              protocol: TCP
@@ -27,7 +27,6 @@ spec:
        - ingress
        - monitoring
        - paperless
-        - paperless-eval # same apps/paperless key plus the shared barman S3 creds
        - trmnl
        - velero
        - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)
@@ -11,6 +11,12 @@ machine:
    # kubelet picks the first non-loopback IP, which could land on the v30iot
    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
    # must stay on mgmt LAN.
+    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
+    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
+    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
+    # provider prefix, which it then probes until the agent restarts (#789).
+    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
    nodeIP:
      validSubnets:
        - 10.2.40.0/24
+        - fd40::/64
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+# Retrieval evaluation for an embedding model, on German documents of the kind
+# this homelab actually files.
+#
+# Companion to scripts/ai-eval.sh, but it cannot work the same way. A chat
+# answer can be read and judged; an embedding is 1024 opaque numbers. What is
+# measurable is whether the right document comes back first, so this builds a
+# small labelled corpus, embeds it, and scores real questions against it.
+#
+# Metrics:
+#   Recall@1  the correct document ranked first — what Paperless's RAG actually
+#             feeds to the classifier
+#   Recall@3  it at least made the shortlist
+#   MRR       mean reciprocal rank, so near-misses are visible rather than
+#             collapsing to pass/fail
+#   margin    cosine gap between the correct document and the best wrong one.
+#             A model that is right by 0.002 is guessing; one that is right by
+#             0.15 has actually separated them.
+#
+# The corpus deliberately contains confusable pairs — two Allianz policies, four
+# documents that are all "Rechnungen", a car insurance and a vehicle
+# registration sharing a number plate. An easy corpus would rate everything 100%
+# and tell you nothing.
+#
+# Usage:
+#   scripts/embed-eval.sh                                    # cluster default
+#   scripts/embed-eval.sh -m some-model
+#   scripts/embed-eval.sh -e https://host/v1/embeddings -m model -k KEY
+#
+# Env equivalents: EMBED_EVAL_ENDPOINT, EMBED_EVAL_MODEL, EMBED_EVAL_KEY.
+#
+# To compare two models on our llama-embed, -m is not enough: swap --hf-repo in
+# kubernetes/apps/llama-server/embed.yaml, let it roll out, then re-run. Note
+# that changing the model also invalidates any vectors Paperless has stored, so
+# comparisons are best made before an index exists.
+#
+# Exit code is the number of queries whose correct document did not rank first.
+set -uo pipefail
+
+ENDPOINT="${EMBED_EVAL_ENDPOINT:-http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings}"
+MODEL="${EMBED_EVAL_MODEL:-qwen3-embedding-0.6b}"
+APIKEY="${EMBED_EVAL_KEY:-}"
+# Run curl from inside this pod when the endpoint is a cluster-internal name.
+# llama-embed has no ingress by design, so there is nothing to reach from a
+# workstation.
+VIA_POD="${EMBED_EVAL_VIA_POD:-}"
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -e|--endpoint) ENDPOINT="$2"; shift 2 ;;
+        -m|--model)    MODEL="$2";    shift 2 ;;
+        -k|--key)      APIKEY="$2";   shift 2 ;;
+        -p|--via-pod)  VIA_POD="$2";  shift 2 ;;
+        -h|--help)     sed -n '2,34p' "$0"; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+CASES="$HERE/embed-eval"
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+for f in "$CASES/corpus.jsonl" "$CASES/queries.jsonl"; do
+    [ -r "$f" ] || { echo "missing fixture: $f" >&2; exit 2; }
+done
+
+# Default to reaching the Service from a pod that is allowed to, unless the
+# caller pointed us somewhere externally reachable.
+if [ -z "$VIA_POD" ] && [ "${ENDPOINT#*svc.cluster.local}" != "$ENDPOINT" ]; then
+    VIA_POD="deploy/llama-server -n ai"
+fi
+
+# embed <text> -> the vector as a JSON array on stdout
+embed() {
+    local payload out
+    payload=$(jq -nc --arg m "$MODEL" --arg t "$1" '{model:$m,input:$t}')
+    if [ -n "$VIA_POD" ]; then
+        # stderr is kept and surfaced on failure rather than discarded: a
+        # silently swallowed kubectl or curl error is indistinguishable from a
+        # model that returned nothing.
+        # shellcheck disable=SC2086
+        out=$(kubectl exec $VIA_POD -- curl -sS -m 120 \
+              -H 'Content-Type: application/json' \
+              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
+              -d "$payload" "$ENDPOINT" 2>"$WORK/err.txt")
+    else
+        out=$(curl -sS -m 120 -H 'Content-Type: application/json' \
+              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
+              -d "$payload" "$ENDPOINT")
+    fi
+    printf '%s' "$out" | jq -c '.data[0].embedding // empty'
+}
+
+echo "endpoint: $ENDPOINT"
+echo "model:    $MODEL"
+[ -n "$VIA_POD" ] && echo "via pod:  $VIA_POD"
+echo ""
+
+# ---- embed the corpus ------------------------------------------------------
+DOCS=$(wc -l < "$CASES/corpus.jsonl" | tr -d ' ')
+echo "embedding $DOCS documents..."
+: > "$WORK/vecs.jsonl"
+start=$(date +%s)
+while IFS= read -r line; do
+    id=$(printf '%s' "$line" | jq -r '.id')
+    text=$(printf '%s' "$line" | jq -r '.text')
+    vec=$(embed "$text")
+    if [ -z "$vec" ]; then
+        echo "FAILED to embed '$id' — endpoint returned no vector." >&2
+        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
+        exit 2
+    fi
+    jq -nc --arg id "$id" --argjson v "$vec" '{id:$id, vec:$v}' >> "$WORK/vecs.jsonl"
+done < "$CASES/corpus.jsonl"
+elapsed=$(( $(date +%s) - start ))
+DIM=$(head -1 "$WORK/vecs.jsonl" | jq '.vec | length')
+echo "  $DOCS documents, ${DIM} dimensions, ${elapsed}s"
+echo ""
+
+# ---- score the queries -----------------------------------------------------
+# Cosine similarity in jq. The vectors llama.cpp returns are normalised, but the
+# full form costs nothing and keeps this correct against servers that do not.
+rank() { # queryvec -> "id score" lines, best first
+    jq -s -r --argjson q "$1" '
+        def dot($a; $b): reduce range(0; $a|length) as $i (0; . + ($a[$i] * $b[$i]));
+        def norm($a): (reduce ($a[]) as $x (0; . + ($x * $x))) | sqrt;
+        map({id: .id, s: (dot(.vec; $q) / ((norm(.vec) * norm($q)) + 1e-12))})
+        | sort_by(-.s)[]
+        | "\(.id) \(.s)"
+    ' "$WORK/vecs.jsonl"
+}
+
+MISS=0; TOTAL=0; HIT1=0; HIT3=0; MRRSUM=0
+printf '%-46s %-22s %5s %8s\n' QUERY EXPECTED RANK MARGIN
+printf '%s\n' "------------------------------------------------------------------------------------"
+while IFS= read -r line; do
+    q=$(printf '%s' "$line" | jq -r '.q')
+    want=$(printf '%s' "$line" | jq -r '.want')
+    qvec=$(embed "$q")
+    if [ -z "$qvec" ]; then
+        echo "FAILED to embed query: $q" >&2
+        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
+        exit 2
+    fi
+    rank "$qvec" > "$WORK/ranked.txt"
+
+    pos=$(grep -n "^$want " "$WORK/ranked.txt" | cut -d: -f1)
+    pos=${pos:-999}
+    TOTAL=$((TOTAL + 1))
+    [ "$pos" -eq 1 ] && HIT1=$((HIT1 + 1)) || MISS=$((MISS + 1))
+    [ "$pos" -le 3 ] && HIT3=$((HIT3 + 1))
+    MRRSUM=$(awk -v s="$MRRSUM" -v p="$pos" 'BEGIN{printf "%.6f", s + 1/p}')
+
+    # Margin: the correct document's score minus the best incorrect one. On a
+    # miss this is negative, which says how badly it lost.
+    want_s=$(awk -v w="$want" '$1==w{print $2; exit}' "$WORK/ranked.txt")
+    best_other=$(awk -v w="$want" '$1!=w{print $2; exit}' "$WORK/ranked.txt")
+    margin=$(awk -v a="${want_s:-0}" -v b="${best_other:-0}" 'BEGIN{printf "%+.4f", a-b}')
+
+    if [ "$pos" -eq 1 ]; then mark=" "; else mark="!"; fi
+    printf '%s%-45.45s %-22s %5s %8s\n' "$mark" "$q" "$want" "$pos" "$margin"
+    if [ "$pos" -ne 1 ]; then
+        printf '   got instead: %s\n' "$(head -1 "$WORK/ranked.txt" | cut -d' ' -f1)"
+    fi
+done < "$CASES/queries.jsonl"
+
+R1=$(awk -v h="$HIT1" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
+R3=$(awk -v h="$HIT3" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
+MRR=$(awk -v s="$MRRSUM" -v t="$TOTAL" 'BEGIN{printf "%.3f", s/t}')
+
+echo ""
+echo "========================================"
+printf '  RESULT model=%s dim=%s recall@1=%s%% recall@3=%s%% mrr=%s\n' \
+    "$MODEL" "$DIM" "$R1" "$R3" "$MRR"
+echo "========================================"
+exit "$MISS"
@@ -0,0 +1,12 @@
+{"id":"stadtwerke-strom","text":"Stadtwerke Muenchen GmbH. Jahresrechnung Strom. Rechnungsnummer 4711. Abrechnungszeitraum 01.01.2025 bis 31.12.2025. Zaehlernummer 1EMH0012345678. Verbrauch 2450 kWh zum Arbeitspreis von 31,2 Cent je kWh. Grundpreis 9,90 EUR monatlich. Zu zahlender Betrag 812,40 EUR faellig am 15.02.2026."}
+{"id":"telekom-mobilfunk","text":"Telekom Deutschland GmbH. Ihre Mobilfunkrechnung fuer Januar. Kundennummer 0891234567. Rufnummer 0170 1234567. Tarif MagentaMobil M mit 20 GB Datenvolumen. Grundpreis 39,95 EUR. Verbindungen ins Ausland 4,20 EUR. Rechnungsbetrag 44,15 EUR wird per Lastschrift eingezogen."}
+{"id":"finanzamt-steuer","text":"Finanzamt Muenchen Abteilung II. Bescheid ueber Einkommensteuer und Solidaritaetszuschlag fuer 2025. Steuernummer 143/256/78901. Zu versteuerndes Einkommen 54.320 EUR. Festgesetzte Einkommensteuer 8.412 EUR. Bereits geleistete Vorauszahlungen 9.100 EUR. Erstattungsbetrag 688 EUR. Rechtsbehelfsbelehrung siehe Seite 3."}
+{"id":"allianz-hausrat","text":"Allianz Versicherungs-AG. Versicherungsschein Hausratversicherung. Policennummer HR-88231. Versicherte Wohnflaeche 96 Quadratmeter. Versicherungssumme 65.000 EUR. Eingeschlossen sind Feuer, Leitungswasser, Sturm und Einbruchdiebstahl. Fahrraddiebstahl bis 1.500 EUR mitversichert. Jahresbeitrag 214 EUR."}
+{"id":"allianz-kfz","text":"Allianz Versicherungs-AG. Beitragsrechnung Kraftfahrtversicherung. Policennummer KFZ-55190. Amtliches Kennzeichen M-AB 1234. Fahrzeug VW Golf VIII. Vollkasko mit 500 EUR Selbstbeteiligung, Teilkasko 150 EUR. Schadenfreiheitsklasse SF 12. Jahresbeitrag 638 EUR."}
+{"id":"mietvertrag","text":"Mietvertrag ueber Wohnraum. Vermieter Immobilien Wagner GbR, Mieter Konstantin Renner. Mietobjekt Dreizimmerwohnung im zweiten Obergeschoss, 78 Quadratmeter, Lindenstrasse 14. Mietbeginn 01.04.2024. Kaltmiete 1.180 EUR monatlich, Betriebskostenvorauszahlung 240 EUR. Kaution drei Nettokaltmieten."}
+{"id":"arbeitsvertrag","text":"Arbeitsvertrag zwischen der Muster Technik GmbH und dem Arbeitnehmer. Taetigkeit als Softwareentwickler ab 01.09.2024. Bruttojahresgehalt 78.000 EUR zahlbar in zwoelf Monatsraten. Woechentliche Arbeitszeit 40 Stunden. Urlaubsanspruch 30 Arbeitstage im Kalenderjahr. Probezeit sechs Monate."}
+{"id":"aok-krankenkasse","text":"AOK Bayern. Beitragsbescheid zur gesetzlichen Kranken- und Pflegeversicherung. Versichertennummer A123456789. Beitragspflichtige Einnahmen 4.500 EUR monatlich. Allgemeiner Beitragssatz 14,6 Prozent zuzueglich Zusatzbeitrag 1,7 Prozent. Pflegeversicherung 3,4 Prozent. Monatlicher Gesamtbeitrag 887,50 EUR."}
+{"id":"sparkasse-kontoauszug","text":"Stadtsparkasse Muenchen. Kontoauszug Nummer 3 zum Girokonto IBAN DE89 7015 0000 0012 3456 78. Alter Kontostand 2.340,12 EUR. Gutschrift Gehalt 4.210,00 EUR. Lastschrift Miete 1.420,00 EUR. Dauerauftrag Sparplan 300,00 EUR. Neuer Kontostand 4.830,12 EUR."}
+{"id":"amazon-bestellung","text":"Amazon EU S.a.r.l. Bestellbestaetigung. Bestellnummer 302-4455667-8899001. Artikel Logitech MX Master 3S Wireless Maus, Menge 1, Preis 89,99 EUR. Versand durch Amazon, Lieferung voraussichtlich am 04.09. Zahlungsart Kreditkarte. Rechnungsbetrag inklusive Mehrwertsteuer 89,99 EUR."}
+{"id":"handwerker-rechnung","text":"Sanitaer Bergmeister GmbH. Rechnung fuer erbrachte Leistungen. Rechnungsnummer 2026-0417. Austausch eines defekten Heizungsthermostats sowie Entlueftung von drei Heizkoerpern. Arbeitszeit 2,5 Stunden zu 68 EUR. Material 42,30 EUR. Nettobetrag 212,30 EUR, Umsatzsteuer 19 Prozent, Gesamtbetrag 252,64 EUR."}
+{"id":"kfz-zulassung","text":"Landeshauptstadt Muenchen Kreisverwaltungsreferat. Zulassungsbescheinigung Teil I. Amtliches Kennzeichen M-AB 1234. Fahrzeugklasse Personenkraftwagen. Hersteller Volkswagen, Handelsbezeichnung Golf. Erstzulassung 14.03.2022. Hubraum 1498 Kubikzentimeter, Leistung 110 kW. Naechste Hauptuntersuchung 03.2028."}
@@ -0,0 +1,12 @@
+{"q":"Wie hoch war mein Stromverbrauch im letzten Jahr?","want":"stadtwerke-strom","note":"paraphrase: asks about consumption, the document says Verbrauch kWh"}
+{"q":"Was kostet mein Handyvertrag monatlich?","want":"telekom-mobilfunk","note":"paraphrase: Handyvertrag vs Mobilfunkrechnung/Tarif"}
+{"q":"Bescheid vom Finanzamt ueber die Einkommensteuer","want":"finanzamt-steuer","note":"near-literal, should be easy"}
+{"q":"Welchen Versicherungsschutz habe ich fuer mein Auto?","want":"allianz-kfz","note":"HARD: two Allianz policies. Must not pick the Hausrat one."}
+{"q":"Ist mein Fahrrad gegen Diebstahl versichert?","want":"allianz-hausrat","note":"HARD: same sender as kfz, and bicycles are a detail deep in the text"}
+{"q":"Wie hoch ist die Kaltmiete meiner Wohnung?","want":"mietvertrag","note":"must beat sparkasse, which also mentions Miete as a Lastschrift"}
+{"q":"Wie viele Urlaubstage stehen mir zu?","want":"arbeitsvertrag","note":"paraphrase: Urlaubsanspruch"}
+{"q":"Was zahle ich monatlich an die Krankenkasse?","want":"aok-krankenkasse","note":"Krankenkasse vs Kranken- und Pflegeversicherung"}
+{"q":"Wie viel Geld ist auf meinem Girokonto?","want":"sparkasse-kontoauszug","note":"Kontostand"}
+{"q":"Rechnung fuer die Reparatur der Heizung","want":"handwerker-rechnung","note":"HARD: several documents are Rechnungen; must pick the Sanitaer one"}
+{"q":"Wann muss mein Auto zum TUEV?","want":"kfz-zulassung","note":"HARD: TUEV never appears; the document says Hauptuntersuchung. Pure semantics, and must not be pulled to the Kfz-Versicherung by the shared plate number."}
+{"q":"Wann wurde meine Maus geliefert?","want":"amazon-bestellung","note":"colloquial vs Bestellbestaetigung/Lieferung"}
@@ -1,8 +1,8 @@
-# Measuring Paperless suggestions against the real corpus
+# Comparing chat models on Paperless suggestions

-Compares the two engines Paperless ships — the scikit-learn classifier and the
-LLM path — over documents whose labels a human already chose, in the
-`paperless-eval` namespace rather than against the live archive.
+Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
+against documents a human already filed, in the `paperless-eval` namespace
+rather than against the live archive.

 **Applied by hand, not by Flux.** A Job is immutable, so re-running a
 configuration means deleting and recreating it, and a Flux-managed Job would
@@ -11,54 +11,156 @@ simply come back. The namespace itself *is* managed

 ## Running one

-```sh
-kubectl -n paperless-eval create configmap paperless-eval-script \
-  --from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
-  | kubectl apply -f -
+Screen first, always:

-kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
-kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
+```sh
+./run-candidate.sh qwen3-8b 5          # ~5 min to a rejection, ~10 to a pass
+kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval
+
+kubectl -n paperless-eval delete job eval-qwen3-8b
+./run-candidate.sh qwen3-8b            # the real measurement, 50 documents
 ```

+Three of the first four candidates could not do the task at all, and each was
+left running for an hour before anyone looked. eval.py now stops after three
+consecutive failures from the first document, so a screening run rejects a
+hopeless candidate in about five minutes.
+
+Screening uses genuine paperless requests. That matters — `probe-toolcall.sh`
+is cheaper still and worth a first pass, but it passed two candidates that then
+failed on document one, because a synthesised request cannot reproduce the RAG
+neighbours and taxonomy candidate lists that make a real prompt what it is.
+
+Never compare numbers across sample sizes. A screen is a gate; the comparison
+happens at 50.
+
+Candidates live in `candidates.tsv`; the runner substitutes the model-specific
+values into `job-template.yaml` and refreshes the `eval.py` ConfigMap. Adding a
+candidate is a line in that file, not a copy of the manifest — the sidecar
+configuration deliberately exists once.
+
+Two run at a time: the pod anti-affinity puts one per node, and only two nodes
+have a free iGPU. A third waits.
+
+Re-running after an edit needs the old Job gone first:
+`kubectl -n paperless-eval delete job eval-<label>`.
+
 Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
 document) and `<label>.summary.json`, and in the pod log.

-Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
-the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
+## What is being compared

-## Adding a candidate
+The candidates against each other. The scikit-learn classifier runs in every
+job as a fixed reference line — it costs 15 seconds and keeps running in
+production whatever the LLM does, so `llm_rescued` against `llm_only_wrong`
+says what *this* model adds on top of it.

-Copy `job-qwen35.yaml`, then change four things: the Job name, the
-`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
-the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
-`eval` container (it names the output files).
-
-Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
-and k8s-2 have a free iGPU. A third waits.
+Do not read the classifier's own score as a quality bar. `train()` fits on every
+non-inbox document with no holdout, so the evaluation documents are in its
+training set and its numbers are memorisation. That does not matter for ranking
+models — it is the same constant under all of them.

 ## Reading the output

-`complement` in the summary is the number the exercise exists for. The two
-engines are not alternatives: the classifier costs 0.6 s per document and keeps
-running whatever the LLM does, so what matters is what the LLM adds where the
-classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
-percentages cannot show that; 74% against 72% could overlap almost entirely or
-barely at all.
-
 `correspondent_offered` against `correspondent_recognised` is the
 `MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
-rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
-stored "Generali AG".
+rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
+"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.

 `type_specific` against `type_catchall` splits on whether the true type is
 "Sonstige", which holds 1133 of ~3900 documents and is the one label with no
-semantic content. The classifier can memorise it; a model reading the document
-has to infer "none of the above".
+semantic content. A model reading the document has to infer "none of the above".
+
+`failed` counts documents where the model produced nothing usable. With
+`--n-predict 768` in place, that is mostly non-termination: paperless reports
+the truncated response as a flat `Invalid AI configuration` and discards the
+content, so what the model was doing is visible only in the llama-server log.
+
+## Results so far
+
+| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
+|---|---|---|---|---|---|---|---|
+| *classifier (reference)* | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 18 |
+| Qwen3.6-35B-A3B Q4_K_M | 72.9 | 77.1 | 85.4 | 94.9 | 44.4 | 2 | 5494 |
+| Ministral-3 3B Q4_K_M | 71.4 | 76.2 | 76.2 | 83.3 | 33.3 | **29** | 5511 |
+| Ministral-3 8B Q4_K_M | — | — | — | — | — | 3/3 | aborted |
+| Granite 3.3 8B Q4_K_M | — | — | — | — | — | all | not measurable |
+
+Read the Ministral-3 3B row with its failure count, not without it. It scored 21
+documents and failed 29 — and on those 21 its correspondent numbers sit within
+the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents
+it could answer at all, which is a set selected for being easy, and the
+classifier does better on them too (11 of 21 both right against 30 of 48).
+A model that declines three documents in five has no usable accuracy number.
+
+Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two
+of those four "broken" are the difflib gap above, not model errors.
+
+### How big a difference means anything
+
+Qwen3.6 was measured twice over the same 50 documents, once with the deprecated
+--chat-template-kwargs and once with --reasoning off:
+
+    70.8 / 75.0 / 83.3 / 94.9 / 33.3   two failures
+    72.9 / 77.1 / 85.4 / 94.9 / 44.4   two failures
+
+Every one of those gaps is a single document out of 48 — the catchall one is
+one document out of nine. So that is the noise floor: **a difference between
+two candidates smaller than about two documents says nothing.** Decoding is
+sampled, and the same model on the same input does not give the same answer
+twice.
+
+Worth knowing before reading a 2-point lead as a result.
+
+## The instrument is checked, not assumed
+
+Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the
+documents, every ten documents, and again if it aborts. If Paris does not come
+back, the run stops and says so.
+
+This exists because a whole day of results turned out to be measuring a broken
+instrument. On this Vulkan stack a prompt past some model-specific length stops
+llama.cpp producing language — it repeats one token, " de de de" or
+" phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop"
+and nothing in the log. The slot then stays broken for every request after it
+until the pod restarts.
+
+So a run against a poisoned slot reports fifty failures that are one event, and
+reads exactly like a model that cannot classify. Six candidates were written up
+that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris."
+perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its
+recorded classification scores described a corrupted slot and nothing else.
+
+**A run whose canary breaks is void, not a result.** Failures adjacent to the
+break are one fault. Use `probe-length.sh` to find where the model's boundary
+actually sits before judging it — paperless needs ~5300 tokens.
+
+## Bounds of the harness, not of the model
+
+Two numbers in here are mine and both have produced misleading results:
+
+`--n-predict` caps generation server-side, default 768. A candidate that stops
+exactly there has hit MY bound. Six candidates were written up as incapable on
+that signal before anyone checked whether they would have finished at 900
+tokens. A working answer takes 241-291 tokens, so 768 looked generous — and
+"looked generous" is not evidence.
+
+`EVAL_TIMEOUT` bounds the client, default 300 s. Raising the token cap alone
+just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of
+generation, so the run fails with TimeoutError instead. Raise both together.
+
+    ./run-candidate.sh <label> <sample> <n_predict> <timeout>
+    ./run-candidate.sh gemma4-26b-a4b 3 4096 900
+
+The Q8 precision comparison had to be thrown away over this. Q8's uncapped
+answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same
+task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run
+looked like "Q8 classifies worse" when it measured "Q8 is wordier than my
+bound". Before comparing two configurations, check that neither is sitting on
+the cap.

 ## What to watch for

-Two things make a result look like a bad model when it is not:
-
 **Prompt length.** Past roughly 14k tokens this stack stops producing language
 and starts producing garbage or nothing, without any error — and a single such
 request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
@@ -67,6 +169,30 @@ with their RAG neighbours. A run that starts scoring zeros mid-way is more
 likely this than the model.

 **Ground truth.** It is the current filing, and over ten years conventions
-change. A disagreement with an old document is not automatically an error. The
-comparison between engines is the durable result; the absolute numbers are
-indicative.
+change. A disagreement with an old document is not automatically an error.
+
+## The models volume fills up
+
+Each candidate leaves its weights on the shared `llama-models` PVC — 2.5 GB for
+a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB
+and nine screened candidates filled it completely.
+
+A full volume does not announce itself. llama-server cannot write its metadata
+cache, so it cannot resolve the repository listing, and says:
+
+    no GGUF files found in repository <repo>
+    Available GGUF files:
+      - <whatever it had cached before the disk filled>
+
+which reads as a wrong model reference. The pod then sits in
+`Init:CrashLoopBackOff`, which from outside is indistinguishable from a slow
+34 GB download. Two jobs spent three hours that way.
+
+**Check `df -h /models` before believing anything llama-server says about a
+repository.** Clear out rejected candidates when the volume gets tight — the
+weights are re-downloadable, so deleting them costs only bandwidth:
+
+```sh
+kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
+  sh -c 'cd /models && du -sh * | sort -h'
+```
@@ -0,0 +1,238 @@
+# label	hf-repo	hf-file	alias	mem-request	mem-limit
+#
+# Every candidate must do function calling: paperless calls the model through
+# chat_with_tools(..., tool_required=True) with DocumentClassifierSchema as a
+# tool, not via JSON mode. A model whose chat template has no tool support does
+# not score badly here, it fails outright.
+#
+# Instruct variants, never Reasoning ones. Qwen3.6 spent two of fifty documents
+# generating until something cut it off, and a model tuned to think first does
+# that by design. The one exception is Qwen3.6 itself, which is measured with
+# thinking switched off.
+#
+# Sizes. An earlier version of this note said "a smaller dense model gains at
+# both ends" and chose the Ministral pair on that basis. Measured, it is wrong:
+#
+#     Ministral-3 8B dense, ~4.7 GB weights   7.1-8.2 tok/s
+#     Qwen3.6 35B-A3B MoE,  ~20 GB weights    11 tok/s
+#
+# The largest model is the fastest one here. But the bandwidth explanation I
+# gave for it is wrong, and measuring Q8 disproved it:
+#
+#     Qwen3.6-35B-A3B Q4_K_M   ~21 GB   11.0 tok/s
+#     Qwen3.6-35B-A3B Q8_0     ~34 GB   10.3 tok/s
+#
+# Doubling the bytes per active parameter cost ten percent, not half. So decode
+# is NOT bandwidth-bound for this MoE — it is compute-bound on many small
+# per-expert matmuls, which also explains the earlier measurement of 21.4 GB/s
+# effective against a 53.9 GB/s ceiling: it was already leaving 60% of the bus
+# unused.
+#
+# Consequences: the bytes-per-token formula predicts dense models and not this
+# one, and higher precision is nearly free on an MoE here — about 10% slower
+# for double the weight precision.
+#
+# So the order to prefer is: MoE with few active parameters, then dense 3-9B,
+# then dense 10-14B only for an exceptional model (~4 tok/s, roughly three
+# hours for 50 documents), and nothing dense above 14B.
+qwen35	ggml-org/Qwen3.6-35B-A3B-GGUF	Qwen3.6-35B-A3B-Q4_K_M.gguf	qwen3.6-35b-a3b	24Gi	32Gi
+#
+# The same model at higher precision, to find out what Q4 costs. Everything in
+# this file is Q4_K_M, so the whole comparison could be sitting on quantisation
+# damage nobody has measured — and there is a concrete suspicion, not just a
+# worry: Ministral-3's leak of [THINK] and [TOOL_CALLS] as literal text is the
+# kind of fault a 4-bit quantisation can produce, and Q8 would separate "the
+# model cannot do this" from "Q4 broke it".
+#
+# ~34 GB resident against Q4's ~21, and roughly 6 tok/s against 11, so about
+# double the wall clock. Read the result against the noise floor in README.md:
+# below about two documents of difference, this says nothing.
+qwen35-q8	ggml-org/Qwen3.6-35B-A3B-GGUF	Qwen3.6-35B-A3B-Q8_0.gguf	qwen3.6-35b-a3b-q8	40Gi	48Gi
+#
+# --- MoE, few active parameters: the shape that wins on this hardware ---
+qwen3-30b-a3b	bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF	Qwen_Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf	qwen3-30b-a3b	22Gi	28Gi
+nemotron3-30b-a3b	bartowski/nvidia_Nemotron-3-Nano-30B-A3B-GGUF	nvidia_Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf	nemotron3-30b-a3b	28Gi	34Gi
+glm47-flash	bartowski/zai-org_GLM-4.7-Flash-GGUF	zai-org_GLM-4.7-Flash-Q4_K_M.gguf	glm-4.7-flash	20Gi	26Gi
+gemma4-26b-a4b	unsloth/gemma-4-26B-A4B-it-GGUF	gemma-4-26B-A4B-it-UD-Q4_K_M.gguf	gemma-4-26b-a4b	20Gi	26Gi
+qwen35-35b-a3b	bartowski/Qwen_Qwen3.5-35B-A3B-GGUF	Qwen_Qwen3.5-35B-A3B-Q4_K_M.gguf	qwen3.5-35b-a3b	24Gi	30Gi
+nemotron35-30b-a3b	bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf	nemotron3.5-30b-a3b	28Gi	34Gi
+#
+# --- dense, small, with German named in the model's own card ---
+glm4-9b	bartowski/glm-4-9b-chat-GGUF	glm-4-9b-chat-Q4_K_M.gguf	glm-4-9b	10Gi	14Gi
+commandr7b	bartowski/c4ai-command-r7b-12-2024-GGUF	c4ai-command-r7b-12-2024-Q4_K_M.gguf	command-r7b	10Gi	14Gi
+qwen25-7b	bartowski/Qwen2.5-7B-Instruct-GGUF	Qwen2.5-7B-Instruct-Q4_K_M.gguf	qwen2.5-7b	10Gi	14Gi
+llama31-8b	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf	llama-3.1-8b	10Gi	14Gi
+nemotron-nano-12b	bartowski/nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF	nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q4_K_M.gguf	nemotron-nano-12b	12Gi	16Gi
+phi4-mini	bartowski/microsoft_Phi-4-mini-instruct-GGUF	microsoft_Phi-4-mini-instruct-Q4_K_M.gguf	phi-4-mini	8Gi	12Gi
+#
+# --- dense, small, no German claim but no thinking risk either ---
+qwen3-4b-2507	bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF	Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf	qwen3-4b	8Gi	12Gi
+qwen35-9b	bartowski/Qwen_Qwen3.5-9B-GGUF	Qwen_Qwen3.5-9B-Q4_K_M.gguf	qwen3.5-9b	10Gi	14Gi
+#
+# WHAT THE RESEARCH RULED OUT, so nobody re-derives it
+#
+# Hard-excluded on tool calling, chat template inspected directly rather than
+# assumed — the string "tool" appears zero times in any of them:
+#   Gemma 3, every size (4B/12B/27B)   phi-4 14B (the mini DOES have it)
+#   Cohere Aya, all variants           InternLM 2.5 / 3
+#
+# Hard-excluded on German, stated by the vendor rather than merely missing:
+#   Falcon 3 — "supports EN, FR, ES, PT", German is not on the list
+#   EXAONE — the small ones are English/Korean; the ones with German are 33B+
+#
+# German fine-tunes are a dead end, and for one consistent reason: SauerkrautLM,
+# DiscoLM, LeoLM, Occiglot and Teuken all overwrite their base model's chat
+# template with a minimal ChatML loop that drops tool calling. VAGOsolutions
+# does this even on bases that have it (Llama-3.1, Qwen2.5). Assume any German
+# tune is tool-incapable until its own tokenizer_config.json says otherwise.
+#
+# On llama.cpp's tool parsing, which explains the Granite failure: current
+# builds have no per-vendor parser table any more. common/chat.cpp exposes
+# PEG_SIMPLE / PEG_NATIVE / PEG_GEMMA4 / PEG_MINIMAX_M3 and derives the grammar
+# from the model's own template. Hermes-style <tool_call> JSON is a first-class
+# tested fixture in tests/test-chat.cpp; Granite's format is not. So this is not
+# "Granite is unsupported" so much as "Granite's template shape is outside what
+# the auto-parser was validated against".
+#
+# SCREENING RESULTS, and a pattern that reframes the whole search
+#
+#   qwen35        MoE 35B total / 3B active   works, 2 of 50 failed
+#   qwen3-8b      dense 8B                    4 of 5 failed
+#   hermes4-14b   dense 14B                   3 of 3 failed, aborted
+#   ministral8b   dense 8B                    3 of 3 failed, aborted
+#   ministral3b   dense 3B                    29 of 50 failed
+#   granite8b     dense 8B                    every document failed
+#
+# Five dense models from 3B to 14B cannot produce a valid call against
+# paperless's 14-field schema; the one mixture-of-experts model can. Hermes is
+# the cleanest of these failures: its <tool_call> JSON is an explicitly tested
+# fixture in llama.cpp's own test suite, so the parser demonstrably handles the
+# format and the failure has to be the model.
+#
+# Working hypothesis: holding a long grammar-constrained structure needs model
+# capacity, and what supplies it is TOTAL parameters, not active ones. If that
+# is right, "go smaller for speed" is a dead end here — and the consolation is
+# that the fastest shape on this hardware, MoE with few active parameters, is
+# also the only one that works.
+#
+# qwen3-30b-a3b tests it directly: 30B total, 3B active.
+#
+# CORRECTION: Gemma is not excluded after all. Gemma *3* has no tool support —
+# "tool" appears zero times in its chat template at every size, checked
+# directly. Gemma *4* 26B-A4B does have it, documented in the template, plus
+# "140+ languages pretrained, 35+ out of the box". Excluding a family by its
+# previous generation was sloppy.
+#
+# Two of these carry a Vulkan risk worth checking in the llama-server log
+# before trusting a run: Nemotron-3.5 interleaves real Mamba2 layers and
+# Qwen3.5 uses DeltaNet linear attention, and llama.cpp's Vulkan SSM shaders
+# are less exercised on Iris Xe than on Arc. Issue #19957 reported exactly this
+# for Qwen3.5-35B-A3B and it turned out to be a stale build, fixed months
+# before b10731 — but confirm layers are actually offloaded rather than
+# silently falling back to CPU, which would read as a slow model.
+#
+# Ruled out after checking: ERNIE-4.5-21B-A3B (only the Thinking variant has
+# tool calling; the instruct one has none), Ling-3.0-flash (124B total),
+# Granite-4.0-H-Small (9B active), DeepSeek-V2-Lite and OLMoE (under 20B
+# total), Cohere North Mini (code-only, no ggml-org/bartowski conversion,
+# cohere2_moe support in llama.cpp unverified), gpt-oss-20b (a reasoning-effort
+# model in Harmony format rather than plain instruct).
+#
+# THE MoE HYPOTHESIS IS DEAD. qwen3-30b-a3b — MoE, 30B total, 3B active, the
+# shape that was supposed to explain everything — scored 1 of 5 and failed 4,
+# indistinguishable from the dense 8B.
+#
+# So neither architecture nor size predicts success:
+#
+#   Qwen3.6-35B-A3B   MoE 35B/3B   Apr 2026   2 of 50 failed
+#   Qwen3-30B-A3B     MoE 30B/3B   older      4 of 5 failed
+#   Qwen3-8B          dense 8B     older      4 of 5 failed
+#   Hermes-4-14B      dense 14B    —          3 of 3 failed
+#
+# The only thing correlating with success so far is how new and how strong the
+# model is. That is an unsatisfying hypothesis because it predicts nothing
+# cheaply — but it is what the data says, and two architectural theories have
+# already died here.
+#
+# qwen35-35b-a3b isolates it: same architecture, same total and active
+# parameters as the working baseline, one generation older. If it fails, the
+# generation is the variable and shape is noise.
+#
+# qwen35-35b-a3b PASSES the screen: 5 of 5, no failures, 584 s. So the
+# generation hypothesis is dead too — Qwen3.5 is a generation OLDER than the
+# working Qwen3.6 and handles the schema cleanly.
+#
+# What the two working models share is 35B total parameters; the one failing
+# MoE has 30B. But that is confounded — Qwen3-30B-A3B is both smaller and
+# older, and three data points cannot separate the two.
+#
+#   Qwen3.6-35B-A3B   35B total   newest   works
+#   Qwen3.5-35B-A3B   35B total   older    works
+#   Qwen3-30B-A3B     30B total   oldest   4 of 5 failed
+#
+# glm47-flash separates them: 30B total, newest generation. If it works,
+# generation is what matters and 35B is not a threshold. If it fails, total
+# size is, and the search should stop looking below ~35B.
+#
+# PREFILL SPEED IS A SELECTION CRITERION AND I WAS NOT TRACKING IT.
+# Measured on this hardware, prompt processing varies more than sixfold across
+# architectures, and it is roughly half of a run's wall clock:
+#
+#   dense 8B (Ministral)        240 tok/s
+#   Qwen3.6-35B-A3B             100 tok/s
+#   GLM-4.7-Flash                31 tok/s   decode 4.9 tok/s
+#
+# GLM-4.7-Flash is disqualified on that alone. At 31 tok/s a 5300-token prompt
+# takes 168 s, times ~2.6 requests per document, is over seven minutes per
+# document and about six hours for a run — before considering whether its
+# answers are any good.
+#
+# It also shows why the 300 s REQUEST_TIMEOUT needs care: 4453 prompt tokens at
+# 31.5 tok/s plus 768 generated at 4.9 tok/s is 298 s, so a slow model turns a
+# content failure into a TimeoutError and muddies the diagnosis. Read the
+# llama-server timings, not just the eval error.
+#
+# glm47-flash: aborted, 3 of 3 — but every one a TimeoutError, not an invalid
+# answer, so this does NOT settle the size-versus-generation question it was
+# started to settle. It failed on speed. Document 1 did reach the --n-predict
+# cap without producing a call, but 2 and 3 were cut off by the 300 s client
+# bound before anything could be concluded about them.
+#
+# Disqualified regardless: 31 tok/s prefill and 4.9 tok/s decode is over seven
+# minutes per document and about six hours for a full run.
+#
+# gemma4-26b-a4b is the better discriminator — 26B total, well under 35, and
+# newest generation. Works means generation and quality matter rather than a
+# 35B threshold; fails is consistent with total size being the line.
+#
+# gemma4-26b-a4b: aborted, 3 of 3, all HTTP 400 rather than timeouts, and the
+# server log shows it ran to the --n-predict cap on every one. So it is the
+# same capability failure as the small dense models, not a template mismatch.
+# (Prefill 45-50 tok/s, decode 6 tok/s — also not fast.)
+#
+# THE SIZE THRESHOLD SURVIVED ITS FIRST REAL TEST. gemma-4 was chosen
+# specifically because it could refute it — newest generation, 26B total — and
+# it did not:
+#
+#   Qwen3.6-35B-A3B   35B   newest    works
+#   Qwen3.5-35B-A3B   35B   older     works
+#   Qwen3-30B-A3B     30B   oldest    fails
+#   gemma-4-26B-A4B   26B   NEWEST    fails
+#   Hermes-4-14B      14B   —         fails
+#   dense 3-8B        —     —         fail
+#
+# Two generations work at 35B total; everything at 30B and below fails,
+# including the newest model in the field. Operating rule until something
+# contradicts it: do not spend a screening slot on anything under ~35B total.
+#
+# Not pinned further on purpose. Whether the line sits at 31B or 34B is
+# interesting and not decision-relevant — there are two working candidates and
+# the useful next number is a full 50-document run of the second one.
+#
+# Re-enabled for length probing, not for scoring. Their classification numbers
+# were taken against a slot that may already have been corrupted, so the only
+# honest thing to do with them is find out where each one's boundary sits.
+hermes4-14b	bartowski/NousResearch_Hermes-4-14B-GGUF	NousResearch_Hermes-4-14B-Q4_K_M.gguf	hemes-4-14b	14Gi	20Gi
+qwen3-8b	bartowski/Qwen_Qwen3-8B-GGUF	Qwen_Qwen3-8B-Q4_K_M.gguf	qwen3-8b	12Gi	16Gi
+ministral8b	bartowski/mistralai_Ministral-3-8B-Instruct-2512-GGUF	mistralai_Ministral-3-8B-Instruct-2512-Q4_K_M.gguf	ministral-3-8b	12Gi	16Gi
+granite8b	bartowski/ibm-granite_granite-3.3-8b-instruct-GGUF	ibm-granite_granite-3.3-8b-instruct-Q4_K_M.gguf	granite-3.3-8b	12Gi	16Gi
@@ -86,7 +86,12 @@ OUT_DIR = os.environ.get("EVAL_OUT", "/results")
 # 300 s against the ~73 s a healthy document takes end to end, prompt included.
 # It does work: llama-server logs `stop: cancel task` on the disconnect and
 # starts the next request 0.4 s later.
-REQUEST_TIMEOUT = 300
+REQUEST_TIMEOUT = int(os.environ.get("EVAL_TIMEOUT", "300"))
+
+# Consecutive failures from the very first document that mean the engine is
+# unusable rather than merely weak. Three, because one could be a single hard
+# document and two could be coincidence.
+ABORT_AFTER = 3

 CATCHALL_TYPE = "Sonstige"

@@ -96,20 +101,37 @@ def log(msg: str) -> None:


 def api_token(conn) -> str:
-    """A token for this run, created in the database.
+    """A token for this run, against a user belonging to this run alone.

    The API needs one and this database is a disposable clone, so the run
-    issues its own against a superuser rather than depending on a token that
-    happened to come across in the restore.
+    issues its own rather than depending on a token that happened to come
+    across in the restore.
+
+    The per-label user is not tidiness. Django's authtoken holds one token per
+    user, so an earlier version took the first superuser and deleted its
+    existing token first — which meant the second of two candidates running
+    concurrently silently revoked the first one's credentials. That surfaced as
+    a mid-run `HTTP 401: Invalid token` sitting among genuine model failures,
+    where it reads like one more broken document rather than the harness
+    sabotaging itself. Both jobs share this database; only the user can differ.
    """
    token = secrets.token_hex(20)
+    username = f"eval-{LABEL}"
    with conn.cursor() as cur:
-        cur.execute(
-            "select id from auth_user where is_superuser and is_active order by id limit 1",
-        )
+        cur.execute("select id from auth_user where username = %s", (username,))
        row = cur.fetchone()
        if row is None:
-            raise SystemExit("no active superuser in the restored database")
+            cur.execute(
+                """
+                insert into auth_user (password, is_superuser, username,
+                                       first_name, last_name, email, is_staff,
+                                       is_active, date_joined)
+                values ('!unusable', true, %s, '', '', '', true, true, now())
+                returning id
+                """,
+                (username,),
+            )
+            row = cur.fetchone()
        user_id = row[0]
        cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
        cur.execute(
@@ -247,9 +269,56 @@ def score_document(payload: dict, doc: dict) -> dict:
    }


+LLM_ENDPOINT = os.environ.get("PAPERLESS_AI_LLM_ENDPOINT", "http://127.0.0.1:8080/v1")
+
+
+def canary() -> tuple[bool, str]:
+    """Ask the model something it cannot get wrong, and check it did not.
+
+    The instrument has to be shown working before its readings mean anything,
+    and this one can fail silently in a way that looks like a bad model. On
+    this Vulkan stack a prompt past some model-specific length stops llama.cpp
+    producing language: it repeats one token — " de de de", " phosphorus
+    phosphorus", whitespace — at HTTP 200 with finish_reason "stop". The slot
+    stays broken for every request after it until the pod restarts.
+
+    So a run that measured 50 documents against a poisoned slot reports 50
+    failures that are really one event, and reads as a model that cannot
+    classify. That is exactly what happened before this existed.
+
+    Run before the documents and again after: healthy then broken means the
+    failures in between are one fault, not evidence about the model.
+    """
+    body = json.dumps({
+        "model": os.environ.get("PAPERLESS_AI_LLM_MODEL", "probe"),
+        "max_tokens": 40,
+        "messages": [{"role": "user", "content": "Nenne die Hauptstadt von Frankreich."}],
+    }).encode()
+    req = urllib.request.Request(
+        f"{LLM_ENDPOINT}/chat/completions",
+        data=body,
+        headers={"Content-Type": "application/json"},
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            payload = json.loads(resp.read())
+        text = payload["choices"][0]["message"].get("content") or ""
+        text += payload["choices"][0]["message"].get("reasoning_content") or ""
+        return ("Paris" in text), text[:80]
+    except Exception as err:  # noqa: BLE001
+        return False, f"{type(err).__name__}: {err}"
+
+
 def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
    log(f"=== {engine}: {len(docs)} documents")
    rows = []
+    if engine == "ai":
+        ok, text = canary()
+        log(f"  canary before: {'OK' if ok else 'BROKEN'} — {text!r}")
+        if not ok:
+            log("  The model is not answering a trivial question correctly, so")
+            log("  nothing measured after this describes its classification.")
+            return [{"engine": engine, "canary_before": False, "canary_text": text}]
    for n, doc in enumerate(docs, 1):
        payload, error, elapsed = fetch(
            f"/api/documents/{doc['id']}/{suffix}",
@@ -264,6 +333,17 @@ def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
            "catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
        }
        if payload is None:
+            # An authentication failure is never a fact about the model, and
+            # counted alongside real ones it reads like one more broken
+            # document. Stop loudly instead: something revoked this run's
+            # credentials mid-flight, and every number after it is worthless.
+            if error and error.startswith("HTTP 401"):
+                raise SystemExit(
+                    f"[eval] FATAL: {error}\n"
+                    f"[eval] This run's token stopped working part-way through. "
+                    f"Django's authtoken holds one token per user, so another "
+                    f"job sharing this database and this user revoked it.",
+                )
            row["error"] = error
            log(f"  {doc['id']}: {error}")
        else:
@@ -274,8 +354,82 @@ def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
                row["degenerate"] = broken
                log(f"  {doc['id']}: DEGENERATE — {broken}")
        rows.append(row)
+
+        # Clear the slot between documents so the next one is measured on a
+        # clean server. The corruption this guards against is stateful: one bad
+        # request leaves the slot returning a single repeated token for
+        # everything after it, and without this a run counts that once per
+        # remaining document.
+        if engine == "ai":
+            try:
+                urllib.request.urlopen(
+                    urllib.request.Request(
+                        f"{LLM_ENDPOINT.rsplit('/v1', 1)[0]}/slots/0?action=erase",
+                        data=b"",
+                        method="POST",
+                    ),
+                    timeout=30,
+                )
+            except Exception as err:  # noqa: BLE001 — best effort, never fatal
+                if n == 1:
+                    log(f"  slot erase unavailable ({type(err).__name__}) — "
+                        f"failures after a corrupt document will not be independent")
+
        if n % 10 == 0:
            log(f"  {n}/{len(docs)}")
+            # Re-check the instrument, not just the progress. A slot corrupted
+            # part-way through turns every remaining document into a failure
+            # that says nothing about the model, and scattered failures — 29 of
+            # 50 in one earlier run — are exactly the shape that a
+            # start-of-run check alone would miss.
+            if engine == "ai":
+                ok, text = canary()
+                if not ok:
+                    log(f"  CANARY BROKEN at {n}/{len(docs)} — {text!r}")
+                    log("  The model can no longer answer a trivial question, so")
+                    log("  everything from here would measure a dead slot. Stopping.")
+                    log("  Failures recorded before this point are still usable;")
+                    log("  the ones adjacent to it are one fault, not several.")
+                    rows.append({"engine": engine, "canary_failed_at": n, "canary_text": text})
+                    break
+
+        # Stop early when a model cannot do the task at all, rather than
+        # spending an hour writing the same error fifty times. Three candidates
+        # in a row failed this way — Granite because llama.cpp cannot read its
+        # tool calls back, both Ministral sizes because they fill the tool
+        # call's first string with their own [THINK]/[TOOL_CALLS] control
+        # tokens until the token cap. In each case document one already said
+        # so, and the remaining forty-nine said nothing new.
+        #
+        # Deliberately not a ratio: a model that fails a third of the time is a
+        # result worth measuring in full. This only catches the case where
+        # nothing has ever worked.
+        if len(rows) == ABORT_AFTER and all("error" in r for r in rows):
+            ok, text = canary()
+            log(f"  canary after {ABORT_AFTER} failures: {'OK' if ok else 'BROKEN'} — {text!r}")
+            if not ok:
+                log("  The slot is corrupted, so these are ONE fault and not")
+                log(f"  {ABORT_AFTER} independent results. Probe the model's prompt-length")
+                log("  boundary with probe-length.sh before judging it.")
+            log(f"  ABORT — first {ABORT_AFTER} documents all failed:")
+            log(f"    {rows[0]['error']}")
+            log("  Nothing has succeeded, so the rest would measure nothing.")
+            log("  Check the model's tool calling before re-running:")
+            log("    curl -s $ENDPOINT/v1/chat/completions -d @probe.json")
+            log("  A usable candidate answers with finish_reason 'tool_calls'.")
+            break
+
+    # And once at the end, whatever happened. A short run whose first
+    # document succeeds trips neither the abort check nor the every-tenth
+    # one, so without this a five-document screen can finish having never
+    # re-verified the instrument it just used.
+    if engine == "ai":
+        ok, text = canary()
+        log(f"  canary after: {'OK' if ok else 'BROKEN'} — {text!r}")
+        if not ok:
+            log("  Healthy before, broken after: the failures in this run are")
+            log("  one fault and not independent results.")
+            rows.append({"engine": engine, "canary_after": False, "canary_text": text})
    return rows


@@ -16,7 +16,7 @@
 apiVersion: batch/v1
 kind: Job
 metadata:
-  name: eval-qwen35
+  name: eval-__LABEL__
  namespace: paperless-eval
  labels:
    app.kubernetes.io/name: paperless-eval
@@ -107,9 +107,28 @@ spec:
          restartPolicy: Always
          args:
            - --hf-repo
-            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
+            - __HF_REPO__
+            # The exact filename rather than the :QUANT shorthand. The
+            # shorthand matches by substring across the whole repo and repos do
+            # carry several files under one tag — ggml-org ships mmproj-, mtp-
+            # and dflash- variants at Q8_0 beside the model — so naming the file
+            # removes a real ambiguity and costs nothing.
+            #
+            # It is NOT what caused the three-hour crashloop that prompted this,
+            # and an earlier version of this comment claimed it was. The real
+            # cause was the llama-models volume being 100% full: llama-server
+            # could not write its metadata cache, so it could not resolve the
+            # repo listing, and reported
+            #     no GGUF files found in repository <repo>
+            # followed by a truncated list of what it had cached before the
+            # disk filled. That reads as a bad reference and is a full disk.
+            #
+            # If a model will not load, check `df -h /models` before believing
+            # anything llama-server says about the repository.
+            - --hf-file
+            - __HF_FILE__
            - --alias
-            - qwen3.6-35b-a3b
+            - __ALIAS__
            - --n-gpu-layers
            - "99"
            - --threads
@@ -131,21 +150,38 @@ spec:
            # healthy 11 tok/s, and three of the first eleven documents did it
            # — 900 s each, most of an hour spent on four documents.
            #
-            # 768 is generous against the ~300 a correct answer takes, and a
-            # truncated response is better evidence than a timeout: it shows
-            # what the model was doing instead of stopping.
+            # 768 is generous against the ~300 a correct answer takes. Note
+            # that paperless reports a truncated response as a flat
+            # "Invalid AI configuration" and discards the content, so what the
+            # model was actually doing is visible only in this container's log.
            - --n-predict
-            - "768"
+            - "__N_PREDICT__"
            # Without this, Qwen3.6 spends the response on reasoning_content
            # and returns empty content — 983 tokens into a classification that
            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
            # slow or broken model and is neither.
            #
-            # A candidate whose template has no such switch has to be measured
-            # as it comes, and that is a fair part of its result: paperless
-            # gives the model no token budget and waits.
-            - --chat-template-kwargs
-            - '{"enable_thinking": false}'
+            # NOT --chat-template-kwargs '{"enable_thinking": false}'. That is
+            # deprecated, llama-server says so at startup, and it is not the
+            # harmless no-op it looks like on a template that never reads the
+            # variable: injecting it into Mistral's render context produced
+            # token salad — "1UnComplete1Self1UnCompleteSDirect11S..." — on a
+            # 681-token prompt, while the identical request against a server
+            # without the flag returned a clean tool call. It reads exactly
+            # like a broken model and is a broken prompt.
+            - --reasoning
+            - "off"
+            # Enables POST /slots/0?action=erase, which the eval calls between
+            # documents. Without it the endpoint answers 501.
+            #
+            # This is what makes the measurements independent. A request that
+            # corrupts the slot otherwise takes every document after it, so a
+            # run reports N failures for one fault — "4 of 5 failed" was really
+            # "document 2 broke the slot". Erasing costs the prefix cache and
+            # some prefill speed, which is the right trade for numbers that
+            # mean what they say.
+            - --slot-save-path
+            - /tmp/slots
            - --no-webui
          env:
            - name: LLAMA_CACHE
@@ -156,20 +192,27 @@ spec:
          volumeMounts:
            - name: models
              mountPath: /models
-          # ~20 GB of weights off CephFS, so a generous budget: 60 * 20s.
+          # 60 minutes. Sized for a FIRST DOWNLOAD from HuggingFace, not for
+          # reading a warm PVC — an earlier version allowed 20 minutes on the
+          # latter reasoning and killed GLM-4.7-Flash mid-download at 17.8 GB.
+          # The container then restarts and begins again, so too small a budget
+          # is not "slow start", it is a loop that never finishes.
+          #
+          # The kill shows up as exitCode 137 with reason "Error" rather than
+          # OOMKilled, which is how to tell a probe kill from a memory limit.
          startupProbe:
            httpGet:
              path: /health
              port: llm
            periodSeconds: 20
-            failureThreshold: 60
+            failureThreshold: 180
          resources:
            requests:
              cpu: "1"
-              memory: 24Gi
+              memory: __MEM_REQ__
              gpu.intel.com/i915: "1"
            limits:
-              memory: 32Gi
+              memory: __MEM_LIM__
              gpu.intel.com/i915: "1"
          securityContext:
            allowPrivilegeEscalation: false
@@ -209,7 +252,7 @@ spec:
            - name: PAPERLESS_AI_LLM_ENDPOINT
              value: http://127.0.0.1:8080/v1
            - name: PAPERLESS_AI_LLM_MODEL
-              value: qwen3.6-35b-a3b
+              value: __ALIAS__
            - name: PAPERLESS_REDIS
              value: redis://127.0.0.1:6379
            - name: PAPERLESS_SECRET_KEY
@@ -265,9 +308,15 @@ spec:
          command: ["python3", "/eval/eval.py"]
          env:
            - name: EVAL_LABEL
-              value: qwen35
+              value: __LABEL__
            - name: EVAL_SAMPLE
-              value: "50"
+              value: "__SAMPLE__"
+            # Raised alongside --n-predict for a diagnostic run. The two bounds
+            # interact: lifting the token cap alone just moves the failure from
+            # "hit the cap" to "hit the clock", which is what happened the first
+            # time gemma-4 was retried at 4096.
+            - name: EVAL_TIMEOUT
+              value: "__TIMEOUT__"
            - name: PGHOST
              value: pprleval-pg-rw
            - name: PGDATABASE
@@ -0,0 +1,149 @@
+#!/bin/sh
+# Where does this model's output stop being language?
+#
+#   ./probe-length.sh <label>
+#
+# THE TEST THAT SHOULD HAVE COME FIRST. On this Vulkan/Iris Xe stack every
+# model has a prompt length past which llama.cpp stops producing language and
+# starts repeating a single token — " de de de", " the the the", " phosphorus
+# phosphorus", or plain whitespace. It is silent: HTTP 200, finish_reason
+# "stop" or "length", nothing in the log.
+#
+# It is also STATEFUL. One request past the boundary leaves the slot broken for
+# every request after it, until the pod restarts. So in a 50-document run the
+# first bad document takes the rest with it, and "failed 3 of 3" is one
+# measurement, not three.
+#
+# A real paperless request is ~5300 tokens once the document, its RAG
+# neighbours and the taxonomy lists are in. So the only question that matters
+# about a candidate is whether its boundary is above that — and answering it
+# takes minutes, against ninety for a classification run that would have
+# measured the same thing badly.
+#
+# Measured so far:
+#   Qwen3.6-35B-A3B   coherent at 13588, broken at 15086
+#   gemma-4-26B-A4B   coherent at ~350, whitespace at 2334
+set -eu
+
+label="${1:?usage: probe-length.sh <label>   (labels are in candidates.tsv)}"
+here="$(dirname "$0")"
+ns=paperless-eval
+pod="lenprobe-$(echo "$label" | tr 'A-Z._' 'a-z--' | cut -c1-40)"
+
+row="$(grep -v '^#' "$here/candidates.tsv" | awk -v l="$label" '$1==l')"
+[ -n "$row" ] || { echo "unknown candidate '$label'" >&2; exit 1; }
+repo="$(echo "$row"   | cut -f2)"
+hffile="$(echo "$row" | cut -f3)"
+memreq="$(echo "$row" | cut -f5)"
+memlim="$(echo "$row" | cut -f6)"
+
+cleanup() { kubectl -n "$ns" delete pod "$pod" --wait=false >/dev/null 2>&1 || true; }
+trap cleanup EXIT INT TERM
+cleanup; sleep 2
+
+kubectl -n "$ns" apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: $pod, namespace: $ns}
+spec:
+  restartPolicy: Never
+  containers:
+    - name: llama-server
+      image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+      args: ["--hf-repo","$repo","--hf-file","$hffile","--alias","probe",
+             "--n-gpu-layers","99","--threads","4","--ctx-size","16384",
+             "--parallel","1","--ubatch-size","1024","--n-predict","200","--no-webui"]
+      env: [{name: LLAMA_CACHE, value: /models}]
+      volumeMounts: [{name: models, mountPath: /models}]
+      resources:
+        requests: {cpu: "1", memory: $memreq, gpu.intel.com/i915: "1"}
+        limits: {memory: $memlim, gpu.intel.com/i915: "1"}
+      securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: ["ALL"]}}
+  volumes:
+    - name: models
+      persistentVolumeClaim: {claimName: llama-models}
+EOF
+
+# Say which of the two it is doing. There are only two iGPUs, so a probe
+# started while both are busy sits Pending indefinitely — and an earlier run
+# spent an hour reporting nothing before giving up with "never became healthy",
+# which reads as a broken model rather than a queue.
+echo "waiting for $label"
+i=0
+until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do
+  i=$((i+1))
+  if [ $(( i % 8 )) -eq 0 ]; then
+    ph=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.phase}' 2>&1)
+    why=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")].message}' 2>&1)
+    echo "  ${ph}${why:+ — $why}"
+    [ "$ph" = "Pending" ] && [ -n "$why" ] && { echo "FAIL: cannot schedule — free a GPU first"; exit 1; }
+  fi
+  [ "$i" -gt 240 ] && { echo "FAIL: never became healthy after 60 min"; exit 1; }
+  sleep 15
+done
+
+# Fresh pod per size. The defect poisons the slot, so probing sizes in sequence
+# against one server would measure the first bad request and nothing after it.
+for reps in 4 20 40 70 110 160; do
+  kubectl -n "$ns" delete pod "$pod" >/dev/null 2>&1 || true
+  sleep 2
+  kubectl -n "$ns" apply -f - >/dev/null <<EOF2
+apiVersion: v1
+kind: Pod
+metadata: {name: $pod, namespace: $ns}
+spec:
+  restartPolicy: Never
+  containers:
+    - name: llama-server
+      image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+      args: ["--hf-repo","$repo","--hf-file","$hffile","--alias","probe",
+             "--n-gpu-layers","99","--threads","4","--ctx-size","16384",
+             "--parallel","1","--ubatch-size","1024","--n-predict","200","--no-webui"]
+      env: [{name: LLAMA_CACHE, value: /models}]
+      volumeMounts: [{name: models, mountPath: /models}]
+      resources:
+        requests: {cpu: "1", memory: $memreq, gpu.intel.com/i915: "1"}
+        limits: {memory: $memlim, gpu.intel.com/i915: "1"}
+      securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: ["ALL"]}}
+  volumes:
+    - name: models
+      persistentVolumeClaim: {claimName: llama-models}
+EOF2
+  until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do sleep 10; done
+
+  filler=""; n=0
+  while [ $n -lt $reps ]; do
+    filler="${filler}Die Rechnung wurde am fuenften Maerz ausgestellt und umfasst mehrere Positionen aus dem laufenden Wartungsvertrag. "
+    n=$((n+1))
+  done
+  printf '{"model":"probe","max_tokens":200,"messages":[{"role":"user","content":"%s Nenne die Hauptstadt von Frankreich."}]}' "$filler" > /tmp/lp.json
+  kubectl -n "$ns" cp /tmp/lp.json "$pod":/tmp/lp.json >/dev/null 2>&1 || true
+  r=$(kubectl -n "$ns" exec "$pod" -- curl -s --max-time 400 -X POST http://127.0.0.1:8080/v1/chat/completions \
+        -H 'Content-Type: application/json' -d @/tmp/lp.json)
+  pt=$(printf '%s' "$r" | grep -oE '"prompt_tokens":[0-9]+' | head -1 | cut -d: -f2)
+  # Test for DEGENERATION, not for the right answer. Two earlier versions of
+  # this check asked whether "Paris" came back and failed a working model
+  # twice: once because a thinking model leaves `content` empty and puts its
+  # answer in `reasoning_content`, then because 64 tokens is not enough for one
+  # to think its way to the answer at all.
+  #
+  # What actually marks the defect is a single token repeated — " de de de",
+  # " phosphorus phosphorus", whitespace. That signature survives truncation
+  # and does not care which field the model writes to or whether it finished,
+  # so it tests the thing itself instead of a proxy for it.
+  txt=$(printf '%s' "$r" | grep -oE '"(reasoning_)?content":"[^"]*' | sed 's/"[a-z_]*content":"//' | tr '\n' ' ')
+  rep=$(printf '%s' "$txt" | tr -s ' ' '\n' | grep -c . || echo 0)
+  top=$(printf '%s' "$txt" | tr -s ' ' '\n' | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')
+  : "${top:=0}"
+  if [ "$rep" -gt 8 ] && [ $(( top * 2 )) -gt "$rep" ]; then
+    verdict="BROKEN"      # one token is more than half the output
+  elif [ "$rep" -le 1 ]; then
+    verdict="EMPTY"       # nothing came back at all — also not healthy
+  else
+    verdict="OK"
+  fi
+  printf '%-6s prompt=%-6s %s  %s\n' "$verdict" "${pt:-?}" "$(printf '%.50s' "$txt")" ""
+  [ "$verdict" != "OK" ] && { echo; echo "boundary is below ${pt:-?} prompt tokens"; echo "paperless needs ~5300 — $([ "${pt:-0}" -lt 6000 ] && echo 'DISQUALIFIED' || echo 'still above the bar')"; exit 0; }
+done
+echo
+echo "coherent through the largest size probed — boundary is above that"
@@ -0,0 +1,95 @@
+#!/bin/sh
+# Can this model make one valid tool call against paperless's real schema?
+#
+#   ./probe-toolcall.sh ggml-org/Ministral-3-8B-Instruct-2512-GGUF:Q8_0
+#
+# Two minutes, against ninety for a full run. Every candidate must pass this
+# before it is worth measuring, because three in a row failed here and the
+# remaining forty-nine documents of each run said nothing the first one had not.
+#
+# WHAT THIS CAN AND CANNOT TELL YOU
+#
+# A FAIL is conclusive: the model cannot make the call, do not spend a run.
+# A PASS is not. This payload has been too easy twice — first with three fields
+# instead of fourteen, then with a 1191-token prompt instead of production's
+# ~5300 — and each time a pass sent a ninety-minute run after a model that
+# failed on document one.
+#
+# It carries the real DocumentClassifierSchema and ~5000 tokens now, but it
+# still cannot reproduce what actually inflates a paperless prompt: the RAG
+# neighbours and the full taxonomy candidate lists, hundreds of names with IDs
+# the model must copy exactly and align by position. Document 1970 is 1976
+# characters and still produced a 5333-token prompt.
+#
+# So treat this as a cheap smoke test, not a gate. The real gate is
+# ABORT_AFTER in eval.py, which stops a run after three failed documents —
+# five minutes lost instead of ninety, using the genuine request.
+set -eu
+
+repo="${1:?usage: probe-toolcall.sh <hf-repo:quant> [extra llama-server args...]}"
+shift || true
+here="$(dirname "$0")"
+ns=paperless-eval
+pod="toolprobe-$(echo "$repo" | tr 'A-Z/:._' 'a-z----' | cut -c1-40 | sed 's/-*$//')"
+
+cleanup() { kubectl -n "$ns" delete pod "$pod" --wait=false >/dev/null 2>&1 || true; }
+trap cleanup EXIT INT TERM
+cleanup
+sleep 2
+
+echo "probing $repo"
+kubectl -n "$ns" apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: $pod
+spec:
+  restartPolicy: Never
+  containers:
+    - name: llama-server
+      image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+      args: ["--hf-repo","$repo","--alias","probe","--n-gpu-layers","99",
+             "--threads","4","--ctx-size","16384","--parallel","1",
+             "--ubatch-size","1024","--n-predict","768","--reasoning","off",
+             "--no-webui"$(for a in "$@"; do printf ',"%s"' "$a"; done)]
+      env: [{name: LLAMA_CACHE, value: /models}]
+      volumeMounts: [{name: models, mountPath: /models}]
+      resources:
+        requests: {cpu: "1", memory: 12Gi, gpu.intel.com/i915: "1"}
+        limits: {memory: 20Gi, gpu.intel.com/i915: "1"}
+      securityContext:
+        allowPrivilegeEscalation: false
+        capabilities: {drop: ["ALL"]}
+  volumes:
+    - name: models
+      persistentVolumeClaim: {claimName: llama-models}
+EOF
+
+echo "waiting for the model to load (a first download takes minutes)"
+i=0
+until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do
+  i=$((i + 1))
+  [ "$i" -gt 120 ] && { echo "FAIL: server never became healthy"; kubectl -n "$ns" logs "$pod" --tail=20; exit 1; }
+  sleep 10
+done
+
+sed "s/\"MODEL\"/\"probe\"/" "$here/toolcall-probe.json" > /tmp/tcp.json
+kubectl -n "$ns" cp /tmp/tcp.json "$pod":/tmp/probe.json >/dev/null 2>&1 || true
+r=$(kubectl -n "$ns" exec "$pod" -- curl -s --max-time 300 \
+      -X POST http://127.0.0.1:8080/v1/chat/completions \
+      -H 'Content-Type: application/json' -d @/tmp/probe.json)
+
+finish=$(printf '%s' "$r" | grep -oE '"finish_reason":"[a-z_]+"' | head -1 | cut -d'"' -f4)
+echo
+echo "finish_reason: ${finish:-<none>}"
+printf '%s' "$r" | grep -oE '"arguments":"[^"]{0,300}' | head -1 || true
+echo
+
+# finish_reason "tool_calls" is the whole test. "length" means it never stopped,
+# "stop" with no tool_calls means the server could not read the call back.
+if [ "$finish" = "tool_calls" ]; then
+  echo "PASS — usable, worth a full run"
+else
+  echo "FAIL — do not spend a run on this until the cause is understood"
+  exit 1
+fi
@@ -0,0 +1,90 @@
+#!/bin/sh
+# Start one candidate's measurement.
+#
+#   ./run-candidate.sh ministral8b
+#
+# Exists so the sidecar configuration lives in one file. Four copies of a
+# 300-line Job manifest is how --chat-template-kwargs ended up on the isolation
+# pod but not on the first eval job, and the run that followed measured empty
+# completions for two hours before anyone noticed.
+set -eu
+
+label="${1:?usage: run-candidate.sh <label> [sample]   (labels are in candidates.tsv)}"
+
+# Screen before you measure. A candidate that cannot do the task at all is
+# rejected by eval.py's ABORT_AFTER within three documents, so a screening run
+# costs about five minutes; a full run costs ninety. Three of the first four
+# candidates were rejections, and each one was allowed to run for an hour or
+# more before anyone looked.
+#
+#     ./run-candidate.sh <label> 5     screen
+#     ./run-candidate.sh <label>       measure, 50 documents
+#
+# Screening uses genuine paperless requests, which matters: the standalone
+# probe-toolcall.sh passed two candidates that then failed on document one,
+# because a synthesised request cannot reproduce the RAG neighbours and the
+# taxonomy candidate lists that make a real prompt what it is.
+#
+# Never compare numbers across sample sizes. A screen is a gate, not a
+# measurement; the comparison happens at 50.
+sample="${2:-50}"
+
+# Third argument raises the server-side generation cap for a diagnostic run.
+#
+# The default 768 is MINE, not a property of any model, and every candidate
+# recorded as failing at exactly 768 tokens is failing against my bound rather
+# than demonstrably failing the task. A model that needs 900 tokens for a valid
+# call looks identical to one that never terminates. Raise it and find out
+# before calling a model incapable:
+#
+#     ./run-candidate.sh gemma4-26b-a4b 5 4096
+npredict="${3:-768}"
+
+# Client-side bound, raised with the cap. At 6 tok/s a 4096-token budget needs
+# 620 s of generation alone, so leaving this at 300 turns the diagnostic into a
+# different failure rather than an answer.
+timeout="${4:-300}"
+here="$(dirname "$0")"
+
+row="$(grep -v '^#' "$here/candidates.tsv" | awk -v l="$label" '$1==l')"
+if [ -z "$row" ]; then
+  echo "unknown candidate '$label'. Known:" >&2
+  grep -v '^#' "$here/candidates.tsv" | awk 'NF{print "  " $1}' >&2
+  exit 1
+fi
+
+repo="$(echo "$row"   | cut -f2)"
+hffile="$(echo "$row" | cut -f3)"
+alias="$(echo "$row"  | cut -f4)"
+memreq="$(echo "$row" | cut -f5)"
+memlim="$(echo "$row" | cut -f6)"
+
+if [ -z "$hffile" ]; then
+  echo "candidate '$label' has no hf-file column — refusing to guess." >&2
+  echo "Resolve the exact .gguf filename from the HF API and add it." >&2
+  exit 1
+fi
+
+# The script is mounted from a ConfigMap, so it has to be current before the
+# Job starts. Shared by every candidate — they run the same measurement.
+kubectl -n paperless-eval create configmap paperless-eval-script \
+  --from-file=eval.py="$here/eval.py" --dry-run=client -o yaml | kubectl apply -f -
+
+if [ "$sample" -lt 50 ]; then
+  echo "SCREENING $label at $sample documents — a gate, not a measurement"
+fi
+echo "candidate=$label repo=$repo alias=$alias mem=$memreq/$memlim sample=$sample n_predict=$npredict timeout=$timeout"
+sed -e "s#__LABEL__#$label#g" \
+    -e "s#__HF_REPO__#$repo#g" \
+    -e "s#__ALIAS__#$alias#g" \
+    -e "s#__MEM_REQ__#$memreq#g" \
+    -e "s#__MEM_LIM__#$memlim#g" \
+    -e "s#__SAMPLE__#$sample#g" \
+    -e "s#__HF_FILE__#$hffile#g" \
+    -e "s#__N_PREDICT__#$npredict#g" \
+    -e "s#__TIMEOUT__#$timeout#g" \
+    "$here/job-template.yaml" | kubectl apply -f -
+
+echo
+echo "follow:  kubectl -n paperless-eval logs -f job/eval-$label -c eval"
+echo "re-run:  kubectl -n paperless-eval delete job eval-$label   # a Job is immutable"