fix(paperless-eval): bound generation server-side; the model does not always stop

Three of the first eleven documents in a run never terminated. One reached
10269 tokens on a classification that needs ~300, at a perfectly healthy
11 tok/s — so it read as slowness and was neither slow nor a context problem.

llama-server's --n-predict defaults to -1, infinity, and paperless sets no
token budget of its own and waits. Nothing anywhere bounded the request. Four
documents cost most of an hour.

--n-predict 768 on the sidecar caps it, and a truncated response is better
evidence than a timeout: it shows what the model was doing instead of
stopping. The client timeout drops to 300 s and becomes a backstop.

Also:

- the degeneracy check fired on 'Rechnung 2025-5425541' — 8 letters in 21
  characters, below the 40% Latin threshold. Digits and punctuation are normal
  in a title; look for characters from other scripts directly instead of
  inferring them from a shortage of Latin ones.

- the ctx-size comment still carried the claim bisection disproved in #777,
  that 32768 corrupts inference. The boundary is prompt length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml

@@ -58,30 +58,10 @@ spec:
           # VeleroBackupFailed alert fires on the first failure regardless — so
           # nothing is hidden, only the escalation to critical is deferred
           # until a second cycle confirms it.
-          #
-          # The `and on(schedule) velero_backup_last_status` gate exists because
-          # velero_backup_last_successful_timestamp is an in-process gauge that
-          # velero does NOT retract when a Schedule is deleted: the series just
-          # stops being updated, ages past 60h and pages critical forever, for a
-          # schedule that no longer exists. That is what decommissioning
-          # netbird-backup-daily did on 2026-09-01.
-          #
-          # velero_backup_last_status is the same shape but IS retracted on
-          # Schedule deletion, so it works as a liveness set for "this schedule
-          # still exists". It hides nothing: velero repopulates it for every
-          # existing Schedule on its first scrape after a restart (verified
-          # across the three pods of the 2026-08-27 rollout), and it is written
-          # on failed runs too (0) as well as successful ones (1) — so a
-          # schedule that exists and is failing, or that exists and has stopped
-          # firing, still carries the series and still alerts.
           expr: |
-            (
-              time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
-              and
-              velero_backup_last_successful_timestamp{schedule!=""} > 0
-            )
-            and on(schedule)
-            velero_backup_last_status
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
+            and
+            velero_backup_last_successful_timestamp{schedule!=""} > 0
           for: 10m
           labels:
             severity: critical

kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml

@@ -937,7 +937,7 @@ subjects:
 # with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
 stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
   name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
             secretKeyRef:
               key: SIDECAR_IMAGE
               name: plugin-barman-cloud-kg5c2795f7
-        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
+        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
         name: barman-cloud
         ports:
         - containerPort: 9090

scripts/paperless-eval/eval.py

@@ -78,10 +78,15 @@ ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
 LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
 OUT_DIR = os.environ.get("EVAL_OUT", "/results")
 
-# Long enough that a slow document is measured rather than abandoned. A client
-# timeout does not stop the generation behind it, and with a single llama.cpp
-# slot an abandoned request holds the queue for everything after it.
-REQUEST_TIMEOUT = 900
+# A backstop, not the real bound. Generation is capped server-side by
+# --n-predict on the llama-server sidecar, which is what actually stops a model
+# that will not terminate; this catches everything else — a wedged sidecar, a
+# request that never reaches it.
+#
+# 300 s against the ~73 s a healthy document takes end to end, prompt included.
+# It does work: llama-server logs `stop: cancel task` on the disconnect and
+# starts the next request 0.4 s later.
+REQUEST_TIMEOUT = 300
 
 CATCHALL_TYPE = "Sonstige"
 
@@ -165,7 +170,9 @@ def floor(conn) -> float:
             """,
         )
         top, total = cur.fetchone()
-    return 100.0 * top / total if total else 0.0
+    # float() on both: psycopg returns Postgres numerics as Decimal, and
+    # mixing that with a float literal raises TypeError rather than coercing.
+    return 100.0 * float(top) / float(total) if total else 0.0
 
 
 def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
@@ -198,6 +205,32 @@ def recognised(payload: dict, field: str, want_name: str) -> bool:
     return any(want <= normalise(name) for name in payload.get(field) or [])
 
 
+def degenerate(payload: dict) -> str | None:
+    """Whether this looks like broken inference rather than a wrong guess.
+
+    Past roughly 14k prompt tokens this stack stops producing language — first
+    empty completions, then runs of digits and CJK characters — and one such
+    request poisons the llama.cpp slot until the pod restarts. Nothing in the
+    HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
+
+    Scored as a wrong answer it would silently depress a candidate's numbers,
+    and because the slot stays poisoned it would take the rest of the run with
+    it. Flagged instead, so the failure is visible as itself.
+    """
+    title = payload.get("title") or ""
+    if not title.strip():
+        return "empty title"
+    # Digits, punctuation and whitespace are normal in a document title —
+    # "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
+    # earlier version of this check. What broken inference produces is
+    # characters from other scripts, so look for those directly rather than
+    # inferring them from a shortage of Latin ones.
+    foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
+    if foreign > 0.2 * len(title):
+        return f"non-latin title: {title[:40]!r}"
+    return None
+
+
 def score_document(payload: dict, doc: dict) -> dict:
     offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
     offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
@@ -235,6 +268,11 @@ def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
             log(f"  {doc['id']}: {error}")
         else:
             row.update(score_document(payload, doc))
+            # Only the AI path returns a title; the classifier answers with ids
+            # alone, so an "empty title" there is normal rather than broken.
+            if engine == "ai" and (broken := degenerate(payload)) is not None:
+                row["degenerate"] = broken
+                log(f"  {doc['id']}: DEGENERATE — {broken}")
         rows.append(row)
         if n % 10 == 0:
             log(f"  {n}/{len(docs)}")
@@ -246,17 +284,23 @@ def pct(hits: int, total: int) -> float:
 
 
 def summarise(rows: list[dict], engine: str) -> dict:
-    scored = [r for r in rows if r["engine"] == engine and "error" not in r]
-    failed = [r for r in rows if r["engine"] == engine and "error" in r]
+    mine = [r for r in rows if r["engine"] == engine]
+    # Degenerate rows are excluded from the accuracy figures: they measure the
+    # backend giving up, not the model choosing wrongly, and leaving them in
+    # would quietly punish whichever candidate happened to trip the slot.
+    scored = [r for r in mine if "error" not in r and "degenerate" not in r]
+    failed = [r for r in mine if "error" in r]
+    broken = [r for r in mine if "degenerate" in r]
     specific = [r for r in scored if not r["catchall_type"]]
     catchall = [r for r in scored if r["catchall_type"]]
     return {
         "engine": engine,
         "scored": len(scored),
         "failed": len(failed),
-        "seconds_total": round(
-            sum(r["seconds"] for r in rows if r["engine"] == engine),
-        ),
+        # Non-zero means the run is suspect from that point on, not that the
+        # model is bad — the slot stays poisoned until the pod restarts.
+        "degenerate": len(broken),
+        "seconds_total": round(sum(r["seconds"] for r in mine)),
         "correspondent_offered": pct(
             sum(r["correspondent_offered"] for r in scored), len(scored)
         ),
@@ -281,7 +325,8 @@ def complement(rows: list[dict]) -> dict | None:
     """
     by_engine = {"classic": {}, "ai": {}}
     for row in rows:
-        if "error" not in row and row["engine"] in by_engine:
+        usable = "error" not in row and "degenerate" not in row
+        if usable and row["engine"] in by_engine:
             by_engine[row["engine"]][row["document_id"]] = row
     both = set(by_engine["classic"]) & set(by_engine["ai"])
     if not both or not by_engine["ai"]:

scripts/paperless-eval/job-qwen35.yaml

@@ -69,9 +69,27 @@ spec:
             - -c
             - |
               set -e
-              cp -a /golden/. /data/
-              echo "seeded /data:"
-              du -sh /data/index /data/llm_index 2>/dev/null || true
+              # -r, not -a: preserving ownership needs CHOWN, which this
+              # container does not have, and cp then exits non-zero after
+              # copying the files anyway. Running as 1000 makes the copy land
+              # with the ownership paperless needs — it writes to the LLM index
+              # while serving.
+              cp -r /golden/. /data/
+              du -sh /data/index /data/llm_index
+              # Refuse to start rather than measure the wrong thing. An empty
+              # llm_index makes retrieve_similar_nodes return nothing, and the
+              # run would report a RAG result having never retrieved anything —
+              # which is exactly what the first attempt at this job did, because
+              # the golden volume had not been populated yet. The index is
+              # ~125 MiB once built; anything under 1 MiB is not one.
+              bytes=$(du -sb /data/llm_index | cut -f1)
+              if [ "$bytes" -lt 1000000 ]; then
+                echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
+                echo "smoke-test deployment first:" >&2
+                echo "  kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
+                echo "    python manage.py document_llmindex rebuild" >&2
+                exit 1
+              fi
           volumeMounts:
             - name: golden
               mountPath: /golden
@@ -79,6 +97,8 @@ spec:
             - name: data
               mountPath: /data
           securityContext:
+            runAsUser: 1000
+            runAsGroup: 1000
             allowPrivilegeEscalation: false
             capabilities:
               drop: ["ALL"]
@@ -94,17 +114,38 @@ spec:
             - "99"
             - --threads
             - "4"
-            # The production ceiling, and for the same reason: 32768 per slot
-            # corrupts inference on this Vulkan stack — 1025 tokens of "!"
-            # against a 650-token prompt, with /props still reporting the
-            # requested size. A candidate needing more context than this cannot
-            # be measured here, and that is itself a result.
+            # The production ceiling. Not because the flag value itself breaks
+            # anything — bisection showed the corruption boundary is prompt
+            # length, around 14000 tokens, whatever --ctx-size says — but
+            # because measuring against a different ceiling than production
+            # runs would not describe production.
             - --ctx-size
             - "16384"
             - --parallel
             - "1"
             - --ubatch-size
             - "1024"
+            # The default is -1, infinity: nothing on the server caps how long
+            # a request generates, and paperless sets no budget of its own and
+            # waits. A classification that needs ~300 tokens ran to 10269 at a
+            # healthy 11 tok/s, and three of the first eleven documents did it
+            # — 900 s each, most of an hour spent on four documents.
+            #
+            # 768 is generous against the ~300 a correct answer takes, and a
+            # truncated response is better evidence than a timeout: it shows
+            # what the model was doing instead of stopping.
+            - --n-predict
+            - "768"
+            # Without this, Qwen3.6 spends the response on reasoning_content
+            # and returns empty content — 983 tokens into a classification that
+            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
+            # slow or broken model and is neither.
+            #
+            # A candidate whose template has no such switch has to be measured
+            # as it comes, and that is a fair part of its result: paperless
+            # gives the model no token budget and waits.
+            - --chat-template-kwargs
+            - '{"enable_thinking": false}'
             - --no-webui
           env:
             - name: LLAMA_CACHE