fix(paperless-eval): bound generation server-side; the model does not always stop

Three of the first eleven documents in a run never terminated. One reached
10269 tokens on a classification that needs ~300, at a perfectly healthy
11 tok/s — so it read as slowness and was neither slow nor a context problem.

llama-server's --n-predict defaults to -1, infinity, and paperless sets no
token budget of its own and waits. Nothing anywhere bounded the request. Four
documents cost most of an hour.

--n-predict 768 on the sidecar caps it, and a truncated response is better
evidence than a timeout: it shows what the model was doing instead of
stopping. The client timeout drops to 300 s and becomes a backstop.

Also:

- the degeneracy check fired on 'Rechnung 2025-5425541' — 8 letters in 21
  characters, below the 40% Latin threshold. Digits and punctuation are normal
  in a title; look for characters from other scripts directly instead of
  inferring them from a shortage of Latin ones.

- the ctx-size comment still carried the claim bisection disproved in #777,
  that 32768 corrupts inference. The boundary is prompt length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/llama-server/deployment.yaml

@@ -75,19 +75,37 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 16k, and going above it corrupts inference rather than failing.
-            # At 32768 per slot this server answered a 650-token classification
-            # prompt with 1025 tokens of "!" — the KV cache producing numbers
-            # rather than language. It does not crash, it does not warn, and
-            # the OpenAI response looks well-formed, so the damage only shows
-            # in what callers get back: Paperless ran past its 300 s timeout,
-            # and Home Assistant got an empty completion and a hallucinated
-            # device list.
+            # 16k is room to spare, not a safety limit. The limit that matters
+            # is not set here at all.
             #
-            # The same Vulkan/Mesa stack already fails hard at --ubatch-size
-            # 2048; this is the quiet version of that limit. Raise this only
-            # against a generation that is read, not against a server that
-            # merely starts.
+            # Isolated by bisection on a scratch pod: this stack breaks on
+            # PROMPT length, and the value of --ctx-size makes no difference to
+            # it. Measured at --ctx-size 65536 with a single slot, so with four
+            # times the headroom:
+            #
+            #     13588 prompt tokens   coherent answer
+            #     15086                 one token, immediate EOS, empty content
+            #     16088                 CJK characters and digit runs
+            #
+            # A single request past that boundary then poisons the slot: the
+            # same 13588-token prompt that answered cleanly a minute earlier
+            # returns one empty token afterwards, and stays broken until the
+            # pod restarts. That is why it looked intermittent, and why a
+            # 650-token classification once came back as 1025 tokens of "!" —
+            # it followed Home Assistant's ~19k prompts through the same slot.
+            #
+            # Nothing reports any of it: finish_reason "stop", a well-formed
+            # OpenAI response, HTTP 200, /health green.
+            #
+            # So the operating rule is about callers, not this flag. Paperless
+            # sits near 5k and is safe; Home Assistant's Assist prompt measured
+            # 18910 and never had a chance. An earlier version of this comment
+            # blamed --ctx-size 32768, which the bisection disproves — 32768
+            # and 65536 both answer correctly as long as the prompt stays under
+            # the boundary.
+            #
+            # The same stack also fails hard at --ubatch-size 2048, which is
+            # the loud version of a limit this one keeps quiet about.
             #
             - --ctx-size
             - "16384"

kubernetes/apps/paperless-eval/kustomization.yaml

@@ -1,17 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: paperless-eval
+# The instance only. The measurement harness lives in scripts/paperless-eval/
+# and is applied by hand — a Job is immutable, re-running one means deleting and
+# recreating it, and Flux would just put the old one back. Nothing here should
+# converge on a finished measurement.
 resources:
   - secrets.yaml
   - cluster.yaml
   - pvcs.yaml
   - redis.yaml
   - paperless.yaml
-  - job-qwen35.yaml
-configMapGenerator:
-  # Hash-suffixed, so editing eval.py produces a new ConfigMap name and any Job
-  # referencing it has to be recreated rather than silently running the old
-  # script against a new name.
-  - name: paperless-eval-script
-    files:
-      - eval.py

scripts/paperless-eval/README.md

@@ -0,0 +1,72 @@
+# Measuring Paperless suggestions against the real corpus
+
+Compares the two engines Paperless ships — the scikit-learn classifier and the
+LLM path — over documents whose labels a human already chose, in the
+`paperless-eval` namespace rather than against the live archive.
+
+**Applied by hand, not by Flux.** A Job is immutable, so re-running a
+configuration means deleting and recreating it, and a Flux-managed Job would
+simply come back. The namespace itself *is* managed
+(`kubernetes/apps/paperless-eval/`); only the measurement is not.
+
+## Running one
+
+```sh
+kubectl -n paperless-eval create configmap paperless-eval-script \
+  --from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
+  | kubectl apply -f -
+
+kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
+kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
+```
+
+Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
+document) and `<label>.summary.json`, and in the pod log.
+
+Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
+the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
+
+## Adding a candidate
+
+Copy `job-qwen35.yaml`, then change four things: the Job name, the
+`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
+the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
+`eval` container (it names the output files).
+
+Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
+and k8s-2 have a free iGPU. A third waits.
+
+## Reading the output
+
+`complement` in the summary is the number the exercise exists for. The two
+engines are not alternatives: the classifier costs 0.6 s per document and keeps
+running whatever the LLM does, so what matters is what the LLM adds where the
+classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
+percentages cannot show that; 74% against 72% could overlap almost entirely or
+barely at all.
+
+`correspondent_offered` against `correspondent_recognised` is the
+`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
+rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
+stored "Generali AG".
+
+`type_specific` against `type_catchall` splits on whether the true type is
+"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
+semantic content. The classifier can memorise it; a model reading the document
+has to infer "none of the above".
+
+## What to watch for
+
+Two things make a result look like a bad model when it is not:
+
+**Prompt length.** Past roughly 14k tokens this stack stops producing language
+and starts producing garbage or nothing, without any error — and a single such
+request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
+near 5k, but the 252 documents over 14000 characters push toward the boundary
+with their RAG neighbours. A run that starts scoring zeros mid-way is more
+likely this than the model.
+
+**Ground truth.** It is the current filing, and over ten years conventions
+change. A disagreement with an old document is not automatically an error. The
+comparison between engines is the durable result; the absolute numbers are
+indicative.

scripts/paperless-eval/eval.py

@@ -78,10 +78,15 @@ ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
 LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
 OUT_DIR = os.environ.get("EVAL_OUT", "/results")
 
-# Long enough that a slow document is measured rather than abandoned. A client
-# timeout does not stop the generation behind it, and with a single llama.cpp
-# slot an abandoned request holds the queue for everything after it.
-REQUEST_TIMEOUT = 900
+# A backstop, not the real bound. Generation is capped server-side by
+# --n-predict on the llama-server sidecar, which is what actually stops a model
+# that will not terminate; this catches everything else — a wedged sidecar, a
+# request that never reaches it.
+#
+# 300 s against the ~73 s a healthy document takes end to end, prompt included.
+# It does work: llama-server logs `stop: cancel task` on the disconnect and
+# starts the next request 0.4 s later.
+REQUEST_TIMEOUT = 300
 
 CATCHALL_TYPE = "Sonstige"
 
@@ -165,7 +170,9 @@ def floor(conn) -> float:
             """,
         )
         top, total = cur.fetchone()
-    return 100.0 * top / total if total else 0.0
+    # float() on both: psycopg returns Postgres numerics as Decimal, and
+    # mixing that with a float literal raises TypeError rather than coercing.
+    return 100.0 * float(top) / float(total) if total else 0.0
 
 
 def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
@@ -198,6 +205,32 @@ def recognised(payload: dict, field: str, want_name: str) -> bool:
     return any(want <= normalise(name) for name in payload.get(field) or [])
 
 
+def degenerate(payload: dict) -> str | None:
+    """Whether this looks like broken inference rather than a wrong guess.
+
+    Past roughly 14k prompt tokens this stack stops producing language — first
+    empty completions, then runs of digits and CJK characters — and one such
+    request poisons the llama.cpp slot until the pod restarts. Nothing in the
+    HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
+
+    Scored as a wrong answer it would silently depress a candidate's numbers,
+    and because the slot stays poisoned it would take the rest of the run with
+    it. Flagged instead, so the failure is visible as itself.
+    """
+    title = payload.get("title") or ""
+    if not title.strip():
+        return "empty title"
+    # Digits, punctuation and whitespace are normal in a document title —
+    # "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
+    # earlier version of this check. What broken inference produces is
+    # characters from other scripts, so look for those directly rather than
+    # inferring them from a shortage of Latin ones.
+    foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
+    if foreign > 0.2 * len(title):
+        return f"non-latin title: {title[:40]!r}"
+    return None
+
+
 def score_document(payload: dict, doc: dict) -> dict:
     offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
     offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
@@ -235,6 +268,11 @@ def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
             log(f"  {doc['id']}: {error}")
         else:
             row.update(score_document(payload, doc))
+            # Only the AI path returns a title; the classifier answers with ids
+            # alone, so an "empty title" there is normal rather than broken.
+            if engine == "ai" and (broken := degenerate(payload)) is not None:
+                row["degenerate"] = broken
+                log(f"  {doc['id']}: DEGENERATE — {broken}")
         rows.append(row)
         if n % 10 == 0:
             log(f"  {n}/{len(docs)}")
@@ -246,17 +284,23 @@ def pct(hits: int, total: int) -> float:
 
 
 def summarise(rows: list[dict], engine: str) -> dict:
-    scored = [r for r in rows if r["engine"] == engine and "error" not in r]
-    failed = [r for r in rows if r["engine"] == engine and "error" in r]
+    mine = [r for r in rows if r["engine"] == engine]
+    # Degenerate rows are excluded from the accuracy figures: they measure the
+    # backend giving up, not the model choosing wrongly, and leaving them in
+    # would quietly punish whichever candidate happened to trip the slot.
+    scored = [r for r in mine if "error" not in r and "degenerate" not in r]
+    failed = [r for r in mine if "error" in r]
+    broken = [r for r in mine if "degenerate" in r]
     specific = [r for r in scored if not r["catchall_type"]]
     catchall = [r for r in scored if r["catchall_type"]]
     return {
         "engine": engine,
         "scored": len(scored),
         "failed": len(failed),
-        "seconds_total": round(
-            sum(r["seconds"] for r in rows if r["engine"] == engine),
-        ),
+        # Non-zero means the run is suspect from that point on, not that the
+        # model is bad — the slot stays poisoned until the pod restarts.
+        "degenerate": len(broken),
+        "seconds_total": round(sum(r["seconds"] for r in mine)),
         "correspondent_offered": pct(
             sum(r["correspondent_offered"] for r in scored), len(scored)
         ),
@@ -281,7 +325,8 @@ def complement(rows: list[dict]) -> dict | None:
     """
     by_engine = {"classic": {}, "ai": {}}
     for row in rows:
-        if "error" not in row and row["engine"] in by_engine:
+        usable = "error" not in row and "degenerate" not in row
+        if usable and row["engine"] in by_engine:
             by_engine[row["engine"]][row["document_id"]] = row
     both = set(by_engine["classic"]) & set(by_engine["ai"])
     if not both or not by_engine["ai"]:

scripts/paperless-eval/job-qwen35.yaml

@@ -69,9 +69,27 @@ spec:
             - -c
             - |
               set -e
-              cp -a /golden/. /data/
-              echo "seeded /data:"
-              du -sh /data/index /data/llm_index 2>/dev/null || true
+              # -r, not -a: preserving ownership needs CHOWN, which this
+              # container does not have, and cp then exits non-zero after
+              # copying the files anyway. Running as 1000 makes the copy land
+              # with the ownership paperless needs — it writes to the LLM index
+              # while serving.
+              cp -r /golden/. /data/
+              du -sh /data/index /data/llm_index
+              # Refuse to start rather than measure the wrong thing. An empty
+              # llm_index makes retrieve_similar_nodes return nothing, and the
+              # run would report a RAG result having never retrieved anything —
+              # which is exactly what the first attempt at this job did, because
+              # the golden volume had not been populated yet. The index is
+              # ~125 MiB once built; anything under 1 MiB is not one.
+              bytes=$(du -sb /data/llm_index | cut -f1)
+              if [ "$bytes" -lt 1000000 ]; then
+                echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
+                echo "smoke-test deployment first:" >&2
+                echo "  kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
+                echo "    python manage.py document_llmindex rebuild" >&2
+                exit 1
+              fi
           volumeMounts:
             - name: golden
               mountPath: /golden
@@ -79,6 +97,8 @@ spec:
             - name: data
               mountPath: /data
           securityContext:
+            runAsUser: 1000
+            runAsGroup: 1000
             allowPrivilegeEscalation: false
             capabilities:
               drop: ["ALL"]
@@ -94,17 +114,38 @@ spec:
             - "99"
             - --threads
             - "4"
-            # The production ceiling, and for the same reason: 32768 per slot
-            # corrupts inference on this Vulkan stack — 1025 tokens of "!"
-            # against a 650-token prompt, with /props still reporting the
-            # requested size. A candidate needing more context than this cannot
-            # be measured here, and that is itself a result.
+            # The production ceiling. Not because the flag value itself breaks
+            # anything — bisection showed the corruption boundary is prompt
+            # length, around 14000 tokens, whatever --ctx-size says — but
+            # because measuring against a different ceiling than production
+            # runs would not describe production.
             - --ctx-size
             - "16384"
             - --parallel
             - "1"
             - --ubatch-size
             - "1024"
+            # The default is -1, infinity: nothing on the server caps how long
+            # a request generates, and paperless sets no budget of its own and
+            # waits. A classification that needs ~300 tokens ran to 10269 at a
+            # healthy 11 tok/s, and three of the first eleven documents did it
+            # — 900 s each, most of an hour spent on four documents.
+            #
+            # 768 is generous against the ~300 a correct answer takes, and a
+            # truncated response is better evidence than a timeout: it shows
+            # what the model was doing instead of stopping.
+            - --n-predict
+            - "768"
+            # Without this, Qwen3.6 spends the response on reasoning_content
+            # and returns empty content — 983 tokens into a classification that
+            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
+            # slow or broken model and is neither.
+            #
+            # A candidate whose template has no such switch has to be measured
+            # as it comes, and that is a fair part of its result: paperless
+            # gives the model no token budget and waits.
+            - --chat-template-kwargs
+            - '{"enable_thinking": false}'
             - --no-webui
           env:
             - name: LLAMA_CACHE