# Comparing chat models on Paperless suggestions

Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
against documents a human already filed, in the `paperless-eval` namespace
rather than against the live archive.

**Applied by hand, not by Flux.** A Job is immutable, so re-running a
configuration means deleting and recreating it, and a Flux-managed Job would
simply come back. The namespace itself *is* managed
(`kubernetes/apps/paperless-eval/`); only the measurement is not.

## Running one

Screen first, always:

```sh
./run-candidate.sh qwen3-8b 5          # ~5 min to a rejection, ~10 to a pass
kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval

kubectl -n paperless-eval delete job eval-qwen3-8b
./run-candidate.sh qwen3-8b            # the real measurement, 50 documents
```

Three of the first four candidates could not do the task at all, and each was
left running for an hour before anyone looked. eval.py now stops after three
consecutive failures from the first document, so a screening run rejects a
hopeless candidate in about five minutes.

Screening uses genuine paperless requests. That matters — `probe-toolcall.sh`
is cheaper still and worth a first pass, but it passed two candidates that then
failed on document one, because a synthesised request cannot reproduce the RAG
neighbours and taxonomy candidate lists that make a real prompt what it is.

Never compare numbers across sample sizes. A screen is a gate; the comparison
happens at 50.

Candidates live in `candidates.tsv`; the runner substitutes the model-specific
values into `job-template.yaml` and refreshes the `eval.py` ConfigMap. Adding a
candidate is a line in that file, not a copy of the manifest — the sidecar
configuration deliberately exists once.

Two run at a time: the pod anti-affinity puts one per node, and only two nodes
have a free iGPU. A third waits.

Re-running after an edit needs the old Job gone first:
`kubectl -n paperless-eval delete job eval-<label>`.

Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
document) and `<label>.summary.json`, and in the pod log.

## What is being compared

The candidates against each other. The scikit-learn classifier runs in every
job as a fixed reference line — it costs 15 seconds and keeps running in
production whatever the LLM does, so `llm_rescued` against `llm_only_wrong`
says what *this* model adds on top of it.

Do not read the classifier's own score as a quality bar. `train()` fits on every
non-inbox document with no holdout, so the evaluation documents are in its
training set and its numbers are memorisation. That does not matter for ranking
models — it is the same constant under all of them.

## Reading the output

`correspondent_offered` against `correspondent_recognised` is the
`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.

`type_specific` against `type_catchall` splits on whether the true type is
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
semantic content. A model reading the document has to infer "none of the above".

`failed` counts documents where the model produced nothing usable. With
`--n-predict 768` in place, that is mostly non-termination: paperless reports
the truncated response as a flat `Invalid AI configuration` and discards the
content, so what the model was doing is visible only in the llama-server log.

## Results so far

| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
|---|---|---|---|---|---|---|---|
| *classifier (reference)* | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 18 |
| Qwen3.6-35B-A3B Q4_K_M | 72.9 | 77.1 | 85.4 | 94.9 | 44.4 | 2 | 5494 |
| Ministral-3 3B Q4_K_M | 71.4 | 76.2 | 76.2 | 83.3 | 33.3 | **29** | 5511 |
| Ministral-3 8B Q4_K_M | — | — | — | — | — | 3/3 | aborted |
| Granite 3.3 8B Q4_K_M | — | — | — | — | — | all | not measurable |

Read the Ministral-3 3B row with its failure count, not without it. It scored 21
documents and failed 29 — and on those 21 its correspondent numbers sit within
the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents
it could answer at all, which is a set selected for being easy, and the
classifier does better on them too (11 of 21 both right against 30 of 48).
A model that declines three documents in five has no usable accuracy number.

Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two
of those four "broken" are the difflib gap above, not model errors.

### How big a difference means anything

Qwen3.6 was measured twice over the same 50 documents, once with the deprecated
--chat-template-kwargs and once with --reasoning off:

    70.8 / 75.0 / 83.3 / 94.9 / 33.3   two failures
    72.9 / 77.1 / 85.4 / 94.9 / 44.4   two failures

Every one of those gaps is a single document out of 48 — the catchall one is
one document out of nine. So that is the noise floor: **a difference between
two candidates smaller than about two documents says nothing.** Decoding is
sampled, and the same model on the same input does not give the same answer
twice.

Worth knowing before reading a 2-point lead as a result.

## The instrument is checked, not assumed

Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the
documents, every ten documents, and again if it aborts. If Paris does not come
back, the run stops and says so.

This exists because a whole day of results turned out to be measuring a broken
instrument. On this Vulkan stack a prompt past some model-specific length stops
llama.cpp producing language — it repeats one token, " de de de" or
" phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop"
and nothing in the log. The slot then stays broken for every request after it
until the pod restarts.

So a run against a poisoned slot reports fifty failures that are one event, and
reads exactly like a model that cannot classify. Six candidates were written up
that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris."
perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its
recorded classification scores described a corrupted slot and nothing else.

**A run whose canary breaks is void, not a result.** Failures adjacent to the
break are one fault. Use `probe-length.sh` to find where the model's boundary
actually sits before judging it — paperless needs ~5300 tokens.

## Bounds of the harness, not of the model

Two numbers in here are mine and both have produced misleading results:

`--n-predict` caps generation server-side, default 768. A candidate that stops
exactly there has hit MY bound. Six candidates were written up as incapable on
that signal before anyone checked whether they would have finished at 900
tokens. A working answer takes 241-291 tokens, so 768 looked generous — and
"looked generous" is not evidence.

`EVAL_TIMEOUT` bounds the client, default 300 s. Raising the token cap alone
just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of
generation, so the run fails with TimeoutError instead. Raise both together.

    ./run-candidate.sh <label> <sample> <n_predict> <timeout>
    ./run-candidate.sh gemma4-26b-a4b 3 4096 900

The Q8 precision comparison had to be thrown away over this. Q8's uncapped
answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same
task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run
looked like "Q8 classifies worse" when it measured "Q8 is wordier than my
bound". Before comparing two configurations, check that neither is sitting on
the cap.

## What to watch for

**Prompt length.** Past roughly 14k tokens this stack stops producing language
and starts producing garbage or nothing, without any error — and a single such
request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
near 5k, but the 252 documents over 14000 characters push toward the boundary
with their RAG neighbours. A run that starts scoring zeros mid-way is more
likely this than the model.

**Ground truth.** It is the current filing, and over ten years conventions
change. A disagreement with an old document is not automatically an error.

## The models volume fills up

Each candidate leaves its weights on the shared `llama-models` PVC — 2.5 GB for
a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB
and nine screened candidates filled it completely.

A full volume does not announce itself. llama-server cannot write its metadata
cache, so it cannot resolve the repository listing, and says:

    no GGUF files found in repository <repo>
    Available GGUF files:
      - <whatever it had cached before the disk filled>

which reads as a wrong model reference. The pod then sits in
`Init:CrashLoopBackOff`, which from outside is indistinguishable from a slow
34 GB download. Two jobs spent three hours that way.

**Check `df -h /models` before believing anything llama-server says about a
repository.** Clear out rejected candidates when the volume gets tight — the
weights are re-downloadable, so deleting them costs only bandwidth:

```sh
kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
  sh -c 'cd /models && du -sh * | sort -h'
```