homelab/scripts/paperless-eval

Comparing chat models on Paperless suggestions

Which local LLM is worth running behind paperless-ngx's AI suggestions, measured against documents a human already filed, in the paperless-eval namespace rather than against the live archive.

Applied by hand, not by Flux. A Job is immutable, so re-running a configuration means deleting and recreating it, and a Flux-managed Job would simply come back. The namespace itself is managed (kubernetes/apps/paperless-eval/); only the measurement is not.

Running one

Screen first, always:

./run-candidate.sh qwen3-8b 5          # ~5 min to a rejection, ~10 to a pass
kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval

kubectl -n paperless-eval delete job eval-qwen3-8b
./run-candidate.sh qwen3-8b            # the real measurement, 50 documents

Three of the first four candidates could not do the task at all, and each was left running for an hour before anyone looked. eval.py now stops after three consecutive failures from the first document, so a screening run rejects a hopeless candidate in about five minutes.

Screening uses genuine paperless requests. That matters — probe-toolcall.sh is cheaper still and worth a first pass, but it passed two candidates that then failed on document one, because a synthesised request cannot reproduce the RAG neighbours and taxonomy candidate lists that make a real prompt what it is.

Never compare numbers across sample sizes. A screen is a gate; the comparison happens at 50.

Candidates live in candidates.tsv; the runner substitutes the model-specific values into job-template.yaml and refreshes the eval.py ConfigMap. Adding a candidate is a line in that file, not a copy of the manifest — the sidecar configuration deliberately exists once.

Two run at a time: the pod anti-affinity puts one per node, and only two nodes have a free iGPU. A third waits.

Re-running after an edit needs the old Job gone first: kubectl -n paperless-eval delete job eval-<label>.

Results land on the paperless-eval-results PVC as <label>.jsonl (one row per document) and <label>.summary.json, and in the pod log.

What is being compared

The candidates against each other. The scikit-learn classifier runs in every job as a fixed reference line — it costs 15 seconds and keeps running in production whatever the LLM does, so llm_rescued against llm_only_wrong says what this model adds on top of it.

Do not read the classifier's own score as a quality bar. train() fits on every non-inbox document with no holdout, so the evaluation documents are in its training set and its numbers are memorisation. That does not matter for ranking models — it is the same constant under all of them.

Reading the output

correspondent_offered against correspondent_recognised is the MATCH_THRESHOLD = 0.8 gap: the model names the right entity and difflib rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored "GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.

type_specific against type_catchall splits on whether the true type is "Sonstige", which holds 1133 of ~3900 documents and is the one label with no semantic content. A model reading the document has to infer "none of the above".

failed counts documents where the model produced nothing usable. With --n-predict 768 in place, that is mostly non-termination: paperless reports the truncated response as a flat Invalid AI configuration and discards the content, so what the model was doing is visible only in the llama-server log.

Results so far

model	corr. offered	corr. recognised	type	specific	catchall	failed	seconds
classifier (reference)	72.0	72.0	100.0	100.0	100.0	0	18
Qwen3.6-35B-A3B Q4_K_M	72.9	77.1	85.4	94.9	44.4	2	5494
Ministral-3 3B Q4_K_M	71.4	76.2	76.2	83.3	33.3	29	5511
Ministral-3 8B Q4_K_M	—	—	—	—	—	3/3	aborted
Granite 3.3 8B Q4_K_M	—	—	—	—	—	all	not measurable

Read the Ministral-3 3B row with its failure count, not without it. It scored 21 documents and failed 29 — and on those 21 its correspondent numbers sit within the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents it could answer at all, which is a set selected for being easy, and the classifier does better on them too (11 of 21 both right against 30 of 48). A model that declines three documents in five has no usable accuracy number.

Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two of those four "broken" are the difflib gap above, not model errors.

How big a difference means anything

Qwen3.6 was measured twice over the same 50 documents, once with the deprecated --chat-template-kwargs and once with --reasoning off:

70.8 / 75.0 / 83.3 / 94.9 / 33.3   two failures
72.9 / 77.1 / 85.4 / 94.9 / 44.4   two failures

Every one of those gaps is a single document out of 48 — the catchall one is one document out of nine. So that is the noise floor: a difference between two candidates smaller than about two documents says nothing. Decoding is sampled, and the same model on the same input does not give the same answer twice.

Worth knowing before reading a 2-point lead as a result.

The instrument is checked, not assumed

Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the documents, every ten documents, and again if it aborts. If Paris does not come back, the run stops and says so.

This exists because a whole day of results turned out to be measuring a broken instrument. On this Vulkan stack a prompt past some model-specific length stops llama.cpp producing language — it repeats one token, " de de de" or " phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop" and nothing in the log. The slot then stays broken for every request after it until the pod restarts.

So a run against a poisoned slot reports fifty failures that are one event, and reads exactly like a model that cannot classify. Six candidates were written up that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris." perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its recorded classification scores described a corrupted slot and nothing else.

A run whose canary breaks is void, not a result. Failures adjacent to the break are one fault. Use probe-length.sh to find where the model's boundary actually sits before judging it — paperless needs ~5300 tokens.

Bounds of the harness, not of the model

Two numbers in here are mine and both have produced misleading results:

--n-predict caps generation server-side, default 768. A candidate that stops exactly there has hit MY bound. Six candidates were written up as incapable on that signal before anyone checked whether they would have finished at 900 tokens. A working answer takes 241-291 tokens, so 768 looked generous — and "looked generous" is not evidence.

EVAL_TIMEOUT bounds the client, default 300 s. Raising the token cap alone just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of generation, so the run fails with TimeoutError instead. Raise both together.

./run-candidate.sh <label> <sample> <n_predict> <timeout>
./run-candidate.sh gemma4-26b-a4b 3 4096 900

The Q8 precision comparison had to be thrown away over this. Q8's uncapped answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run looked like "Q8 classifies worse" when it measured "Q8 is wordier than my bound". Before comparing two configurations, check that neither is sitting on the cap.

What to watch for

Prompt length. Past roughly 14k tokens this stack stops producing language and starts producing garbage or nothing, without any error — and a single such request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit near 5k, but the 252 documents over 14000 characters push toward the boundary with their RAG neighbours. A run that starts scoring zeros mid-way is more likely this than the model.

Ground truth. It is the current filing, and over ten years conventions change. A disagreement with an old document is not automatically an error.

The models volume fills up

Each candidate leaves its weights on the shared llama-models PVC — 2.5 GB for a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB and nine screened candidates filled it completely.

A full volume does not announce itself. llama-server cannot write its metadata cache, so it cannot resolve the repository listing, and says:

no GGUF files found in repository <repo>
Available GGUF files:
  - <whatever it had cached before the disk filled>

which reads as a wrong model reference. The pod then sits in Init:CrashLoopBackOff, which from outside is indistinguishable from a slow 34 GB download. Two jobs spent three hours that way.

Check df -h /models before believing anything llama-server says about a repository. Clear out rejected candidates when the volume gets tight — the weights are re-downloadable, so deleting them costs only bandwidth:

kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
  sh -c 'cd /models && du -sh * | sort -h'