claude-botandClaude Opus 5
220bf6bfc3
The tests have to get better, and this is the change that would have caught
today's whole error class on document one instead of after six candidates.
Every AI run now asks "Nenne die Hauptstadt von Frankreich" before the
documents, every ten documents, and again when it aborts. No Paris, no run.
Why it goes mid-run and not only at the start: failures are not always
clustered. Ministral-3 3B failed 29 of 50 scattered through the run, which a
start-of-run check would have passed straight through. A slot that dies at
document 12 otherwise produces 38 more "failures" that are one event.
What the harness did wrong, so it is written down rather than relearned:
- it measured classification accuracy while a confound (slot corruption)
dominated, and never established that it could tell the two apart
- it counted sequential failures on a shared slot as independent evidence
- it recorded status codes and token counts and never once looked at what the
model actually emitted — twice the answer was one curl away
- it treated my own bounds, --n-predict 768 and a 300 s timeout, as
properties of the models hitting them
A run whose canary breaks is void, not a result. Failures adjacent to the break
are one fault, and probe-length.sh says where the model's boundary really is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
199 lines
9.3 KiB
Markdown
199 lines
9.3 KiB
Markdown
# Comparing chat models on Paperless suggestions
|
|
|
|
Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
|
|
against documents a human already filed, in the `paperless-eval` namespace
|
|
rather than against the live archive.
|
|
|
|
**Applied by hand, not by Flux.** A Job is immutable, so re-running a
|
|
configuration means deleting and recreating it, and a Flux-managed Job would
|
|
simply come back. The namespace itself *is* managed
|
|
(`kubernetes/apps/paperless-eval/`); only the measurement is not.
|
|
|
|
## Running one
|
|
|
|
Screen first, always:
|
|
|
|
```sh
|
|
./run-candidate.sh qwen3-8b 5 # ~5 min to a rejection, ~10 to a pass
|
|
kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval
|
|
|
|
kubectl -n paperless-eval delete job eval-qwen3-8b
|
|
./run-candidate.sh qwen3-8b # the real measurement, 50 documents
|
|
```
|
|
|
|
Three of the first four candidates could not do the task at all, and each was
|
|
left running for an hour before anyone looked. eval.py now stops after three
|
|
consecutive failures from the first document, so a screening run rejects a
|
|
hopeless candidate in about five minutes.
|
|
|
|
Screening uses genuine paperless requests. That matters — `probe-toolcall.sh`
|
|
is cheaper still and worth a first pass, but it passed two candidates that then
|
|
failed on document one, because a synthesised request cannot reproduce the RAG
|
|
neighbours and taxonomy candidate lists that make a real prompt what it is.
|
|
|
|
Never compare numbers across sample sizes. A screen is a gate; the comparison
|
|
happens at 50.
|
|
|
|
Candidates live in `candidates.tsv`; the runner substitutes the model-specific
|
|
values into `job-template.yaml` and refreshes the `eval.py` ConfigMap. Adding a
|
|
candidate is a line in that file, not a copy of the manifest — the sidecar
|
|
configuration deliberately exists once.
|
|
|
|
Two run at a time: the pod anti-affinity puts one per node, and only two nodes
|
|
have a free iGPU. A third waits.
|
|
|
|
Re-running after an edit needs the old Job gone first:
|
|
`kubectl -n paperless-eval delete job eval-<label>`.
|
|
|
|
Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
|
|
document) and `<label>.summary.json`, and in the pod log.
|
|
|
|
## What is being compared
|
|
|
|
The candidates against each other. The scikit-learn classifier runs in every
|
|
job as a fixed reference line — it costs 15 seconds and keeps running in
|
|
production whatever the LLM does, so `llm_rescued` against `llm_only_wrong`
|
|
says what *this* model adds on top of it.
|
|
|
|
Do not read the classifier's own score as a quality bar. `train()` fits on every
|
|
non-inbox document with no holdout, so the evaluation documents are in its
|
|
training set and its numbers are memorisation. That does not matter for ranking
|
|
models — it is the same constant under all of them.
|
|
|
|
## Reading the output
|
|
|
|
`correspondent_offered` against `correspondent_recognised` is the
|
|
`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
|
|
rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
|
|
"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.
|
|
|
|
`type_specific` against `type_catchall` splits on whether the true type is
|
|
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
|
|
semantic content. A model reading the document has to infer "none of the above".
|
|
|
|
`failed` counts documents where the model produced nothing usable. With
|
|
`--n-predict 768` in place, that is mostly non-termination: paperless reports
|
|
the truncated response as a flat `Invalid AI configuration` and discards the
|
|
content, so what the model was doing is visible only in the llama-server log.
|
|
|
|
## Results so far
|
|
|
|
| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
|
|
|---|---|---|---|---|---|---|---|
|
|
| *classifier (reference)* | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 18 |
|
|
| Qwen3.6-35B-A3B Q4_K_M | 72.9 | 77.1 | 85.4 | 94.9 | 44.4 | 2 | 5494 |
|
|
| Ministral-3 3B Q4_K_M | 71.4 | 76.2 | 76.2 | 83.3 | 33.3 | **29** | 5511 |
|
|
| Ministral-3 8B Q4_K_M | — | — | — | — | — | 3/3 | aborted |
|
|
| Granite 3.3 8B Q4_K_M | — | — | — | — | — | all | not measurable |
|
|
|
|
Read the Ministral-3 3B row with its failure count, not without it. It scored 21
|
|
documents and failed 29 — and on those 21 its correspondent numbers sit within
|
|
the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents
|
|
it could answer at all, which is a set selected for being easy, and the
|
|
classifier does better on them too (11 of 21 both right against 30 of 48).
|
|
A model that declines three documents in five has no usable accuracy number.
|
|
|
|
Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two
|
|
of those four "broken" are the difflib gap above, not model errors.
|
|
|
|
### How big a difference means anything
|
|
|
|
Qwen3.6 was measured twice over the same 50 documents, once with the deprecated
|
|
--chat-template-kwargs and once with --reasoning off:
|
|
|
|
70.8 / 75.0 / 83.3 / 94.9 / 33.3 two failures
|
|
72.9 / 77.1 / 85.4 / 94.9 / 44.4 two failures
|
|
|
|
Every one of those gaps is a single document out of 48 — the catchall one is
|
|
one document out of nine. So that is the noise floor: **a difference between
|
|
two candidates smaller than about two documents says nothing.** Decoding is
|
|
sampled, and the same model on the same input does not give the same answer
|
|
twice.
|
|
|
|
Worth knowing before reading a 2-point lead as a result.
|
|
|
|
## The instrument is checked, not assumed
|
|
|
|
Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the
|
|
documents, every ten documents, and again if it aborts. If Paris does not come
|
|
back, the run stops and says so.
|
|
|
|
This exists because a whole day of results turned out to be measuring a broken
|
|
instrument. On this Vulkan stack a prompt past some model-specific length stops
|
|
llama.cpp producing language — it repeats one token, " de de de" or
|
|
" phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop"
|
|
and nothing in the log. The slot then stays broken for every request after it
|
|
until the pod restarts.
|
|
|
|
So a run against a poisoned slot reports fifty failures that are one event, and
|
|
reads exactly like a model that cannot classify. Six candidates were written up
|
|
that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris."
|
|
perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its
|
|
recorded classification scores described a corrupted slot and nothing else.
|
|
|
|
**A run whose canary breaks is void, not a result.** Failures adjacent to the
|
|
break are one fault. Use `probe-length.sh` to find where the model's boundary
|
|
actually sits before judging it — paperless needs ~5300 tokens.
|
|
|
|
## Bounds of the harness, not of the model
|
|
|
|
Two numbers in here are mine and both have produced misleading results:
|
|
|
|
`--n-predict` caps generation server-side, default 768. A candidate that stops
|
|
exactly there has hit MY bound. Six candidates were written up as incapable on
|
|
that signal before anyone checked whether they would have finished at 900
|
|
tokens. A working answer takes 241-291 tokens, so 768 looked generous — and
|
|
"looked generous" is not evidence.
|
|
|
|
`EVAL_TIMEOUT` bounds the client, default 300 s. Raising the token cap alone
|
|
just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of
|
|
generation, so the run fails with TimeoutError instead. Raise both together.
|
|
|
|
./run-candidate.sh <label> <sample> <n_predict> <timeout>
|
|
./run-candidate.sh gemma4-26b-a4b 3 4096 900
|
|
|
|
The Q8 precision comparison had to be thrown away over this. Q8's uncapped
|
|
answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same
|
|
task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run
|
|
looked like "Q8 classifies worse" when it measured "Q8 is wordier than my
|
|
bound". Before comparing two configurations, check that neither is sitting on
|
|
the cap.
|
|
|
|
## What to watch for
|
|
|
|
**Prompt length.** Past roughly 14k tokens this stack stops producing language
|
|
and starts producing garbage or nothing, without any error — and a single such
|
|
request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
|
|
near 5k, but the 252 documents over 14000 characters push toward the boundary
|
|
with their RAG neighbours. A run that starts scoring zeros mid-way is more
|
|
likely this than the model.
|
|
|
|
**Ground truth.** It is the current filing, and over ten years conventions
|
|
change. A disagreement with an old document is not automatically an error.
|
|
|
|
## The models volume fills up
|
|
|
|
Each candidate leaves its weights on the shared `llama-models` PVC — 2.5 GB for
|
|
a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB
|
|
and nine screened candidates filled it completely.
|
|
|
|
A full volume does not announce itself. llama-server cannot write its metadata
|
|
cache, so it cannot resolve the repository listing, and says:
|
|
|
|
no GGUF files found in repository <repo>
|
|
Available GGUF files:
|
|
- <whatever it had cached before the disk filled>
|
|
|
|
which reads as a wrong model reference. The pod then sits in
|
|
`Init:CrashLoopBackOff`, which from outside is indistinguishable from a slow
|
|
34 GB download. Two jobs spent three hours that way.
|
|
|
|
**Check `df -h /models` before believing anything llama-server says about a
|
|
repository.** Clear out rejected candidates when the volume gets tight — the
|
|
weights are re-downloadable, so deleting them costs only bandwidth:
|
|
|
|
```sh
|
|
kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
|
|
sh -c 'cd /models && du -sh * | sort -h'
|
|
```
|