claude-botandClaude Opus 5
a7bba7acbb
The point of this harness is comparing chat models, and it was shaped for comparing engines. Four candidates meant four 300-line copies of the same Job, which is exactly how --chat-template-kwargs ended up on the isolation pod but not on the first eval job — two hours of empty completions before anyone looked. So: job-template.yaml holds the sidecar configuration once, candidates.tsv holds the five values that differ, and run-candidate.sh substitutes them. Adding a model is a line, not a file. Candidates, all first-party GGUFs verified to exist with a Q4_K_M: ministral8b mistralai/Ministral-3-8B-Instruct-2512 ministral3b mistralai/Ministral-3-3B-Instruct-2512 granite8b ibm-granite/granite-3.3-8b-instruct Instruct, never Reasoning — a model tuned to think first does by design what cost Qwen3.6 two of fifty documents. Sizes are chosen because decode is bandwidth-bound at ~11 tok/s and prefill was 51% of the first run's wall clock, so a smaller dense model gains at both ends; 3B against 8B is the size ablation, granite against ministral the family one. The classifier stays in every run as a fixed reference line, not as a quality bar: train() fits on every non-inbox document with no holdout, so the evaluation documents are in its training set and its 100% on document type is memorisation. Irrelevant to ranking models — same constant under all of them — and the README now says so rather than inviting the comparison. First result, Qwen3.6-35B-A3B Q4_K_M over 50 documents: correspondent 70.8 offered / 75.0 recognised against the classifier's 72.0, document type 83.3 against 100, and 33.3 against 100 on the "Sonstige" catch-all. 5 rescued against 5 broken — but two of those five are difflib rejecting a correct answer, "GOTOOLS GmbH" against the stored "GOTOOLS", an exact prefix scoring ~0.74 under MATCH_THRESHOLD = 0.8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.7 KiB
Comparing chat models on Paperless suggestions
Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
against documents a human already filed, in the paperless-eval namespace
rather than against the live archive.
Applied by hand, not by Flux. A Job is immutable, so re-running a
configuration means deleting and recreating it, and a Flux-managed Job would
simply come back. The namespace itself is managed
(kubernetes/apps/paperless-eval/); only the measurement is not.
Running one
./run-candidate.sh ministral8b
kubectl -n paperless-eval logs -f job/eval-ministral8b -c eval
Candidates live in candidates.tsv; the runner substitutes the model-specific
values into job-template.yaml and refreshes the eval.py ConfigMap. Adding a
candidate is a line in that file, not a copy of the manifest — the sidecar
configuration deliberately exists once.
Two run at a time: the pod anti-affinity puts one per node, and only two nodes have a free iGPU. A third waits.
Re-running after an edit needs the old Job gone first:
kubectl -n paperless-eval delete job eval-<label>.
Results land on the paperless-eval-results PVC as <label>.jsonl (one row per
document) and <label>.summary.json, and in the pod log.
What is being compared
The candidates against each other. The scikit-learn classifier runs in every
job as a fixed reference line — it costs 15 seconds and keeps running in
production whatever the LLM does, so llm_rescued against llm_only_wrong
says what this model adds on top of it.
Do not read the classifier's own score as a quality bar. train() fits on every
non-inbox document with no holdout, so the evaluation documents are in its
training set and its numbers are memorisation. That does not matter for ranking
models — it is the same constant under all of them.
Reading the output
correspondent_offered against correspondent_recognised is the
MATCH_THRESHOLD = 0.8 gap: the model names the right entity and difflib
rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.
type_specific against type_catchall splits on whether the true type is
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
semantic content. A model reading the document has to infer "none of the above".
failed counts documents where the model produced nothing usable. With
--n-predict 768 in place, that is mostly non-termination: paperless reports
the truncated response as a flat Invalid AI configuration and discards the
content, so what the model was doing is visible only in the llama-server log.
Results so far
| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
|---|---|---|---|---|---|---|---|
| classifier (reference) | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 15 |
| Qwen3.6-35B-A3B Q4_K_M | 70.8 | 75.0 | 83.3 | 94.9 | 33.3 | 2 | 5206 |
Qwen3.6 complement: 29 both right, 5 rescued, 5 broken, 9 both wrong — but two of the five "broken" are the difflib gap above, not model errors.
What to watch for
Prompt length. Past roughly 14k tokens this stack stops producing language and starts producing garbage or nothing, without any error — and a single such request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit near 5k, but the 252 documents over 14000 characters push toward the boundary with their RAG neighbours. A run that starts scoring zeros mid-way is more likely this than the model.
Ground truth. It is the current filing, and over ten years conventions change. A disagreement with an old document is not automatically an error.