claude-botandClaude Opus 5
c2f47e58bc
I wrote in the last commit that passing '{"enable_thinking": false}' to every
candidate was a no-op on templates that never read the variable. That is
wrong, and it cost a run.
Ministral-3-8B produced token salad on a 681-token prompt — far below the ~14k
corruption boundary, so not that defect:
1UnComplete1Self1UnCompleteSDirect11S1S111Document1CompleteFromS11Un...
Every document failed, and each one generated to the --n-predict cap. The
identical request against a server started without the flag returned a clean
tool call and the correct answer. Plain German was fine throughout — "Die
Hauptstadt von Frankreich ist Paris" — so it only ever broke with tools, which
is what made it look like a model that cannot do function calling.
llama-server prints "Setting 'enable_thinking' via --chat-template-kwargs is
deprecated. Use --reasoning on / --reasoning off instead." at startup. It means
it. --reasoning off is verified to give Ministral a proper tool_calls response
before this went near a real run.
Qwen3.6's template does consume enable_thinking, so its measured result stands
— but its run used the old flag, and the configuration should be uniform
across candidates before the numbers are compared. Worth a re-run.
granite8b is commented out of candidates.tsv, and this one is not my flag: it
classifies correctly and llama.cpp b10731 simply cannot read its tool calls
back, returning them as content with finish_reason "stop". Reproduced on an
idle server both with and without the flag. Details are in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comparing chat models on Paperless suggestions
Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
against documents a human already filed, in the paperless-eval namespace
rather than against the live archive.
Applied by hand, not by Flux. A Job is immutable, so re-running a
configuration means deleting and recreating it, and a Flux-managed Job would
simply come back. The namespace itself is managed
(kubernetes/apps/paperless-eval/); only the measurement is not.
Running one
./run-candidate.sh ministral8b
kubectl -n paperless-eval logs -f job/eval-ministral8b -c eval
Candidates live in candidates.tsv; the runner substitutes the model-specific
values into job-template.yaml and refreshes the eval.py ConfigMap. Adding a
candidate is a line in that file, not a copy of the manifest — the sidecar
configuration deliberately exists once.
Two run at a time: the pod anti-affinity puts one per node, and only two nodes have a free iGPU. A third waits.
Re-running after an edit needs the old Job gone first:
kubectl -n paperless-eval delete job eval-<label>.
Results land on the paperless-eval-results PVC as <label>.jsonl (one row per
document) and <label>.summary.json, and in the pod log.
What is being compared
The candidates against each other. The scikit-learn classifier runs in every
job as a fixed reference line — it costs 15 seconds and keeps running in
production whatever the LLM does, so llm_rescued against llm_only_wrong
says what this model adds on top of it.
Do not read the classifier's own score as a quality bar. train() fits on every
non-inbox document with no holdout, so the evaluation documents are in its
training set and its numbers are memorisation. That does not matter for ranking
models — it is the same constant under all of them.
Reading the output
correspondent_offered against correspondent_recognised is the
MATCH_THRESHOLD = 0.8 gap: the model names the right entity and difflib
rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.
type_specific against type_catchall splits on whether the true type is
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
semantic content. A model reading the document has to infer "none of the above".
failed counts documents where the model produced nothing usable. With
--n-predict 768 in place, that is mostly non-termination: paperless reports
the truncated response as a flat Invalid AI configuration and discards the
content, so what the model was doing is visible only in the llama-server log.
Results so far
| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
|---|---|---|---|---|---|---|---|
| classifier (reference) | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 15 |
| Qwen3.6-35B-A3B Q4_K_M | 70.8 | 75.0 | 83.3 | 94.9 | 33.3 | 2 | 5206 |
Qwen3.6 complement: 29 both right, 5 rescued, 5 broken, 9 both wrong — but two of the five "broken" are the difflib gap above, not model errors.
What to watch for
Prompt length. Past roughly 14k tokens this stack stops producing language and starts producing garbage or nothing, without any error — and a single such request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit near 5k, but the 252 documents over 14000 characters push toward the boundary with their RAG neighbours. A run that starts scoring zeros mid-way is more likely this than the model.
Ground truth. It is the current filing, and over ten years conventions change. A disagreement with an old document is not automatically an error.