claude-botandClaude Opus 5
c47c2e6bc1
A Job is immutable and a measurement is finished when it finishes — neither is something Flux should converge on. Re-running a configuration means deleting and recreating the Job, and a managed one comes straight back. Worse, eval.py was a kustomize-generated ConfigMap, so editing the script between runs and applying it by hand would have been reverted on the next reconcile: the iteration loop the harness exists for was closed off by where it was put. The namespace stays managed. It holds a database clone, PVCs and a network policy — state that should converge and should vanish when the Kustomization is removed. Only the harness moves to scripts/, alongside the other tools nobody deploys, with a README covering how to run one, how to add a candidate, and how to read the output. Two failure modes are written down there rather than left to be rediscovered: prompts past ~14k tokens make this stack emit garbage silently and poison the slot until restart, and ground truth is a filing convention that has drifted over ten years, so the comparison between engines is the durable result rather than the absolute numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring Paperless suggestions against the real corpus
Compares the two engines Paperless ships — the scikit-learn classifier and the
LLM path — over documents whose labels a human already chose, in the
paperless-eval namespace rather than against the live archive.
Applied by hand, not by Flux. A Job is immutable, so re-running a
configuration means deleting and recreating it, and a Flux-managed Job would
simply come back. The namespace itself is managed
(kubernetes/apps/paperless-eval/); only the measurement is not.
Running one
kubectl -n paperless-eval create configmap paperless-eval-script \
--from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
| kubectl apply -f -
kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
Results land on the paperless-eval-results PVC as <label>.jsonl (one row per
document) and <label>.summary.json, and in the pod log.
Re-running after editing eval.py means re-creating the ConfigMap and deleting
the Job first — kubectl -n paperless-eval delete job eval-qwen35.
Adding a candidate
Copy job-qwen35.yaml, then change four things: the Job name, the
--hf-repo/--alias on the llama-server sidecar, PAPERLESS_AI_LLM_MODEL on
the paperless sidecar (it must match the alias), and EVAL_LABEL on the
eval container (it names the output files).
Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1 and k8s-2 have a free iGPU. A third waits.
Reading the output
complement in the summary is the number the exercise exists for. The two
engines are not alternatives: the classifier costs 0.6 s per document and keeps
running whatever the LLM does, so what matters is what the LLM adds where the
classifier is wrong — llm_rescued against llm_only_wrong. Two aggregate
percentages cannot show that; 74% against 72% could overlap almost entirely or
barely at all.
correspondent_offered against correspondent_recognised is the
MATCH_THRESHOLD = 0.8 gap: the model names the right entity and difflib
rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
stored "Generali AG".
type_specific against type_catchall splits on whether the true type is
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
semantic content. The classifier can memorise it; a model reading the document
has to infer "none of the above".
What to watch for
Two things make a result look like a bad model when it is not:
Prompt length. Past roughly 14k tokens this stack stops producing language and starts producing garbage or nothing, without any error — and a single such request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit near 5k, but the 252 documents over 14000 characters push toward the boundary with their RAG neighbours. A run that starts scoring zeros mid-way is more likely this than the model.
Ground truth. It is the current filing, and over ten years conventions change. A disagreement with an old document is not automatically an error. The comparison between engines is the durable result; the absolute numbers are indicative.