claude-botandClaude Opus 5
c47c2e6bc1
A Job is immutable and a measurement is finished when it finishes — neither is something Flux should converge on. Re-running a configuration means deleting and recreating the Job, and a managed one comes straight back. Worse, eval.py was a kustomize-generated ConfigMap, so editing the script between runs and applying it by hand would have been reverted on the next reconcile: the iteration loop the harness exists for was closed off by where it was put. The namespace stays managed. It holds a database clone, PVCs and a network policy — state that should converge and should vanish when the Kustomization is removed. Only the harness moves to scripts/, alongside the other tools nobody deploys, with a README covering how to run one, how to add a candidate, and how to read the output. Two failure modes are written down there rather than left to be rediscovered: prompts past ~14k tokens make this stack emit garbage silently and poison the slot until restart, and ground truth is a filing convention that has drifted over ten years, so the comparison between engines is the durable result rather than the absolute numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
# Measuring Paperless suggestions against the real corpus
|
|
|
|
Compares the two engines Paperless ships — the scikit-learn classifier and the
|
|
LLM path — over documents whose labels a human already chose, in the
|
|
`paperless-eval` namespace rather than against the live archive.
|
|
|
|
**Applied by hand, not by Flux.** A Job is immutable, so re-running a
|
|
configuration means deleting and recreating it, and a Flux-managed Job would
|
|
simply come back. The namespace itself *is* managed
|
|
(`kubernetes/apps/paperless-eval/`); only the measurement is not.
|
|
|
|
## Running one
|
|
|
|
```sh
|
|
kubectl -n paperless-eval create configmap paperless-eval-script \
|
|
--from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
|
|
| kubectl apply -f -
|
|
|
|
kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
|
|
kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
|
|
```
|
|
|
|
Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
|
|
document) and `<label>.summary.json`, and in the pod log.
|
|
|
|
Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
|
|
the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
|
|
|
|
## Adding a candidate
|
|
|
|
Copy `job-qwen35.yaml`, then change four things: the Job name, the
|
|
`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
|
|
the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
|
|
`eval` container (it names the output files).
|
|
|
|
Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
|
|
and k8s-2 have a free iGPU. A third waits.
|
|
|
|
## Reading the output
|
|
|
|
`complement` in the summary is the number the exercise exists for. The two
|
|
engines are not alternatives: the classifier costs 0.6 s per document and keeps
|
|
running whatever the LLM does, so what matters is what the LLM adds where the
|
|
classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
|
|
percentages cannot show that; 74% against 72% could overlap almost entirely or
|
|
barely at all.
|
|
|
|
`correspondent_offered` against `correspondent_recognised` is the
|
|
`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
|
|
rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
|
|
stored "Generali AG".
|
|
|
|
`type_specific` against `type_catchall` splits on whether the true type is
|
|
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
|
|
semantic content. The classifier can memorise it; a model reading the document
|
|
has to infer "none of the above".
|
|
|
|
## What to watch for
|
|
|
|
Two things make a result look like a bad model when it is not:
|
|
|
|
**Prompt length.** Past roughly 14k tokens this stack stops producing language
|
|
and starts producing garbage or nothing, without any error — and a single such
|
|
request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
|
|
near 5k, but the 252 documents over 14000 characters push toward the boundary
|
|
with their RAG neighbours. A run that starts scoring zeros mid-way is more
|
|
likely this than the model.
|
|
|
|
**Ground truth.** It is the current filing, and over ten years conventions
|
|
change. A disagreement with an old document is not automatically an error. The
|
|
comparison between engines is the durable result; the absolute numbers are
|
|
indicative.
|