konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Merge pull request 'fix(paperless-eval): three faults the first job run exposed' (#778) from eval-script-fixes into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Details

Browse Source 
Reviewed-on: k.renner/homelab#778
This commit was merged in pull request #778.
This commit is contained in:
k.renner
2026-09-05 07:29:40 +00:00
parent a358e8cfd5 8123021ea4
commit 19dbd05783
8 changed files with 1077 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/paperless-eval/README.md
+162 -36
View File
@@ -1,8 +1,8 @@
# Measuring Paperless suggestions against the real corpus
# Comparing chat models on Paperless suggestions
Compares the two engines Paperless ships — the scikit-learn classifier and the
LLM path — over documents whose labels a human already chose, in the
`paperless-eval` namespace rather than against the live archive.
Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
against documents a human already filed, in the `paperless-eval` namespace
rather than against the live archive.
**Applied by hand, not by Flux.** A Job is immutable, so re-running a
configuration means deleting and recreating it, and a Flux-managed Job would
@@ -11,54 +11,156 @@ simply come back. The namespace itself *is* managed
## Running one
```sh
kubectl -n paperless-eval create configmap paperless-eval-script \
--from-file=eval.py=scripts/paperless-eval/eval.py --dry-run=client -o yaml \
| kubectl apply -f -
Screen first, always:
kubectl apply -f scripts/paperless-eval/job-qwen35.yaml
kubectl -n paperless-eval logs -f job/eval-qwen35 -c eval
```sh
./run-candidate.sh qwen3-8b 5 # ~5 min to a rejection, ~10 to a pass
kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval
kubectl -n paperless-eval delete job eval-qwen3-8b
./run-candidate.sh qwen3-8b # the real measurement, 50 documents
```
Three of the first four candidates could not do the task at all, and each was
left running for an hour before anyone looked. eval.py now stops after three
consecutive failures from the first document, so a screening run rejects a
hopeless candidate in about five minutes.
Screening uses genuine paperless requests. That matters — `probe-toolcall.sh`
is cheaper still and worth a first pass, but it passed two candidates that then
failed on document one, because a synthesised request cannot reproduce the RAG
neighbours and taxonomy candidate lists that make a real prompt what it is.
Never compare numbers across sample sizes. A screen is a gate; the comparison
happens at 50.
Candidates live in `candidates.tsv`; the runner substitutes the model-specific
values into `job-template.yaml` and refreshes the `eval.py` ConfigMap. Adding a
candidate is a line in that file, not a copy of the manifest — the sidecar
configuration deliberately exists once.
Two run at a time: the pod anti-affinity puts one per node, and only two nodes
have a free iGPU. A third waits.
Re-running after an edit needs the old Job gone first:
`kubectl -n paperless-eval delete job eval-<label>`.
Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
document) and `<label>.summary.json`, and in the pod log.
Re-running after editing `eval.py` means re-creating the ConfigMap and deleting
the Job first — `kubectl -n paperless-eval delete job eval-qwen35`.
## What is being compared
## Adding a candidate
The candidates against each other. The scikit-learn classifier runs in every
job as a fixed reference line — it costs 15 seconds and keeps running in
production whatever the LLM does, so `llm_rescued` against `llm_only_wrong`
says what *this* model adds on top of it.
Copy `job-qwen35.yaml`, then change four things: the Job name, the
`--hf-repo`/`--alias` on the `llama-server` sidecar, `PAPERLESS_AI_LLM_MODEL` on
the `paperless` sidecar (it must match the alias), and `EVAL_LABEL` on the
`eval` container (it names the output files).
Two run in parallel — the pod anti-affinity puts one per node, and only k8s-1
and k8s-2 have a free iGPU. A third waits.
Do not read the classifier's own score as a quality bar. `train()` fits on every
non-inbox document with no holdout, so the evaluation documents are in its
training set and its numbers are memorisation. That does not matter for ranking
models — it is the same constant under all of them.
## Reading the output
`complement` in the summary is the number the exercise exists for. The two
engines are not alternatives: the classifier costs 0.6 s per document and keeps
running whatever the LLM does, so what matters is what the LLM adds where the
classifier is wrong — `llm_rescued` against `llm_only_wrong`. Two aggregate
percentages cannot show that; 74% against 72% could overlap almost entirely or
barely at all.
`correspondent_offered` against `correspondent_recognised` is the
`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
rejects it, as with "Generali Deutschland Krankenversicherung AG" against the
stored "Generali AG".
rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.
`type_specific` against `type_catchall` splits on whether the true type is
"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
semantic content. The classifier can memorise it; a model reading the document
has to infer "none of the above".
semantic content. A model reading the document has to infer "none of the above".
`failed` counts documents where the model produced nothing usable. With
`--n-predict 768` in place, that is mostly non-termination: paperless reports
the truncated response as a flat `Invalid AI configuration` and discards the
content, so what the model was doing is visible only in the llama-server log.
## Results so far
| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
|---|---|---|---|---|---|---|---|
| *classifier (reference)* | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 18 |
| Qwen3.6-35B-A3B Q4_K_M | 72.9 | 77.1 | 85.4 | 94.9 | 44.4 | 2 | 5494 |
| Ministral-3 3B Q4_K_M | 71.4 | 76.2 | 76.2 | 83.3 | 33.3 | **29** | 5511 |
| Ministral-3 8B Q4_K_M | — | — | — | — | — | 3/3 | aborted |
| Granite 3.3 8B Q4_K_M | — | — | — | — | — | all | not measurable |
Read the Ministral-3 3B row with its failure count, not without it. It scored 21
documents and failed 29 — and on those 21 its correspondent numbers sit within
the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents
it could answer at all, which is a set selected for being easy, and the
classifier does better on them too (11 of 21 both right against 30 of 48).
A model that declines three documents in five has no usable accuracy number.
Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two
of those four "broken" are the difflib gap above, not model errors.
### How big a difference means anything
Qwen3.6 was measured twice over the same 50 documents, once with the deprecated
--chat-template-kwargs and once with --reasoning off:
70.8 / 75.0 / 83.3 / 94.9 / 33.3 two failures
72.9 / 77.1 / 85.4 / 94.9 / 44.4 two failures
Every one of those gaps is a single document out of 48 — the catchall one is
one document out of nine. So that is the noise floor: **a difference between
two candidates smaller than about two documents says nothing.** Decoding is
sampled, and the same model on the same input does not give the same answer
twice.
Worth knowing before reading a 2-point lead as a result.
## The instrument is checked, not assumed
Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the
documents, every ten documents, and again if it aborts. If Paris does not come
back, the run stops and says so.
This exists because a whole day of results turned out to be measuring a broken
instrument. On this Vulkan stack a prompt past some model-specific length stops
llama.cpp producing language — it repeats one token, " de de de" or
" phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop"
and nothing in the log. The slot then stays broken for every request after it
until the pod restarts.
So a run against a poisoned slot reports fifty failures that are one event, and
reads exactly like a model that cannot classify. Six candidates were written up
that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris."
perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its
recorded classification scores described a corrupted slot and nothing else.
**A run whose canary breaks is void, not a result.** Failures adjacent to the
break are one fault. Use `probe-length.sh` to find where the model's boundary
actually sits before judging it — paperless needs ~5300 tokens.
## Bounds of the harness, not of the model
Two numbers in here are mine and both have produced misleading results:
`--n-predict` caps generation server-side, default 768. A candidate that stops
exactly there has hit MY bound. Six candidates were written up as incapable on
that signal before anyone checked whether they would have finished at 900
tokens. A working answer takes 241-291 tokens, so 768 looked generous — and
"looked generous" is not evidence.
`EVAL_TIMEOUT` bounds the client, default 300 s. Raising the token cap alone
just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of
generation, so the run fails with TimeoutError instead. Raise both together.
./run-candidate.sh <label> <sample> <n_predict> <timeout>
./run-candidate.sh gemma4-26b-a4b 3 4096 900
The Q8 precision comparison had to be thrown away over this. Q8's uncapped
answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same
task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run
looked like "Q8 classifies worse" when it measured "Q8 is wordier than my
bound". Before comparing two configurations, check that neither is sitting on
the cap.
## What to watch for
Two things make a result look like a bad model when it is not:
**Prompt length.** Past roughly 14k tokens this stack stops producing language
and starts producing garbage or nothing, without any error — and a single such
request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
@@ -67,6 +169,30 @@ with their RAG neighbours. A run that starts scoring zeros mid-way is more
likely this than the model.
**Ground truth.** It is the current filing, and over ten years conventions
change. A disagreement with an old document is not automatically an error. The
comparison between engines is the durable result; the absolute numbers are
indicative.
change. A disagreement with an old document is not automatically an error.
## The models volume fills up
Each candidate leaves its weights on the shared `llama-models` PVC — 2.5 GB for
a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB
and nine screened candidates filled it completely.
A full volume does not announce itself. llama-server cannot write its metadata
cache, so it cannot resolve the repository listing, and says:
no GGUF files found in repository <repo>
Available GGUF files:
- <whatever it had cached before the disk filled>
which reads as a wrong model reference. The pod then sits in
`Init:CrashLoopBackOff`, which from outside is indistinguishable from a slow
34 GB download. Two jobs spent three hours that way.
**Check `df -h /models` before believing anything llama-server says about a
repository.** Clear out rejected candidates when the volume gets tight — the
weights are re-downloadable, so deleting them costs only bandwidth:
```sh
kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
sh -c 'cd /models && du -sh * | sort -h'
```
scripts/paperless-eval/candidates.tsv
+238
View File
@@ -0,0 +1,238 @@
# label hf-repo hf-file alias mem-request mem-limit
#
# Every candidate must do function calling: paperless calls the model through
# chat_with_tools(..., tool_required=True) with DocumentClassifierSchema as a
# tool, not via JSON mode. A model whose chat template has no tool support does
# not score badly here, it fails outright.
#
# Instruct variants, never Reasoning ones. Qwen3.6 spent two of fifty documents
# generating until something cut it off, and a model tuned to think first does
# that by design. The one exception is Qwen3.6 itself, which is measured with
# thinking switched off.
#
# Sizes. An earlier version of this note said "a smaller dense model gains at
# both ends" and chose the Ministral pair on that basis. Measured, it is wrong:
#
# Ministral-3 8B dense, ~4.7 GB weights 7.1-8.2 tok/s
# Qwen3.6 35B-A3B MoE, ~20 GB weights 11 tok/s
#
# The largest model is the fastest one here. But the bandwidth explanation I
# gave for it is wrong, and measuring Q8 disproved it:
#
# Qwen3.6-35B-A3B Q4_K_M ~21 GB 11.0 tok/s
# Qwen3.6-35B-A3B Q8_0 ~34 GB 10.3 tok/s
#
# Doubling the bytes per active parameter cost ten percent, not half. So decode
# is NOT bandwidth-bound for this MoE — it is compute-bound on many small
# per-expert matmuls, which also explains the earlier measurement of 21.4 GB/s
# effective against a 53.9 GB/s ceiling: it was already leaving 60% of the bus
# unused.
#
# Consequences: the bytes-per-token formula predicts dense models and not this
# one, and higher precision is nearly free on an MoE here — about 10% slower
# for double the weight precision.
#
# So the order to prefer is: MoE with few active parameters, then dense 3-9B,
# then dense 10-14B only for an exceptional model (~4 tok/s, roughly three
# hours for 50 documents), and nothing dense above 14B.
qwen35 ggml-org/Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-Q4_K_M.gguf qwen3.6-35b-a3b 24Gi 32Gi
#
# The same model at higher precision, to find out what Q4 costs. Everything in
# this file is Q4_K_M, so the whole comparison could be sitting on quantisation
# damage nobody has measured — and there is a concrete suspicion, not just a
# worry: Ministral-3's leak of [THINK] and [TOOL_CALLS] as literal text is the
# kind of fault a 4-bit quantisation can produce, and Q8 would separate "the
# model cannot do this" from "Q4 broke it".
#
# ~34 GB resident against Q4's ~21, and roughly 6 tok/s against 11, so about
# double the wall clock. Read the result against the noise floor in README.md:
# below about two documents of difference, this says nothing.
qwen35-q8 ggml-org/Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-Q8_0.gguf qwen3.6-35b-a3b-q8 40Gi 48Gi
#
# --- MoE, few active parameters: the shape that wins on this hardware ---
qwen3-30b-a3b bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF Qwen_Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf qwen3-30b-a3b 22Gi 28Gi
nemotron3-30b-a3b bartowski/nvidia_Nemotron-3-Nano-30B-A3B-GGUF nvidia_Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf nemotron3-30b-a3b 28Gi 34Gi
glm47-flash bartowski/zai-org_GLM-4.7-Flash-GGUF zai-org_GLM-4.7-Flash-Q4_K_M.gguf glm-4.7-flash 20Gi 26Gi
gemma4-26b-a4b unsloth/gemma-4-26B-A4B-it-GGUF gemma-4-26B-A4B-it-UD-Q4_K_M.gguf gemma-4-26b-a4b 20Gi 26Gi
qwen35-35b-a3b bartowski/Qwen_Qwen3.5-35B-A3B-GGUF Qwen_Qwen3.5-35B-A3B-Q4_K_M.gguf qwen3.5-35b-a3b 24Gi 30Gi
nemotron35-30b-a3b bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf nemotron3.5-30b-a3b 28Gi 34Gi
#
# --- dense, small, with German named in the model's own card ---
glm4-9b bartowski/glm-4-9b-chat-GGUF glm-4-9b-chat-Q4_K_M.gguf glm-4-9b 10Gi 14Gi
commandr7b bartowski/c4ai-command-r7b-12-2024-GGUF c4ai-command-r7b-12-2024-Q4_K_M.gguf command-r7b 10Gi 14Gi
qwen25-7b bartowski/Qwen2.5-7B-Instruct-GGUF Qwen2.5-7B-Instruct-Q4_K_M.gguf qwen2.5-7b 10Gi 14Gi
llama31-8b bartowski/Meta-Llama-3.1-8B-Instruct-GGUF Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf llama-3.1-8b 10Gi 14Gi
nemotron-nano-12b bartowski/nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q4_K_M.gguf nemotron-nano-12b 12Gi 16Gi
phi4-mini bartowski/microsoft_Phi-4-mini-instruct-GGUF microsoft_Phi-4-mini-instruct-Q4_K_M.gguf phi-4-mini 8Gi 12Gi
#
# --- dense, small, no German claim but no thinking risk either ---
qwen3-4b-2507 bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf qwen3-4b 8Gi 12Gi
qwen35-9b bartowski/Qwen_Qwen3.5-9B-GGUF Qwen_Qwen3.5-9B-Q4_K_M.gguf qwen3.5-9b 10Gi 14Gi
#
# WHAT THE RESEARCH RULED OUT, so nobody re-derives it
#
# Hard-excluded on tool calling, chat template inspected directly rather than
# assumed — the string "tool" appears zero times in any of them:
# Gemma 3, every size (4B/12B/27B) phi-4 14B (the mini DOES have it)
# Cohere Aya, all variants InternLM 2.5 / 3
#
# Hard-excluded on German, stated by the vendor rather than merely missing:
# Falcon 3 — "supports EN, FR, ES, PT", German is not on the list
# EXAONE — the small ones are English/Korean; the ones with German are 33B+
#
# German fine-tunes are a dead end, and for one consistent reason: SauerkrautLM,
# DiscoLM, LeoLM, Occiglot and Teuken all overwrite their base model's chat
# template with a minimal ChatML loop that drops tool calling. VAGOsolutions
# does this even on bases that have it (Llama-3.1, Qwen2.5). Assume any German
# tune is tool-incapable until its own tokenizer_config.json says otherwise.
#
# On llama.cpp's tool parsing, which explains the Granite failure: current
# builds have no per-vendor parser table any more. common/chat.cpp exposes
# PEG_SIMPLE / PEG_NATIVE / PEG_GEMMA4 / PEG_MINIMAX_M3 and derives the grammar
# from the model's own template. Hermes-style <tool_call> JSON is a first-class
# tested fixture in tests/test-chat.cpp; Granite's format is not. So this is not
# "Granite is unsupported" so much as "Granite's template shape is outside what
# the auto-parser was validated against".
#
# SCREENING RESULTS, and a pattern that reframes the whole search
#
# qwen35 MoE 35B total / 3B active works, 2 of 50 failed
# qwen3-8b dense 8B 4 of 5 failed
# hermes4-14b dense 14B 3 of 3 failed, aborted
# ministral8b dense 8B 3 of 3 failed, aborted
# ministral3b dense 3B 29 of 50 failed
# granite8b dense 8B every document failed
#
# Five dense models from 3B to 14B cannot produce a valid call against
# paperless's 14-field schema; the one mixture-of-experts model can. Hermes is
# the cleanest of these failures: its <tool_call> JSON is an explicitly tested
# fixture in llama.cpp's own test suite, so the parser demonstrably handles the
# format and the failure has to be the model.
#
# Working hypothesis: holding a long grammar-constrained structure needs model
# capacity, and what supplies it is TOTAL parameters, not active ones. If that
# is right, "go smaller for speed" is a dead end here — and the consolation is
# that the fastest shape on this hardware, MoE with few active parameters, is
# also the only one that works.
#
# qwen3-30b-a3b tests it directly: 30B total, 3B active.
#
# CORRECTION: Gemma is not excluded after all. Gemma *3* has no tool support —
# "tool" appears zero times in its chat template at every size, checked
# directly. Gemma *4* 26B-A4B does have it, documented in the template, plus
# "140+ languages pretrained, 35+ out of the box". Excluding a family by its
# previous generation was sloppy.
#
# Two of these carry a Vulkan risk worth checking in the llama-server log
# before trusting a run: Nemotron-3.5 interleaves real Mamba2 layers and
# Qwen3.5 uses DeltaNet linear attention, and llama.cpp's Vulkan SSM shaders
# are less exercised on Iris Xe than on Arc. Issue #19957 reported exactly this
# for Qwen3.5-35B-A3B and it turned out to be a stale build, fixed months
# before b10731 — but confirm layers are actually offloaded rather than
# silently falling back to CPU, which would read as a slow model.
#
# Ruled out after checking: ERNIE-4.5-21B-A3B (only the Thinking variant has
# tool calling; the instruct one has none), Ling-3.0-flash (124B total),
# Granite-4.0-H-Small (9B active), DeepSeek-V2-Lite and OLMoE (under 20B
# total), Cohere North Mini (code-only, no ggml-org/bartowski conversion,
# cohere2_moe support in llama.cpp unverified), gpt-oss-20b (a reasoning-effort
# model in Harmony format rather than plain instruct).
#
# THE MoE HYPOTHESIS IS DEAD. qwen3-30b-a3b — MoE, 30B total, 3B active, the
# shape that was supposed to explain everything — scored 1 of 5 and failed 4,
# indistinguishable from the dense 8B.
#
# So neither architecture nor size predicts success:
#
# Qwen3.6-35B-A3B MoE 35B/3B Apr 2026 2 of 50 failed
# Qwen3-30B-A3B MoE 30B/3B older 4 of 5 failed
# Qwen3-8B dense 8B older 4 of 5 failed
# Hermes-4-14B dense 14B — 3 of 3 failed
#
# The only thing correlating with success so far is how new and how strong the
# model is. That is an unsatisfying hypothesis because it predicts nothing
# cheaply — but it is what the data says, and two architectural theories have
# already died here.
#
# qwen35-35b-a3b isolates it: same architecture, same total and active
# parameters as the working baseline, one generation older. If it fails, the
# generation is the variable and shape is noise.
#
# qwen35-35b-a3b PASSES the screen: 5 of 5, no failures, 584 s. So the
# generation hypothesis is dead too — Qwen3.5 is a generation OLDER than the
# working Qwen3.6 and handles the schema cleanly.
#
# What the two working models share is 35B total parameters; the one failing
# MoE has 30B. But that is confounded — Qwen3-30B-A3B is both smaller and
# older, and three data points cannot separate the two.
#
# Qwen3.6-35B-A3B 35B total newest works
# Qwen3.5-35B-A3B 35B total older works
# Qwen3-30B-A3B 30B total oldest 4 of 5 failed
#
# glm47-flash separates them: 30B total, newest generation. If it works,
# generation is what matters and 35B is not a threshold. If it fails, total
# size is, and the search should stop looking below ~35B.
#
# PREFILL SPEED IS A SELECTION CRITERION AND I WAS NOT TRACKING IT.
# Measured on this hardware, prompt processing varies more than sixfold across
# architectures, and it is roughly half of a run's wall clock:
#
# dense 8B (Ministral) 240 tok/s
# Qwen3.6-35B-A3B 100 tok/s
# GLM-4.7-Flash 31 tok/s decode 4.9 tok/s
#
# GLM-4.7-Flash is disqualified on that alone. At 31 tok/s a 5300-token prompt
# takes 168 s, times ~2.6 requests per document, is over seven minutes per
# document and about six hours for a run — before considering whether its
# answers are any good.
#
# It also shows why the 300 s REQUEST_TIMEOUT needs care: 4453 prompt tokens at
# 31.5 tok/s plus 768 generated at 4.9 tok/s is 298 s, so a slow model turns a
# content failure into a TimeoutError and muddies the diagnosis. Read the
# llama-server timings, not just the eval error.
#
# glm47-flash: aborted, 3 of 3 — but every one a TimeoutError, not an invalid
# answer, so this does NOT settle the size-versus-generation question it was
# started to settle. It failed on speed. Document 1 did reach the --n-predict
# cap without producing a call, but 2 and 3 were cut off by the 300 s client
# bound before anything could be concluded about them.
#
# Disqualified regardless: 31 tok/s prefill and 4.9 tok/s decode is over seven
# minutes per document and about six hours for a full run.
#
# gemma4-26b-a4b is the better discriminator — 26B total, well under 35, and
# newest generation. Works means generation and quality matter rather than a
# 35B threshold; fails is consistent with total size being the line.
#
# gemma4-26b-a4b: aborted, 3 of 3, all HTTP 400 rather than timeouts, and the
# server log shows it ran to the --n-predict cap on every one. So it is the
# same capability failure as the small dense models, not a template mismatch.
# (Prefill 45-50 tok/s, decode 6 tok/s — also not fast.)
#
# THE SIZE THRESHOLD SURVIVED ITS FIRST REAL TEST. gemma-4 was chosen
# specifically because it could refute it — newest generation, 26B total — and
# it did not:
#
# Qwen3.6-35B-A3B 35B newest works
# Qwen3.5-35B-A3B 35B older works
# Qwen3-30B-A3B 30B oldest fails
# gemma-4-26B-A4B 26B NEWEST fails
# Hermes-4-14B 14B — fails
# dense 3-8B — — fail
#
# Two generations work at 35B total; everything at 30B and below fails,
# including the newest model in the field. Operating rule until something
# contradicts it: do not spend a screening slot on anything under ~35B total.
#
# Not pinned further on purpose. Whether the line sits at 31B or 34B is
# interesting and not decision-relevant — there are two working candidates and
# the useful next number is a full 50-document run of the second one.
#
# Re-enabled for length probing, not for scoring. Their classification numbers
# were taken against a slot that may already have been corrupted, so the only
# honest thing to do with them is find out where each one's boundary sits.
hermes4-14b bartowski/NousResearch_Hermes-4-14B-GGUF NousResearch_Hermes-4-14B-Q4_K_M.gguf hemes-4-14b 14Gi 20Gi
qwen3-8b bartowski/Qwen_Qwen3-8B-GGUF Qwen_Qwen3-8B-Q4_K_M.gguf qwen3-8b 12Gi 16Gi
ministral8b bartowski/mistralai_Ministral-3-8B-Instruct-2512-GGUF mistralai_Ministral-3-8B-Instruct-2512-Q4_K_M.gguf ministral-3-8b 12Gi 16Gi
granite8b bartowski/ibm-granite_granite-3.3-8b-instruct-GGUF ibm-granite_granite-3.3-8b-instruct-Q4_K_M.gguf granite-3.3-8b 12Gi 16Gi
Can't render this file because it contains an unexpected character in line 13 and column 47.
scripts/paperless-eval/eval.py
+217 -18
View File
@@ -78,10 +78,20 @@ ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
OUT_DIR = os.environ.get("EVAL_OUT", "/results")
# Long enough that a slow document is measured rather than abandoned. A client
# timeout does not stop the generation behind it, and with a single llama.cpp
# slot an abandoned request holds the queue for everything after it.
REQUEST_TIMEOUT = 900
# A backstop, not the real bound. Generation is capped server-side by
# --n-predict on the llama-server sidecar, which is what actually stops a model
# that will not terminate; this catches everything else — a wedged sidecar, a
# request that never reaches it.
#
# 300 s against the ~73 s a healthy document takes end to end, prompt included.
# It does work: llama-server logs `stop: cancel task` on the disconnect and
# starts the next request 0.4 s later.
REQUEST_TIMEOUT = int(os.environ.get("EVAL_TIMEOUT", "300"))
# Consecutive failures from the very first document that mean the engine is
# unusable rather than merely weak. Three, because one could be a single hard
# document and two could be coincidence.
ABORT_AFTER = 3
CATCHALL_TYPE = "Sonstige"
@@ -91,20 +101,37 @@ def log(msg: str) -> None:
def api_token(conn) -> str:
"""A token for this run, created in the database.
"""A token for this run, against a user belonging to this run alone.
The API needs one and this database is a disposable clone, so the run
issues its own against a superuser rather than depending on a token that
happened to come across in the restore.
issues its own rather than depending on a token that happened to come
across in the restore.
The per-label user is not tidiness. Django's authtoken holds one token per
user, so an earlier version took the first superuser and deleted its
existing token first — which meant the second of two candidates running
concurrently silently revoked the first one's credentials. That surfaced as
a mid-run `HTTP 401: Invalid token` sitting among genuine model failures,
where it reads like one more broken document rather than the harness
sabotaging itself. Both jobs share this database; only the user can differ.
"""
token = secrets.token_hex(20)
username = f"eval-{LABEL}"
with conn.cursor() as cur:
cur.execute(
"select id from auth_user where is_superuser and is_active order by id limit 1",
)
cur.execute("select id from auth_user where username = %s", (username,))
row = cur.fetchone()
if row is None:
raise SystemExit("no active superuser in the restored database")
cur.execute(
"""
insert into auth_user (password, is_superuser, username,
first_name, last_name, email, is_staff,
is_active, date_joined)
values ('!unusable', true, %s, '', '', '', true, true, now())
returning id
""",
(username,),
)
row = cur.fetchone()
user_id = row[0]
cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
cur.execute(
@@ -165,7 +192,9 @@ def floor(conn) -> float:
""",
)
top, total = cur.fetchone()
return 100.0 * top / total if total else 0.0
# float() on both: psycopg returns Postgres numerics as Decimal, and
# mixing that with a float literal raises TypeError rather than coercing.
return 100.0 * float(top) / float(total) if total else 0.0
def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
@@ -198,6 +227,32 @@ def recognised(payload: dict, field: str, want_name: str) -> bool:
return any(want <= normalise(name) for name in payload.get(field) or [])
def degenerate(payload: dict) -> str | None:
"""Whether this looks like broken inference rather than a wrong guess.
Past roughly 14k prompt tokens this stack stops producing language — first
empty completions, then runs of digits and CJK characters — and one such
request poisons the llama.cpp slot until the pod restarts. Nothing in the
HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
Scored as a wrong answer it would silently depress a candidate's numbers,
and because the slot stays poisoned it would take the rest of the run with
it. Flagged instead, so the failure is visible as itself.
"""
title = payload.get("title") or ""
if not title.strip():
return "empty title"
# Digits, punctuation and whitespace are normal in a document title —
# "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
# earlier version of this check. What broken inference produces is
# characters from other scripts, so look for those directly rather than
# inferring them from a shortage of Latin ones.
foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
if foreign > 0.2 * len(title):
return f"non-latin title: {title[:40]!r}"
return None
def score_document(payload: dict, doc: dict) -> dict:
offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
@@ -214,9 +269,56 @@ def score_document(payload: dict, doc: dict) -> dict:
}
LLM_ENDPOINT = os.environ.get("PAPERLESS_AI_LLM_ENDPOINT", "http://127.0.0.1:8080/v1")
def canary() -> tuple[bool, str]:
"""Ask the model something it cannot get wrong, and check it did not.
The instrument has to be shown working before its readings mean anything,
and this one can fail silently in a way that looks like a bad model. On
this Vulkan stack a prompt past some model-specific length stops llama.cpp
producing language: it repeats one token — " de de de", " phosphorus
phosphorus", whitespace — at HTTP 200 with finish_reason "stop". The slot
stays broken for every request after it until the pod restarts.
So a run that measured 50 documents against a poisoned slot reports 50
failures that are really one event, and reads as a model that cannot
classify. That is exactly what happened before this existed.
Run before the documents and again after: healthy then broken means the
failures in between are one fault, not evidence about the model.
"""
body = json.dumps({
"model": os.environ.get("PAPERLESS_AI_LLM_MODEL", "probe"),
"max_tokens": 40,
"messages": [{"role": "user", "content": "Nenne die Hauptstadt von Frankreich."}],
}).encode()
req = urllib.request.Request(
f"{LLM_ENDPOINT}/chat/completions",
data=body,
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=120) as resp:
payload = json.loads(resp.read())
text = payload["choices"][0]["message"].get("content") or ""
text += payload["choices"][0]["message"].get("reasoning_content") or ""
return ("Paris" in text), text[:80]
except Exception as err: # noqa: BLE001
return False, f"{type(err).__name__}: {err}"
def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
log(f"=== {engine}: {len(docs)} documents")
rows = []
if engine == "ai":
ok, text = canary()
log(f" canary before: {'OK' if ok else 'BROKEN'} — {text!r}")
if not ok:
log(" The model is not answering a trivial question correctly, so")
log(" nothing measured after this describes its classification.")
return [{"engine": engine, "canary_before": False, "canary_text": text}]
for n, doc in enumerate(docs, 1):
payload, error, elapsed = fetch(
f"/api/documents/{doc['id']}/{suffix}",
@@ -231,13 +333,103 @@ def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
"catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
}
if payload is None:
# An authentication failure is never a fact about the model, and
# counted alongside real ones it reads like one more broken
# document. Stop loudly instead: something revoked this run's
# credentials mid-flight, and every number after it is worthless.
if error and error.startswith("HTTP 401"):
raise SystemExit(
f"[eval] FATAL: {error}\n"
f"[eval] This run's token stopped working part-way through. "
f"Django's authtoken holds one token per user, so another "
f"job sharing this database and this user revoked it.",
)
row["error"] = error
log(f" {doc['id']}: {error}")
else:
row.update(score_document(payload, doc))
# Only the AI path returns a title; the classifier answers with ids
# alone, so an "empty title" there is normal rather than broken.
if engine == "ai" and (broken := degenerate(payload)) is not None:
row["degenerate"] = broken
log(f" {doc['id']}: DEGENERATE — {broken}")
rows.append(row)
# Clear the slot between documents so the next one is measured on a
# clean server. The corruption this guards against is stateful: one bad
# request leaves the slot returning a single repeated token for
# everything after it, and without this a run counts that once per
# remaining document.
if engine == "ai":
try:
urllib.request.urlopen(
urllib.request.Request(
f"{LLM_ENDPOINT.rsplit('/v1', 1)[0]}/slots/0?action=erase",
data=b"",
method="POST",
),
timeout=30,
)
except Exception as err: # noqa: BLE001 — best effort, never fatal
if n == 1:
log(f" slot erase unavailable ({type(err).__name__}) — "
f"failures after a corrupt document will not be independent")
if n % 10 == 0:
log(f" {n}/{len(docs)}")
# Re-check the instrument, not just the progress. A slot corrupted
# part-way through turns every remaining document into a failure
# that says nothing about the model, and scattered failures — 29 of
# 50 in one earlier run — are exactly the shape that a
# start-of-run check alone would miss.
if engine == "ai":
ok, text = canary()
if not ok:
log(f" CANARY BROKEN at {n}/{len(docs)} — {text!r}")
log(" The model can no longer answer a trivial question, so")
log(" everything from here would measure a dead slot. Stopping.")
log(" Failures recorded before this point are still usable;")
log(" the ones adjacent to it are one fault, not several.")
rows.append({"engine": engine, "canary_failed_at": n, "canary_text": text})
break
# Stop early when a model cannot do the task at all, rather than
# spending an hour writing the same error fifty times. Three candidates
# in a row failed this way — Granite because llama.cpp cannot read its
# tool calls back, both Ministral sizes because they fill the tool
# call's first string with their own [THINK]/[TOOL_CALLS] control
# tokens until the token cap. In each case document one already said
# so, and the remaining forty-nine said nothing new.
#
# Deliberately not a ratio: a model that fails a third of the time is a
# result worth measuring in full. This only catches the case where
# nothing has ever worked.
if len(rows) == ABORT_AFTER and all("error" in r for r in rows):
ok, text = canary()
log(f" canary after {ABORT_AFTER} failures: {'OK' if ok else 'BROKEN'} — {text!r}")
if not ok:
log(" The slot is corrupted, so these are ONE fault and not")
log(f" {ABORT_AFTER} independent results. Probe the model's prompt-length")
log(" boundary with probe-length.sh before judging it.")
log(f" ABORT — first {ABORT_AFTER} documents all failed:")
log(f" {rows[0]['error']}")
log(" Nothing has succeeded, so the rest would measure nothing.")
log(" Check the model's tool calling before re-running:")
log(" curl -s $ENDPOINT/v1/chat/completions -d @probe.json")
log(" A usable candidate answers with finish_reason 'tool_calls'.")
break
# And once at the end, whatever happened. A short run whose first
# document succeeds trips neither the abort check nor the every-tenth
# one, so without this a five-document screen can finish having never
# re-verified the instrument it just used.
if engine == "ai":
ok, text = canary()
log(f" canary after: {'OK' if ok else 'BROKEN'} — {text!r}")
if not ok:
log(" Healthy before, broken after: the failures in this run are")
log(" one fault and not independent results.")
rows.append({"engine": engine, "canary_after": False, "canary_text": text})
return rows
@@ -246,17 +438,23 @@ def pct(hits: int, total: int) -> float:
def summarise(rows: list[dict], engine: str) -> dict:
scored = [r for r in rows if r["engine"] == engine and "error" not in r]
failed = [r for r in rows if r["engine"] == engine and "error" in r]
mine = [r for r in rows if r["engine"] == engine]
# Degenerate rows are excluded from the accuracy figures: they measure the
# backend giving up, not the model choosing wrongly, and leaving them in
# would quietly punish whichever candidate happened to trip the slot.
scored = [r for r in mine if "error" not in r and "degenerate" not in r]
failed = [r for r in mine if "error" in r]
broken = [r for r in mine if "degenerate" in r]
specific = [r for r in scored if not r["catchall_type"]]
catchall = [r for r in scored if r["catchall_type"]]
return {
"engine": engine,
"scored": len(scored),
"failed": len(failed),
"seconds_total": round(
sum(r["seconds"] for r in rows if r["engine"] == engine),
),
# Non-zero means the run is suspect from that point on, not that the
# model is bad — the slot stays poisoned until the pod restarts.
"degenerate": len(broken),
"seconds_total": round(sum(r["seconds"] for r in mine)),
"correspondent_offered": pct(
sum(r["correspondent_offered"] for r in scored), len(scored)
),
@@ -281,7 +479,8 @@ def complement(rows: list[dict]) -> dict | None:
"""
by_engine = {"classic": {}, "ai": {}}
for row in rows:
if "error" not in row and row["engine"] in by_engine:
usable = "error" not in row and "degenerate" not in row
if usable and row["engine"] in by_engine:
by_engine[row["engine"]][row["document_id"]] = row
both = set(by_engine["classic"]) & set(by_engine["ai"])
if not both or not by_engine["ai"]:
scripts/paperless-eval/job-qwen35.yaml → scripts/paperless-eval/job-template.yaml
+108 -18
View File
@@ -16,7 +16,7 @@
apiVersion: batch/v1
kind: Job
metadata:
name: eval-qwen35
name: eval-__LABEL__
namespace: paperless-eval
labels:
app.kubernetes.io/name: paperless-eval
@@ -69,9 +69,27 @@ spec:
- -c
- |
set -e
cp -a /golden/. /data/
echo "seeded /data:"
du -sh /data/index /data/llm_index 2>/dev/null || true
# -r, not -a: preserving ownership needs CHOWN, which this
# container does not have, and cp then exits non-zero after
# copying the files anyway. Running as 1000 makes the copy land
# with the ownership paperless needs — it writes to the LLM index
# while serving.
cp -r /golden/. /data/
du -sh /data/index /data/llm_index
# Refuse to start rather than measure the wrong thing. An empty
# llm_index makes retrieve_similar_nodes return nothing, and the
# run would report a RAG result having never retrieved anything —
# which is exactly what the first attempt at this job did, because
# the golden volume had not been populated yet. The index is
# ~125 MiB once built; anything under 1 MiB is not one.
bytes=$(du -sb /data/llm_index | cut -f1)
if [ "$bytes" -lt 1000000 ]; then
echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
echo "smoke-test deployment first:" >&2
echo " kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
echo " python manage.py document_llmindex rebuild" >&2
exit 1
fi
volumeMounts:
- name: golden
mountPath: /golden
@@ -79,6 +97,8 @@ spec:
- name: data
mountPath: /data
securityContext:
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
@@ -87,24 +107,81 @@ spec:
restartPolicy: Always
args:
- --hf-repo
- ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
- __HF_REPO__
# The exact filename rather than the :QUANT shorthand. The
# shorthand matches by substring across the whole repo and repos do
# carry several files under one tag — ggml-org ships mmproj-, mtp-
# and dflash- variants at Q8_0 beside the model — so naming the file
# removes a real ambiguity and costs nothing.
#
# It is NOT what caused the three-hour crashloop that prompted this,
# and an earlier version of this comment claimed it was. The real
# cause was the llama-models volume being 100% full: llama-server
# could not write its metadata cache, so it could not resolve the
# repo listing, and reported
# no GGUF files found in repository <repo>
# followed by a truncated list of what it had cached before the
# disk filled. That reads as a bad reference and is a full disk.
#
# If a model will not load, check `df -h /models` before believing
# anything llama-server says about the repository.
- --hf-file
- __HF_FILE__
- --alias
- qwen3.6-35b-a3b
- __ALIAS__
- --n-gpu-layers
- "99"
- --threads
- "4"
# The production ceiling, and for the same reason: 32768 per slot
# corrupts inference on this Vulkan stack — 1025 tokens of "!"
# against a 650-token prompt, with /props still reporting the
# requested size. A candidate needing more context than this cannot
# be measured here, and that is itself a result.
# The production ceiling. Not because the flag value itself breaks
# anything — bisection showed the corruption boundary is prompt
# length, around 14000 tokens, whatever --ctx-size says — but
# because measuring against a different ceiling than production
# runs would not describe production.
- --ctx-size
- "16384"
- --parallel
- "1"
- --ubatch-size
- "1024"
# The default is -1, infinity: nothing on the server caps how long
# a request generates, and paperless sets no budget of its own and
# waits. A classification that needs ~300 tokens ran to 10269 at a
# healthy 11 tok/s, and three of the first eleven documents did it
# — 900 s each, most of an hour spent on four documents.
#
# 768 is generous against the ~300 a correct answer takes. Note
# that paperless reports a truncated response as a flat
# "Invalid AI configuration" and discards the content, so what the
# model was actually doing is visible only in this container's log.
- --n-predict
- "__N_PREDICT__"
# Without this, Qwen3.6 spends the response on reasoning_content
# and returns empty content — 983 tokens into a classification that
# needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
# slow or broken model and is neither.
#
# NOT --chat-template-kwargs '{"enable_thinking": false}'. That is
# deprecated, llama-server says so at startup, and it is not the
# harmless no-op it looks like on a template that never reads the
# variable: injecting it into Mistral's render context produced
# token salad — "1UnComplete1Self1UnCompleteSDirect11S..." — on a
# 681-token prompt, while the identical request against a server
# without the flag returned a clean tool call. It reads exactly
# like a broken model and is a broken prompt.
- --reasoning
- "off"
# Enables POST /slots/0?action=erase, which the eval calls between
# documents. Without it the endpoint answers 501.
#
# This is what makes the measurements independent. A request that
# corrupts the slot otherwise takes every document after it, so a
# run reports N failures for one fault — "4 of 5 failed" was really
# "document 2 broke the slot". Erasing costs the prefix cache and
# some prefill speed, which is the right trade for numbers that
# mean what they say.
- --slot-save-path
- /tmp/slots
- --no-webui
env:
- name: LLAMA_CACHE
@@ -115,20 +192,27 @@ spec:
volumeMounts:
- name: models
mountPath: /models
# ~20 GB of weights off CephFS, so a generous budget: 60 * 20s.
# 60 minutes. Sized for a FIRST DOWNLOAD from HuggingFace, not for
# reading a warm PVC — an earlier version allowed 20 minutes on the
# latter reasoning and killed GLM-4.7-Flash mid-download at 17.8 GB.
# The container then restarts and begins again, so too small a budget
# is not "slow start", it is a loop that never finishes.
#
# The kill shows up as exitCode 137 with reason "Error" rather than
# OOMKilled, which is how to tell a probe kill from a memory limit.
startupProbe:
httpGet:
path: /health
port: llm
periodSeconds: 20
failureThreshold: 60
failureThreshold: 180
resources:
requests:
cpu: "1"
memory: 24Gi
memory: __MEM_REQ__
gpu.intel.com/i915: "1"
limits:
memory: 32Gi
memory: __MEM_LIM__
gpu.intel.com/i915: "1"
securityContext:
allowPrivilegeEscalation: false
@@ -168,7 +252,7 @@ spec:
- name: PAPERLESS_AI_LLM_ENDPOINT
value: http://127.0.0.1:8080/v1
- name: PAPERLESS_AI_LLM_MODEL
value: qwen3.6-35b-a3b
value: __ALIAS__
- name: PAPERLESS_REDIS
value: redis://127.0.0.1:6379
- name: PAPERLESS_SECRET_KEY
@@ -224,9 +308,15 @@ spec:
command: ["python3", "/eval/eval.py"]
env:
- name: EVAL_LABEL
value: qwen35
value: __LABEL__
- name: EVAL_SAMPLE
value: "50"
value: "__SAMPLE__"
# Raised alongside --n-predict for a diagnostic run. The two bounds
# interact: lifting the token cap alone just moves the failure from
# "hit the cap" to "hit the clock", which is what happened the first
# time gemma-4 was retried at 4096.
- name: EVAL_TIMEOUT
value: "__TIMEOUT__"
- name: PGHOST
value: pprleval-pg-rw
- name: PGDATABASE
scripts/paperless-eval/probe-length.sh
Executable
+149
View File
@@ -0,0 +1,149 @@
#!/bin/sh
# Where does this model's output stop being language?
#
# ./probe-length.sh <label>
#
# THE TEST THAT SHOULD HAVE COME FIRST. On this Vulkan/Iris Xe stack every
# model has a prompt length past which llama.cpp stops producing language and
# starts repeating a single token — " de de de", " the the the", " phosphorus
# phosphorus", or plain whitespace. It is silent: HTTP 200, finish_reason
# "stop" or "length", nothing in the log.
#
# It is also STATEFUL. One request past the boundary leaves the slot broken for
# every request after it, until the pod restarts. So in a 50-document run the
# first bad document takes the rest with it, and "failed 3 of 3" is one
# measurement, not three.
#
# A real paperless request is ~5300 tokens once the document, its RAG
# neighbours and the taxonomy lists are in. So the only question that matters
# about a candidate is whether its boundary is above that — and answering it
# takes minutes, against ninety for a classification run that would have
# measured the same thing badly.
#
# Measured so far:
# Qwen3.6-35B-A3B coherent at 13588, broken at 15086
# gemma-4-26B-A4B coherent at ~350, whitespace at 2334
set -eu
label="${1:?usage: probe-length.sh <label> (labels are in candidates.tsv)}"
here="$(dirname "$0")"
ns=paperless-eval
pod="lenprobe-$(echo "$label" | tr 'A-Z._' 'a-z--' | cut -c1-40)"
row="$(grep -v '^#' "$here/candidates.tsv" | awk -v l="$label" '$1==l')"
[ -n "$row" ] || { echo "unknown candidate '$label'" >&2; exit 1; }
repo="$(echo "$row" | cut -f2)"
hffile="$(echo "$row" | cut -f3)"
memreq="$(echo "$row" | cut -f5)"
memlim="$(echo "$row" | cut -f6)"
cleanup() { kubectl -n "$ns" delete pod "$pod" --wait=false >/dev/null 2>&1 || true; }
trap cleanup EXIT INT TERM
cleanup; sleep 2
kubectl -n "$ns" apply -f - >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata: {name: $pod, namespace: $ns}
spec:
restartPolicy: Never
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
args: ["--hf-repo","$repo","--hf-file","$hffile","--alias","probe",
"--n-gpu-layers","99","--threads","4","--ctx-size","16384",
"--parallel","1","--ubatch-size","1024","--n-predict","200","--no-webui"]
env: [{name: LLAMA_CACHE, value: /models}]
volumeMounts: [{name: models, mountPath: /models}]
resources:
requests: {cpu: "1", memory: $memreq, gpu.intel.com/i915: "1"}
limits: {memory: $memlim, gpu.intel.com/i915: "1"}
securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: ["ALL"]}}
volumes:
- name: models
persistentVolumeClaim: {claimName: llama-models}
EOF
# Say which of the two it is doing. There are only two iGPUs, so a probe
# started while both are busy sits Pending indefinitely — and an earlier run
# spent an hour reporting nothing before giving up with "never became healthy",
# which reads as a broken model rather than a queue.
echo "waiting for $label"
i=0
until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do
i=$((i+1))
if [ $(( i % 8 )) -eq 0 ]; then
ph=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.phase}' 2>&1)
why=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")].message}' 2>&1)
echo " ${ph}${why:+ — $why}"
[ "$ph" = "Pending" ] && [ -n "$why" ] && { echo "FAIL: cannot schedule — free a GPU first"; exit 1; }
fi
[ "$i" -gt 240 ] && { echo "FAIL: never became healthy after 60 min"; exit 1; }
sleep 15
done
# Fresh pod per size. The defect poisons the slot, so probing sizes in sequence
# against one server would measure the first bad request and nothing after it.
for reps in 4 20 40 70 110 160; do
kubectl -n "$ns" delete pod "$pod" >/dev/null 2>&1 || true
sleep 2
kubectl -n "$ns" apply -f - >/dev/null <<EOF2
apiVersion: v1
kind: Pod
metadata: {name: $pod, namespace: $ns}
spec:
restartPolicy: Never
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
args: ["--hf-repo","$repo","--hf-file","$hffile","--alias","probe",
"--n-gpu-layers","99","--threads","4","--ctx-size","16384",
"--parallel","1","--ubatch-size","1024","--n-predict","200","--no-webui"]
env: [{name: LLAMA_CACHE, value: /models}]
volumeMounts: [{name: models, mountPath: /models}]
resources:
requests: {cpu: "1", memory: $memreq, gpu.intel.com/i915: "1"}
limits: {memory: $memlim, gpu.intel.com/i915: "1"}
securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: ["ALL"]}}
volumes:
- name: models
persistentVolumeClaim: {claimName: llama-models}
EOF2
until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do sleep 10; done
filler=""; n=0
while [ $n -lt $reps ]; do
filler="${filler}Die Rechnung wurde am fuenften Maerz ausgestellt und umfasst mehrere Positionen aus dem laufenden Wartungsvertrag. "
n=$((n+1))
done
printf '{"model":"probe","max_tokens":200,"messages":[{"role":"user","content":"%s Nenne die Hauptstadt von Frankreich."}]}' "$filler" > /tmp/lp.json
kubectl -n "$ns" cp /tmp/lp.json "$pod":/tmp/lp.json >/dev/null 2>&1 || true
r=$(kubectl -n "$ns" exec "$pod" -- curl -s --max-time 400 -X POST http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' -d @/tmp/lp.json)
pt=$(printf '%s' "$r" | grep -oE '"prompt_tokens":[0-9]+' | head -1 | cut -d: -f2)
# Test for DEGENERATION, not for the right answer. Two earlier versions of
# this check asked whether "Paris" came back and failed a working model
# twice: once because a thinking model leaves `content` empty and puts its
# answer in `reasoning_content`, then because 64 tokens is not enough for one
# to think its way to the answer at all.
#
# What actually marks the defect is a single token repeated — " de de de",
# " phosphorus phosphorus", whitespace. That signature survives truncation
# and does not care which field the model writes to or whether it finished,
# so it tests the thing itself instead of a proxy for it.
txt=$(printf '%s' "$r" | grep -oE '"(reasoning_)?content":"[^"]*' | sed 's/"[a-z_]*content":"//' | tr '\n' ' ')
rep=$(printf '%s' "$txt" | tr -s ' ' '\n' | grep -c . || echo 0)
top=$(printf '%s' "$txt" | tr -s ' ' '\n' | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')
: "${top:=0}"
if [ "$rep" -gt 8 ] && [ $(( top * 2 )) -gt "$rep" ]; then
verdict="BROKEN" # one token is more than half the output
elif [ "$rep" -le 1 ]; then
verdict="EMPTY" # nothing came back at all — also not healthy
else
verdict="OK"
fi
printf '%-6s prompt=%-6s %s %s\n' "$verdict" "${pt:-?}" "$(printf '%.50s' "$txt")" ""
[ "$verdict" != "OK" ] && { echo; echo "boundary is below ${pt:-?} prompt tokens"; echo "paperless needs ~5300 — $([ "${pt:-0}" -lt 6000 ] && echo 'DISQUALIFIED' || echo 'still above the bar')"; exit 0; }
done
echo
echo "coherent through the largest size probed — boundary is above that"
scripts/paperless-eval/probe-toolcall.sh
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/bin/sh
# Can this model make one valid tool call against paperless's real schema?
#
# ./probe-toolcall.sh ggml-org/Ministral-3-8B-Instruct-2512-GGUF:Q8_0
#
# Two minutes, against ninety for a full run. Every candidate must pass this
# before it is worth measuring, because three in a row failed here and the
# remaining forty-nine documents of each run said nothing the first one had not.
#
# WHAT THIS CAN AND CANNOT TELL YOU
#
# A FAIL is conclusive: the model cannot make the call, do not spend a run.
# A PASS is not. This payload has been too easy twice — first with three fields
# instead of fourteen, then with a 1191-token prompt instead of production's
# ~5300 — and each time a pass sent a ninety-minute run after a model that
# failed on document one.
#
# It carries the real DocumentClassifierSchema and ~5000 tokens now, but it
# still cannot reproduce what actually inflates a paperless prompt: the RAG
# neighbours and the full taxonomy candidate lists, hundreds of names with IDs
# the model must copy exactly and align by position. Document 1970 is 1976
# characters and still produced a 5333-token prompt.
#
# So treat this as a cheap smoke test, not a gate. The real gate is
# ABORT_AFTER in eval.py, which stops a run after three failed documents —
# five minutes lost instead of ninety, using the genuine request.
set -eu
repo="${1:?usage: probe-toolcall.sh <hf-repo:quant> [extra llama-server args...]}"
shift || true
here="$(dirname "$0")"
ns=paperless-eval
pod="toolprobe-$(echo "$repo" | tr 'A-Z/:._' 'a-z----' | cut -c1-40 | sed 's/-*$//')"
cleanup() { kubectl -n "$ns" delete pod "$pod" --wait=false >/dev/null 2>&1 || true; }
trap cleanup EXIT INT TERM
cleanup
sleep 2
echo "probing $repo"
kubectl -n "$ns" apply -f - >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata:
name: $pod
spec:
restartPolicy: Never
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
args: ["--hf-repo","$repo","--alias","probe","--n-gpu-layers","99",
"--threads","4","--ctx-size","16384","--parallel","1",
"--ubatch-size","1024","--n-predict","768","--reasoning","off",
"--no-webui"$(for a in "$@"; do printf ',"%s"' "$a"; done)]
env: [{name: LLAMA_CACHE, value: /models}]
volumeMounts: [{name: models, mountPath: /models}]
resources:
requests: {cpu: "1", memory: 12Gi, gpu.intel.com/i915: "1"}
limits: {memory: 20Gi, gpu.intel.com/i915: "1"}
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: ["ALL"]}
volumes:
- name: models
persistentVolumeClaim: {claimName: llama-models}
EOF
echo "waiting for the model to load (a first download takes minutes)"
i=0
until kubectl -n "$ns" exec "$pod" -- curl -sf --max-time 10 http://127.0.0.1:8080/health >/dev/null 2>&1; do
i=$((i + 1))
[ "$i" -gt 120 ] && { echo "FAIL: server never became healthy"; kubectl -n "$ns" logs "$pod" --tail=20; exit 1; }
sleep 10
done
sed "s/\"MODEL\"/\"probe\"/" "$here/toolcall-probe.json" > /tmp/tcp.json
kubectl -n "$ns" cp /tmp/tcp.json "$pod":/tmp/probe.json >/dev/null 2>&1 || true
r=$(kubectl -n "$ns" exec "$pod" -- curl -s --max-time 300 \
-X POST http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' -d @/tmp/probe.json)
finish=$(printf '%s' "$r" | grep -oE '"finish_reason":"[a-z_]+"' | head -1 | cut -d'"' -f4)
echo
echo "finish_reason: ${finish:-<none>}"
printf '%s' "$r" | grep -oE '"arguments":"[^"]{0,300}' | head -1 || true
echo
# finish_reason "tool_calls" is the whole test. "length" means it never stopped,
# "stop" with no tool_calls means the server could not read the call back.
if [ "$finish" = "tool_calls" ]; then
echo "PASS — usable, worth a full run"
else
echo "FAIL — do not spend a run on this until the cause is understood"
exit 1
fi
scripts/paperless-eval/run-candidate.sh
Executable
+90
View File
@@ -0,0 +1,90 @@
#!/bin/sh
# Start one candidate's measurement.
#
# ./run-candidate.sh ministral8b
#
# Exists so the sidecar configuration lives in one file. Four copies of a
# 300-line Job manifest is how --chat-template-kwargs ended up on the isolation
# pod but not on the first eval job, and the run that followed measured empty
# completions for two hours before anyone noticed.
set -eu
label="${1:?usage: run-candidate.sh <label> [sample] (labels are in candidates.tsv)}"
# Screen before you measure. A candidate that cannot do the task at all is
# rejected by eval.py's ABORT_AFTER within three documents, so a screening run
# costs about five minutes; a full run costs ninety. Three of the first four
# candidates were rejections, and each one was allowed to run for an hour or
# more before anyone looked.
#
# ./run-candidate.sh <label> 5 screen
# ./run-candidate.sh <label> measure, 50 documents
#
# Screening uses genuine paperless requests, which matters: the standalone
# probe-toolcall.sh passed two candidates that then failed on document one,
# because a synthesised request cannot reproduce the RAG neighbours and the
# taxonomy candidate lists that make a real prompt what it is.
#
# Never compare numbers across sample sizes. A screen is a gate, not a
# measurement; the comparison happens at 50.
sample="${2:-50}"
# Third argument raises the server-side generation cap for a diagnostic run.
#
# The default 768 is MINE, not a property of any model, and every candidate
# recorded as failing at exactly 768 tokens is failing against my bound rather
# than demonstrably failing the task. A model that needs 900 tokens for a valid
# call looks identical to one that never terminates. Raise it and find out
# before calling a model incapable:
#
# ./run-candidate.sh gemma4-26b-a4b 5 4096
npredict="${3:-768}"
# Client-side bound, raised with the cap. At 6 tok/s a 4096-token budget needs
# 620 s of generation alone, so leaving this at 300 turns the diagnostic into a
# different failure rather than an answer.
timeout="${4:-300}"
here="$(dirname "$0")"
row="$(grep -v '^#' "$here/candidates.tsv" | awk -v l="$label" '$1==l')"
if [ -z "$row" ]; then
echo "unknown candidate '$label'. Known:" >&2
grep -v '^#' "$here/candidates.tsv" | awk 'NF{print " " $1}' >&2
exit 1
fi
repo="$(echo "$row" | cut -f2)"
hffile="$(echo "$row" | cut -f3)"
alias="$(echo "$row" | cut -f4)"
memreq="$(echo "$row" | cut -f5)"
memlim="$(echo "$row" | cut -f6)"
if [ -z "$hffile" ]; then
echo "candidate '$label' has no hf-file column — refusing to guess." >&2
echo "Resolve the exact .gguf filename from the HF API and add it." >&2
exit 1
fi
# The script is mounted from a ConfigMap, so it has to be current before the
# Job starts. Shared by every candidate — they run the same measurement.
kubectl -n paperless-eval create configmap paperless-eval-script \
--from-file=eval.py="$here/eval.py" --dry-run=client -o yaml | kubectl apply -f -
if [ "$sample" -lt 50 ]; then
echo "SCREENING $label at $sample documents — a gate, not a measurement"
fi
echo "candidate=$label repo=$repo alias=$alias mem=$memreq/$memlim sample=$sample n_predict=$npredict timeout=$timeout"
sed -e "s#__LABEL__#$label#g" \
-e "s#__HF_REPO__#$repo#g" \
-e "s#__ALIAS__#$alias#g" \
-e "s#__MEM_REQ__#$memreq#g" \
-e "s#__MEM_LIM__#$memlim#g" \
-e "s#__SAMPLE__#$sample#g" \
-e "s#__HF_FILE__#$hffile#g" \
-e "s#__N_PREDICT__#$npredict#g" \
-e "s#__TIMEOUT__#$timeout#g" \
"$here/job-template.yaml" | kubectl apply -f -
echo
echo "follow: kubectl -n paperless-eval logs -f job/eval-$label -c eval"
echo "re-run: kubectl -n paperless-eval delete job eval-$label # a Job is immutable"
scripts/paperless-eval/toolcall-probe.json
+18
View File
File diff suppressed because one or more lines are too long