fix(paperless-eval): three faults the first job run exposed #778
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#778
Reference in New Issue
Block a user
The Job from #776 ran and failed. That was more useful than it sounds — it surfaced three separate problems, one of which would have produced plausible-looking but wrong results.
It died before scoring anything
psycopg returns Postgres numerics as
Decimal, which does not coerce against a float literal. Both operands are cast now.It seeded nothing, and said so quietly
The
seed-datainitContainer printedseeded /data:and nothing else. The golden volume was empty — the smoke-test deployment had not built the LLM index yet — socp -acopied nothing anddu ... || trueswallowed the evidence.The run would then have measured the no-RAG path and reported it as a RAG result, which is exactly the failure that initContainer exists to prevent. It refuses to start now when
llm_indexis under 1 MiB (it is ~125 MiB once built) and prints the command to build it.It could not tell broken inference from a wrong guess
Past roughly 14k prompt tokens this stack returns empty completions and then runs of digits and CJK characters — status 200,
finish_reason: "stop"— and one such request poisons the llama.cpp slot for everything after it until the pod restarts.Scored as wrong answers, those would quietly depress whichever candidate happened to trip the boundary, and take the rest of that run down with it. A candidate could lose a comparison to a backend fault.
They are now flagged, excluded from the accuracy figures, and counted separately:
A non-zero
degeneratemeans the run is suspect from that point on — not that the model is bad. This matters for the real corpus: 252 of ~3900 documents are over 14000 characters, and their RAG neighbours push them toward the boundary.Note
The LLM index build on the eval instance is running now — ~7100 chunks at ~72/min, so roughly 100 minutes before any AI measurement can start.
🤖 Generated with Claude Code
🤖 Automated review — model
claude-sonnet-4-6(complexity: normal), commite26108966690bfffb55546914602f6ceaf5791af.Summary
Three targeted fixes following the first real job run: a
Decimal/floattype error that crashed scoring, a silent no-op seed that would have measured the wrong path, and a detection layer for broken LLM inference that would otherwise have silently depressed accuracy numbers. All three fixes address confirmed, concrete problems and the logic is sound. No blocking issues.🔴 Blocking
None.
🟡 Suggestions
job-qwen35.yaml:73— informationalducan fail before the helpful FATAL messageThe script runs under
set -e. The line:has no
|| true. If the golden volume was completely unpopulated and neither directory was copied in, thisduexits non-zero and the container stops here — before the FATAL message with thedocument_llmindex rebuildinstructions is ever printed. The user sees a rawdu: cannot access '/data/llm_index': No such file or directoryinstead of the actionable build command.The primary protection still works (the container refuses to proceed), but the exact failure scenario this guard was written for is the one where the diagnostic degrades. Adding
|| trueto the informational line, or using2>/dev/null || true, would ensure the FATAL path is always reached whenllm_indexis absent:eval.py:261—degenerate()is applied to both engines, not justaidegenerate()detects a failure mode (llama.cpp slot poisoning returning digit/CJK garbage) that is specific to the AI engine. Becauserun()calls it regardless ofengine, a legitimate classic-classifier document whose title is mostly numeric (e.g. a document titled2024-001) could be flagged and excluded from thecomplement()analysis for both engines. In practice the threshold (40% letter ratio) is reasonable and classic titles are unlikely to be garbage, but the intent could be made explicit with a guard:This also documents that the check is a llama.cpp-specific heuristic rather than a general data-quality filter.
🟢 Nits
eval.py:218: The generator expressionc.isalpha() and c.isascii() or c in "äöüÄÖÜß"is correct (Python evaluatesandbeforeor), but the implicit precedence is easy to misread. Parenthesising(c.isalpha() and c.isascii()) or c in "äöüÄÖÜß"makes the intent unambiguous at a glance.Tests
No test suite exists in
scripts/paperless-eval/. The newdegenerate()function encodes a concrete threshold (40% letter ratio, specific German umlauts) that could silently shift if the function is adjusted later. A handful of unit tests covering the empty-title case, a pure-digit title, a normal Latin title, and a mixed CJK+digit string would lock the boundary in place. Not a blocker given the eval-script context, but worth adding before the threshold is tuned against real corpus data.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit34c250ad2a6e5004f251283130aec01bfe982267.Summary
This PR fixes three problems surfaced by the first real run of the paperless-eval job (#776): a
Decimal/floatTypeErrorinfloor(), aseed-datainitContainer that silently produced an empty LLM index, and unflagged "degenerate" LLM completions past ~14k prompt tokens that would have been scored as ordinary wrong answers. I read both changed files in full plus the README for context. I did not find a concrete blocking bug — the fixes address the failure modes described in the PR body correctly — but there's one fragile interaction in the new shell guard and a couple of judgment calls in the heuristic worth a second look.🟡 Suggestions
scripts/paperless-eval/job-qwen35.yaml:112— the un-guardeddu -sh /data/index /data/llm_index(the|| truewas deliberately dropped) is what currently makes the size check at line 119-126 safe when/data/llm_indexis missing entirely: if the path doesn't exist, thisdufails andset -eaborts the script before reaching thebytes=$(du -sb /data/llm_index | cut -f1)line. If that line were ever removed or reordered (e.g. because someone finds its output redundant withdu -sbtwo lines below), the bytes check would silently break:du -sbon a nonexistent path prints nothing to stdout, sobytesbecomes an empty string, and[ "$bytes" -lt 1000000 ]errors out ("integer expression expected") rather than failing safe —set -edoesn't abort on a failingifcondition, so the script would fall through topaperlessstarting anyway. This is exactly the class of silent-failure bug this PR is fixing, just relocated. Making the check self-contained (e.g.bytes=$(du -sb /data/llm_index 2>/dev/null | cut -f1 || true); bytes=${bytes:-0}) would remove the dependency on the earlier command's failure mode.scripts/paperless-eval/eval.py:218-219— the letter-ratio heuristic (letters < 0.4 * len(title)) counts only ASCII letters plusäöüÄÖÜß; anything else (digits, punctuation, other accented Latin characters) counts against the ratio. A title that's legitimately number-heavy — an invoice title embedding a reference number or date, e.g."Rechnung 2024-03-15 Nr. 800123456"— could plausibly fall under 0.4 letters and get flagged asnon-latin titleeven though it's a normal (if unhelpfully formatted) LLM answer, not degenerate output. Since I don't have access to realai_suggestionstitle output to check how often this actually happens, I can't confirm this is a live problem — but it's worth validating against the logs from the run mentioned in the PR body (or a handful of real non-degenerate titles) to see if the threshold needs tuning, since a false positive here silently shrinksscoredand inflatesdegeneraterather than corrupting the accuracy numbers (fails safe, but still worth checking).scripts/paperless-eval/README.md— "Reading the output" documentscomplement,correspondent_offered/recognised, andtype_specific/catchall, but not the newdegeneratefield in the summary, even though the PR body treats a non-zerodegenerateas the signal that a run is suspect. Worth a line there so a reader of a.summary.jsonfile (without also reading this PR) knows what it means.🟢 Nits
scripts/paperless-eval/job-qwen35.yaml:112and:119—du -sh /data/llm_indexanddu -sb /data/llm_indexboth run against the same path a few lines apart; could be collapsed (e.g. derive the human-readable size from the byte count) but this is purely cosmetic.scripts/paperless-eval/job-qwen35.yaml:118— the comment says "anything under 1 MiB is not one" but the threshold used (1000000bytes) is decimal 1 MB (~976.6 KiB), not 1 MiB (1,048,576 bytes). Doesn't matter at these thresholds (built index is ~125 MiB) but the comment is technically imprecise.Tests
No automated tests exist for
eval.pyor the job manifest, consistent with the rest ofscripts/paperless-eval/(it's an operator-run script against a live Kubernetes Job, not part of a tested package — there's no test scaffolding elsewhere in this directory). Given that, manual verification via the next job run is the realistic test plan here, and the PR body indicates that run is already in progress. Thedegenerate()heuristic in particular has no unit coverage; a short table-driven test (empty title, all-digits title, CJK title, a normal German title, a number-heavy-but-legitimate title) would be cheap insurance against the false-positive risk noted above and would document the intended behavior better than the docstring alone.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Nachgeschoben, nachdem der Lauf noch einmal gescheitert ist — diesmal nicht am Skript, sondern am Modell.
Es hört nicht auf
Drei der ersten elf Dokumente terminierten nie. Eines erreichte 10269 Token für eine Klassifikation, die rund 300 braucht — bei 11 Token/s, also kerngesund. Von außen sieht das aus wie ein langsames Modell und ist keines.
llama-servers
--n-predictsteht auf -1, unendlich. Paperless setzt kein eigenes Token-Budget und wartet. Damit begrenzte nichts die Anfrage, und vier Dokumente kosteten den größten Teil einer Stunde.--n-predict 768auf dem Sidecar deckelt das. Eine abgeschnittene Antwort ist besseres Material als ein Timeout: sie zeigt, was das Modell stattdessen tat. Der Client-Timeout fällt auf 300 s und wird zum Auffangnetz — er funktioniert, das Log zeigtstop: cancel taskbeim Verbindungsabbruch und 0,4 s später den nächsten Task.Zwei Kleinigkeiten
Die Degeneriert-Prüfung schlug bei 'Rechnung 2025-5425541' an — 8 Buchstaben auf 21 Zeichen, unter der 40-Prozent-Schwelle. Ziffern und Satzzeichen sind in einem Titel normal; sie sucht jetzt direkt nach Zeichen fremder Schriften, statt sie aus einem Mangel an lateinischen zu erschließen.
Und der
--ctx-size-Kommentar trug noch die Behauptung, die #777 per Bisektion widerlegt hat: dass 32768 die Inferenz zerstört. Die Grenze ist die Prompt-Länge.🤖 Automated review — model
claude-sonnet-5(complexity: normal), commitcb751cbedc7b945d90191a53d5517722b809bab8.Summary
This PR fixes three faults surfaced by the first real run of the paperless-eval job: a
Decimal/floatTypeErrorinfloor(), a silently-empty golden volume in theseed-datainitContainer, and (new) detection of degenerate LLM output so a poisoned llama.cpp slot doesn't get scored as wrong answers. TheDecimalfix and the golden-volume guard are both correct and well-targeted. The newdegenerate()detector, however, only catches half of the failure mode the PR itself describes.🔴 Blocking
scripts/paperless-eval/eval.py:220-231—degenerate()doesn't detect the digit-run failure mode it's meant to catch. The PR body and the function's own docstring both describe the corruption as producing "empty completions" then "runs of digits and CJK characters." The implementation only flags non-empty titles whose alphabetic characters are disproportionately non-Latin (c.isalpha() and not (c.isascii() or c in "äöü…")). A title that is a pure digit run — e.g."48592038574029385740293857"— has zero alphabetic characters, soforeignis0, the ratio check never fires, anddegenerate()returnsNone. That document then flows throughscore_document()as an ordinary wrong answer instead of being excluded and counted indegenerate, which is exactly the failure this PR sets out to prevent ("Scored as wrong answers, those would quietly depress whichever candidate happened to trip the boundary"). Since the eval script's whole purpose here is to keep a poisoned-slot document from silently distorting the accuracy figures, and one of the two named corruption patterns can slip through untouched, this should be fixed before the next scored run — e.g. by also flagging titles with a very low alphabetic-character ratio overall (not just a high non-Latin-among-alphabetic ratio), or by checking for long digit runs directly.🟡 Suggestions
scripts/paperless-eval/eval.py:208-231—degenerate()is pure and cheap to unit-test (no network, no k8s needed), and its correctness now directly gates whether a document counts against a candidate's score. Worth a few inline test cases (empty title, normal German title with digits, CJK-heavy title, pure-digit title) once the gap above is closed, so a future retune of the threshold doesn't regress silently.scripts/paperless-eval/job-qwen35.yaml:84— the comment says "anything under 1 MiB is not one" but the check compares against1000000bytes (1 MB decimal, ~4.6% smaller than 1 MiB = 1048576 bytes). Doesn't matter at the actual scale (~125 MiB vs. an empty/near-empty index), just a minor mismatch between comment and code if the threshold is ever tightened.🟢 Nits
scripts/paperless-eval/job-qwen35.yaml:71-92— the rationale for refusing to start on an emptyllm_indexis stated twice (once in the initContainer's top-level comment at line ~59, again inline at line ~79-84). Not wrong, just repeats itself.Tests
No automated tests exist for this script (none did before this PR either — it's an operational k8s eval tool, consistent with the rest of the repo). Given
degenerate()is new, self-contained logic with a real bug (see Blocking above), it's the one piece here that would clearly benefit from a couple of unit tests even in a repo that otherwise doesn't carry them, since its failure mode is silent by design (aNonereturn looks identical to "this document is fine").Recommendation
Request changes — 1 blocking issue.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commita7bba7acbb2f2b6f57235ff69c6e04082dd7a0db.Summary
This PR fixes three faults surfaced by the first real run of the paperless-eval job: a
Decimal/floatcrash infloor(), a silently-empty seed volume that would have made a no-RAG run masquerade as a RAG result, and un-flagged "degenerate" LLM completions past ~14k prompt tokens that would otherwise be scored as ordinary wrong answers. It also generalizes the singlejob-qwen35.yamlinto a templatedjob-template.yaml+candidates.tsv+run-candidate.sh, so adding a model is a one-line data change instead of a manifest copy. I traced each fix against its caller/context in the checkout and didn't find a defect that breaks the stated goal — no blocking issues.🟡 Suggestions
eval.py:220-231(degenerate()) may miss a real failure mode it's meant to catch. The docstring says the broken-inference symptom is "runs of digits and CJK characters," but the detector only counts non-ASCII/non-listed-accented alphabetic characters against title length (eval.py:228-229). A degenerate completion whose title is mostly digits/punctuation with no CJK (e.g. all-digit garbage) would scoreforeign == 0and pass through as a normal answer — exactly the "scored as a wrong answer, quietly depresses a candidate's numbers" failure this PR is trying to eliminate for the CJK case. Worth checking against the actual<label>.jsonlfrom a completed run for titles that are digit-heavy but weren't flaggeddegenerate. If none show up, the empirical basis in the PR body (digits and CJK appearing together, not separately) holds and no change is needed.job-template.yaml:65-92(seed-data): the new byte-count guard only checksllm_index, notindex. The same script also doesdu -sh /data/index /data/llm_index— if the search index (/data/index) were the one left unseeded instead, theduitself would fail loudly underset -e(since removing2>/dev/null || truenow surfaces that), so it's still caught, just with a less specific message than the new "FATAL: llm_index is..." one. Not a defect, just an asymmetry worth a one-line comment if it's intentional (the LLM index is the expensive/silent one; the search index failing to copy would presumably break every request obviously).🟢 Nits
run-candidate.sh:15(row=$(grep -v '^#' ... | awk -v l="$label" '$1==l')) — ifcandidates.tsvever grows a duplicate label,$rowwould hold two lines and the latercut -f2calls would silently concatenate both with a newline, breaking thesedsubstitution with an unhelpful error. Cheap enough to guard withawk 'NF && $1==l{print; c++} END{exit c!=1}'-style uniqueness check, but not worth it unless the file is hand-edited by more than one person.Tests
No automated tests exist for this script or elsewhere in this repo (it's a GitOps/infra repo, not a software project with a test harness), which matches the surrounding convention — the closest thing to a test here is the manual job run itself, and the PR body documents that the fixes were validated against exactly that (the failing run in #776, and the
degenerate: 3example in the actual output). That's consistent with how this directory has always been verified.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
I wrote in the last commit that passing '{"enable_thinking": false}' to every candidate was a no-op on templates that never read the variable. That is wrong, and it cost a run. Ministral-3-8B produced token salad on a 681-token prompt — far below the ~14k corruption boundary, so not that defect: 1UnComplete1Self1UnCompleteSDirect11S1S111Document1CompleteFromS11Un... Every document failed, and each one generated to the --n-predict cap. The identical request against a server started without the flag returned a clean tool call and the correct answer. Plain German was fine throughout — "Die Hauptstadt von Frankreich ist Paris" — so it only ever broke with tools, which is what made it look like a model that cannot do function calling. llama-server prints "Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead." at startup. It means it. --reasoning off is verified to give Ministral a proper tool_calls response before this went near a real run. Qwen3.6's template does consume enable_thinking, so its measured result stands — but its run used the old flag, and the configuration should be uniform across candidates before the numbers are compared. Worth a re-run. granite8b is commented out of candidates.tsv, and this one is not my flag: it classifies correctly and llama.cpp b10731 simply cannot read its tool calls back, returning them as content with finish_reason "stop". Reproduced on an idle server both with and without the flag. Details are in the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
claude-sonnet-5(complexity: normal), commitc2f47e58bc080aab7eb711e1bf74fb05e69cdb38.Summary
Fixes three real problems surfaced by the first job run (a
Decimal/floatcrash infloor(), a silent no-op seed that could measure the no-RAG path and report it as RAG, and undetectable degenerate completions past ~14k prompt tokens), and refactors the per-candidate Job into a template +candidates.tsv+run-candidate.shso adding a model is a one-line diff instead of a manifest copy. The logic changes are small, targeted, and each is backed by a concrete incident described in the PR body. No blocking issues found; two robustness points below are worth a look.🟡 Suggestions
scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexhas no2>/dev/null/|| true(unlike the old code), so if/data/llm_indexdoesn't exist at all (rather than existing-but-tiny), this line fails underset -ewith a raw "cannot access" error before reaching the new byte-count check at line 85. That means the friendlyFATAL: llm_index is ... bytes — build it on the smoke-test deployment firstdiagnostic — the whole point of this fix, per the PR body — would not be shown in exactly the "golden volume never had the LLM index built" scenario that originally motivated it; the initContainer would still fail (so the safety property holds), just with a less actionable message. I couldn't confirm from this repo whether paperless_ai pre-creates an emptyllm_indexdirectory on startup (in which case this never triggers) or only creates it once documents are indexed (in which case it does); worth checking the paperless_ai source or just testing against a genuinely never-built golden volume. If it's a real gap, redirecting stderr on that line (or computingbyteswith2>/dev/nulland defaulting to0) would make the diagnostic reliable in both cases.scripts/paperless-eval/eval.py:228—degenerate()'s allowed-diacritics set (äöüÄÖÜßáéèêíóúàçñ) covers German/French/Spanish but not e.g. Polish/Czech/Nordic diacritics. Low risk given the corpus is described as German/English business correspondence, but a correspondent name using an uncovered diacritic would be misclassified as "non-latin title" (degenerate) rather than scored normally. Worth keeping in mind if a future candidate's real output includes such a name.scripts/paperless-eval/run-candidate.sh:15,22-25— ifcandidates.tsvever grew a duplicate label,rowwould contain two lines andcut -f2..5would return multi-line values, silently embedding a newline into$repo/$alias/etc. and corrupting thesedsubstitution intojob-template.yamlrather than erroring clearly. Self-inflicted (the file is hand-maintained by the same person adding candidates) so low priority, but a one-line[ "$(echo "$row" | wc -l)" -eq 1 ]guard would fail loudly instead.🟢 Nits
scripts/paperless-eval/eval.py:157-175(floor) — theDecimal/floatbug fixed here is exactly the kind of thing a two-line unit test (floor()against a fake cursor, or just a standalone arithmetic assertion) would have caught before it reached the actual job run. Not asking for a test suite for an ops script, just noting it for next time given the PR is explicitly about faults the first run exposed.Tests
No automated tests exist for this script or are added here, consistent with how this directory has always been treated (a hand-run ops tool, not part of the app's test suite). The three fixes were each validated against a real failure from the prior job run per the PR body, which is reasonable verification for this kind of script, though it means the two suggestions above haven't been exercised against the actual failure modes they touch.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Three candidates in a row failed every document, and each time I let it run. The harness now stops after ABORT_AFTER = 3 consecutive failures from the first document, because document one already said everything the remaining forty-nine would. Not a ratio — a model failing a third of the time is a result worth measuring in full. This only catches "nothing has ever worked". What the three were, since none of them is a verdict on the model: granite8b classifies correctly, but llama.cpp b10731 hands its tool call back as content with finish_reason "stop" and no tool_calls array, so llama_index raises "Expected at least one tool call, but got 0". ministral8b given paperless's real 14-field DocumentClassifierSchema, opens ministral3b a valid tool call and then fills its first string with its own control tokens until the cap: {"title": "[TOOL_CALLS][ARGS][THINK][/THINK][THINK]... The grammar allows any character inside a JSON string, so those are "valid" and it never closes. I chased the Ministral failure as a prompt-length problem for an hour on the strength of a three-field probe that worked. It reproduces at a 1191-token prompt with the real schema, and a 5799-token prompt with the simple one gets a clean tool call — the schema is the variable, not the length. So the first thing to establish about a candidate is not German, or speed, or classification quality. It is whether it can emit one valid call against a 14-field schema, and that is a two-minute probe rather than a ninety-minute run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
claude-sonnet-5(complexity: normal), commitb00ca34783f14ea4322a287dc978108c9a19cefb.Summary
This PR fixes three real faults surfaced by the first
paperless-evaljob run: aDecimal/floatTypeErrorinfloor(), a silent no-op seed step that would have quietly measured the no-RAG path, and an unflagged "degenerate" failure mode where the llama.cpp slot poisons after long prompts. It also generalizes the singlejob-qwen35.yamlinto a templatedjob-template.yaml+candidates.tsv+run-candidate.shso future candidates don't require copy-pasting a 300-line manifest. I readeval.py,job-template.yaml,run-candidate.sh, andcandidates.tsvin full against the diff and did not find a blocking defect.🟡 Suggestions
scripts/paperless-eval/eval.py:233—degenerate()'s non-Latin heuristic whitelists only German/French-ish accented characters (äöüÄÖÜßáéèêíóúàçñ). A legitimate correspondent/title using other Latin-adjacent characters (Polish "ł/ę", Czech "ř/č", Turkish "ş/ğ", etc.) would count every such letter as "foreign" and could push a normal title over the0.2 * len(title)threshold, silently excluding a correctly-scored document from the accuracy figures. Given the corpus is described as mostly German business documents this is probably rare in practice, but worth knowing if a candidate'sscoredcount looks abnormally low compared tofailed+degenerate.scripts/paperless-eval/job-template.yaml:77-92—du -sh /data/index /data/llm_index(informational) followed immediately bydu -sb /data/llm_index | cut -f1(the actual gate) runsdutwice over the same tree. Harmless at ~125 MiB, but could be collapsed into one call if you want to shave a bit of edge case complexity — not worth blocking on.scripts/paperless-eval/job-template.yaml:99-101— the newrunAsUser: 1000 / runAsGroup: 1000onseed-datais justified in the comment by "Running as 1000 makes the copy land with the ownership paperless needs." Thepaperlesssidecar's own securityContext (lines 254-259, unchanged) still runs as root at entrypoint and addsCHOWN/DAC_OVERRIDE/FOWNERspecifically so its entrypoint can fix ownership on/dataat startup. If that entrypoint chowns recursively on every boot (common for the paperless-ngx image), the newrunAsUser/runAsGroupaddition may be redundant rather than load-bearing — not a bug, but worth confirming against actual pod behavior on the next run rather than assuming, since the PR body doesn't call this test out explicitly the way it does for the other two fixes.🟢 Nits
scripts/paperless-eval/run-candidate.sh:33-38— thesedsubstitutions use raw$label/$repo/$alias/$memreq/$memliminsides#...#...#gwithout escaping#,&, or\. Not exploitable today since values only ever come from the checked-incandidates.tsv, but if a future candidate row's hf-repo or alias ever contains one of those characters, the substitution would silently produce a broken manifest rather than erroring.Tests
No automated tests exist for this directory (it's an operational scripts folder driven by manual
kubectlruns against a live cluster), consistent with the rest of the repo. Thefloat()fix, theABORT_AFTERearly-exit, and thedegenerate()heuristic are all easy to unit test in isolation (pure functions on plain dicts/Decimal-like inputs) if this script grows further, but given the PR's own description that the fixes were validated by rerunning the actual job, I don't consider added tests blocking here.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Every parallel run so far was quietly broken, and it took a 401 sitting in the middle of three genuine model failures to notice. api_token() took the first superuser and deleted its existing token before inserting a new one, because Django's authtoken holds one token per user. Both candidates share this database, so the second job to start silently revoked the first one's credentials. In the log it appears as 50173: HTTP 401: b'{"detail":"Invalid token."}' among HTTP 400s from the model, where it reads like one more broken document. It was counted toward ABORT_AFTER, so a harness fault helped end a run and would have been recorded as a model verdict. Two changes. Each run now creates its own eval-<label> superuser, so the token belongs to the run and concurrency is safe — only the user can differ, the database cannot. And a 401 is now fatal with an explicit message rather than another row in the error column: it is never a fact about the model, and every number after it is worthless. The qwen35 result predates this and ran alone, so it is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commitb64f971373a3f67b765ed082831d86323a8e1551.Summary
This PR fixes three faults surfaced by the first real eval job run: a
Decimal/floatTypeErrorinfloor(), a silently-empty golden volume that would have measured the no-RAG path and reported it as RAG, and no way to tell a poisoned llama.cpp slot from a genuinely wrong model answer. It also generalizes the eval harness from one hardcoded job (job-qwen35.yaml) to a template driven bycandidates.tsvandrun-candidate.sh, and fixes a real concurrency bug where two candidates running at once would revoke each other's API token. The three described faults are fixed correctly and are well grounded in the diff. I found no blocking defects, but there's a genuine (if now rare) data-loss gap in the new fatal-abort path worth a look before it bites a 90-minute run.🟡 Suggestions
scripts/paperless-eval/eval.py:293-299— On a mid-runHTTP 401,run()raises a bareSystemExitthat propagates straight out ofmain(). Butmain()only writes{LABEL}.jsonl/{LABEL}.summary.jsonafter both engines finish (eval.py:417-434) — nothing is flushed incrementally. So if the token dies partway through theaiengine (e.g. document 45 of 50, each of which can take up toREQUEST_TIMEOUT), the entire run's results are discarded, including the already-completeclassicengine and everyaidocument scored before the failure — not just "every number after it" as the docstring says (eval.py:114-116). Given the whole point of this PR is to stop wasting GPU-hours on unusable runs, losing up to ~90 minutes of already-valid data to an unrelated auth hiccup is exactly the kind of waste this PR is trying to prevent elsewhere (seeABORT_AFTER). Note the per-label user fix elsewhere in this same diff (eval.py:103-142) makes the two-candidates-sharing-a-token race that motivated this path largely closed already, so the trigger is now rare — but when it does fire (e.g. a stale token, manual DB interference), consider writing out whatever rows were already collected before raising, or writing incrementally as documents are scored.scripts/paperless-eval/job-template.yaml:77-92—du -sh /data/index /data/llm_index(line 78) no longer has the|| truethe old version had, and runs underset -e(line 71). If/data/llm_indexdoesn't exist at all yet (rather than existing-but-small),duexits 1 on the missing path and the script aborts right there — before it reaches the byte-count check that prints the intended friendlyFATAL: llm_index is ${bytes} bytes — build it on the smoke-test deployment first...message (lines 86-91). The container still fails closed (the job still won't proceed to measure the wrong thing), so this doesn't reintroduce the bug this PR is fixing, but the diagnostic that's the whole point of the initContainer would be replaced by a baredu: ... No such file or directory. Worth confirming whether the golden PVC'sllm_indexdirectory is guaranteed to exist (even empty) before the index build finishes; if not, seeding an empty dir first or guarding theducall would preserve the intended message.scripts/paperless-eval/probe-toolcall.sh:618-627—kubectl cp /tmp/tcp.json "$pod":/tmp/probe.json >/dev/null 2>&1 || truesilently swallows a copy failure (e.g.tarmissing from the llama.cpp server image, whichkubectl cprequires). If the copy fails, the followingcurl -d @/tmp/probe.jsoninside the pod has no file to read,rends up empty/error text,finishparses to empty, and the script reportsFAIL — do not spend a run on this until the cause is understood— indistinguishable from a real model failure. That's the exact ambiguity the rest of this PR (degenerate(), theABORT_AFTERmessaging, the 401 handling) is careful to avoid elsewhere. Consider dropping the|| trueor checking the copy's exit status explicitly.🟢 Nits
scripts/paperless-eval/job-template.yaml:86— the threshold1000000bytes is decimal (≈0.954 MiB), while the surrounding comments (and the PR body) say "under 1 MiB". Off by ~5%; harmless given the real index is ~125 MiB, but the comment and the constant don't quite agree if anyone tightens it later.Tests
No automated tests are added or exist for this harness, which is consistent with the rest of
scripts/paperless-eval/(it's k8s/GPU-dependent ops tooling, exercised by actually running it, which is how these three faults were found). That said,floor()'sDecimal/floatbug anddegenerate()'s title-heuristic are both pure functions with no I/O — a handful of offline unit tests (aDecimaltotal fromsample(conn)-shaped input, a title with digits/punctuation vs. one with CJK runs) would have caught theTypeErrorbefore a live run did, and would guardsummarise()'s scored/failed/degenerate partitioning going forward.Recommendation
Looks reasonable — no blocking issues; comments above are optional but the
SystemExitdata-loss path is worth a deliberate look.🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit440cbaec90ea31e6fe6b44d389069ab4585dd57c.Summary
This PR fixes the three faults the first
paperless-evaljob run exposed: aDecimal/floatTypeErrorinfloor(), a silent no-op seed step that could make a run measure the no-RAG path while reporting it as RAG, and no way to distinguish a poisoned llama.cpp slot's garbage output from a genuine wrong answer. It also generalizes the eval Job into a template driven bycandidates.tsv/run-candidate.sh, and adds aprobe-toolcall.shsmoke test plus an early-abort mechanism ineval.py. I traced each of the three fixes against the code and they are correct; no blocking issues found.🟡 Suggestions
scripts/paperless-eval/job-template.yaml:85—bytes=$(du -sb /data/llm_index | cut -f1)runs underset -ewith no|| trueand no existence check first. If/golden/llm_indexis entirely absent (rather than just under the 1 MiB threshold),duitself fails and the container exits ondu's own error message rather than the intended "FATAL: llm_index is ... bytes — build it on the smoke-test deployment first" guidance. Worth a[ -d /data/llm_index ] || bytes=0(or similar) ahead of theducall so the friendly message covers the "missing entirely" case too, not just "too small".scripts/paperless-eval/probe-toolcall.sh:627—kubectl -n "$ns" cp /tmp/tcp.json "$pod":/tmp/probe.json >/dev/null 2>&1 || truesilently swallows a real copy failure (e.g.tarmissing in the probe image, wrong pod name). Ifcpactually fails, the followingcurl -d @/tmp/probe.jsoninside the pod will error on a missing file, which reads like a probing/model problem rather than a harness problem. A quick check after thecp(or dropping the|| trueand lettingset -eucatch it) would make that failure mode clearer.scripts/paperless-eval/eval.py:293-299— on anHTTP 401, the script raisesSystemExitimmediately without writing anything toOUT_DIR, discarding rows already scored for this run (e.g., an entire completedclassicpass if theaipass is the one that hits 401). That's consistent with the stated intent ("every number after it is worthless"), but if you ever want to keep the partial.jsonlfor post-mortem debugging of when the token got revoked, this is the place afinally/early flush would go. Not required — just flagging the tradeoff since it's a change in behavior from before (previously a bad token would just show up as per-document errors).Tests
No test suite exists for this directory (it's operational/ops tooling), consistent with the rest of
scripts/. Theprobe-toolcall.shsmoke test and the newABORT_AFTERearly-exit ineval.pysubstitute reasonably for automated tests here — they were added specifically because the first live run surfaced failures that a test suite couldn't have caught without a real llama.cpp/paperless stack. I verified the three specific fixes by reading the code paths directly (see below) rather than executing the script, since this environment has no sandbox to run it against a real Postgres/paperless instance.floor()(eval.py:179-197):sum(c)over acount(*)grouping returns a Postgresnumeric(→Decimalvia psycopg) whilemax(c)returnsbigint(→int); the original100.0 * top / totalfailed exactly as the traceback in the PR body shows. Wrapping both operands infloat()fixes it, and theif total else 0.0guard still short-circuits before either conversion whentotalisNone/0, so no newTypeError/ZeroDivisionErroris introduced.job-template.yaml:65-104): the old script had2>/dev/null || truearounddu, so acp -athat copied nothing due to missing CHOWN capability failed silently and never blocked startup. The newcp -r(no ownership preservation, so no CHOWN failure) plus the explicitdu -sb/exit 1size gate correctly turns "seeded nothing" into a hard failure beforepaperless(a sidecarrestartPolicy: Alwayscontainer) ever starts.degenerate()/ABORT_AFTER(eval.py:230-253,272-332,339-370,373-406): correctly wired through — flagged rows are excluded fromscoredinsummarise()and from theusableset incomplement(), and counted separately asdegenerate. TheABORT_AFTERchecklen(rows) == ABORT_AFTER and all("error" in r for r in rows)only fires once (sincerowskeeps growing past 3) and only on realerrorrows, notdegenerateones, matching the intent described in the comment.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
CLI default(complexity: complex), commit44636a82686b06270ebe0fdcb7e531cb9ce3875f.Summary
Three fixes to the paperless-eval harness, all grounded in a real failed run: a
Decimal/floatTypeErrorinfloor(), a seed-data initContainer that silently produced empty results, and a newdegenerate()detector that flags llama.cpp responses corrupted by the ~14k-token prompt boundary so they don't get scored as wrong answers. The PR also generalizes the singlejob-qwen35.yamlinto a templatedjob-template.yaml+run-candidate.sh+candidates.tsv, and adds a standaloneprobe-toolcall.shsmoke test. The changes are well-reasoned and each is backed by an observed failure described in the commit/PR body. No blocking issues found.🟡 Suggestions
scripts/paperless-eval/eval.py:293-299— the new HTTP‑401 fast-abort (raise SystemExit(...)) fires from insiderun(), which is called frommain()before results are ever written to disk (the.jsonl/.summary.jsonwrite only happens once, after both engines finish, ateval.py:430-434). Sinceclassicalways runs beforeai(eval.py:418), a 401 partway through theaiengine discards the fully-completedclassicresults and every already-scoredaidocument along with it, not just "everything after" the failure as the error message claims. Given a full run costs on the order of 90 minutes (per the PR description), this is an expensive way to lose otherwise-valid data. Since the per-label user (eval-{LABEL}) this PR introduces is itself the fix for the concurrent-token-revocation race, this path should now be rare — but it's worth writing the results collected so far before theSystemExit, or at least gating the abort so it only discardsairows rather than the whole run.scripts/paperless-eval/probe-toolcall.sh:636—kubectl cp ... || truesilently swallows a failed copy ofprobe.jsoninto the pod.kubectl cprequirestarinside the target container; I could not confirm whetherghcr.io/ggml-org/llama.cpp:server-vulkan-b10731includes it. If it doesn't, the subsequentcurl -d @/tmp/probe.json(line 637) fails to find the file,finish_reasoncomes back empty, and the script reportsFAIL — do not spend a run on this until the cause is understood(line 652) — which would misattribute a script/environment problem to the model, exactly the kind of misleading signal this PR is elsewhere careful to avoid. Worth confirmingtaris present, or copying viakubectl exec ... sh -c 'cat > /tmp/probe.json'instead, which doesn't depend on it.🟢 Nits
scripts/paperless-eval/eval.py:250— the non-Latin heuristic indegenerate()allowlists lowercaseäöüÄÖÜßáéèêíóúàçñbut not their uppercase French/Spanish counterparts (e.g.É,Á,Í). Given the eval corpus is described as German invoices, this is unlikely to matter in practice.scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexno longer has the2>/dev/null || truethe old version had, andset -eis in effect. If/data/indexis ever missing after the copy (as opposed tollm_indexmerely being under the 1 MiB floor), the container fails on this line with a baredu: cannot accessinstead of the friendlierFATAL:message a few lines below — a smaller version of the same "quiet failure" problem this PR is fixing elsewhere, just for a different path.Tests
No automated tests exist for this directory (it's an operational script, not application code), consistent with the rest of the repo. The PR body describes manual verification in progress (an actual eval job run against the fixes). That's appropriate for this kind of change; nothing here seems to call for unit tests.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
CLI default(complexity: complex), commitf6b0789c017ee5c2cc5f92b45118ccbebd12033c.Summary
This PR fixes three concrete problems the first real run of the paperless-eval Job exposed: a
Decimal/floatTypeError that killed scoring before it started (eval.py:197), a silent no-op seed that could report a no-RAG result as a RAG result (job-template.yaml'sseed-datainitContainer), and an inability to distinguish "wrong answer" from "backend fell over" past ~14k prompt tokens (degenerate()ineval.py). It also reworks the single-candidatejob-qwen35.yamlinto a templatedjob-template.yaml+candidates.tsv+run-candidate.sh, adds an early-abort for candidates that fail every document, adds a per-label DB user to stop concurrent runs from revoking each other's API tokens, and adds a standaloneprobe-toolcall.shsmoke test. I traced each of the three claimed fixes against the code and they do what the PR body says. No blocking defects found.🟡 Suggestions
job-template.yaml:78—du -sh /data/index /data/llm_index(no longer followed by|| true) will abort the initContainer with a rawdu: cannot access ... No such file or directoryunderset -eif/goldennever had anllm_indexdirectory at all (as opposed to one that's merely small). That still fails safe — the Job still refuses to start — but it means the friendlyFATAL: llm_index is ${bytes} bytes ...diagnostic added right below it (line 85-92) won't fire in exactly the "index was never built" case the PR's own description describes. Considerdu -sb /data/llm_index 2>/dev/null || echo 0feeding the size check directly, so the polished message always fires regardless of whether the directory exists at all vs. exists-but-small.eval.py:250— the "non-Latin" heuristic indegenerate()allowlists a fixed set of German/Latin diacritics (äöüÄÖÜßáéèêíóúàçñ). Given the corpus spans ten years of real business correspondence, a legitimate title with e.g. Polish/Czech/Nordic diacritics (ł, č, ø, å) not in that list could trip the >20% threshold and get misclassified as "degenerate" (backend garbage) rather than scored as a wrong answer. Low risk given the observed failure mode is CJK/digit runs, but worth a comment noting the allowlist is Latin-only, or switching to a check based on Unicode script (Latin vs. non-Latin) rather than an explicit character allowlist.probe-toolcall.sh:666—kubectl cp ... || truesilently swallows a copy failure (e.g. if the llama.cpp server image lackstar, whichkubectl cpneeds). The subsequentcurl -d @/tmp/probe.jsonwould then fail on a missing file and the script reports a generic "FAIL", which is directionally safe (never a false PASS) but would read as a model failure rather than a probe-infrastructure failure if it ever happened. Not urgent since this is a manual debugging tool with the printedfinish_reasonoutput allowing a human to notice something's off.🟢 Nits
candidates.tsv:169-171— the three disabled candidates (ministral8b,ministral3b,granite8b) are commented out by prefixing the whole line with#, which meansrun-candidate.sh's "unknown candidate" listing won't show them at all as available-but-disabled — a user has to read the file to know they exist. Matches the stated intent ("commented out"), just noting it as the reason they're invisible to the script's own error message.Tests
No automated tests are added or expected here — this is glue/ops tooling (a Python eval script plus shell wrappers around
kubectl), consistent with the rest ofscripts/. The real verification is operational:probe-toolcall.shas a fast pre-check andABORT_AFTERineval.pyas a fast-fail inside a real run. I did not execute either against a live cluster (no cluster access here); I verified thedegenerate(),floor(),api_token(), and abort-loop logic by reading the code paths directly rather than running them, and cross-checked the README's numeric claims (e.g.48 = 50 - 2,30+5+4+9 = 48) for internal consistency, which they satisfy.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
The setup is now good enough to test everything with a chance, so the gate matters more than any single run. run-candidate.sh takes a sample size: ./run-candidate.sh <label> 5 screen ./run-candidate.sh <label> measure With ABORT_AFTER a hopeless candidate is rejected in about five minutes. Three of the first four were rejections, and each was left running for an hour or more first. Screening deliberately uses genuine paperless requests rather than probe-toolcall.sh. The standalone probe is cheaper and still worth a first pass, but it passed two candidates that failed on document one — a synthesised request cannot reproduce the RAG neighbours and the taxonomy candidate lists that make a real prompt what it is. Document 1970 is 1976 characters and still produces a 5333-token request. A screen is a gate, not a measurement. Numbers are never compared across sample sizes; the comparison happens at 50. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit851e1b9ef682aacf9bf9736b0d7a5e3d0cfe280d.Summary
This PR fixes three real problems the first
paperless-evaljob run exposed: aDecimal/floatcrash infloor(), a silent-no-op golden-volume seed, and unflagged degenerate LLM output poisoning the accuracy numbers. It also adds an early-abort for hopeless candidates, a per-label API token/user to stop concurrent jobs from revoking each other's tokens, acandidates.tsv+run-candidate.sh/job-template.yamltemplating setup to replace hand-copied Job manifests, and aprobe-toolcall.shsmoke test. The core fixes are correct and well-reasoned. No blocking issues, but there's a broken quickstart example and a couple of edge cases worth a look.🟡 Suggestions
scripts/paperless-eval/README.md:17-21— the "Running one" quickstart uses candidate labelqwen3-8b(./run-candidate.sh qwen3-8b 5,job/eval-qwen3-8b), butcandidates.tsvonly defines the labelqwen35(the other three entries,ministral8b/ministral3b/granite8b, are commented out). Running the example verbatim hitsrun-candidate.sh's "unknown candidate 'qwen3-8b'" check (scripts/paperless-eval/run-candidate.sh:46-50) and fails immediately. Worth fixing the example toqwen35(or adding aqwen3-8brow) since this is the first thing a reader will copy-paste.scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexno longer has the old2>/dev/null || trueguard, and the script runs underset -e. If/data/llm_indexdoesn't exist at all aftercp -r(rather than existing-but-small, which is the case the new size check at line 85 handles),dufails and the container dies on the baredu: cannot access ... No such file or directoryinstead of reaching the craftedFATAL: llm_index is ${bytes} bytes — build it on the smoke-test deployment firstmessage at line 87. The safety property (refuse to start) still holds either way, but the specific "never built at all" case — arguably the most likely one, sincellm_indexis only created once someone runsdocument_llmindex rebuild— loses the friendly diagnostic this PR is adding. Considerbytes=$(du -sb /data/llm_index 2>/dev/null | cut -f1); bytes=${bytes:-0}so both "missing" and "too small" hit the same message.scripts/paperless-eval/eval.py:288-300— the new HTTP‑401 handling raisesSystemExitstraight out ofrun(), which unwinds throughmain()before the results-writing block (main(),~line 430) ever runs. That means a mid-run token revocation discards not just the in-progress engine's rows but also a fully completed and valid first engine's results still sitting in therowslist (e.g. a completeclassicpass, if the revocation happens during the subsequentaipass) — with no.jsonl/.summary.jsonwritten at all. Givenclassiccosts only ~18s total (per the README table) this is a small loss in practice, but ifaiwere reordered first (or the loop order changes later) a 90-minuteairun's data would be discarded on a token revoked one document from the end. Worth considering whethermain()should catch this and still flush whatever rows were gathered before exiting non-zero.scripts/paperless-eval/eval.py:324—ABORT_AFTERonly ever fires on the first three rows (len(rows) == ABORT_AFTER), and only when those rows have an"error"key. A model that returns degenerate output (garbage tokens, caught bydegenerate()) rather than an HTTP error on every one of its first three documents would not trigger the abort, since degenerate rows carry"degenerate", not"error". Based on the candidates documented incandidates.tsv, the observed total-failure modes (Granite's unreadable tool call, Ministral's control-token overflow) do surface as HTTP errors from paperless, so this may not be a live problem today — but it's a gap between the intent ("stop when nothing has ever worked") and what the check actually covers if a future candidate's failure mode is degenerate output instead.🟢 Nits
scripts/paperless-eval/README.mdreading-the-output section and the module docstring ineval.py:9-16both still describe the two-engine comparison narrative; harmless duplication, not something introduced by this PR to fix.Tests
No automated tests exist for this directory (it's an operational eval harness driven by
kubectl), and this PR doesn't add any, consistent with the existing state. Thefloor()Decimal fix and thedegenerate()heuristic are the two pieces of new logic that would most benefit from a quick unit test (e.g.degenerate()against a title with numbers-only, one with a normal umlaut, and one with CJK/digit runs) if this script ever grows a test harness, but given its throwaway, hand-run nature I wouldn't block on it.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Eight candidates now, every repo:quant verified present via the HF API and every one a ggml-org / bartowski / lmstudio-community / unsloth conversion rather than the vendor's own. The selection rule in this file was wrong and is corrected. It said a smaller dense model gains at both ends, and the Ministral pair was chosen on that basis. Measured: Ministral-3 8B dense, ~4.7 GB weights 7.1-8.2 tok/s Qwen3.6 35B-A3B MoE, ~20 GB weights 11 tok/s The largest model is the fastest one here. Decode is bandwidth-bound on the ACTIVE parameter set, which for an A3B mixture is ~3B however much is resident; total size only has to fit, and 96 GiB per node means it always does. So the order is MoE with few active parameters, then dense 3-9B, then dense 10-14B only for an exceptional model — that is ~4 tok/s, near three hours for 50 documents — and nothing dense above 14B. hermes4-14b is in the list despite being on the wrong side of that line: five screening documents are cheap, and it answers whether llama.cpp's named Hermes tool-call parser handles what Granite's format could not. That answer transfers to smaller Hermes sizes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit0c46a219e30646c19932c1538b1c375508b42642.Summary
This PR fixes three real faults surfaced by the first live job run of the paperless-eval harness: a
Decimal/floatcrash infloor(), a silent no-op when the golden LLM index wasn't seeded, and unflagged "degenerate" (broken-inference) completions past ~14k prompt tokens polluting accuracy numbers. It also adds a candidate-driven job template (candidates.tsv+run-candidate.sh+job-template.yaml), a standalone tool-call screening probe, an early-abort-on-repeated-failure guard ineval.py, and a substantial README rewrite. I read the diff against the full checkout (eval.py,job-template.yaml,candidates.tsv,run-candidate.sh,probe-toolcall.sh). I found no blocking defects — the three stated fixes are each correct and match their described failure modes. There are a few rough edges worth tidying.🟡 Suggestions
scripts/paperless-eval/probe-toolcall.sh:676—kubectl cp ... || truesilently discards akubectl cpfailure. If it fails (e.g. thellama.cpp:server-vulkanimage lackstar, whichkubectl cprequires),/tmp/probe.jsonnever lands in the pod, the followingcurl -d @/tmp/probe.jsonfails to read the file, and the script reportsfinish_reason: <none>→FAIL — do not spend a run on this until the cause is understood(line 692). That reads exactly like a model that can't tool-call, which is precisely the class of silent-failure-masquerading-as-a-real-result this PR is elsewhere trying to eliminate (the seed-data /degenerate()fixes). Worth at least logging when thecpfailed so a tooling problem isn't mistaken for a candidate rejection. I did not verify whether that base image actually lackstar; that's the one check that would confirm this is live rather than theoretical.scripts/paperless-eval/eval.py:293-299— anHTTP 401on theaiengine's loop raisesSystemExitbeforemain()reaches the point where results are written to disk (eval.py:430onward), so any already-completedclassic-engine rows for that run are discarded along with the abortedairows, forcing a full re-run of both engines. Givenclassicis cheap (~0.6s/doc per the README), this is low-cost, but if it's not intentional, writing out what was collected before raising would save theclassichalf of the run.🟢 Nits
scripts/paperless-eval/run-candidate.sh:4andREADME.md's "Running one" section both give usage examples (./run-candidate.sh ministral8b,./run-candidate.sh qwen3-8b 5) that reference candidate labels not present incandidates.tsv(which hasqwen35,qwen3-30b-a3b, etc., but noministral8borqwen3-8b). Copy-pasting either example fails immediately with "unknown candidate". Worth using a label that actually exists, or noting the fictitious label is illustrative.scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexnow runs without|| trueunderset -e. If/data/index(notllm_index) is ever missing post-copy, the script aborts here with a genericdu: cannot accesserror instead of reaching the more informativellm_indexsize check and its FATAL message a few lines down. Minor, sinceindexis expected to always be present in the golden copy.Tests
No automated tests are added or exist for this directory, consistent with the rest of
scripts/paperless-eval/(it's an operationally-run tool against a live cluster/Postgres/llama.cpp stack, applied by hand per the README, not something the repo's CI exercises). The PR body itself documents that the fixes were validated by an actual job run failing and then succeeding differently, which is reasonable validation for this kind of script. Thedegenerate()regression noted in its own comment (digit-heavy titles like "Rechnung 2025-5425541" false-tripping an earlier version) was caught by observation rather than a test — a smallif __name__doctest-style sanity check fordegenerate()andfloor()would be cheap given how easy both are to get subtly wrong again, but is not required by repo convention.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit71a4c1b3a7ae8bc2c830a49268505d7d29ca35d3.Summary
This PR fixes the three faults the first real job run surfaced (
Decimal/floatdivision infloor(), a silently-empty golden volume, and undetectable degenerate LLM output), and bundles in a larger restructuring of the eval tooling: acandidates.tsv+run-candidate.sh/job-template.yamlgenerator to replace hand-copied Job manifests, aprobe-toolcall.shsmoke test, an early-abort mechanism (ABORT_AFTER), and a per-label DB user to stop concurrent runs from revoking each other's tokens. I verified each of the three described fixes against the actual failure it names and they land where the PR body says. No blocking defects found; two non-blocking points worth a look below.🟡 Suggestions
scripts/paperless-eval/eval.py:89—REQUEST_TIMEOUTwas cut from 900s to 300s in the same change that addsABORT_AFTER = 3(line 94, used at line 324: three consecutive"error"rows abort the whole engine run).candidates.tsvexplicitly still recommends trying "dense 10-14B only for an exceptional model (~4 tok/s, roughly three hours for 50 documents)" — i.e. ~216s/doc average. A document near the ~14k-token prompt boundary on one of those candidates plausibly exceeds 300s on prompt processing + up to 768 output tokens at 4 tok/s (~192s of generation alone), which would register as a client-side timeout"error"row. If that happens to be true for the first three sampled documents (sampling order ismd5(id), effectively random per run), a working-but-slow candidate would be wrongly aborted as "unusable" via the same code path meant for tool-calling failures. I did not run this to confirm it actually happens — the fix would be either scopingABORT_AFTERto a specific error kind (tool-call failure) rather than anyerror, or keeping the timeout closer to the old 900s for the slow-dense-model case the tsv still recommends testing.scripts/paperless-eval/README.md:31—./run-candidate.sh qwen3-8b 5is presented as a literal copy-pasteable example, butqwen3-8bis not a label incandidates.tsv(closest entries areqwen3-4b-2507,qwen3-30b-a3b,qwen35-9b,nemotron-nano-12b, etc.). Copy-pasting the example verbatim hitsrun-candidate.sh's "unknown candidate" exit (scripts/paperless-eval/run-candidate.sh:747). Worth pointing at a real label or adding a placeholder note.Tests
No automated tests, consistent with the rest of
scripts/paperless-eval(there were none before this PR either, and this is Kubernetes-ops tooling rather than library code). The changes are validated implicitly by the PR body's description of the failed run that motivated them. Nothing here seems like it would be practically unit-testable given the dependency on a live cluster, Postgres clone, and llama.cpp sidecar.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit11b0def055644adb1c58c58775130922f5c1008a.Summary
This PR fixes three real faults surfaced by the first production run of the paperless-eval harness: a
Decimal/floatTypeErrorinfloor(), a silently-emptyllm_indexthat would have measured the no-RAG path and reported it as RAG, and unflagged "degenerate" completions near the context boundary that would have been scored as ordinary wrong answers. It also adds arun-candidate.sh/candidates.tsvtemplate system replacing per-model job YAML copies, an early-abort mechanism for hopeless candidates, aprobe-toolcall.shsmoke test, and a per-label API token/user fix for a real concurrent-run credential-revocation bug. The core fixes are correct and well-grounded in the failure evidence quoted in the PR body. No blocking issues found; a couple of gaps are worth a look.🟡 Suggestions
scripts/paperless-eval/eval.py:230-253—degenerate()'s own docstring says the failure mode is "first empty completions, then runs of digits and CJK characters," but the implementation only detects empty titles and non-Latin-alphabetic characters (foreign = sum(c.isalpha() and not (...) for c in title)). A title that degenerates into pure digits (e.g."482910482910482910") hasforeign == 0and is not flagged — it falls straight through toscore_document()and gets scored as an ordinary wrong answer, which is exactly the failure this PR is trying to prevent (a poisoned-slot response silently depressing a candidate's numbers). This may be a deliberate tradeoff — the comment at line 245-249 explains digits are normal in real titles ("Rechnung 2025-5425541") and a digit-density check would false-positive on those — but if so, the docstring overclaims what the function actually catches. Worth either tightening the docstring to say "CJK/foreign-script titles only" or adding a narrower digit-run signal (e.g. a very long unbroken digit sequence, which a real title is unlikely to produce) if digit-only degeneration is actually observed in practice.scripts/paperless-eval/probe-toolcall.sh:710-711—kubectl cpis used to stage/tmp/probe.jsoninto the pod, and its failure is swallowed (|| true).kubectl cpshells out totarinside the target container; ifghcr.io/ggml-org/llama.cpp:server-vulkan-b10731doesn't shiptar(plausible for a minimal server image), the copy silently fails, the subsequentcurl -d @/tmp/probe.jsonreads a nonexistent file, and the probe reports a FAIL that has nothing to do with the model's tool-calling ability — exactly the kind of false negative this script exists to prevent. I haven't confirmed whethertaris present in that image. Piping the payload via stdin instead (kubectl exec -i "$pod" -- curl ... -d @- < /tmp/tcp.json) would remove the dependency ontarentirely and is a small, low-risk change sincecurlis already known to work in that container (it's used for the health check and the real request).🟢 Nits
Decimalcrash, the empty index, and degenerate completions) but the diff also changesapi_token()'s user-selection logic to fix a fourth, real bug: two concurrent candidates sharing a superuser token would silently revoke each other's credentials (eval.py:103-142). It's a good fix, just undocumented in the PR description — worth a mention for anyone skimming the summary later.Tests
No automated tests are added or exist for
scripts/paperless-eval/, unlikescripts/ci/test-model-selection.shelsewhere in the repo, which shows there's precedent for offline shell/logic tests in this codebase. Most of this PR's surface (Kubernetes manifests, a live LLM backend) genuinely can't be unit-tested cheaply, which likely explains the gap and is consistent with "applied by hand, not by Flux." That said, the pure logic added here —degenerate(), thefloat()coercion infloor(), andABORT_AFTER's early-break condition inrun()— has no cluster dependency and could be exercised with a handful of table-driven cases (e.g. digit-only title, CJK title, mixed real title with digits, empty title) the waytest-model-selection.shdoes forvalid-model.sh. Not blocking given the nature of this change, but would have caught the digit-only gap noted above.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit043d1af4d4d5579eb34db077e2d8d34ff6cdf7cb.Summary
Fixes three real faults surfaced by the first live job run of the paperless-eval harness: a
Decimal/floatTypeErrorinfloor(), a silently-empty LLM index that would have reported a no-RAG run as a RAG result, and unflagged "degenerate" model output (empty/non-Latin garbage past ~14k prompt tokens) polluting accuracy figures. It also generalizes the singlejob-qwen35.yamlinto a templatedjob-template.yamldriven by a newcandidates.tsv+run-candidate.sh, adds a standaloneprobe-toolcall.shsmoke test, and expands the README with results and methodology notes. All three described bugs are fixed correctly and match the mechanisms described in the PR body — I found no blocking issues, only a few minor robustness/nit points below.🟡 Suggestions
scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexruns without the previous|| true, underset -e. If either directory is ever absent aftercp -r(not just present-but-tiny), the initContainer dies on this line with a rawdu: cannot access ...instead of reaching the friendlyFATAL: llm_index is ... bytesmessage with the rebuild instructions two lines below. In practice this path only matters before the golden PVC has ever been seeded by paperless at all, but since the whole point of this hunk is a clear diagnostic for "the index isn't there", considerdu -sh /data/index /data/llm_index || true(matching the byte-check line, which already tolerates a missing/small index) so the friendly message is reachable in that case too.scripts/paperless-eval/eval.py:384-386—complement()'s guardif not both or not by_engine["ai"]: return None— the second clause is redundant:bothis the intersection of the classic and ai dicts, so ifby_engine["ai"]is empty,bothis already empty andnot bothalone triggers the return.scripts/paperless-eval/probe-toolcall.sh:741—finish=$(... grep -oE '"finish_reason":"[a-z_]+"' ...)assumes compact JSON with no space after the colon. It matches llama-server's current output, but if that ever changes to pretty-printed JSON the regex silently matches nothing and the script reportsFAILinstead of erroring loudly on the parse itself.🟢 Nits
scripts/paperless-eval/eval.py:103-141— the new per-labeleval-{LABEL}user relies onLABELbeing distinct across concurrently-running jobs (true for the documented workflow, where a screen run is deleted before the full run reuses the same label sequentially). Worth a one-line comment noting the sequential-reuse assumption, since a stray concurrent re-run under the same label would hit a unique-constraint race oninsert into auth_user.Tests
No automated tests accompany this change, but that matches the rest of
scripts/— the only test harness in the repo (scripts/ci/test-model-selection.sh) covers a different script, and this directory's other files (run-candidate.sh,job-template.yaml) are un-tested ops tooling by convention. The PR body describes validating the three fixes against an actual failed job run, which is reasonable evidence for scripts that talk to a live Kubernetes cluster/Postgres/llama.cpp and are impractical to unit test. Thedegenerate()heuristic (empty title / >20% non-Latin characters) and thefloor()Decimal fix are simple enough to be low-risk without dedicated tests.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit319968c0f177f580fbabe7230b89578e2a06ffe0.Summary
This PR fixes three real problems surfaced by the first live run of the eval harness: a
Decimal/floatTypeErrorinfloor(), a silent no-op seed that could report a no-RAG run as a RAG result, and unflagged "degenerate" (garbage/empty) completions past the ~14k-token boundary that would otherwise be scored as plain wrong answers. It also adds a screening workflow (run-candidate.sh,probe-toolcall.sh,candidates.tsv) and generalizes the single-candidate Job manifest into a template. The core logic changes are sound and each is backed by a concrete, documented failure mode. One README example is broken against the actual candidate list.🔴 Blocking
None.
🟡 Suggestions
scripts/paperless-eval/README.md:17-21: the quick-start example runs./run-candidate.sh qwen3-8b 5/./run-candidate.sh qwen3-8b, butqwen3-8bis not a label incandidates.tsv(candidates.tsv:88only lists it in the "SCREENING RESULTS" comment as a rejected candidate — it was never an active row, or was removed). Following the README verbatim getsunknown candidate 'qwen3-8b'fromrun-candidate.sh:814-818immediately, not the "~5 min to a rejection" behavior the comment promises. Worth swapping in a label that's actually in the table (or noting explicitly that this is a hypothetical/rejected example).scripts/paperless-eval/eval.py:324(ABORT_AFTERcheck): the abort only fires when the firstABORT_AFTERdocuments are all failures (len(rows) == ABORT_AFTER, checked once). A candidate that succeeds on document 1 but then fails every document from 2 onward will never trip the abort and will run to completion. This matches the docstring's stated intent ("catches the case where nothing has ever worked", not a ratio), so it's not a defect, just a narrower safety net than the nameABORT_AFTERmight suggest — worth a one-line callout in the README's screening section if that gap matters in practice.scripts/paperless-eval/job-template.yaml:77-85: thedu -sh /data/index /data/llm_indexcall lost its old2>/dev/null || trueguard. That's intentional (the PR body calls out the old version "swallowed the evidence"), but underset -e, if/goldennever had anindex/llm_indexdirectory at all (rather than an empty one),duitself fails with "No such file or directory" before reaching the byte-count check, so the friendlierFATAL: llm_index is ${bytes} bytes...message with the rebuild command never prints — the initContainer still fails (safe), just with a less actionable message than the one this PR is adding. Minor, since the fix's main target (empty-but-present directory) is handled correctly.🟢 Nits
scripts/paperless-eval/eval.py:293:if error and error.startswith("HTTP 401")—erroris always a non-Nonestring here (fetch()only setspayload=Nonetogether with a stringerror), so theerror andguard is redundant. Harmless.Tests
No automated tests exist for this harness (script-driven, run against a live Postgres/paperless/llama.cpp stack), consistent with the rest of
scripts/paperless-eval/. The PR's own justification is the failed live run described in the PR body, which is a reasonable substitute given the nature of this tooling — each fix traces to an observed failure (TypeErrortraceback, empty golden volume, poisoned llama.cpp slot) rather than a hypothetical. I did not find a lower-cost way to unit-testfloor()'sDecimalhandling ordegenerate()'s heuristic without a live Postgres connection or sample payloads, but a couple of pure-function cases (e.g.degenerate()given a title with digits/punctuation vs. one with CJK runs) could be pulled out as plainasserts cheaply if this file grows more logic like it.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Two jobs crashlooped for three hours without producing a single document, and I did not notice because the pods sat in Init the whole time — which looks exactly like a slow model download. llama_model_load_from_file_impl: exactly one out metadata, path_model, and file must be defined That reads as a loading error. It is an ambiguous reference. `--hf-repo repo:QUANT` matches by substring across the entire repository, and repos routinely carry several files bearing one tag: ggml-org ships mmproj-, mtp- and dflash- variants at Q8_0 beside the model, bartowski ships kld_results/kld_Q4_K_M.txt beside the weights. Q4_K_M happened to be unambiguous for the baseline, which is why this never surfaced before. candidates.tsv now carries the exact filename as its own column, resolved from the HF API for all fifteen entries, and run-candidate.sh refuses to start a candidate that has none rather than guessing. --hf-file overrides the quant. Also fixing the monitoring gap that hid it. The watch treated PodInitializing as "keep waiting" indefinitely, so a container failing to start was indistinguishable from one still pulling 34 GB. It now checks init container states for CrashLoopBackOff/Error/ImagePullBackOff and reports with the last lines of the failing container's log. Silence was not success, and the tooling should have said so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Correcting the previous commit, which named the wrong cause with some confidence. The llama-models volume was 100% full — 120 GiB, nothing free. llama-server could not write its metadata cache, so it could not resolve the repository listing, and reported no GGUF files found in repository <repo> Available GGUF files: - <whatever it had cached before the disk filled> That reads as a bad model reference, and I read it that way. It is a full disk. Nine screened candidates filled the volume: 2.5 GB for a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. Freed 56.5 GB by removing the weights of candidates already rejected — they are re-downloadable, so that costs only bandwidth. --hf-file stays. Naming the exact file removes a genuine ambiguity (ggml-org ships mmproj-, mtp- and dflash- variants at Q8_0 beside the model) and costs nothing. It just was not the fault here, and the comment claiming it was is rewritten. README now says to check `df -h /models` before believing anything llama-server says about a repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commitfaebeb15944358a9f7cb51fde0a6375699a28153.Summary
This PR fixes three concrete faults exposed by the first real job run of the paperless-eval harness: a
Decimal/floatcrash infloor(), a silent-empty-seed bug in theseed-datainitContainer, and unflagged "degenerate" (poisoned-slot) LLM output being scored as wrong answers. It also generalizes the harness from a single hardcodedjob-qwen35.yamlto a templated, TSV-driven multi-candidate runner (job-template.yaml+run-candidate.sh+candidates.tsv), adds an early-abort for hopeless candidates, and adds a standaloneprobe-toolcall.shsmoke test. I found no blocking defects — the three described fixes are implemented correctly and consistently with the narrative in the PR body and README. A few non-blocking gaps are worth a look.🟡 Suggestions
scripts/paperless-eval/eval.py:324— The early-abort check (if len(rows) == ABORT_AFTER and all("error" in r for r in rows)) only counts rows with anerrorkey. Adegeneraterow (empty/non-Latin title from a poisoned llama.cpp slot) has noerrorkey, so three consecutive degenerate results from document one won't trigger the abort. The PR body itself notes that one degenerate response "poisons the llama.cpp slot for everything after it," which is exactly the scenarioABORT_AFTERexists to catch cheaply — as written, a candidate that goes degenerate immediately would still burn the full ~90-minute run (the degenerate rows are correctly excluded from scoring afterward, so this is a wasted-time issue, not a correctness issue).scripts/paperless-eval/eval.py:250— The Latin-1 allowlist indegenerate()(äöüÄÖÜßáéèêíóúàçñ) includes uppercase Ä/Ö/Ü/ß but only lowercase forms of the other accented letters (é, è, ê, í, ó, ú, à, ç, ñ). A title beginning with an uppercase accented letter outside that set (e.g. "Écoles...", "Íñigo...") would count that character as "foreign." At the current 20%-of-title threshold this is unlikely to misfire alone on a normal-length title, but it's an easy gap to close by uppercasing the allowlist too.scripts/paperless-eval/job-template.yaml:172(--reasoning off) and:157(--n-predict 768) are now baked into the shared template for every candidate, but the rationale in both the template comments andcandidates.tsv(lines ~199-206) frames--reasoning offspecifically as a Qwen3.6 workaround ("the one exception is Qwen3.6 itself, which is measured with thinking switched off"). I can't verify from the checkout alone whether--reasoning offis a harmless no-op on llama-server for models with no reasoning concept at all — worth a quick check in the llama-server log the next time a non-reasoning candidate (e.g.qwen25-7b,llama31-8b) is run, in case it changes chat-template rendering unexpectedly the way--chat-template-kwargsdid (per the adjacent comment about Mistral producing token salad from an unrelated injected kwarg).🟢 Nits
scripts/paperless-eval/run-candidate.sh:882—if [ "$sample" -lt 50 ]will throw a POSIX-shell integer-comparison error if someone passes a non-numeric second argument instead of the documented failure message; not worth guarding given this is a hand-invoked local tool.Tests
No automated tests are added or expected here —
scripts/paperless-eval/is explicitly hand-run tooling ("Applied by hand, not by Flux") outside the CI-testedscripts/ci/path (confirmed via.woodpecker/ci-scripts-test.yml, which only exercisesscripts/ci/test-model-selection.sh). The three bug fixes are instead validated empirically — the README's new "Results so far" table and reproducibility section show concrete numbers from real runs using the fixed harness, which is reasonable evidence theDecimal/floatfix, the seed-data guard, and the degenerate-output flagging all work as described.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit1ae0e59ea604e3e50bcdb9af21d0de07b1f3438d.Summary
Three targeted fixes to the paperless-eval harness (Decimal/float crash in
floor(), an initContainer that now refuses to start on an unseededllm_index, and a newdegenerate()filter for poisoned-slot garbage), plus a larger rework of the runner into acandidates.tsv+run-candidate.sh+job-template.yamlpipeline with an early-abort screen. All three bugs described in the PR body are genuinely fixed for the failure modes shown. No blocking issues found; two points below are worth tightening.🟡 Suggestions
degenerate()only catches non-Latin-script garbage, not the other garbled patterns this same PR documents (scripts/paperless-eval/eval.py:230-253). It flags an empty title or a title with a high fraction of non-ASCII/non-German-diacritic letters. But two things this PR itself calls out as symptoms of a poisoned slot would sail through untouched:isalpha()characters, soforeignstays 0 anddegenerate()returnsNone.job-template.yaml:168("1UnComplete1Self1UnCompleteSDirect11S...") is plain ASCII letters and digits — it would not trip the non-Latin check either.That second one is real evidence from this same PR of a garbled response the current heuristic would miss (mitigated for that specific cause by dropping
--chat-template-kwargs, but nothing guarantees ASCII-only garbage can't recur from a different trigger). Since a missed degenerate document is scored as a plain wrong answer — exactly the outcome this feature exists to prevent — it's worth strengthening the check (e.g. flag titles with no letters at all, or detect character-repetition/no-whitespace runs) rather than relying solely on the foreign-script signal.A 401 mid-run discards every result already collected, including a fully-completed other engine (
scripts/paperless-eval/eval.py:288-299,main()ateval.py:408-435).run()nowraise SystemExits immediately on an HTTP 401, andmain()only writes{LABEL}.jsonl/.summary.jsonafter both engines finish. Ifclassiccompletes cleanly andaihits a 401 on document 30, the 30 already-scored AI rows and the entire classic run are thrown away with nothing written to disk. Given the per-label user fix (eval.py:118-142) is meant to make cross-job token revocation far less likely already, this is a narrow case, but flushing what's already been scored before exiting would avoid losing real API-cost data to a now-rarer failure.🟢 Nits
scripts/paperless-eval/README.md:17-21) uses./run-candidate.sh qwen3-8b 5as the copy-pasteable example, butqwen3-8bisn't a row incandidates.tsv(it only appears in a comment noting it was screened out:candidates.tsv:88). Run verbatim, the example fails withrun-candidate.sh's "unknown candidate" error. Worth swapping in a label that's actually still in the file (e.g.glm4-9b), or a placeholder like<label>.scripts/paperless-eval/probe-toolcall.sh:775-778deletes any stale pod with--wait=falsethen onlysleep 2before re-applying the same pod name; on a slow teardown this could occasionally collide with "already exists". Low impact for a manually-run diagnostic tool.Tests
No automated tests, consistent with the rest of this script-driven, manually-applied directory (the README explicitly documents "applied by hand, not by Flux"). The PR's own validation is a real job run against the cluster, which is the existing convention here and reasonable for this kind of script.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Measured, same model, same documents, only the quantisation differs: Qwen3.6-35B-A3B Q4_K_M ~21 GB 11.0 tok/s Qwen3.6-35B-A3B Q8_0 ~34 GB 10.3 tok/s I predicted ~6 tok/s for Q8 — a halving, on the reasoning that doubling bytes per active parameter doubles what crosses the memory bus. It cost ten percent. So decode is not bandwidth-bound for this MoE at all. It is compute-bound on many small per-expert matmuls, which retroactively explains a measurement I recorded but never followed through: the MoE reached 21.4 GB/s against a 53.9 GB/s ceiling. It was already leaving 60% of the bus idle, so more bytes cost almost nothing. That formula has been steering decisions in this file — it is what ranked candidates by size and what I used to predict runtimes. It predicts dense models and not this one. Practical consequence: higher precision is nearly free on an MoE here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit7658cf8bb6b28fef8ff658b489c2bf1b7e43c19f.Summary
This PR fixes the three faults described in the PR body (the
Decimal/floatdivision crash infloor(), the silently-emptyseed-datainit container, and the inability to distinguish backend garbage from a genuine wrong answer) and, beyond what the description mentions, substantially reworks the eval harness's operational tooling: a per-label auth token/user (fixing a real token-revocation race between concurrent jobs), a screening abort path (ABORT_AFTER),--n-predict/--reasoning offon the llama-server sidecar, and acandidates.tsv+run-candidate.sh+probe-toolcall.shrestructuring that replaces per-model Job manifests with one template. I checked each change against the rest ofeval.pyand the manifests on disk; the fixes described in the PR body are correct. I found no blocking defects, but two of the new heuristics have gaps worth a look, plus a couple of doc/UX nits.🟡 Suggestions
eval.py:242-253—degenerate()doesn't actually catch "runs of digits." The docstring says broken inference produces "first empty completions, then runs of digits and CJK characters," but the implementation only flags an empty title or a high proportion of non-Latin alphabetic characters (c.isalpha()isFalsefor digits). A title that degenerates into pure digit garbage (e.g. repeated digit runs, no CJK) would scoreforeign == 0and pass through as a normal, non-degenerate answer — silently counted as a real wrong answer, which is exactly the failure mode fault #3 in this PR set out to prevent. This looks like a deliberate tradeoff (the adjacent comment explains digit-heavy titles like "Rechnung 2025-5425541" are legitimate and must not false-positive), but it leaves a real gap between the docstring's claimed coverage and what the code checks. Worth either tightening the docstring to say "CJK/non-Latin scripts" specifically, or adding a narrower digit-run heuristic (e.g. a long unbroken run of digits with no separators).scripts/paperless-eval/probe-toolcall.sh:829—kubectl cp ... || truesilently swallows a copy failure.kubectl cpdepends ontarexisting inside the target container, which minimal server images (this isghcr.io/ggml-org/llama.cpp:server-vulkan-b10731) often lack. If the copy silently fails, the followingcurl -d @/tmp/probe.jsonruns against a nonexistent file;curl -swill likely produce empty output,finishwill be empty, and the script reportsFAIL — do not spend a run on this until the cause is understood— misattributing a tooling failure to the model. Since the whole point of this script is to cheaply and reliably reject bad candidates before a 90-minute run, a silent false negative here defeats that purpose. Consider checking thekubectl cpexit code explicitly and failing loudly (distinct from a genuine model FAIL) if it doesn't succeed.🟢 Nits
README.md:17,21— The quickstart example uses./run-candidate.sh qwen3-8b 5, butqwen3-8bis not a row incandidates.tsv(it's only mentioned in the historical screening notes incandidates.tsv:296as a candidate that was screened and rejected, not kept). Running the README's own first command fails immediately withunknown candidate 'qwen3-8b'rather than demonstrating the described "~5 min to a rejection" flow. A currently-present label (e.g.qwen35-9b) would make the example actually runnable.scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexno longer has2>/dev/null || trueandset -eis active, so if the golden PVC never had these directories created at all (as opposed to existing-but-undersized), the init container dies on a baredu: cannot access ...rather than the friendlierFATAL: llm_index is ... bytes — build it on the smoke-test deployment firstmessage a few lines below. Minor — the important case (present-but-empty) is handled well; this only affects the harder edge case of directories missing outright.Tests
This is an operational/ops-script area of the repo (kubectl-driven Jobs and shell scripts) with no existing unit test harness, consistent with the rest of
scripts/. The PR body documents that the fixes were validated against a real, failing job run, which is the appropriate verification method here. The newdegenerate()heuristic and theABORT_AFTERearly-exit have no automated coverage, which is reasonable given the surrounding conventions, but note the digit-run gap above is exactly the kind of thing a couple of inlineassert degenerate(...) == ...sanity checks (even a__main__-guarded self-test, if the repo has a place for that) would have caught cheaply.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
CLI default(complexity: complex), commit43634e46c9838b4fd28bef113e0b97cbae76a4e7.Summary
This PR fixes the three faults exposed by the first real
paperless-evaljob run: aDecimal/floatTypeErrorinfloor(), a silent no-op seed of the LLM index (now hard-gated), and unflagged "degenerate" LLM outputs beyond ~14k prompt tokens that would otherwise be scored as wrong answers. It also addsABORT_AFTERearly-exit, a per-label API-token user (fixing a token-revocation race between concurrent jobs), ajob-qwen35.yaml→job-template.yaml+candidates.tsv+run-candidate.shtemplating scheme, and a newprobe-toolcall.shsmoke test. I traced all three described bugs against the checkout and each fix is correct and matches the diagnosis in the PR body. No blocking issues found.🟡 Suggestions
scripts/paperless-eval/README.md:17,21— The flagship "Running one" example (./run-candidate.sh qwen3-8b 5/./run-candidate.sh qwen3-8b) usesqwen3-8bas the label. That label is not an active row incandidates.tsv— it only appears inside the comment block as a screened-and-rejected candidate (candidates.tsv:296,:345). Running the README's own first example verbatim fails immediately withrun-candidate.sh'sunknown candidate 'qwen3-8b'error (run-candidate.sh:906-910). Worth swapping the example to an actual active label (e.g.qwen35orglm47-flash) so a new reader can copy-paste it successfully.scripts/paperless-eval/job-template.yaml:77-92— Thedu -sh /data/index /data/llm_indexcall lost its old2>/dev/null || truesuppression (compare to the previouscp -aversion). That's intentional for the "refuse to start on a tiny index" case, but if/data/llm_indexdoesn't exist at all aftercp -r(rather than existing-but-small — the scenario the PR description says actually happened: "the golden volume was empty"),duitself errors on the missing path and, underset -e, the initContainer dies right there with a rawdu: cannot access ... No such file or directoryinstead of reaching the craftedFATAL: llm_index is ... bytes — build it on the smoke-test deployment firstmessage. It still fails safe (does not start with a bad index), just with a less actionable message for that specific case. Considermkdir -p /data/llm_indexbefore theducheck, or wrapping just that diagnostic line, to guarantee the friendlier message fires either way.Tests
No automated tests exist for this directory (it's Kubernetes Job manifests + operational shell/Python scripts, consistent with the rest of the repo). The change is validated empirically by the job run described in the PR body rather than by a test suite, which matches how this script has always been verified. Nothing here calls for new automated tests.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
CLI default(complexity: complex), commitec849f2e29c5f654234d1f4b6f26f5e028c39f5b.Summary
This PR fixes the three concrete bugs its description names — the
Decimal/floatTypeErrorinfloor(), the silently-emptyllm_indexseeding, and undetectable degenerate LLM output — and additionally reworks the eval tooling to support many candidates (candidates.tsv,job-template.yaml,run-candidate.sh,probe-toolcall.sh) instead of one copy-pasted Job per model. I readeval.pyandjob-template.yamlin full against the diff; all three described fixes are implemented correctly and match their stated failure modes. No blocking defects found.🟡 Suggestions
scripts/paperless-eval/eval.py:324—ABORT_AFTERonly counts rows with an"error"key (HTTP/transport failures), never"degenerate"rows. A candidate that returns HTTP 200 with garbage content on every document (rather than failing outright) will run the full 50-document sample beforedegeneratecount reveals it's broken, instead of aborting in ~5 minutes like the HTTP-failure case. This matches the current intent (degenerate ≠ error, since a wrong-but-well-formed answer is a real result), but is worth a deliberate call — right now a systematically-poisoned slot burns a full ~90 minute run before the summary flags it.scripts/paperless-eval/README.md:39-41— "a screening run rejects a hopeless candidate in about five minutes" assumes fast failures. If the first three documents time out instead (REQUEST_TIMEOUT = 300ineval.py:89),ABORT_AFTERdoesn't fire until ~15 minutes, not 5 — which is exactly whatcandidates.tsv:391-395documents happening toglm47-flash("every one a TimeoutError"). Worth a one-line caveat so a stalled screen at the 5-minute mark isn't mistaken for a hang.scripts/paperless-eval/probe-toolcall.sh:837-840—cleanupdeletes any leftover pod with--wait=falsethen onlysleep 2beforekubectl applys a pod of the same name. If termination takes longer than 2s (not unusual for a pod holding a GPU/PVC mount), theapplycan transiently fail against the still-terminating pod. Minor, since it's a manual dev tool that can just be re-run.🟢 Nits
scripts/paperless-eval/job-template.yaml:77-78—du -sh /data/index /data/llm_indexno longer has2>/dev/null || true, so underset -ea missing/data/indexnow aborts the initContainer instead of silently continuing. That's clearly intentional (fail loud, per the PR's own theme) but the comment above only explains the-rvs-achange, not why error-suppression was dropped on the following line — worth a half-line note since a future reader will otherwise wonder if it's accidental.Tests
No automated tests exist for this script (consistent with the rest of this ops-tooling directory — it's exercised by running it against the real cluster). The PR body says the fix is being validated by a live run against the eval instance that was in progress at PR creation time; I can't confirm that run's outcome from the checkout. The three fixes themselves are each narrow and directly traceable to a concrete failure mode described in the PR body and reproduced in code (Decimal/float coercion, byte-threshold check, non-Latin/empty-title detection), which is about as much verification as a script like this gets without a real cluster.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
CLI default(complexity: complex), commit47ed4383f1c4ec120cb30eaf17630b7d85af9b89.Summary
This PR fixes the three faults that killed the first real
paperless-evaljob run: aDecimal/floatTypeErrorinfloor(), a silently-empty golden volume that would have scored the no-RAG path as a RAG result, and unscored "degenerate" (poisoned-slot) completions being counted as wrong answers. It also generalizes the singlejob-qwen35.yamlinto a templatedjob-template.yamldriven bycandidates.tsvplus arun-candidate.shrunner, adds an early-abort on 3 consecutive failures, adds a standaloneprobe-toolcall.shsmoke test, and rewrites the README with results and operational notes. I checked each of the three described fixes against the code and they are correct. No blocking defects found; a couple of robustness/doc issues are worth a look.🟡 Suggestions
scripts/paperless-eval/probe-toolcall.sh:77—kubectl cp ... || truesilently swallows a failed copy of the probe payload into the pod. If the copy fails (network hiccup,kubectl cprequiringtarinside a minimalllama.cppserver image, pod not fully ready, etc.), the followingr=$(kubectl exec ... curl -d @/tmp/probe.json)on line 78 either dies immediately underset -ewith no diagnostic (a barecurl: (26) Failed to open/read local data from fileand no PASS/FAIL line), or — if curl doesn't error — produces an empty response that printsFAIL — do not spend a run on thiseven though the model itself was never actually probed. Either way, an infrastructure hiccup gets misreported as a model result (or the script just dies unhelpfully) instead of a clear "copy failed" message. Worth either checking thekubectl cpexit status explicitly and failing loudly, or retrying it.scripts/paperless-eval/README.md"Running one" example (~line 30) uses./run-candidate.sh qwen3-8b 5/./run-candidate.sh qwen3-8bas the walkthrough command.qwen3-8bis not present incandidates.tsv— it's only mentioned in the file's comments as a screened-and-rejected candidate ("qwen3-8b dense 8B 4 of 5 failed"). Running the README's own first example verbatim will hitrun-candidate.sh's "unknown candidate" exit (scripts/paperless-eval/run-candidate.sh:961-965). Worth swapping the example for a label that actually exists incandidates.tsv(e.g.qwen35-9bornemotron3-30b-a3b).scripts/paperless-eval/job-template.yaml:77-86—du -sh /data/index /data/llm_index(no longer|| true) and the followingdu -sb /data/llm_indexboth run underset -eand will abort the initContainer with a rawdu: cannot access ...if/data/llm_indexdoesn't exist at all (as opposed to existing-but-small). That still achieves the goal of refusing to start, just without the friendlyFATAL: llm_index is ... bytesmessage this block was written to produce — minor, since the outcome (fail closed) is the same either way.🟢 Nits
scripts/paperless-eval/probe-toolcall.sh:51builds--hf-repo "$repo"still accepting the oldorg/Model-GGUF:QUANTshorthand the README/job-template.yamlcomments now explicitly steer away from ("The exact filename rather than the:QUANTshorthand... repos do carry several files under one tag"). Not wrong for a quick smoke test, but slightly inconsistent with the rest of the PR's stated preference.Tests
No automated tests, consistent with the rest of this directory (it's an operational eval harness, not application code, and there's no existing test scaffold to extend). The three fixes described in the PR body were validated against a real job run per the description. The newly-added pure functions in
eval.py(floor,degenerate,pct,summarise,complement) are easy to unit test in isolation if this script grows further, but I wouldn't block on adding that now.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Fair challenge I had not put to myself: --n-predict 768 is a bound I chose this morning because Qwen3.6 once ran to 10269 tokens and cost 900 s a document. It is not a property of any model. Which means every candidate recorded as failing at exactly 768 generated tokens — Hermes-4-14B, gemma-4-26B-A4B, both Ministrals, Granite, Qwen3-8B — is failing against MY bound, and I wrote them up as incapable. A model that needs 900 tokens to emit a valid tool call is indistinguishable, in that record, from one that never terminates at all. run-candidate.sh takes the cap as a third argument: ./run-candidate.sh gemma4-26b-a4b 5 4096 gemma-4 is re-running at 4096 against the same three documents it failed, since it is the newest model in the field and the cleanest failure of the set. If it produces valid calls given room, six rejections in this file are wrong and the size threshold I committed an hour ago is built on an artefact. If it still runs to the new cap, the cap only changed how long it took to find out. Worth the node time either way. A working answer takes 241-291 tokens, so 768 looked generous — but "looked generous" is not evidence, and I should have tested the bound before drawing conclusions from models hitting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit4ba68b91aac5d18e9a34600d2cfe24b81d645075.Summary
Three fixes to
scripts/paperless-eval, each grounded in a failure the PR body describes from an actual job run: aDecimal/floatTypeErrorinfloor(), a silently-emptyseed-datainitContainer that would have scored the no-RAG path as a RAG result, and unflagged "degenerate" completions past ~14k prompt tokens that would have poisoned a candidate's accuracy numbers. It also generalizes the single-candidate Job manifest into a template driven bycandidates.tsvand a newrun-candidate.sh, adds a screening abort (ABORT_AFTER) and a standaloneprobe-toolcall.sh, and fixes a real concurrency bug inapi_token()(two jobs sharing one superuser's token could revoke each other's credentials mid-run). I traced each of the three headline fixes against the code and they check out. No blocking issues found.🟡 Suggestions
scripts/paperless-eval/candidates.tsv:249— the commented-out# qwen3-30b-a3brow still uses the old 5-field format (repo:quantcombined, no separatehf-filecolumn), unlike every active row's 6-field format thatrun-candidate.shparses by fixed position (cut -f2..6). If someone uncomments it expecting it to work, the columns silently misalign (e.g. the alias lands in the mem-request slot) rather than failing loudly —run-candidate.sh's ownhf-fileguard (run-candidate.sh:984) only catches a missing file, not one that's off by a column. Worth a one-line note that it needs updating to the current format first, or just deleting it.scripts/paperless-eval/probe-toolcall.sh:903—kubectl cp ... || trueswallows a copy failure. If it fails, the subsequentcurl -d @/tmp/probe.jsoninside the pod reads a nonexistent file, and underset -euthe script dies on an opaquekubectl exec/curl error rather than a clear "could not stage the probe payload" message.scripts/paperless-eval/run-candidate.sh:957,968—sampleandnpredictaren't validated as numeric before being spliced into[ "$sample" -lt 50 ](run-candidate.sh:995) and into the manifest. A typo'd non-numeric argument produces a cryptic[: -lt: ... expectedshell error instead of the script's own usage message.🟢 Nits
scripts/paperless-eval/job-template.yaml:86— the "under 1 MiB" comment (job-template.yaml:84) actually checks< 1000000bytes (≈976.6 KiB, not 1 MiB). Harmless since the real index is ~125 MiB either way, but the comment and the constant don't quite agree.Tests
No automated tests, but that's consistent with the rest of
scripts/paperless-eval(the pre-existingeval.pyhad none either) and the nature of the change — these scripts drive a live k8s cluster, GPU-backed llama.cpp sidecars, and real HuggingFace downloads, which aren't meaningfully unit-testable without heavy mocking. Thefloor()Decimal fix, thedegenerate()heuristic, and theABORT_AFTERearly-stop are all plain Python and could in principle get narrow unit tests (e.g.degenerate()against the "Rechnung 2025-5425541" false-positive case the docstring calls out), but given the script's single-file, no-test precedent I wouldn't block on it.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Retrying gemma-4 at --n-predict 4096 did not answer the question — it moved the wall. At 6.6 tok/s a 4096-token budget needs 620 s of generation, and the client bound was still 300, so the run failed with TimeoutError instead of hitting the cap. Two bounds of mine, and lifting one just exposed the other. EVAL_TIMEOUT is now an environment variable and run-candidate.sh takes it as a fourth argument, so a diagnostic can raise both together: ./run-candidate.sh gemma4-26b-a4b 3 4096 900 What the aborted attempt did show: gemma-4 reached 1634 generated tokens and was still going when the clock ran out. A model needing 900 tokens for a valid call would have finished long before that, so this looks like genuine non-termination rather than a cap set too tight — but "looks like" is exactly the reasoning that put 768 in here unexamined, so it is running properly now. The same caveat applies to the Q8 comparison and I have not resolved it. Q8's uncapped responses run 460-520 tokens against Q4's 280-443, roughly 60% longer, and 20 of its requests hit 768 against Q4's one. So "Q8 performs worse" may be "Q8 is more verbose and my cap catches it", which is a much weaker claim. That comparison needs redoing at a raised cap before its numbers mean anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit5f2955cddef458fe71109c7bbe2937258e552054.Summary
This PR fixes the three faults that surfaced when the paperless-eval Job from #776 actually ran: a
Decimal/floatTypeErrorinfloor(), a seed-data init container that silently produced an empty (unRAGged) index, and no way to distinguish genuine model failures from a known llama.cpp degenerate-output mode past ~14k prompt tokens. It also adds a screening/abort mechanism, a candidate-driven job template (replacing per-model YAML copies), a tool-calling probe script, and a substantially rewritten README. The three fixes themselves are implemented correctly and match the failure modes described in the PR body. I found no blocking defects in the executed code paths; the issues below are all in the surrounding tooling/docs and are non-blocking.🟡 Suggestions
scripts/paperless-eval/README.md:17-21— The flagship "Running one" example uses./run-candidate.sh qwen3-8b 5/eval-qwen3-8b, butqwen3-8bis not a row incandidates.tsv(it only appears in the file's screening-results comments as a rejected candidate — "qwen3-8b dense 8B 4 of 5 failed"). Copy-pasting the documented example fails immediately withunknown candidate 'qwen3-8b'. Swap the example for one of the actual entries (e.g.qwen35orgemma4-26b-a4b).scripts/paperless-eval/candidates.tsv:56-70— The file's own closing notes (lines ~208, ~226) conclude "do not spend a screening slot on anything under ~35B total" and recordgemma4-26b-a4bas already tested and conclusively failed (aborted 3/3, same capability failure as the small dense models). Despite that,gemma4-26b-a4b(line 56) and eight dense candidates well under 35B (glm4-9b,commandr7b,qwen25-7b,llama31-8b,nemotron-nano-12b,phi4-mini,qwen3-4b-2507,qwen35-9b, lines 61-70) are left active (uncommented) rather than commented out like the other rejected candidates in the same file (qwen3-30b-a3b,glm47-flash). Runningrun-candidate.shagainst any of these silently re-wastes a screening slot the file's own research explicitly says to avoid — worth commenting them out (or moving to the "ruled out" section) to match the file's stated conclusion.scripts/paperless-eval/job-template.yaml:77-92—du -sh /data/index /data/llm_index(no|| trueany more) runs before the size check. If/data/llm_indexis entirely absent rather than merely small — the exact scenario this PR's narrative describes ("the golden volume was empty...cp -acopied nothing") —set -ekills the container on this line with a rawdu: cannot access ... No such file or directory, before reaching the new FATAL message that tells the operator to rundocument_llmindex rebuild. The init container still fails safe (good), but the diagnostic this hunk was written to add doesn't fire in the one case it was designed for. Guarding with[ -d /data/llm_index ] && du -sb ... || bytes=0(or similar) would preserve the guidance in that case too.scripts/paperless-eval/probe-toolcall.sh:932-933—kubectl cp ... || trueswallows a copy failure silently. If the copy fails (pod not fully ready, no tar in image, etc.), the subsequentcurl -d @/tmp/probe.jsonruns against a missing file and the script reportsFAIL — do not spend a run on this, indistinguishable from a genuine model tool-calling failure. Given the header explicitly says "A FAIL is conclusive", an infra hiccup here could cause a viable candidate to be wrongly discarded. Worth checking thecpexit code (or verifying the file landed) before treating a non-tool_callsresult as conclusive.🟢 Nits
eval.py:250— thedegenerate()allowlist (äöüÄÖÜßáéèêíóúàçñ) is tuned for German correspondence; a legitimate title in another script (Polish/Czech diacritics, etc.) would false-positive as "non-latin". Already an accepted, documented tradeoff for this corpus, so just flagging for awareness rather than asking for a change.Tests
No automated tests exist for
eval.py(this is an ops script driven by one-off Jobs, and that's consistent with the rest ofscripts/paperless-eval/), so I'm not asking for a test suite here. The three bug fixes (floor()'sDecimal/floatcoercion, thedegenerate()classifier, and the seed-data size guard) were each validated against a real failure the PR body quotes, which is reasonable evidence for scripts of this kind. The only place I'd flag as genuinely worth a quick manual smoke-check before relying on it is the seed-data init container's behavior when/data/llm_indexis completely missing (see Suggestions above) — that's the one path that isn't exercised by "index exists but is too small."Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Everything this harness measured about failing candidates was the wrong thing. On this Vulkan/Iris Xe stack every model has a prompt length past which llama.cpp stops producing language and repeats a single token — " de de de", " the the the", " phosphorus phosphorus", or whitespace. Silent: HTTP 200, finish_reason "stop", nothing in the log. And stateful: one request past the boundary breaks the slot for everything after it until the pod restarts. Which means a 50-document run measures where each model's boundary sits relative to paperless's ~5300-token request, and nothing else. "Failed 3 of 3" is one measurement, not three — the first document took the other two with it. Demonstrated on gemma-4-26B-A4B, which I had written up as failing on capability: fresh pod, "Nenne die Hauptstadt von Frankreich" -> "Die Hauptstadt von Frankreich ist Paris." same pod after one 5000-token request -> " phosphorus phosphorus..." fresh pod, 2334-token prompt, no tools at all -> whitespace The model is fine. Its boundary is under 2334 tokens, so it can never serve a paperless request, and every classification number I recorded for it described a corrupted slot. So the size threshold, the generation theory and the capability rankings are all withdrawn. What correlated with success was having a boundary above 5300 — Qwen3.6 measured coherent at 13588 — not being larger or newer. probe-length.sh answers that in minutes instead of ninety. A fresh pod per size, because probing sizes in sequence against one server would measure the first bad request and nothing after it. The galling part is that this defect is already documented in my own notes from this morning, boundary and slot-poisoning both, and I did not apply it when reading the results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit85768cd7e49901669ace3fcef2c8decd3f207ad9.Summary
This PR fixes three real defects surfaced by the first live run of the paperless-eval Job: a
Decimal/floatTypeErrorthat crashedfloor()before any scoring happened, an initContainer that silently seeded nothing when the golden LLM index wasn't built, and a newdegenerate()check that separates "backend gave up past ~14k prompt tokens" from a genuine wrong answer. It also generalizes the single hardcodedjob-qwen35.yamlinto a templatedjob-template.yamldriven bycandidates.tsvand a newrun-candidate.sh, and adds two standalone diagnostic probes (probe-length.sh,probe-toolcall.sh). I read the diff against the full checkout, includingeval.pyin full andjob-template.yamlin full. All three headline fixes are correct and match the failure modes described in the PR body. No blocking issues found.🟡 Suggestions
scripts/paperless-eval/eval.py:307-315(run()'sABORT_AFTERcheck): the early-abort condition isall("error" in r for r in rows), which only counts hard HTTP/transport failures, notdegeneraterows. Per the PR's own description, one degenerate response "poisons the llama.cpp slot until the pod restarts," so if the first few documents come back degenerate rather than erroring (plausible whenever early documents in the sample already sit near the 14k-token boundary via long RAG neighbours), the loop won't trigger the early-abort and will burn the full sample against an already-poisoned slot. The reported numbers would still be correct (degenerate rows are excluded fromscored), but this reintroduces exactly the "hour spent on a hopeless run" problemABORT_AFTERwas written to prevent. Worth considering whether the abort condition should also fire when the firstABORT_AFTERrows are allerror-or-degenerate.scripts/paperless-eval/probe-length.sh:957(inside thefor reps in 4 20 40 70 110 160loop): the per-iteration readiness waithas no failure threshold, unlike the initial wait a few lines above it (bounded at 240 tries) and unlike the equivalent wait in
probe-toolcall.sh:1054-1058(bounded at 120 tries). If a later pod in the loop fails to come up (resource contention, GPU busy, scheduling failure), this script hangs indefinitely with no diagnostic output, rather than failing with a message the way every other wait in this PR does.🟢 Nits
scripts/paperless-eval/run-candidate.sh:1162-1171andprobe-length.sh's pod manifests interpolatecandidates.tsvfields ($repo,$alias, etc.) directly intosedsubstitutions and YAML heredocs. Sincecandidates.tsvis a reviewed, repo-local file this isn't a real security concern, but a future field value containing&,#, or unescaped YAML-significant characters would silently corrupt the generated manifest rather than error out clearly.Tests
No automated tests exist for this directory (it's operational tooling, not application code), consistent with the rest of the repo's
scripts/conventions — there's no existing test harness this PR is expected to plug into. The fixes are the kind that are naturally validated by the live Job run the PR body references (and the "Note" in the README says that run is in progress). I did not execute any of the scripts or the eval Job myself; thefloor()fix, thedegenerate()logic, and the seed-data byte-threshold check are straightforward enough to verify by reading, and I traced each through the surrounding code (api_token,run,summarise,complementineval.py; the fulljob-template.yaml) to confirm no other caller depends on the old behavior.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
The tests have to get better, and this is the change that would have caught today's whole error class on document one instead of after six candidates. Every AI run now asks "Nenne die Hauptstadt von Frankreich" before the documents, every ten documents, and again when it aborts. No Paris, no run. Why it goes mid-run and not only at the start: failures are not always clustered. Ministral-3 3B failed 29 of 50 scattered through the run, which a start-of-run check would have passed straight through. A slot that dies at document 12 otherwise produces 38 more "failures" that are one event. What the harness did wrong, so it is written down rather than relearned: - it measured classification accuracy while a confound (slot corruption) dominated, and never established that it could tell the two apart - it counted sequential failures on a shared slot as independent evidence - it recorded status codes and token counts and never once looked at what the model actually emitted — twice the answer was one curl away - it treated my own bounds, --n-predict 768 and a 300 s timeout, as properties of the models hitting them A run whose canary breaks is void, not a result. Failures adjacent to the break are one fault, and probe-length.sh says where the model's boundary really is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit73db641d4732655f6fd41a6a8f396f08031c7e45.Summary
This PR fixes three real problems found when
job-qwen35.yamlwas first run: aDecimal/floatcrash infloor(), a silently-empty golden volume in the seed-data initContainer, and unflagged "degenerate" (poisoned-slot) completions being scored as wrong answers. It also adds a canary probe to detect a broken llama.cpp slot, anABORT_AFTERearly-exit for hopeless candidates, templatizes the Job manifest (job-template.yaml+run-candidate.sh+candidates.tsv), and adds two standalone probe scripts. The three faults described in the PR body are fixed correctly, but the new canary machinery introduces an unhandled crash of its own insummarise()/complement()whenever the canary actually fires — exactly the scenario this PR exists to handle gracefully.🔴 Blocking
scripts/paperless-eval/eval.py:321and:372— the two rowsrun()appends when the canary fails ({"engine": engine, "canary_before": False, ...}and{"engine": engine, "canary_failed_at": n, ...}) have noerror,degenerate,catchall_type,document_id, or scoring keys.summarise()(eval.py:409-410, filtering intoscoredvia"error" not in r and "degenerate" not in r) will pick up such a row and then crash ateval.py:415,specific = [r for r in scored if not r["catchall_type"]], withKeyError: 'catchall_type'.complement()(eval.py:451,by_engine[row["engine"]][row["document_id"]] = row) has the same problem viarow["document_id"]. Both are called unconditionally frommain()(summarieslist comprehension runs first, so that's the one that actually fires) whenever"ai"is inENGINES— there is no guard skipping a canary-failure marker row.Concretely: if the canary fails before the AI loop starts, or mid-run at the 10-document check, the job crashes with an unhandled
KeyErrortraceback instead of writing<label>.summary.json/<label>.jsonlat all — not even the classic-engine results that were already collected get written, since the crash happens while buildingresultinmain(), before either file is opened. This is a regression relative to even the pre-PR crash-prone script: the log lines written just before ("The model is not answering a trivial question correctly, so nothing measured after this describes its classification.") promise a controlled outcome that the code doesn't actually deliver. Given the PR's own stated goal is to avoid a run silently reporting a poisoned-slot as a model result, this is the one scenario most likely to be hit in practice (six candidates in the PR's own history hit exactly this), and it currently ends in a stack trace with zero output rather than a usabledegenerate/canarysummary.Fix direction: give the canary-failure rows the same keys the rest of the pipeline expects (or filter them out before
scored/specific/catchall/by_engineconstruction, and surface the canary failure as a top-level field inresultinstead of a row).🟡 Suggestions
scripts/paperless-eval/eval.py:288-289—canary()readsPAPERLESS_AI_LLM_MODEL(default"probe") for the request body's"model"field, but theevalcontainer injob-template.yaml(lines 293-309) never setsPAPERLESS_AI_LLM_MODELorPAPERLESS_AI_LLM_ENDPOINT— only thepaperlesssidecar does. TheLLM_ENDPOINTdefault (http://127.0.0.1:8080/v1) happens to be correct since all containers share the pod network and llama-server listens on 8080 (job-template.yaml:179-180), so this works, but the canary always sends"model": "probe"even though the candidate's real alias (e.g.qwen3.6-35b-a3b) is what's actually loaded. Harmless as long as llama-server ignores themodelfield for a single-loaded-model server (as it typically does), but worth passingPAPERLESS_AI_LLM_MODEL/PAPERLESS_AI_LLM_ENDPOINTthrough explicitly to theevalcontainer so this isn't relying on an implicit default matching an unrelated hardcoded value.scripts/paperless-eval/eval.py:453-467(api_token) — the new per-label user is looked up byselect ... where username = %sand, if absent, inserted, with no unique constraint/locking discussed. If the same label were ever run concurrently (e.g., a screen and a full run for the same candidate started without deleting the first Job, which the README does warn against), the two could race between theselectandinsert, potentially producing two rows or an insert conflict. Low risk given the documented single-Job-per-label workflow, but worth a comment oron conflict do nothingif it's meant to be safe against that.scripts/paperless-eval/probe-length.sh/probe-toolcall.sh— bothkubectl exec "$pod" -- curl ...against theghcr.io/ggml-org/llama.cpp:server-vulkan-b10731image. Worth confirming that image actually shipscurl(llama.cpp server images are often minimal); if not, both probes fail immediately with "executable file not found" rather than a useful message.🟢 Nits
scripts/paperless-eval/probe-toolcall.sh:54— building the JSONargsarray by string-concatenatingprintf ',"%s"' "$a"for extra CLI args doesn't escape embedded quotes/backslashes in"$@". Only matters for hand-typed extra args, so low impact.candidates.tsvcarries a very long research log as comments (lines 246-482). Useful history, but consider trimming to the operative conclusions if this file is expected to grow with more candidates — it's already larger than the code it configures.Tests
No automated tests exist for
eval.pyin this repo (it's an operational script run via Kubernetes Job), consistent with the rest of the directory. Given that, the canary-failure code path (the Blocking issue above) is exactly the kind of edge case that would have been caught by a small unit test aroundsummarise()/complement()with a canary-failure row — worth adding given how easy it is to construct (see the two dict literals ateval.py:321and:372).Recommendation
Request changes — 1 blocking issue.
🤖 Automated review — model
CLI default(complexity: complex), commitf54b1dc7fad4f25fb3776cad41725bc5871c8fd5.Summary
This PR fixes three problems surfaced by the first real job run: a
Decimal/floatcrash infloor(), a silent no-op in theseed-datainitContainer that could measure the no-RAG path and call it RAG, and a newdegenerate()check pluscanary()instrument to stop a poisoned llama.cpp slot from being scored as model failures. It also addscandidates.tsv+run-candidate.sh+job-template.yamlto stop hand-copying the Job manifest, and two standalone probe scripts. The fixes for the three described faults are correct, but the new canary safety net itself has a bug that crashes the script in exactly the scenario it exists to handle gracefully.🔴 Blocking
scripts/paperless-eval/eval.py:321and:372— the sentinel rows returned/appended when the canary check fails ({"engine": engine, "canary_before": False, ...}and{"engine": engine, "canary_failed_at": n, ...}) have neither an"error"nor a"degenerate"key.summarise()(line 412) treats "no error, no degenerate" as scored, then at line 415 doesspecific = [r for r in scored if not r["catchall_type"]]— but the sentinel row has no"catchall_type"key at all, so this raises an unhandledKeyError: 'catchall_type'.Concretely: if the "before" canary fails (model broken/wedged from the start of the
airun),run()returns immediately with just the sentinel row, andmain()'s call tosummarise(rows, "ai")crashes before anything is written to/results. The mid-run canary failure (line 372, triggered every 10 documents) hits the identical crash after appending its own sentinel row to a batch of otherwise-valid rows.This defeats the PR's own stated goal — the README says "If Paris does not come back, the run stops and says so" — but what actually happens is the process dies with a Python traceback instead of writing a clean summary/jsonl showing the canary failure. It's the same class of bug this PR set out to fix in
floor()(crash before scoring anything), reintroduced by the new canary path.Fix direction: either give the sentinel dict the fields
summarise()/complement()expect (e.g."catchall_type","seconds","document_id", defaulted), or explicitly exclude rows carrying"canary_before"/"canary_failed_at"frommine/scoredinsummarise()and from theusablecheck incomplement()(scripts/paperless-eval/eval.py:449).🟡 Suggestions
scripts/paperless-eval/job-template.yaml:78—du -sh /data/index /data/llm_indexruns underset -ewith no|| truenow (the old blanket suppression was correctly identified as swallowing the real problem). But if/data/indexwere ever absent, this line itself would abort the initContainer before reaching the informativellm_indexbyte-count check a few lines down, so the failure would show a baredu: cannot accessinstead of the intended "FATAL: llm_index is N bytes — build it..." message. Worth confirming the golden PVC always ships/data/index, or moving thisduafter the size check (or scoping|| trueto just this diagnostic line, not the size gate).scripts/paperless-eval/probe-length.sh:1041— the fixed rep sequence (4 20 40 70 110 160) with the ~115-character filler sentence tops out around a few thousand prompt tokens atreps=160by rough token-count arithmetic, well short of the 13588/15086-token boundary the script's own header comment (lines 988-990) cites for Qwen3.6. If that estimate holds, a model with a boundary in that range would print "coherent through the largest size probed" without the script ever having tested anywhere near where it actually breaks. Worth confirming against theprompt_tokensthe script itself prints at the top rep count, and extending the rep list if it doesn't reach into five figures for the models this is meant to clear.🟢 Nits
scripts/paperless-eval/candidates.tsv:478—hermes4-14b's alias column ishemes-4-14b(missing the "r"). It's used consistently for both--aliasandPAPERLESS_AI_LLM_MODELso it won't break anything, but it'll show up misspelled in pod logs/results.Tests
No automated tests exist for this directory, consistent with the rest of these ops scripts (they're driven against a live k8s cluster and aren't easily unit-tested). That said, the Blocking issue above is exactly the kind of defect a two-line local smoke test would catch — calling
summarise()/complement()directly against a syntheticrowslist containing a canary-sentinel dict, without needing a cluster.Recommendation
Request changes — 1 blocking issue.
🤖 Automated review — model
CLI default(complexity: complex), commit3f50fa4e6069298a08ca508af585f10a33fa2ff7.Summary
This PR fixes three real problems found by the first live job run (a
Decimal/floatcrash infloor(), a silently-empty golden volume, and undetectable "poisoned slot" garbage output), and adds substantial new tooling (run-candidate.sh,probe-length.sh,probe-toolcall.sh,candidates.tsv) plus a heavily-rewritten README. Thefloor()fix and the initContainer volume guard are correct and well-reasoned. However, the new canary-check machinery ineval.pyintroduces a fresh crash bug that fires in exactly the scenario it exists to handle gracefully (a broken/poisoned inference slot) — this is blocking.🔴 Blocking
scripts/paperless-eval/eval.py:230-411(run()) /eval.py:419-450(summarise()) /eval.py:453-486(complement()) — a canary failure crashes the whole run with an unhandledKeyErrorinstead of producing the intended clean report.When
canary()fails,run()appends (or, on the "before" check, exclusively returns) a sentinel dict shaped like{"engine": "ai", "canary_before": False, "canary_text": "..."}(same forcanary_failed_at/canary_after, ateval.py:321,:372,:411). These sentinel rows have none of the keys a real document row has (document_id,seconds,catchall_type,correspondent_offered, ...).summarise()buildsmine = [r for r in rows if r["engine"] == engine]and then, before ever reaching thereturn, executes:scoredstill contains the sentinel row (it has neither"error"nor"degenerate"), so this line raisesKeyError: 'catchall_type'as soon as any canary check fails.complement()has the identical problem one line further down (row["document_id"]oneval.py:463), though execution never reaches it becausesummarise()is called first inside theresult = {...}literal inmain()(eval.py:502-508), sosummarise()always crashes first.I confirmed this by reading the code path directly (a sandboxed
python3wasn't available to execute a repro here, but the trace is unambiguous: the sentinel dicts genuinely lack the keysspecific/complementindex into, and nothing filters them out before those comprehensions run).Consequences:
canary_before: false(or similar) in the output JSON, the pod crashes with a traceback and no.summary.json/.jsonlis written at all.classicran first and completed successfully (the defaultEVAL_ENGINES=classic,aiorder), that already-computed classic-engine result is also discarded, because the crash happens while buildingresultinmain(), before either output file is written (eval.py:510-514).eval.py:315-321) is the easiest to hit — it fires before a single document is scored, e.g. right after a fresh candidate's llama-server pod comes up still wedged.Fix direction: exclude non-document sentinel rows from the comprehensions in
summarise()/complement()(e.g. filter on"document_id" in r, or track canary outcomes in a separate structure fromrowsrather than appending them into the same list).🟡 Suggestions
scripts/paperless-eval/job-template.yaml:77-92(seed-datainitContainer) — the friendlyFATALdiagnostic can be skipped by a rawduerror in exactly the scenario the comment describes. The script runs underset -eand the un-suppresseddu -sh /data/index /data/llm_index(line 78, no longer wrapped in2>/dev/null || trueas it was before this PR) will abort the script with GNU du's own "cannot access ... No such file or directory" message if/data/llm_indexdoesn't exist at all yet — which is exactly the "smoke-test deployment had not built the LLM index yet" case called out in the PR description as the original incident. In that case, execution never reaches the craftedFATAL: llm_index is ${bytes} bytes ...message with the rebuild command. The container still correctly refuses to proceed (so no silent bad measurement), but the more helpful diagnostic doesn't fire for the specific case it was written for. Considerdu -sh /data/index /data/llm_index 2>&1 || truefor the informational print, keeping the hard failure only in thebytes -lt 1000000check (which already handles "doesn't exist" viadu -sbreturning 0/empty — worth double-checking thatcut -f1on adu -sbfailure doesn't yield an empty string that then breaks the-ltcomparison the same way).scripts/paperless-eval/eval.py:275-309(canary()) — relies on two env vars (PAPERLESS_AI_LLM_ENDPOINT,PAPERLESS_AI_LLM_MODEL) that aren't set on theevalcontainer (job-template.yaml:298-322); they're only set on thepaperlesssidecar (job-template.yaml:241-244). This works today only because the defaults (http://127.0.0.1:8080/v1,"probe") happen to be correct for this pod layout and llama-server apparently doesn't validate themodelfield against its loaded alias. Fine in practice, but worth a comment noting the defaults are load-bearing, since the two env vars silently diverging from the sidecar's actual config wouldn't be caught by anything.🟢 Nits
canary()call sites inrun()(before / every-10 / abort-after / end) repeat very similar logging boilerplate. Not necessary to change, but a smallreport_canary(where, ok, text)helper would remove some duplication if this grows further.Tests
No automated tests exist for this script (consistent with the rest of the repo — these are manually-invoked operational tools), and the sample-cost of running one (real k8s pods, model downloads, tens of minutes) makes a full integration test impractical here. That said, the blocking bug above is exactly the kind of thing a two-line unit test for
summarise()/complement()with a synthetic canary-failure row would have caught immediately, and is worth adding given how easy it is to hit.Recommendation
Request changes — 1 blocking issue.
The measurements were coupled. One request that corrupts the slot leaves it returning a single repeated token for everything after it, so a run counted that fault once per remaining document. "Qwen3-8B failed 4 of 5" was really "document 1970 succeeded, document 363 broke the slot, and the remaining three measured the wreckage". Proved mid-run rather than inferred: with two documents failed, the model answered "Nenne die Hauptstadt von Frankreich" with " \" \" \" \" \" \" \" \" \" \" ... llama-server clears a slot on POST /slots/0?action=erase, but answers 501 without --slot-save-path, so the flag goes on the sidecar and eval.py calls it after every AI document. It costs the prefix cache and some prefill speed, which is the right trade for numbers that mean what they say. Also worth recording: a synthetic reproduction of the poisoning failed. The real paperless request corrupts the slot; my own 5000-token probe with the same 14-field schema does not, and the canary stayed healthy through it. Whatever triggers this lives in the parts of a real request I was not reproducing — the system prompt, the RAG context, the full taxonomy lists. Third time today that a synthesised test passed where the real one fails, so the harness should stop trying to stand in for the real request and instead detect when it has gone wrong, which is what the canary now does. Re-screening qwen3-8b to find out whether it scores more than one of five once the documents are actually independent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit1ec6825b66ad7854e1b531a871acf9b917ec9c8b.Summary
This PR fixes the three faults from the first real job run (a
Decimal/floatcrash infloor(), a silently-empty golden volume, and undetectable "poisoned slot" garbage output), and adds substantial new tooling around candidate screening (run-candidate.sh,probe-toolcall.sh,probe-length.sh,candidates.tsv) plus a canary/degenerate-output detection system ineval.py. The core fixes are sound and well-reasoned, but the new canary-failure paths ineval.pyintroduce a real crash bug that defeats the PR's own goal of failing gracefully and visibly when the instrument breaks.🔴 Blocking
scripts/paperless-eval/eval.py:321,393,432—run()appends "sentinel" rows for a broken canary (canary_before,canary_failed_at,canary_after) that contain onlyengine/text fields and are missingseconds(anderror/degenerate/catchall_type/etc). These rows flow intomain()'srowslist (eval.py:521) and then intosummarise(rows, "ai")(eval.py:527).In
summarise()(eval.py:441-457):A sentinel row has neither
errornordegenerate, so it lands inmineand also inscored. The very next line,sum(r["seconds"] for r in mine), raisesKeyError: 'seconds'for that row before the dict literal even gets tocorrespondent_offered/catchall_type(which would alsoKeyError, e.g. onspecific = [r for r in scored if not r["catchall_type"]]).Concretely: any run where the pre-flight canary fails (
return [{"engine": engine, "canary_before": False, ...}]at line 321) causesmain()to crash with an unhandledKeyError— no.summary.json, no.jsonl, and the JSON summary never reaches the log. The same happens if the periodic (every-10-docs) or post-run canary check fires. This is exactly the "poisoned slot" / "broken instrument" scenario the whole PR is built to detect and report cleanly, so the one case this feature exists for is the one that currently crashes the script uninformatively instead of producing the intendedcanary_before: Falsesignal in a usable summary.Fix direction: either give these sentinel rows a
seconds: 0(and keep them out ofscored/failed/brokensome other way — they already lackerror/degenerateso they wrongly count as "scored"), or filter canary sentinel rows out ofmineinsummarise()explicitly (e.g. skip rows carryingcanary_before/canary_failed_at/canary_after) before computing any aggregate.🟡 Suggestions
scripts/paperless-eval/eval.py:407—if len(rows) == ABORT_AFTER and all("error" in r for r in rows): this only counts harderrorrows, notdegenerateones. If the first three documents come back as degenerate garbage (poisoned slot from document one, before the per-document slot-erase logic and the every-10 canary catch up), the abort-after-3 short-circuit won't fire, and the run continues until the next canary checkpoint at document 10. Given the PR's own motivating incident (a poisoned slot silently producing wrong-looking output), it may be worth includingdegeneratealongsideerrorin this check, or explicitly noting why it's intentionally excluded.scripts/paperless-eval/eval.py:353—degenerate()is only invoked forengine == "ai", and only afterpayload is not None, so a document where the AI response itself is HTTP-erroring is still just counted aserror, which is correct — no change needed here, just confirming this path is fine (noted so the review doesn't look incomplete on this point).🟢 Nits
scripts/paperless-eval/candidates.tsv— this file has grown into an extensive research log (over 200 lines of prose comments documenting the whole model search) embedded in a TSV meant to beawk-parsed by two shell scripts. It works, and the trail of reasoning is genuinely useful, but consider whether aNOTES.mdalongside the TSV would separate "data the scripts consume" from "narrative for humans" more cleanly. Not blocking — purely a maintainability observation.Tests
There is no automated test suite for this script (none existed before this PR either), and given it's a one-shot operational tool driven by a real Kubernetes cluster and live LLM backends, that's consistent with the rest of the repo's approach to this directory. The blocking bug above is exactly the kind of thing a small unit test around
summarise()with a canary-sentinel row would have caught cheaply, since it requires no cluster access — worth adding a minimal test (or at least apython3 -csmoke check in the PR description) forsummarise()/complement()given how easy it is for new row shapes to silently break the aggregate-computing assumptions about which dict keys are always present.Recommendation
Request changes — 1 blocking issue.
🤖 Automated review — model
CLI default(complexity: complex), commit8123021ea4bd509a1134c3c1d7d1227c641cf7c8.Summary
This PR fixes three faults surfaced by the first real eval Job run: a
Decimal/floatcrash infloor(), a silent no-op seed step that could report a no-RAG run as RAG, and undetected "degenerate" completions from a poisoned llama.cpp slot. TheDecimaland seeding fixes are correct and well-targeted. However, the new canary-check machinery ineval.pyintroduces a crash bug that fires in exactly the scenario it exists to handle — a broken/poisoned slot — which is significant given the PR's own stated motivation.🔴 Blocking
scripts/paperless-eval/eval.py:321,393,432+448— the canary-failure marker rows appended torows(canary_before,canary_failed_at,canary_after) only carryengine/canary_text-style keys, not the standard per-document fields (catchall_type,seconds,correspondent_offered, etc.).summarise()filters these rows intoscored(they have neither"error"nor"degenerate"), then doesspecific = [r for r in scored if not r["catchall_type"]]at line 448 — an unconditionalKeyError: 'catchall_type'for any marker row that reaches it. Sinceresult = {..., "summaries": [summarise(...) ...], ...}is built before the.jsonl/.summary.jsonfiles are written inmain(), this exception propagates out ofmain()unhandled: the run ends in a Python traceback with no summary.json and no jsonl at all, instead of the clean "canary broken, run is void" report the feature is meant to produce. This isn't a rare edge case — it's the direct consequence of the exact failure mode (poisoned slot / degenerate output) that this PR was written to detect and report gracefully. A quick trace:run()returns[{"engine": "ai", "canary_before": False, "canary_text": ...}]the moment the pre-flight canary fails (line 321), and that single row is enough to crashsummarise(rows, "ai")on the very first call.scripts/paperless-eval/eval.py:484(complement()) — same root cause:by_engine[row["engine"]][row["document_id"]] = rowassumes every "usable" row (no"error"/"degenerate") has a"document_id", which the canary marker rows don't. This would raise the same class ofKeyErrorifsummarise()didn't already crash first — worth fixing alongside it (e.g. give marker rows a distinct, always-checked shape, or filter them out ofscored/by_engineexplicitly rather than relying on absence of"error"/"degenerate").🟡 Suggestions
scripts/paperless-eval/probe-length.sh:135—rep=$(printf '%s' "$txt" | tr -s ' ' '\n' | grep -c . || echo 0). When$txtis genuinely empty,grep -c .prints0to stdout and exits 1 (no matching lines), so|| echo 0also fires —repends up as the two-line string"0\n0"instead of0. The subsequent[ "$rep" -gt 8 ]/[ "$rep" -le 1 ]then fail with "integer expression expected" rather than correctly selecting theEMPTYverdict branch, so the one case this script most needs to catch (truly empty output) is exactly the one this line mishandles. Simplify torep=$(... | grep -c .)(drop the redundant|| echo 0, sincegrep -calready prints0on no match) or usegrep -c . || truewith the count read from stdout only.scripts/paperless-eval/job-template.yaml:74-77— the comment says running theseed-datainitContainer as uid 1000 is "what makes the copy land with the ownership paperless needs," but thepaperlesscontainer's own securityContext comment (line 289-290) says its entrypoint "starts as root, chowns the volumes and drops to uid 1000" regardless. If that's accurate, therunAsUser/runAsGroup: 1000addition here is redundant with the chown paperless already performs — not wrong, just worth confirming it isn't papering over a different problem (e.g. it was added to fix a real failure, in which case the paperless-entrypoint chown comment may be stale, or vice versa).🟢 Nits
scripts/paperless-eval/eval.py:340catchesHTTP 401specifically to detect token revocation; this is a reasonable heuristic but is silently defeated if the API ever returns 401 for an unrelated reason (e.g. a transient auth-middleware hiccup) — acceptable given the documented incident it targets, just noting it's a heuristic, not a strict token-identity check.Tests
There are no automated tests for
eval.py(consistent with the rest ofscripts/, which is untested ops tooling), so this shipped as a live Job run rather than being caught pre-merge — which is exactly how theDecimal/floatbug infloor()was found originally, per the PR description. The new blocking bug (canary marker rows crashingsummarise()) is easy to catch with a narrow unit test that doesn't need Postgres or Kubernetes: callsummarise()andcomplement()directly with arowslist containing one of the three marker-row shapes and assert they don't raise. Given how centralrun()'s canary logic now is to the script's core promise ("a run whose canary breaks is void, not a result"), that's worth adding even without a broader test harness.Recommendation
Request changes — 1 blocking issue (the second finding shares its root cause and should be fixed in the same pass).
Abschluss — was dieser PR am Ende ist, und was von den Ergebnissen bleibt
Der Titel beschreibt den ersten von vielen Befunden. Tatsächlich enthält der Branch die Messumgebung und die Erkenntnis, dass ihre erste Ergebnisrunde ungültig war.
Der Befund, der alles andere relativiert
llama.cpp b10731 kann auf diesem Vulkan-Stack einen Slot dauerhaft zerstören. Danach liefert jede Anfrage ein einzelnes wiederholtes Token —
" de de de"," phosphorus phosphorus", Leerzeichen — bei HTTP 200 undfinish_reason: "stop", ohne Logeintrag.Der Harness fährt einen Slot für fünfzig aufeinanderfolgende Dokumente. Das erste Dokument, das ihn vergiftet, nimmt alle folgenden mit. "3 von 3 gescheitert" ist ein Vorfall, nicht drei Messungen.
Bewiesen mitten im Lauf: gemma-4-26B-A4B beantwortet auf frischem Pod "Nenne die Hauptstadt von Frankreich" korrekt und liefert nach einer einzigen langen Anfrage nur noch
phosphorus phosphorus— auch auf dieselbe triviale Frage.Zurückgezogen
Drei Schlussfolgerungen, die ich hier committet hatte, sind hinfällig, weil sie auf genau diesen Daten beruhten: die MoE-These, die Generationsthese und die 35-Milliarden-Schwelle. Letztere hatte sogar einen gezielten Widerlegungsversuch überstanden — der Widerlegungskandidat produzierte aber bloß Müll aus einem kaputten Slot.
Sechs Modelle stehen als "unfähig" in der Historie, deren Bewertung nicht trägt.
Was gemessen bleibt
Qwen3.6-35B-A3B Q4_K_M über 50 Dokumente, zweimal, konsistent: Korrespondent 72,9 angeboten / 77,1 erkannt, Typ 85,4, zwei Ausfälle. Komplement 30 gemeinsam richtig, 5 gerettet, 4 verschlechtert. Rauschgrenze rund ein Dokument — Unterschiede unter zwei Dokumenten bedeuten nichts.
Dazu: der Klassifikator trainiert ohne Holdout, seine 100 % sind auswendig gelernt. difflib verwirft bei 0,8 korrekte Antworten wie "GOTOOLS GmbH" gegen "GOTOOLS". Q8 kostet zehn Prozent Geschwindigkeit, nicht die Hälfte — Dekodierung ist bei diesem MoE rechen- und nicht bandbreitenbegrenzt.
Was der Harness jetzt kann
Kanarienvogel vor dem Lauf, alle zehn Dokumente und am Ende: eine Frage mit bekannter Antwort. Kommt Paris nicht zurück, ist der Lauf ungültig statt ein Ergebnis.
Slot-Löschung zwischen Dokumenten über
--slot-save-pathplusPOST /slots/0?action=erase, damit Fehlschläge unabhängig werden.Abbruch nach drei Fehlschlägen, Sieben mit fünf Dokumenten, exakte Dateinamen statt
:QUANT, einstellbare Token- und Zeitgrenzen — beides sind Eigenschaften des Harness und nicht der Modelle, was hier lange verwechselt wurde.Empfehlung
Der Branch ist als Werkzeug mergebar. Die Kandidatentabelle in
candidates.tsvist es als Ergebnis nicht — sie müsste mit der Slot-Löschung neu erhoben werden. Die zurückgezogenen Schlüsse stehen als solche markiert im Repo, nicht stillschweigend überschrieben.