Retrying gemma-4 at --n-predict 4096 did not answer the question — it moved the
wall. At 6.6 tok/s a 4096-token budget needs 620 s of generation, and the
client bound was still 300, so the run failed with TimeoutError instead of
hitting the cap. Two bounds of mine, and lifting one just exposed the other.
EVAL_TIMEOUT is now an environment variable and run-candidate.sh takes it as a
fourth argument, so a diagnostic can raise both together:
./run-candidate.sh gemma4-26b-a4b 3 4096 900
What the aborted attempt did show: gemma-4 reached 1634 generated tokens and
was still going when the clock ran out. A model needing 900 tokens for a valid
call would have finished long before that, so this looks like genuine
non-termination rather than a cap set too tight — but "looks like" is exactly
the reasoning that put 768 in here unexamined, so it is running properly now.
The same caveat applies to the Q8 comparison and I have not resolved it. Q8's
uncapped responses run 460-520 tokens against Q4's 280-443, roughly 60% longer,
and 20 of its requests hit 768 against Q4's one. So "Q8 performs worse" may be
"Q8 is more verbose and my cap catches it", which is a much weaker claim. That
comparison needs redoing at a raised cap before its numbers mean anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fair challenge I had not put to myself: --n-predict 768 is a bound I chose this
morning because Qwen3.6 once ran to 10269 tokens and cost 900 s a document. It
is not a property of any model.
Which means every candidate recorded as failing at exactly 768 generated
tokens — Hermes-4-14B, gemma-4-26B-A4B, both Ministrals, Granite, Qwen3-8B — is
failing against MY bound, and I wrote them up as incapable. A model that needs
900 tokens to emit a valid tool call is indistinguishable, in that record, from
one that never terminates at all.
run-candidate.sh takes the cap as a third argument:
./run-candidate.sh gemma4-26b-a4b 5 4096
gemma-4 is re-running at 4096 against the same three documents it failed, since
it is the newest model in the field and the cleanest failure of the set. If it
produces valid calls given room, six rejections in this file are wrong and the
size threshold I committed an hour ago is built on an artefact. If it still
runs to the new cap, the cap only changed how long it took to find out.
Worth the node time either way. A working answer takes 241-291 tokens, so 768
looked generous — but "looked generous" is not evidence, and I should have
tested the bound before drawing conclusions from models hitting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two jobs crashlooped for three hours without producing a single document, and
I did not notice because the pods sat in Init the whole time — which looks
exactly like a slow model download.
llama_model_load_from_file_impl: exactly one out metadata, path_model,
and file must be defined
That reads as a loading error. It is an ambiguous reference. `--hf-repo
repo:QUANT` matches by substring across the entire repository, and repos
routinely carry several files bearing one tag: ggml-org ships mmproj-, mtp- and
dflash- variants at Q8_0 beside the model, bartowski ships
kld_results/kld_Q4_K_M.txt beside the weights. Q4_K_M happened to be
unambiguous for the baseline, which is why this never surfaced before.
candidates.tsv now carries the exact filename as its own column, resolved from
the HF API for all fifteen entries, and run-candidate.sh refuses to start a
candidate that has none rather than guessing. --hf-file overrides the quant.
Also fixing the monitoring gap that hid it. The watch treated PodInitializing
as "keep waiting" indefinitely, so a container failing to start was
indistinguishable from one still pulling 34 GB. It now checks init container
states for CrashLoopBackOff/Error/ImagePullBackOff and reports with the last
lines of the failing container's log.
Silence was not success, and the tooling should have said so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The setup is now good enough to test everything with a chance, so the gate
matters more than any single run. run-candidate.sh takes a sample size:
./run-candidate.sh <label> 5 screen
./run-candidate.sh <label> measure
With ABORT_AFTER a hopeless candidate is rejected in about five minutes. Three
of the first four were rejections, and each was left running for an hour or
more first.
Screening deliberately uses genuine paperless requests rather than
probe-toolcall.sh. The standalone probe is cheaper and still worth a first
pass, but it passed two candidates that failed on document one — a synthesised
request cannot reproduce the RAG neighbours and the taxonomy candidate lists
that make a real prompt what it is. Document 1970 is 1976 characters and still
produces a 5333-token request.
A screen is a gate, not a measurement. Numbers are never compared across
sample sizes; the comparison happens at 50.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The point of this harness is comparing chat models, and it was shaped for
comparing engines. Four candidates meant four 300-line copies of the same Job,
which is exactly how --chat-template-kwargs ended up on the isolation pod but
not on the first eval job — two hours of empty completions before anyone
looked.
So: job-template.yaml holds the sidecar configuration once, candidates.tsv
holds the five values that differ, and run-candidate.sh substitutes them.
Adding a model is a line, not a file.
Candidates, all first-party GGUFs verified to exist with a Q4_K_M:
ministral8b mistralai/Ministral-3-8B-Instruct-2512
ministral3b mistralai/Ministral-3-3B-Instruct-2512
granite8b ibm-granite/granite-3.3-8b-instruct
Instruct, never Reasoning — a model tuned to think first does by design what
cost Qwen3.6 two of fifty documents. Sizes are chosen because decode is
bandwidth-bound at ~11 tok/s and prefill was 51% of the first run's wall clock,
so a smaller dense model gains at both ends; 3B against 8B is the size
ablation, granite against ministral the family one.
The classifier stays in every run as a fixed reference line, not as a quality
bar: train() fits on every non-inbox document with no holdout, so the
evaluation documents are in its training set and its 100% on document type is
memorisation. Irrelevant to ranking models — same constant under all of them —
and the README now says so rather than inviting the comparison.
First result, Qwen3.6-35B-A3B Q4_K_M over 50 documents: correspondent 70.8
offered / 75.0 recognised against the classifier's 72.0, document type 83.3
against 100, and 33.3 against 100 on the "Sonstige" catch-all. 5 rescued
against 5 broken — but two of those five are difflib rejecting a correct
answer, "GOTOOLS GmbH" against the stored "GOTOOLS", an exact prefix scoring
~0.74 under MATCH_THRESHOLD = 0.8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 5