claude-botandClaude Opus 5
7136a39f3a
Retrying gemma-4 at --n-predict 4096 did not answer the question — it moved the
wall. At 6.6 tok/s a 4096-token budget needs 620 s of generation, and the
client bound was still 300, so the run failed with TimeoutError instead of
hitting the cap. Two bounds of mine, and lifting one just exposed the other.
EVAL_TIMEOUT is now an environment variable and run-candidate.sh takes it as a
fourth argument, so a diagnostic can raise both together:
./run-candidate.sh gemma4-26b-a4b 3 4096 900
What the aborted attempt did show: gemma-4 reached 1634 generated tokens and
was still going when the clock ran out. A model needing 900 tokens for a valid
call would have finished long before that, so this looks like genuine
non-termination rather than a cap set too tight — but "looks like" is exactly
the reasoning that put 768 in here unexamined, so it is running properly now.
The same caveat applies to the Q8 comparison and I have not resolved it. Q8's
uncapped responses run 460-520 tokens against Q4's 280-443, roughly 60% longer,
and 20 of its requests hit 768 against Q4's one. So "Q8 performs worse" may be
"Q8 is more verbose and my cap catches it", which is a much weaker claim. That
comparison needs redoing at a raised cap before its numbers mean anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
91 lines
3.6 KiB
Bash
Executable File
91 lines
3.6 KiB
Bash
Executable File
#!/bin/sh
|
|
# Start one candidate's measurement.
|
|
#
|
|
# ./run-candidate.sh ministral8b
|
|
#
|
|
# Exists so the sidecar configuration lives in one file. Four copies of a
|
|
# 300-line Job manifest is how --chat-template-kwargs ended up on the isolation
|
|
# pod but not on the first eval job, and the run that followed measured empty
|
|
# completions for two hours before anyone noticed.
|
|
set -eu
|
|
|
|
label="${1:?usage: run-candidate.sh <label> [sample] (labels are in candidates.tsv)}"
|
|
|
|
# Screen before you measure. A candidate that cannot do the task at all is
|
|
# rejected by eval.py's ABORT_AFTER within three documents, so a screening run
|
|
# costs about five minutes; a full run costs ninety. Three of the first four
|
|
# candidates were rejections, and each one was allowed to run for an hour or
|
|
# more before anyone looked.
|
|
#
|
|
# ./run-candidate.sh <label> 5 screen
|
|
# ./run-candidate.sh <label> measure, 50 documents
|
|
#
|
|
# Screening uses genuine paperless requests, which matters: the standalone
|
|
# probe-toolcall.sh passed two candidates that then failed on document one,
|
|
# because a synthesised request cannot reproduce the RAG neighbours and the
|
|
# taxonomy candidate lists that make a real prompt what it is.
|
|
#
|
|
# Never compare numbers across sample sizes. A screen is a gate, not a
|
|
# measurement; the comparison happens at 50.
|
|
sample="${2:-50}"
|
|
|
|
# Third argument raises the server-side generation cap for a diagnostic run.
|
|
#
|
|
# The default 768 is MINE, not a property of any model, and every candidate
|
|
# recorded as failing at exactly 768 tokens is failing against my bound rather
|
|
# than demonstrably failing the task. A model that needs 900 tokens for a valid
|
|
# call looks identical to one that never terminates. Raise it and find out
|
|
# before calling a model incapable:
|
|
#
|
|
# ./run-candidate.sh gemma4-26b-a4b 5 4096
|
|
npredict="${3:-768}"
|
|
|
|
# Client-side bound, raised with the cap. At 6 tok/s a 4096-token budget needs
|
|
# 620 s of generation alone, so leaving this at 300 turns the diagnostic into a
|
|
# different failure rather than an answer.
|
|
timeout="${4:-300}"
|
|
here="$(dirname "$0")"
|
|
|
|
row="$(grep -v '^#' "$here/candidates.tsv" | awk -v l="$label" '$1==l')"
|
|
if [ -z "$row" ]; then
|
|
echo "unknown candidate '$label'. Known:" >&2
|
|
grep -v '^#' "$here/candidates.tsv" | awk 'NF{print " " $1}' >&2
|
|
exit 1
|
|
fi
|
|
|
|
repo="$(echo "$row" | cut -f2)"
|
|
hffile="$(echo "$row" | cut -f3)"
|
|
alias="$(echo "$row" | cut -f4)"
|
|
memreq="$(echo "$row" | cut -f5)"
|
|
memlim="$(echo "$row" | cut -f6)"
|
|
|
|
if [ -z "$hffile" ]; then
|
|
echo "candidate '$label' has no hf-file column — refusing to guess." >&2
|
|
echo "Resolve the exact .gguf filename from the HF API and add it." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# The script is mounted from a ConfigMap, so it has to be current before the
|
|
# Job starts. Shared by every candidate — they run the same measurement.
|
|
kubectl -n paperless-eval create configmap paperless-eval-script \
|
|
--from-file=eval.py="$here/eval.py" --dry-run=client -o yaml | kubectl apply -f -
|
|
|
|
if [ "$sample" -lt 50 ]; then
|
|
echo "SCREENING $label at $sample documents — a gate, not a measurement"
|
|
fi
|
|
echo "candidate=$label repo=$repo alias=$alias mem=$memreq/$memlim sample=$sample n_predict=$npredict timeout=$timeout"
|
|
sed -e "s#__LABEL__#$label#g" \
|
|
-e "s#__HF_REPO__#$repo#g" \
|
|
-e "s#__ALIAS__#$alias#g" \
|
|
-e "s#__MEM_REQ__#$memreq#g" \
|
|
-e "s#__MEM_LIM__#$memlim#g" \
|
|
-e "s#__SAMPLE__#$sample#g" \
|
|
-e "s#__HF_FILE__#$hffile#g" \
|
|
-e "s#__N_PREDICT__#$npredict#g" \
|
|
-e "s#__TIMEOUT__#$timeout#g" \
|
|
"$here/job-template.yaml" | kubectl apply -f -
|
|
|
|
echo
|
|
echo "follow: kubectl -n paperless-eval logs -f job/eval-$label -c eval"
|
|
echo "re-run: kubectl -n paperless-eval delete job eval-$label # a Job is immutable"
|