First live pick after #893 chose claude-sonnet-4-5 at 0.23 over
claude-sonnet-4-6: same-family models split the probability for a
mid-weight task, and "prefer newer" is a policy, not a judgment. New
scripts/ci/choose-model.sh applies it to the distribution: among
candidates within CHOOSE_MODEL_MARGIN (0.7) of the best probability the
most recently released wins; a clear best stays; NONE yields nothing.
The picker logs the top probabilities. Four policy tests added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The picker ran a full claude -p in plan mode per selection and parsed
JSON out of prose. With TYPESAFE_API_KEY set it now sends the task and
the catalog rows as state and asks one Choice over the model ids, each
with its catalog row as criterion plus NONE; the id is validated against
the pristine catalog as before. No key or a failed request falls through
to the claude picker unchanged.
Measured on the two-model fixture (Opus 5 / Haiku 4.5): issue-fix and
security-fix -> Opus 0.99, alert triage -> Opus 0.62, wiki audit ->
Haiku 0.94, update audit -> Haiku 0.46, distill -> Haiku 0.63; about a
second per pick. Dry-run test added to test-model-selection.sh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Addresses the 14aa6415 review.
- pick-model.sh: the picker relied on `--allowedTools ""` to be tool-less, but an
empirical check showed an empty allowlist does NOT block tools (the Write tool
still ran) -- it is treated as "unset" and the default toolset stays live. Add
`--permission-mode plan`, which IS verified to block Write while still producing
the selection JSON. --allowedTools "" + the Read deny-list remain as additional
layers. Corrected the comments in pick-model.sh and the picker settings.json
that claimed the empty allowlist made it tool-less.
- .woodpecker/ci-scripts-test.yml: new pull_request-triggered pipeline (homelab
only) that runs scripts/ci/test-model-selection.sh, so a PR weakening the
injection gate fails checks immediately instead of only on the next scheduled
e2e cron (closes the coverage-timing gap the review flagged).
- Pin a recorded real /v1/models response (scripts/ci/testdata/v1-models-sample.json)
and assert fetch-models.sh maps it correctly (opus effort=true, haiku
effort=false), documenting the verified response shape.
Deferred with rationale: hard-failing a pipeline when the degraded-schema warning
fires -- the warning already reaches the job log (curl/jq stderr is no longer
suppressed), and hard-failing on a benign API schema addition would break all
model selection; graceful degradation to name+date is the deliberate choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the a7890e0a review.
- pr-review.yml: strip REVIEW_TOKEN from the valid-model.sh call too (it runs a
script from $HOMELAB, branch-supplied for a homelab PR), and guard it with
`|| true` so a missing script can't kill the step -- restoring the stated "a
run never dies on selection" invariant. For a complex/security-sensitive
review, ignore the (injectable) triage pick and run on the CLI default
(strongest configured) model.
- valid-model.sh: reject a leading dash at the shape check (making the header
accurate) so an option-like value is refused even without a catalog.
- fetch-models.sh: --retry 2 --retry-connrefused so a transient blip doesn't
drop the run to the default; add a FETCH_MODELS_BODY_FILE test seam.
- pick-model.sh: pass --settings (new ci/agents/model-picker/settings.json,
Read deny-list) as belt-and-suspenders behind --allowedTools "".
- test-model-selection.sh: add fetch-models.sh mapping tests (well-formed body
-> non-null capability columns; degraded body -> loud warning) via the seam,
and wire the whole suite into .woodpecker/e2e.yml so it runs in CI.
- Condense the duplicated model-selection comment in the four fixed jobs to a
one-line pointer to pick-model.sh.
- PR description: state the cost / no-fixed-ceiling property change.
Deferred with rationale (in the PR thread): selected-model in the four crons'
posted artifacts (already logged per run), pagination past limit=1000 (API max;
has_more warns), and unit-testing pick-model.sh's JSON extraction (its output
always passes through the tested valid-model.sh gate, and the picker prompt is
developer-controlled, not attacker input).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the remaining pr-review items.
- Extract the untrusted-model validation out of the pipeline YAML into
scripts/ci/valid-model.sh (single safe-token + catalog-membership gate) and
reuse it from both pick-model.sh and pr-review.yml, so the security-critical
logic lives in one tested place. Add scripts/ci/test-model-selection.sh, an
offline fixture test (no network / claude / Woodpecker) that pins the
behaviour: valid ids pass; hallucinated, empty, space-, newline-, leading-
dash-, and metachar-injected values are rejected; empty catalog rejects all.
This is the regression test for the flag-injection fix.
- pr-review triage cost: hand the triage the COMPLETE changed-file list
(files.txt) plus a diff capped at 4000 lines (diff.triage.patch) instead of
the full up-to-20k review diff, so classifying a PR on the default model stays
cheap. stats.txt still carries the true totals; triage-prompt.md updated.
- Doc nits: note the new helpers are POSIX `sh` (not `bash` like the older
scripts/ci/*.sh), and that fetch-models.sh's json mode is a debug affordance.
Deferred with rationale (also in the PR thread):
- Model id in the four cron jobs' posted artifacts: selection (including the
fallback-to-default) is already logged per run in the CI output, the
operational surface for this; threading it through four different posted
formats is a separate observability follow-up.
- --settings sandbox on the picker: it is granted no tools (--allowedTools ""),
so there is nothing for a Read-deny sandbox to constrain -- the other agents'
sandboxes exist to scope tools they DO have.
- Pagination past limit=1000: that is the API's max page size; >1000 models is
not a real scenario, and the has_more warning turns even that into a visible
log line rather than silent truncation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review triage agent reads the attacker-controlled PR diff and writes
triage.json; its Write tool is not path-scoped, so a prompt-injected diff can
make it emit any `model` string. The catalog validation used
`grep -qxF -- "$MODEL"`, and grep -F treats a MULTI-LINE operand as
newline-separated patterns: a value like
"claude-sonnet-5\n--dangerously-skip-permissions" matches on its first line,
passes validation, and -- via the unquoted `${MODEL:+--model $MODEL}` expansion
-- word-splits into an extra argv entry, injecting a CLI flag into the review
agent's own `claude` invocation (defeating its --allowedTools allowlist and Read
sandbox, with CLAUDE_CODE_OAUTH_TOKEN in that process's env).
Removing map_model() (which returned one of three literal ids) is what opened
this; restore the invariant with a single-token guard before the expansion:
`case "$MODEL" in ""|*[!A-Za-z0-9._-]*) MODEL= ;; esac`. Verified it turns the
newline/space/hallucinated payloads into a plain `claude -p` (CLI default) while
still passing real ids. Mirrored the guard in pick-model.sh as defence in depth
(its grep -oE already keeps CHOICE single-line, so it was not exploitable).
Also from the review:
- pr-review.yml: correct the HOMELAB comment -- the fetch-models.sh call is
env -u REVIEW_TOKEN'd, so CLAUDE_CODE_OAUTH_TOKEN (not REVIEW_TOKEN) is what
branch-supplied shell there can see.
- pick-model.sh / triage-prompt.md: stop asking the picker to weigh "cost" the
catalog does not expose; reason from the visible capability signals (context /
max_output / effort|thinking|code_exec) as the scale proxy instead.
- fetch-models.sh: drop 2>/dev/null on curl so its -S error reaches the log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pick-model.sh: stop swallowing fetch-models.sh and claude stderr so the real
cause of a fallback (no token / HTTP 401 / parse error / auth) reaches the job
log instead of just "no catalog available"; time-bound the selection call with
a self-probing `timeout` (the wrapper is skipped unless a probe confirms the
image's timeout takes coreutils "SECS CMD" syntax, so it can never silently
route every run to the default); strip PUSH_TOKEN and REVIEW_TOKEN as well --
issue-fix binds its bot PAT a second time as PUSH_TOKEN, which survived the
original `env -u GITEA_TOKEN`; `grep -qxF --`.
- fetch-models.sh: warn loudly when every model returns a null context window,
so a /v1/models schema change surfaces as a message rather than silently
reducing both selectors to name+date guessing.
- pr-review.yml: `env -u REVIEW_TOKEN` on the fetch-models.sh call (it needs only
CLAUDE_CODE_OAUTH_TOKEN), matching the agent calls below it; `grep -qxF --`.
Deferred with rationale: hard-bounding the triage's diff input (kept as prompt
guidance -- a hard cap risks its security-sensitivity judgment) and an offline
fallback-path test harness (tracked as a follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fetch-models.sh: request ?limit=1000 so older/cheaper models are not dropped
by the API's 20-item default page; warn on has_more. Add a --table mode so
the catalog format lives in ONE place (pick-model.sh and pr-review.yml now
consume it instead of each re-deriving the jq table). Clean up the HTTP-000
double-print and surface jq parse errors instead of hiding them.
- pick-model.sh: run the picker with GITEA_TOKEN stripped from its env and
without acceptEdits (it is granted no tools); consume fetch-models.sh --table.
- pr-review.yml: print model `CLI default` instead of an empty backtick pair on
the fallback path; document that the fetch-models.sh call runs branch-supplied
shell (not a new exposure -- Woodpecker already runs the pipeline YAML from
the PR branch).
- triage-prompt.md: read stats + the changed-file list first and pull the full
diff only when needed, so the triage (now on a capable default model) does not
read a 20k-line diff just to classify a trivial PR.
Verified against the live catalog: all 11 currently-listed ids run under
`claude --model`, and the capability fields (context/effort/thinking/...) are
present in the OAuth /v1/models response -- so the reviewer's blocking concern
(a listed-but-CLI-unusable id) has no live trigger, and the fallback-to-default
path still covers it if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review, update-audit, security-scan, alert-monitor and issue-fix
pipelines each pinned a dated model id (--model claude-sonnet-5, plus
pr-review's haiku/sonnet/opus map_model allowlist). Those go stale every
time a new model ships and bake in assumptions about which model families
exist, so a newly released model could never be chosen without a code edit.
Replace all of it with selection driven only by what is available right now:
- scripts/ci/fetch-models.sh queries the catalog the CLI itself uses
(Anthropic GET /v1/models, via the existing CLAUDE_CODE_OAUTH_TOKEN and the
required anthropic-beta: oauth-2025-04-20 header) and emits each model with
its objective capability signals -- context, max_output, effort, thinking,
code_exec, release date -- with no id patterns and no family names.
- scripts/ci/pick-model.sh lets a triage agent choose the best-fit id from
that catalog for a one-line task description, weighing every listed model
on its merits. pr-review's existing per-PR triage agent now does the same,
reading the live catalog instead of emitting a fixed haiku/sonnet/opus tier.
No model id -- and no model family -- is baked into any pipeline. A model
released tomorrow, of any family, becomes eligible the day the API lists it.
The bootstrap selection call runs on the CLI default model, and if the catalog
is ever unreachable --model is simply omitted (CLI default) rather than falling
back to a named model, so a run never dies over model selection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude-botandClaude Fable 5.1