First live pick after #893 chose claude-sonnet-4-5 at 0.23 over
claude-sonnet-4-6: same-family models split the probability for a
mid-weight task, and "prefer newer" is a policy, not a judgment. New
scripts/ci/choose-model.sh applies it to the distribution: among
candidates within CHOOSE_MODEL_MARGIN (0.7) of the best probability the
most recently released wins; a clear best stays; NONE yields nothing.
The picker logs the top probabilities. Four policy tests added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The picker ran a full claude -p in plan mode per selection and parsed
JSON out of prose. With TYPESAFE_API_KEY set it now sends the task and
the catalog rows as state and asks one Choice over the model ids, each
with its catalog row as criterion plus NONE; the id is validated against
the pristine catalog as before. No key or a failed request falls through
to the claude picker unchanged.
Measured on the two-model fixture (Opus 5 / Haiku 4.5): issue-fix and
security-fix -> Opus 0.99, alert triage -> Opus 0.62, wiki audit ->
Haiku 0.94, update audit -> Haiku 0.46, distill -> Haiku 0.63; about a
second per pick. Dry-run test added to test-model-selection.sh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Addresses the 14aa6415 review.
- pick-model.sh: the picker relied on `--allowedTools ""` to be tool-less, but an
empirical check showed an empty allowlist does NOT block tools (the Write tool
still ran) -- it is treated as "unset" and the default toolset stays live. Add
`--permission-mode plan`, which IS verified to block Write while still producing
the selection JSON. --allowedTools "" + the Read deny-list remain as additional
layers. Corrected the comments in pick-model.sh and the picker settings.json
that claimed the empty allowlist made it tool-less.
- .woodpecker/ci-scripts-test.yml: new pull_request-triggered pipeline (homelab
only) that runs scripts/ci/test-model-selection.sh, so a PR weakening the
injection gate fails checks immediately instead of only on the next scheduled
e2e cron (closes the coverage-timing gap the review flagged).
- Pin a recorded real /v1/models response (scripts/ci/testdata/v1-models-sample.json)
and assert fetch-models.sh maps it correctly (opus effort=true, haiku
effort=false), documenting the verified response shape.
Deferred with rationale: hard-failing a pipeline when the degraded-schema warning
fires -- the warning already reaches the job log (curl/jq stderr is no longer
suppressed), and hard-failing on a benign API schema addition would break all
model selection; graceful degradation to name+date is the deliberate choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review triage agent runs with an unscoped Write tool and its cwd contains
./.pr-review (-> $D), so a prompt-injected diff could make it overwrite
./.pr-review/models.txt and then name a smuggled token as its `model`. Validation
re-read models.txt FROM DISK after the triage ran, so the catalog-membership
check was defeated by the agent's own writeback -- letting an attacker force a
weaker real model, or a syntactically-valid fake id that makes the review's
`claude --model <fake>` fail and silently suppress the review entirely.
Validate against $MODELS_TABLE -- the catalog captured in a shell variable BEFORE
the triage runs -- instead of re-reading the file the triage can overwrite. This
mirrors pick-model.sh, which already validates against its in-memory table before
its (tool-less) picker runs. The triage still reads models.txt for its own pick;
tampering only misleads that pick, which is then rejected against the pristine
snapshot -> CLI default (so the review still runs, it can't be suppressed).
Verified: a token present only in a tampered on-disk file but absent from the
pre-triage snapshot is now rejected (it was accepted before). Added a regression
test to scripts/ci/test-model-selection.sh, and fixed the header comment to note
the complex-review override as another path to the CLI default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the a7890e0a review.
- pr-review.yml: strip REVIEW_TOKEN from the valid-model.sh call too (it runs a
script from $HOMELAB, branch-supplied for a homelab PR), and guard it with
`|| true` so a missing script can't kill the step -- restoring the stated "a
run never dies on selection" invariant. For a complex/security-sensitive
review, ignore the (injectable) triage pick and run on the CLI default
(strongest configured) model.
- valid-model.sh: reject a leading dash at the shape check (making the header
accurate) so an option-like value is refused even without a catalog.
- fetch-models.sh: --retry 2 --retry-connrefused so a transient blip doesn't
drop the run to the default; add a FETCH_MODELS_BODY_FILE test seam.
- pick-model.sh: pass --settings (new ci/agents/model-picker/settings.json,
Read deny-list) as belt-and-suspenders behind --allowedTools "".
- test-model-selection.sh: add fetch-models.sh mapping tests (well-formed body
-> non-null capability columns; degraded body -> loud warning) via the seam,
and wire the whole suite into .woodpecker/e2e.yml so it runs in CI.
- Condense the duplicated model-selection comment in the four fixed jobs to a
one-line pointer to pick-model.sh.
- PR description: state the cost / no-fixed-ceiling property change.
Deferred with rationale (in the PR thread): selected-model in the four crons'
posted artifacts (already logged per run), pagination past limit=1000 (API max;
has_more warns), and unit-testing pick-model.sh's JSON extraction (its output
always passes through the tested valid-model.sh gate, and the picker prompt is
developer-controlled, not attacker input).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the remaining pr-review items.
- Extract the untrusted-model validation out of the pipeline YAML into
scripts/ci/valid-model.sh (single safe-token + catalog-membership gate) and
reuse it from both pick-model.sh and pr-review.yml, so the security-critical
logic lives in one tested place. Add scripts/ci/test-model-selection.sh, an
offline fixture test (no network / claude / Woodpecker) that pins the
behaviour: valid ids pass; hallucinated, empty, space-, newline-, leading-
dash-, and metachar-injected values are rejected; empty catalog rejects all.
This is the regression test for the flag-injection fix.
- pr-review triage cost: hand the triage the COMPLETE changed-file list
(files.txt) plus a diff capped at 4000 lines (diff.triage.patch) instead of
the full up-to-20k review diff, so classifying a PR on the default model stays
cheap. stats.txt still carries the true totals; triage-prompt.md updated.
- Doc nits: note the new helpers are POSIX `sh` (not `bash` like the older
scripts/ci/*.sh), and that fetch-models.sh's json mode is a debug affordance.
Deferred with rationale (also in the PR thread):
- Model id in the four cron jobs' posted artifacts: selection (including the
fallback-to-default) is already logged per run in the CI output, the
operational surface for this; threading it through four different posted
formats is a separate observability follow-up.
- --settings sandbox on the picker: it is granted no tools (--allowedTools ""),
so there is nothing for a Read-deny sandbox to constrain -- the other agents'
sandboxes exist to scope tools they DO have.
- Pagination past limit=1000: that is the API's max page size; >1000 models is
not a real scenario, and the has_more warning turns even that into a visible
log line rather than silent truncation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude-botandClaude Fable 5.1