eval-script-fixes
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  claude-botandClaude Opus 4.8
|
1992fabb12 |
fix(ci): make model-picker actually tool-less; gate tests on PRs
Addresses the
|
||
|
claude-botandClaude Opus 4.8
|
14aa641560 |
fix(ci): validate pr-review model pick against pre-triage catalog snapshot
ci/woodpecker/pr/pr-review Pipeline was successful
The pr-review triage agent runs with an unscoped Write tool and its cwd contains ./.pr-review (-> $D), so a prompt-injected diff could make it overwrite ./.pr-review/models.txt and then name a smuggled token as its `model`. Validation re-read models.txt FROM DISK after the triage ran, so the catalog-membership check was defeated by the agent's own writeback -- letting an attacker force a weaker real model, or a syntactically-valid fake id that makes the review's `claude --model <fake>` fail and silently suppress the review entirely. Validate against $MODELS_TABLE -- the catalog captured in a shell variable BEFORE the triage runs -- instead of re-reading the file the triage can overwrite. This mirrors pick-model.sh, which already validates against its in-memory table before its (tool-less) picker runs. The triage still reads models.txt for its own pick; tampering only misleads that pick, which is then rejected against the pristine snapshot -> CLI default (so the review still runs, it can't be suppressed). Verified: a token present only in a tampered on-disk file but absent from the pre-triage snapshot is now rejected (it was accepted before). Added a regression test to scripts/ci/test-model-selection.sh, and fixed the header comment to note the complex-review override as another path to the CLI default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
44a429685c |
fix(ci): round-3 pr-review hardening — REVIEW_TOKEN, no-die selection, tests
ci/woodpecker/pr/pr-review Pipeline was successful
Addresses the
|
||
|
claude-botandClaude Opus 4.8
|
a7890e0aa2 |
test(ci): extract + unit-test model validation; cut triage cost; doc nits
ci/woodpecker/pr/pr-review Pipeline was successful
Addresses the remaining pr-review items. - Extract the untrusted-model validation out of the pipeline YAML into scripts/ci/valid-model.sh (single safe-token + catalog-membership gate) and reuse it from both pick-model.sh and pr-review.yml, so the security-critical logic lives in one tested place. Add scripts/ci/test-model-selection.sh, an offline fixture test (no network / claude / Woodpecker) that pins the behaviour: valid ids pass; hallucinated, empty, space-, newline-, leading- dash-, and metachar-injected values are rejected; empty catalog rejects all. This is the regression test for the flag-injection fix. - pr-review triage cost: hand the triage the COMPLETE changed-file list (files.txt) plus a diff capped at 4000 lines (diff.triage.patch) instead of the full up-to-20k review diff, so classifying a PR on the default model stays cheap. stats.txt still carries the true totals; triage-prompt.md updated. - Doc nits: note the new helpers are POSIX `sh` (not `bash` like the older scripts/ci/*.sh), and that fetch-models.sh's json mode is a debug affordance. Deferred with rationale (also in the PR thread): - Model id in the four cron jobs' posted artifacts: selection (including the fallback-to-default) is already logged per run in the CI output, the operational surface for this; threading it through four different posted formats is a separate observability follow-up. - --settings sandbox on the picker: it is granted no tools (--allowedTools ""), so there is nothing for a Read-deny sandbox to constrain -- the other agents' sandboxes exist to scope tools they DO have. - Pagination past limit=1000: that is the API's max page size; >1000 models is not a real scenario, and the has_more warning turns even that into a visible log line rather than silent truncation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
f333dfd606 |
fix(ci): block flag injection via agent-supplied model id (pr-review)
ci/woodpecker/pr/pr-review Pipeline is running
The pr-review triage agent reads the attacker-controlled PR diff and writes
triage.json; its Write tool is not path-scoped, so a prompt-injected diff can
make it emit any `model` string. The catalog validation used
`grep -qxF -- "$MODEL"`, and grep -F treats a MULTI-LINE operand as
newline-separated patterns: a value like
"claude-sonnet-5\n--dangerously-skip-permissions" matches on its first line,
passes validation, and -- via the unquoted `${MODEL:+--model $MODEL}` expansion
-- word-splits into an extra argv entry, injecting a CLI flag into the review
agent's own `claude` invocation (defeating its --allowedTools allowlist and Read
sandbox, with CLAUDE_CODE_OAUTH_TOKEN in that process's env).
Removing map_model() (which returned one of three literal ids) is what opened
this; restore the invariant with a single-token guard before the expansion:
`case "$MODEL" in ""|*[!A-Za-z0-9._-]*) MODEL= ;; esac`. Verified it turns the
newline/space/hallucinated payloads into a plain `claude -p` (CLI default) while
still passing real ids. Mirrored the guard in pick-model.sh as defence in depth
(its grep -oE already keeps CHOICE single-line, so it was not exploitable).
Also from the review:
- pr-review.yml: correct the HOMELAB comment -- the fetch-models.sh call is
env -u REVIEW_TOKEN'd, so CLAUDE_CODE_OAUTH_TOKEN (not REVIEW_TOKEN) is what
branch-supplied shell there can see.
- pick-model.sh / triage-prompt.md: stop asking the picker to weigh "cost" the
catalog does not expose; reason from the visible capability signals (context /
max_output / effort|thinking|code_exec) as the scale proxy instead.
- fetch-models.sh: drop 2>/dev/null on curl so its -S error reaches the log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
claude-botandClaude Opus 4.8
|
9d36087857 |
fix(ci): second-round pr-review hardening for model selection
ci/woodpecker/pr/pr-review Pipeline was successful
- pick-model.sh: stop swallowing fetch-models.sh and claude stderr so the real cause of a fallback (no token / HTTP 401 / parse error / auth) reaches the job log instead of just "no catalog available"; time-bound the selection call with a self-probing `timeout` (the wrapper is skipped unless a probe confirms the image's timeout takes coreutils "SECS CMD" syntax, so it can never silently route every run to the default); strip PUSH_TOKEN and REVIEW_TOKEN as well -- issue-fix binds its bot PAT a second time as PUSH_TOKEN, which survived the original `env -u GITEA_TOKEN`; `grep -qxF --`. - fetch-models.sh: warn loudly when every model returns a null context window, so a /v1/models schema change surfaces as a message rather than silently reducing both selectors to name+date guessing. - pr-review.yml: `env -u REVIEW_TOKEN` on the fetch-models.sh call (it needs only CLAUDE_CODE_OAUTH_TOKEN), matching the agent calls below it; `grep -qxF --`. Deferred with rationale: hard-bounding the triage's diff input (kept as prompt guidance -- a hard cap risks its security-sensitivity judgment) and an offline fallback-path test harness (tracked as a follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
b329c432c5 |
fix(ci): address pr-review feedback on dynamic model selection
ci/woodpecker/pr/pr-review Pipeline was successful
- fetch-models.sh: request ?limit=1000 so older/cheaper models are not dropped by the API's 20-item default page; warn on has_more. Add a --table mode so the catalog format lives in ONE place (pick-model.sh and pr-review.yml now consume it instead of each re-deriving the jq table). Clean up the HTTP-000 double-print and surface jq parse errors instead of hiding them. - pick-model.sh: run the picker with GITEA_TOKEN stripped from its env and without acceptEdits (it is granted no tools); consume fetch-models.sh --table. - pr-review.yml: print model `CLI default` instead of an empty backtick pair on the fallback path; document that the fetch-models.sh call runs branch-supplied shell (not a new exposure -- Woodpecker already runs the pipeline YAML from the PR branch). - triage-prompt.md: read stats + the changed-file list first and pull the full diff only when needed, so the triage (now on a capable default model) does not read a 20k-line diff just to classify a trivial PR. Verified against the live catalog: all 11 currently-listed ids run under `claude --model`, and the capability fields (context/effort/thinking/...) are present in the OAuth /v1/models response -- so the reviewer's blocking concern (a listed-but-CLI-unusable id) has no live trigger, and the fallback-to-default path still covers it if that ever changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Opus 4.8
|
ffe3be4b54 |
feat(ci): select agent model from the live catalog, not hardcoded ids
ci/woodpecker/pr/pr-review Pipeline was successful
The pr-review, update-audit, security-scan, alert-monitor and issue-fix pipelines each pinned a dated model id (--model claude-sonnet-5, plus pr-review's haiku/sonnet/opus map_model allowlist). Those go stale every time a new model ships and bake in assumptions about which model families exist, so a newly released model could never be chosen without a code edit. Replace all of it with selection driven only by what is available right now: - scripts/ci/fetch-models.sh queries the catalog the CLI itself uses (Anthropic GET /v1/models, via the existing CLAUDE_CODE_OAUTH_TOKEN and the required anthropic-beta: oauth-2025-04-20 header) and emits each model with its objective capability signals -- context, max_output, effort, thinking, code_exec, release date -- with no id patterns and no family names. - scripts/ci/pick-model.sh lets a triage agent choose the best-fit id from that catalog for a one-line task description, weighing every listed model on its merits. pr-review's existing per-PR triage agent now does the same, reading the live catalog instead of emitting a fixed haiku/sonnet/opus tier. No model id -- and no model family -- is baked into any pipeline. A model released tomorrow, of any family, becomes eligible the day the API lists it. The bootstrap selection call runs on the CLI default model, and if the catalog is ever unreachable --model is simply omitted (CLI default) rather than falling back to a named model, so a run never dies over model selection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
claude-botandClaude Fable 5
|
90aeaf1ef8 |
feat(ci-agents): give triage/fix agents verified read-only live diagnostics
Both wrong assessments (#249 openbao 'sealed', #250 three untested hypotheses while one 'kubectl logs' showed the 401) share one root cause: the agents diagnose blind, so they narrate plausible fiction. Labeling unverified claims (previous commit) treats the symptom; this gives them the means to verify: - kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount ci-diagnostics with a read-only ClusterRole (workloads, events, pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets) plus a namespaced services/proxy Role for the Prometheus and Alertmanager HTTP APIs. - scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from the admin kubeconfig (no new long-lived secret), builds .ci-diag/kubeconfig, copies the static kubectl binary for the jq-only ci-claude image, smoke-tests with the restricted identity, and degrades to snapshot-only mode when the SA isn't reconciled. - alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool, Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other binary; RBAC forbids mutation and secret reads. - prompts: verification is now mandatory, not aspirational — 'a hypothesis you could have settled with one read-only command but didn't is a defect in your output'; fix agent must test the issue's claims before building on them; minimal-quote rule against dumping logs into public issues. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei |
||
 k.rennerandClaude Opus 4.8
|
0ce60164e1 |
ci: decommission migration-only rebuild-k8s; point access tooling at Hetzner
rebuild-k8s.yml was the one-shot PVE->K8s migration/rebuild pipeline; it is no longer needed and still pointed at the retired Garage S3 on the now-off nas-1. Remove it (DR restore is via Velero + flux bootstrap). The live cluster-access path is kept and migrated off Garage onto Hetzner Object Storage (renner-uno-k8s-secrets, fsn1.your-objectstorage.com, region fsn1, hetzner_s3_* keypair): - scripts/ci/fetch-kubeconfig.sh: Garage -> Hetzner; env vars renamed GARAGE_SECRETS_* -> S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY. - .woodpecker/e2e.yml, e2e-test.yml: pass hetzner_s3_access_key/secret_key (the live cron e2e pipeline needs this — it was broken against dead nas-1). - scripts/import-cluster-access.sh: Garage -> Hetzner endpoint/bucket/region. NOTE: the workstation aws-cli "secrets" profile must now hold the Hetzner Object Storage keypair. Also: delete the orphaned test-garage-smoke play + pipeline, and fix the two dangling rebuild-k8s.yml references (zfs-tuning play comment; update-audit prompt Talos-pin file list). Assumes cluster-access objects (kubeconfig.sops/talosconfig.sops) were migrated into renner-uno-k8s-secrets; with rebuild-k8s gone, nothing auto-refreshes them, which is fine for a stable cluster. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
k.rennerandClaude Opus 4.7
|
b4975fabbb |
ci: store cluster access in Garage instead of git
The rebuild-k8s pipeline previously SOPS-encrypted talosconfig and kubeconfig and pushed them back to master. The git push raced with concurrent commits and forced workstation users to `git pull --rebase` after every rebuild. Replace the transport with Garage's k8s-secrets bucket (s3://k8s-secrets/cluster-access/), reusing the existing secrets-key that the install-garage-synology playbook already provisions. SOPS encryption is unchanged — only the storage layer moves. - New scripts/ci/fetch-kubeconfig.sh helper: 9 CI steps that previously inlined `sops -d kubernetes/access/kubeconfig.sops` now call this. - scripts/import-cluster-access.sh rewritten to pull both files from Garage; no more git fetch/checkout. - export-access step uploads via aws s3 cp; gitea_token + git config removed. - All consuming Woodpecker steps gain garage_secrets_key_id / garage_secrets_secret_key from_secret env vars, and bump to ci-k8s:2026-05-01 (the tag with aws-cli baked in). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |