fix(ci): make model-picker actually tool-less; gate tests on PRs

Addresses the 14aa6415 review.

- pick-model.sh: the picker relied on `--allowedTools ""` to be tool-less, but an
  empirical check showed an empty allowlist does NOT block tools (the Write tool
  still ran) -- it is treated as "unset" and the default toolset stays live. Add
  `--permission-mode plan`, which IS verified to block Write while still producing
  the selection JSON. --allowedTools "" + the Read deny-list remain as additional
  layers. Corrected the comments in pick-model.sh and the picker settings.json
  that claimed the empty allowlist made it tool-less.
- .woodpecker/ci-scripts-test.yml: new pull_request-triggered pipeline (homelab
  only) that runs scripts/ci/test-model-selection.sh, so a PR weakening the
  injection gate fails checks immediately instead of only on the next scheduled
  e2e cron (closes the coverage-timing gap the review flagged).
- Pin a recorded real /v1/models response (scripts/ci/testdata/v1-models-sample.json)
  and assert fetch-models.sh maps it correctly (opus effort=true, haiku
  effort=false), documenting the verified response shape.

Deferred with rationale: hard-failing a pipeline when the degraded-schema warning
fires -- the warning already reaches the job log (curl/jq stderr is no longer
suppressed), and hard-failing on a benign API schema addition would break all
model selection; graceful degradation to name+date is the deliberate choice.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.woodpecker/ci-scripts-test.yml

@@ -0,0 +1,22 @@
+# Gate the CI helper-script unit tests on every homelab pull_request.
+#
+# scripts/ci/test-model-selection.sh is the regression test for the model-
+# selection injection gate (valid-model.sh) and the fetch-models.sh response
+# mapping. It is ALSO run on the daily/weekly crons via e2e.yml, but that would
+# only catch a weakening of the gate up to a week after it merged — by which
+# time pick-model.sh / pr-review.yml may have already run the weakened gate
+# against adversarial PR input. Running it here fails the offending PR's checks
+# before it can merge.
+#
+# Only in homelab (scripts/ci/ lives here); the file is not deployed to the
+# mirrored repos, so it does not run on their PRs. The test is offline (no token,
+# no network, no cluster) — the ci-claude image just provides sh + jq, and it is
+# already pulled by pr-review.yml on the same event, so the runner has it cached.
+when:
+  - event: pull_request
+
+steps:
+  - name: model-selection-tests
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    commands:
+      - sh scripts/ci/test-model-selection.sh

ci/agents/model-picker/settings.json

@@ -1,9 +1,9 @@
 {
   "//": "Read sandbox for the model-picker agent (scripts/ci/pick-model.sh). The",
-  "//2": "picker is invoked with --allowedTools \"\" so it holds no tools at all,",
-  "//3": "but its process still carries CLAUDE_CODE_OAUTH_TOKEN, so this deny-list",
-  "//4": "is defence in depth: if an empty --allowedTools were ever treated as",
-  "//5": "'unset' and the default toolset re-enabled, Read still cannot reach the",
+  "//2": "picker runs in --permission-mode plan (verified to block the Write tool;",
+  "//3": "an empty --allowedTools does NOT), so it cannot modify anything. Its",
+  "//4": "process still carries CLAUDE_CODE_OAUTH_TOKEN, so this Read deny-list is",
+  "//5": "defence in depth on top of plan mode, keeping any read tool away from the",
   "//6": "credential-bearing paths. Mirrors ci/agents/pr-review/settings.json.",
   "permissions": {
     "deny": [

scripts/ci/pick-model.sh

@@ -61,14 +61,17 @@ EOF
 )"
 
 # The tiny selection call runs on the CLI's own default model (no --model): we
-# will not name a family even to bootstrap the picker. No tools are granted
-# (--allowedTools ""), and a Read deny-list is loaded too (--settings) as belt-
-# and-suspenders in case an empty allowlist were ever treated as "unset". The
-# forge PATs a caller may hold (GITEA_TOKEN / PUSH_TOKEN / REVIEW_TOKEN) are
-# stripped from its env — the picker has no reason to carry a write-scoped
-# credential. The call is time-bounded so a hung CLI can't stall the pipeline (on
-# timeout `|| true` leaves RAW empty and the fallback fires), and stderr is left
-# intact so an auth failure or rate-limit is visible in the log.
+# will not name a family even to bootstrap the picker. It runs in
+# --permission-mode plan, which makes the agent unable to modify anything —
+# VERIFIED empirically that this blocks the Write tool, whereas `--allowedTools
+# ""` ALONE does NOT (an empty allowlist is treated as "unset" and the default
+# toolset stays live). So plan mode is the actual guarantee; `--allowedTools ""`
+# and the Read deny-list (--settings) are additional layers. The forge PATs a
+# caller may hold (GITEA_TOKEN / PUSH_TOKEN / REVIEW_TOKEN) are stripped from its
+# env — the picker has no reason to carry a write-scoped credential. The call is
+# time-bounded so a hung CLI can't stall the pipeline (on timeout `|| true`
+# leaves RAW empty and the fallback fires), and stderr is left intact so an auth
+# failure or rate-limit is visible in the log.
 TIMEOUT=""
 if command -v timeout >/dev/null 2>&1 && timeout 1 true >/dev/null 2>&1; then
   TIMEOUT="timeout 120"   # coreutils-style "timeout SECS CMD"; probe confirms it
@@ -76,7 +79,7 @@ fi
 SETTINGS="$HERE/../../ci/agents/model-picker/settings.json"
 [ -f "$SETTINGS" ] || SETTINGS=""
 RAW="$(printf '%s' "$PROMPT" | env -u GITEA_TOKEN -u PUSH_TOKEN -u REVIEW_TOKEN \
-        $TIMEOUT claude ${SETTINGS:+--settings $SETTINGS} --allowedTools "" -p || true)"
+        $TIMEOUT claude --permission-mode plan ${SETTINGS:+--settings $SETTINGS} --allowedTools "" -p || true)"
 
 CHOICE="$(printf '%s' "$RAW" \
   | grep -oE '"model"[[:space:]]*:[[:space:]]*"[^"]+"' \

scripts/ci/test-model-selection.sh

@@ -69,6 +69,21 @@ case "$warn" in
   *) printf 'FAIL  degraded-schema warning: got [%s]\n' "$warn"; fails=$((fails + 1)) ;;
 esac
 
+# Recorded REAL /v1/models response (the shape verified live, 2026-07-25) mapped
+# through the same seam — pins the actual field layout the selectors depend on,
+# so a change to what fetch-models.sh extracts is caught here.
+rows="$(FETCH_MODELS_BODY_FILE="$HERE/testdata/v1-models-sample.json" sh "$HERE/fetch-models.sh" --table 2>/dev/null || true)"
+opusline="$(printf '%s\n' "$rows" | grep 'claude-opus-5' || true)"
+haikuline="$(printf '%s\n' "$rows" | grep 'claude-haiku-4-5' || true)"
+o=FAIL; h=FAIL
+case "$opusline" in *"context=1000000"*"effort=true"*"thinking=true"*) o=ok ;; esac
+case "$haikuline" in *"context=200000"*"effort=false"*) h=ok ;; esac
+if [ "$o" = ok ] && [ "$h" = ok ]; then
+  printf 'ok    recorded real /v1/models fixture maps correctly (opus effort=true, haiku effort=false)\n'
+else
+  printf 'FAIL  recorded fixture: opus=[%s] haiku=[%s]\n' "$opusline" "$haikuline"; fails=$((fails + 1))
+fi
+
 if [ "$fails" -eq 0 ]; then
   echo "All model-selection tests passed."
 else

scripts/ci/testdata/v1-models-sample.json

@@ -0,0 +1,45 @@
+{
+  "data": [
+    {
+      "type": "model",
+      "id": "claude-opus-5",
+      "display_name": "Claude Opus 5",
+      "created_at": "2026-07-24T00:00:00Z",
+      "max_input_tokens": 1000000,
+      "max_tokens": 128000,
+      "capabilities": {
+        "batch": { "supported": true },
+        "citations": { "supported": true },
+        "code_execution": { "supported": true },
+        "context_management": { "supported": true },
+        "effort": { "supported": true, "low": { "supported": true }, "medium": { "supported": true }, "high": { "supported": true }, "xhigh": { "supported": true }, "max": { "supported": true } },
+        "image_input": { "supported": true },
+        "pdf_input": { "supported": true },
+        "structured_outputs": { "supported": true },
+        "thinking": { "supported": true }
+      }
+    },
+    {
+      "type": "model",
+      "id": "claude-haiku-4-5-20251001",
+      "display_name": "Claude Haiku 4.5",
+      "created_at": "2025-10-15T00:00:00Z",
+      "max_input_tokens": 200000,
+      "max_tokens": 64000,
+      "capabilities": {
+        "batch": { "supported": true },
+        "citations": { "supported": true },
+        "code_execution": { "supported": false },
+        "context_management": { "supported": true },
+        "effort": { "supported": false },
+        "image_input": { "supported": true },
+        "pdf_input": { "supported": true },
+        "structured_outputs": { "supported": true },
+        "thinking": { "supported": true }
+      }
+    }
+  ],
+  "has_more": false,
+  "first_id": "claude-opus-5",
+  "last_id": "claude-haiku-4-5-20251001"
+}