Flux source URL, flux-git rotator verify repo, review-bot REPOS, pipeline API,
clone and push URLs, the heating-can source label and the review-bot and
kuma-sync Go module paths. Merge right after the transfer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All wiki interaction now goes through the wiki service API; CI agents that
distill and consolidate by writing the notes repo directly are retired.
Removes .woodpecker/wiki-maintenance.yml and ci/agents/wiki-maintenance/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run.sh asks a Mistral agent (mistral-large-4-0) through the Conversations API. The agent reads the repo and the PR through the read-only Gitea connector, so the review step no longer holds a Claude token or a local checkout. The Claude CLI invocation, the model picker and the context clones stay in run.sh as comments. The answer is cut at the first heading and empty or None-only sections are dropped before the review event is derived. A tool call that is not read-only fails the run and nothing is posted. A failed request now fails the pipeline.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The event is derived by the pipeline from the report's severity sections:
Blocking -> REQUEST_CHANGES; none of Blocking/Suggestions/Unverified on a
non-empty diff -> APPROVE; otherwise COMMENT.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop the complexity triage (it downgraded most PRs to a small model) and pick
the model via scripts/ci/pick-model.sh from the live /v1/models catalog,
steered to the strongest and newest. Falls back to the CLI default.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Move the review logic out of .woodpecker/pr-review.yml into
ci/agents/pr-review/run.sh; the per-repo file becomes a logic-free stub that
clones homelab@master and runs it, so the four copies cannot drift.
Prompt: unverified suspicions must be checked (checkout + docs) and are never
blocking. Review agent gets WebFetch/WebSearch.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Run 10608 aborted before its first edit: the agent calls
~/Notes/bin/wiki-snapshot.sh and git -C ~/Notes rev-parse/status, the
rules carried only /root/Notes. Both spellings are allowed for the vault
scripts and the read-only git subcommands on the vault checkout, config is
denied there too, and the consolidate step derives its --allowedTools
from the settings file instead of a second copy. Verified headless:
rev-parse, status and wiki-delete.sh allowed, config denied.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review on #927: the git -C /root/source/ prefix let config and push
through; rm /root/Notes/wiki/ matched path traversal. Allowed git forms
are now log/show/diff/status/rev-parse/ls-files per repo (both the ~ and
/root spelling), config is denied for those forms too, rm is denied and
notes are deleted through the vault's wiki-delete.sh, which validates the
slug. Verified headless: log allowed, config denied, push not allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pass that verifies notes against sources, repos and the live cluster
and edits them still ran on the workstation timer, with no TypeSafe key,
no cost record, and no run on days the machine was off. It joins the
nightly pipeline after the audit: full clones of homelab and homeassistant
under ~/source, read-only kubectl via the ci-diagnostics token, its own
permission scope, JSON output so the report and total_cost_usd land in
the step log. push-notes also commits the rotation state and pushes the
agent's own snapshot commits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The audit agent had no executable path in CI (Bash limited to the two
retrieval tools), wrote itself scripts it could not run and reported one
of seven checks (runs 10262, 10362). The checks are mechanical, so
notes/bin/wiki_audit.py does them without a model. push-notes adds only
wiki/, raw/ and output/, so scratch files in the checkout root are no
longer committed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replaces the workstation's wiki-maintenance.timer. The pipeline clones
k.renner/notes, runs wiki-distill over raw/ and wiki-audit over wiki/
as two separate `claude --agent` runs (no background tasks in print
mode, which killed the workstation job at the 600 s ceiling), then
commits and pushes as Claude Bot; the push webhook refreshes wiki-index.
The agent definitions now live in the vault (.claude/agents/), so CI
and workstation share them. The agents and their retrieval scripts
address the vault as ~/Notes; each agent step links $HOME/Notes to the
checkout. ci-claude gains python3 and py3-yaml for wiki_ask.py.
Secrets on the repo: claude_code_oauth_token (existing), typesafe_api_key,
wiki_bot_token (claude-bot, write:repository), wiki_index_basic_auth.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#476 merged first, which removed edge_open3e_pkg, edge_e3oncan_repo and
edge_e3oncan_version in favour of the image. The audit entry added here still
told the agent to read those, so it would have gone looking for variables
that no longer exist and reported nothing -- the same silent non-auditing the
entry was written to prevent.
Repointed at where the pins actually live now: OPEN3E_REF and E3ONCAN_REF as
ARGs in ci/images/heating-can/Containerfile. The awkward specifics carry over
unchanged, since they are properties of the upstreams rather than of how we
package them: open3e publishes no tags so the comparison must be against
commits, and E3onCAN's pin is an untagged commit later than its newest tag.
Added a note the previous version could not have made: heating_can_image is
:latest, so the node follows whatever CI last built. The Containerfile ARGs
are what actually fix the versions, and bumping one is what requires a
rebuild -- otherwise a reader could reasonably conclude the tag is the pin.
The pin commit itself is now a no-op against master and survives only as the
audit registration; its conflict on heating-can/defaults resolved in favour of
the image.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
open3e was installed from an unpinned git URL, so the heating bridge
silently tracked upstream main: whatever HEAD happened to be on the day a
converge ran was what ended up driving the heat pump. Now pinned to
6553c104, which was simultaneously upstream HEAD and the copy already
installed on edge-1, so the pin froze the running state and changed nothing.
A commit rather than a tag because open3e publishes none -- its tags API
returns an empty array. The package self-reports 0.7.5 at that commit, which
is useful context but is not a tag.
Honest about what this does NOT fix: pip still re-clones and rebuilds a VCS
requirement even when the SHA is pinned, because it cannot compare an
installed VCS dist against a requested URL without fetching it first.
Verified directly on the host:
pip install --dry-run "git+...@6553c104"
-> Cloning ... Resolved ... to commit 6553c104
Installing build dependencies: started
So the task still reports changed and still restarts open3e on every
converge. Pinning buys deliberate versioning, not idempotence.
Pinning without an update path only converts churn into silent rot, so both
services are now registered with the weekly update-audit -- which had been
auditing neither, despite E3onCAN having been pinned since July. The audit
prompt warns that anything not listed is silently not audited, and these two
were exactly that.
The new scope entry records the awkward specifics so the agent does not
report normal states as problems: open3e has no tags at all, so it must
compare commits against the default branch; E3onCAN does tag, but sparsely,
and its pin sits between the newest tag and HEAD.
Also corrected while there: edge_e3oncan_version was commented "# V0.5.2",
which does not exist upstream. The repo's only tag is v0.5.1 (6f43b3fb), and
461dcbdf is a later untagged commit from 2026-06-21, "Fixed: MQTT client ID
collision caused stale retained values; added publish error logging".
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I proposed widening ImageUpdateAutomation's `update.path` to `./` and adding
a woodpecker-cli ImagePolicy so automation would own the lint pin the way it
owns the server and agent. Checking that before building it killed the plan,
and corrected two things I had asserted without evidence.
WRONG #1 — my reason for hesitating. I claimed the 7 ansible-vault encrypted
`.yml` files would break the walk and take all 33 automated images down with
them. That was a guess about kyaml I never verified. Objects that are not
Kubernetes manifests are SKIPPED, not fatal.
WRONG #2 — and this is the one that matters: the plan cannot work at all.
Image-automation only patches YAML that is a Kubernetes manifest. Maintainer,
fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
YAML manifest must have `apiVersion` and `kind` fields." The docs agree —
markers go "in any Kubernetes custom resource stored in Git". A Woodpecker
pipeline file is plain YAML with no apiVersion/kind, so a marker in it is
silently ignored.
That is a property of the FILE, not of `update.path`. Widening the path buys
exactly nothing and would drag 197 unrelated ansible/esphome YAMLs into the
walk. So no policy, no second automation, no path change — all three would
have been motion that looked like a fix and changed nothing. The failure mode
would have been the quiet kind this repo keeps getting bitten by: a marker
sitting in the file looking maintained, never once updated.
What actually closes the gap, given the pin must stay manual:
* README gains a "Not enrolled (structurally impossible)" section holding
the quote and the explicit "do not try widening the path" warning, so the
next person does not re-derive this.
* update-audit item 5 now covers `.woodpecker/*.yml` step images and is told
to flag woodpecker-cli whenever it trails the DEPLOYED SERVER, not merely
the newest CLI release — the server auto-bumps and the CLI does not, so
that skew is the thing worth reporting. Also told to compare only the
vX.Y.Z part (the tag carries `-alpine`; the plain tag is scratch with no
shell and would break the step) and to ignore the `next-*`/`pull_*` dev
tags, which are the newest by date on that repo and are not releases.
* the step comment states the real constraint instead of my earlier wrong
one, which told the reader that widening the path was the fix.
Verified: v3.16.0-alpine is the newest 3.x alpine release tag, and matches the
deployed server v3.16.0. Config still parses; step image unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses the 14aa6415 review.
- pick-model.sh: the picker relied on `--allowedTools ""` to be tool-less, but an
empirical check showed an empty allowlist does NOT block tools (the Write tool
still ran) -- it is treated as "unset" and the default toolset stays live. Add
`--permission-mode plan`, which IS verified to block Write while still producing
the selection JSON. --allowedTools "" + the Read deny-list remain as additional
layers. Corrected the comments in pick-model.sh and the picker settings.json
that claimed the empty allowlist made it tool-less.
- .woodpecker/ci-scripts-test.yml: new pull_request-triggered pipeline (homelab
only) that runs scripts/ci/test-model-selection.sh, so a PR weakening the
injection gate fails checks immediately instead of only on the next scheduled
e2e cron (closes the coverage-timing gap the review flagged).
- Pin a recorded real /v1/models response (scripts/ci/testdata/v1-models-sample.json)
and assert fetch-models.sh maps it correctly (opus effort=true, haiku
effort=false), documenting the verified response shape.
Deferred with rationale: hard-failing a pipeline when the degraded-schema warning
fires -- the warning already reaches the job log (curl/jq stderr is no longer
suppressed), and hard-failing on a benign API schema addition would break all
model selection; graceful degradation to name+date is the deliberate choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the a7890e0a review.
- pr-review.yml: strip REVIEW_TOKEN from the valid-model.sh call too (it runs a
script from $HOMELAB, branch-supplied for a homelab PR), and guard it with
`|| true` so a missing script can't kill the step -- restoring the stated "a
run never dies on selection" invariant. For a complex/security-sensitive
review, ignore the (injectable) triage pick and run on the CLI default
(strongest configured) model.
- valid-model.sh: reject a leading dash at the shape check (making the header
accurate) so an option-like value is refused even without a catalog.
- fetch-models.sh: --retry 2 --retry-connrefused so a transient blip doesn't
drop the run to the default; add a FETCH_MODELS_BODY_FILE test seam.
- pick-model.sh: pass --settings (new ci/agents/model-picker/settings.json,
Read deny-list) as belt-and-suspenders behind --allowedTools "".
- test-model-selection.sh: add fetch-models.sh mapping tests (well-formed body
-> non-null capability columns; degraded body -> loud warning) via the seam,
and wire the whole suite into .woodpecker/e2e.yml so it runs in CI.
- Condense the duplicated model-selection comment in the four fixed jobs to a
one-line pointer to pick-model.sh.
- PR description: state the cost / no-fixed-ceiling property change.
Deferred with rationale (in the PR thread): selected-model in the four crons'
posted artifacts (already logged per run), pagination past limit=1000 (API max;
has_more warns), and unit-testing pick-model.sh's JSON extraction (its output
always passes through the tested valid-model.sh gate, and the picker prompt is
developer-controlled, not attacker input).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the remaining pr-review items.
- Extract the untrusted-model validation out of the pipeline YAML into
scripts/ci/valid-model.sh (single safe-token + catalog-membership gate) and
reuse it from both pick-model.sh and pr-review.yml, so the security-critical
logic lives in one tested place. Add scripts/ci/test-model-selection.sh, an
offline fixture test (no network / claude / Woodpecker) that pins the
behaviour: valid ids pass; hallucinated, empty, space-, newline-, leading-
dash-, and metachar-injected values are rejected; empty catalog rejects all.
This is the regression test for the flag-injection fix.
- pr-review triage cost: hand the triage the COMPLETE changed-file list
(files.txt) plus a diff capped at 4000 lines (diff.triage.patch) instead of
the full up-to-20k review diff, so classifying a PR on the default model stays
cheap. stats.txt still carries the true totals; triage-prompt.md updated.
- Doc nits: note the new helpers are POSIX `sh` (not `bash` like the older
scripts/ci/*.sh), and that fetch-models.sh's json mode is a debug affordance.
Deferred with rationale (also in the PR thread):
- Model id in the four cron jobs' posted artifacts: selection (including the
fallback-to-default) is already logged per run in the CI output, the
operational surface for this; threading it through four different posted
formats is a separate observability follow-up.
- --settings sandbox on the picker: it is granted no tools (--allowedTools ""),
so there is nothing for a Read-deny sandbox to constrain -- the other agents'
sandboxes exist to scope tools they DO have.
- Pagination past limit=1000: that is the API's max page size; >1000 models is
not a real scenario, and the has_more warning turns even that into a visible
log line rather than silent truncation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review triage agent reads the attacker-controlled PR diff and writes
triage.json; its Write tool is not path-scoped, so a prompt-injected diff can
make it emit any `model` string. The catalog validation used
`grep -qxF -- "$MODEL"`, and grep -F treats a MULTI-LINE operand as
newline-separated patterns: a value like
"claude-sonnet-5\n--dangerously-skip-permissions" matches on its first line,
passes validation, and -- via the unquoted `${MODEL:+--model $MODEL}` expansion
-- word-splits into an extra argv entry, injecting a CLI flag into the review
agent's own `claude` invocation (defeating its --allowedTools allowlist and Read
sandbox, with CLAUDE_CODE_OAUTH_TOKEN in that process's env).
Removing map_model() (which returned one of three literal ids) is what opened
this; restore the invariant with a single-token guard before the expansion:
`case "$MODEL" in ""|*[!A-Za-z0-9._-]*) MODEL= ;; esac`. Verified it turns the
newline/space/hallucinated payloads into a plain `claude -p` (CLI default) while
still passing real ids. Mirrored the guard in pick-model.sh as defence in depth
(its grep -oE already keeps CHOICE single-line, so it was not exploitable).
Also from the review:
- pr-review.yml: correct the HOMELAB comment -- the fetch-models.sh call is
env -u REVIEW_TOKEN'd, so CLAUDE_CODE_OAUTH_TOKEN (not REVIEW_TOKEN) is what
branch-supplied shell there can see.
- pick-model.sh / triage-prompt.md: stop asking the picker to weigh "cost" the
catalog does not expose; reason from the visible capability signals (context /
max_output / effort|thinking|code_exec) as the scale proxy instead.
- fetch-models.sh: drop 2>/dev/null on curl so its -S error reaches the log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fetch-models.sh: request ?limit=1000 so older/cheaper models are not dropped
by the API's 20-item default page; warn on has_more. Add a --table mode so
the catalog format lives in ONE place (pick-model.sh and pr-review.yml now
consume it instead of each re-deriving the jq table). Clean up the HTTP-000
double-print and surface jq parse errors instead of hiding them.
- pick-model.sh: run the picker with GITEA_TOKEN stripped from its env and
without acceptEdits (it is granted no tools); consume fetch-models.sh --table.
- pr-review.yml: print model `CLI default` instead of an empty backtick pair on
the fallback path; document that the fetch-models.sh call runs branch-supplied
shell (not a new exposure -- Woodpecker already runs the pipeline YAML from
the PR branch).
- triage-prompt.md: read stats + the changed-file list first and pull the full
diff only when needed, so the triage (now on a capable default model) does not
read a 20k-line diff just to classify a trivial PR.
Verified against the live catalog: all 11 currently-listed ids run under
`claude --model`, and the capability fields (context/effort/thinking/...) are
present in the OAuth /v1/models response -- so the reviewer's blocking concern
(a listed-but-CLI-unusable id) has no live trigger, and the fallback-to-default
path still covers it if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review, update-audit, security-scan, alert-monitor and issue-fix
pipelines each pinned a dated model id (--model claude-sonnet-5, plus
pr-review's haiku/sonnet/opus map_model allowlist). Those go stale every
time a new model ships and bake in assumptions about which model families
exist, so a newly released model could never be chosen without a code edit.
Replace all of it with selection driven only by what is available right now:
- scripts/ci/fetch-models.sh queries the catalog the CLI itself uses
(Anthropic GET /v1/models, via the existing CLAUDE_CODE_OAUTH_TOKEN and the
required anthropic-beta: oauth-2025-04-20 header) and emits each model with
its objective capability signals -- context, max_output, effort, thinking,
code_exec, release date -- with no id patterns and no family names.
- scripts/ci/pick-model.sh lets a triage agent choose the best-fit id from
that catalog for a one-line task description, weighing every listed model
on its merits. pr-review's existing per-PR triage agent now does the same,
reading the live catalog instead of emitting a fixed haiku/sonnet/opus tier.
No model id -- and no model family -- is baked into any pipeline. A model
released tomorrow, of any family, becomes eligible the day the API lists it.
The bootstrap selection call runs on the CLI default model, and if the catalog
is ever unreachable --model is simply omitted (CLI default) rather than falling
back to a named model, so a run never dies over model selection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The triage + review agents run with REVIEW_TOKEN and CLAUDE_CODE_OAUTH_TOKEN in
their environment, process attacker-controlled PR title/body/diff, and their
output is posted verbatim as a public PR comment. Without a Read sandbox a
prompt-injected diff could read /proc/self/environ (the tokens) or .git/config
(Woodpecker's clone credential) and get it published. Every other agent
pipeline already denies this via --settings; pr-review had none.
Add ci/agents/pr-review/settings.json (mirrors issue-fix/update-audit, gitea
#197) denying Read on /proc,/root,/home,/etc,.git, and pass --settings
$PROMPTS/settings.json to both claude invocations (resolves alongside the
prompts, so it works for the repos that fetch prompts from the homelab clone).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a pull_request event Woodpecker's built-in clone already checks the repo out
at the PR head in the workspace, so the agents read that directly for context.
Restores full-context review (the prior commit over-corrected to diff-only)
while keeping the manual, token-bearing clone gone. Prompts point at the
working-directory checkout instead of ./work/target.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clone existed only to hand review_bot_token to git, which persisted it in
work/target/.git/config — a file the review agent (Read/Glob/Grep, output
posted publicly) could read. Nothing needed the checkout that the API-fetched
diff + metadata don't already provide, so drop the clone entirely: no token on
disk, no work/target, no cleanup. The PR + diff are still read over the API
with ephemeral curl Authorization headers (never written to disk).
Prompts updated to match: the agents review the diff alone (no ./work/target),
and are told to flag, not guess, when a judgement needs code they can't see.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both agent steps hold Gitea write credentials in their environment
(issue-fix holds a push token too), but neither passed --settings, so
their Read tool was unrestricted — unlike update-audit, which got a
Read sandbox for exactly this reason (gitea issue #197).
That left a complete exfil path: Read /proc/self/environ yields the
token, Write puts it in the agent's output file, and the pipeline posts
that output verbatim as a Gitea issue body / PR body on a PUBLIC repo.
Add the same deny-list used by update-audit (/proc, /root, /home, /etc,
.git) to both agents and wire --settings into each invocation. The repo
checkout, .ci-alert-monitor/ and /tmp/alert-fix/ stay readable, which is
all either agent needs.
This is defense-in-depth only. The root cause — CI pipelines holding an
admin-owned token at all — is tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New Woodpecker cron pipeline that mirrors the alert-monitor / issue-fix
fleet pattern (cron -> headless `claude` -> Gitea API), but reviews open
pull requests instead of triaging alerts:
- .woodpecker/pr-review.yml: hourly `pr-review` cron. Discovers open PRs
across k.renner/{homelab,shelly-manager,authentik-samba-bridge,
homeassistant}, skips drafts/WIP and PRs already reviewed at their
current head sha (dedup marker), caps at 5/run. For each: clones the PR
head, has a cheap Haiku triage agent pick the model by complexity
(haiku/sonnet/opus; security-sensitive paths force opus), runs the
review with that model, and posts a detailed report as a Gitea PR
review (COMMENT, or REQUEST_CHANGES on a real defect — never APPROVE).
- ci/agents/pr-review/triage-prompt.md: model-selection rubric.
- ci/agents/pr-review/prompt.md: the reviewer's task + report format.
Runs under a dedicated `review-bot` Gitea account (secret
review_bot_token) so reviews are visibly bot-authored and independently
revocable. Static review only — no cluster access; agent tools are
Read/Glob/Grep/Write with no Bash.
Not yet active: needs the review-bot account + token, the review_bot_token
Woodpecker secret, and the `pr-review` cron registered (see file header).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The live 15-min snapshot structurally misses anything that fires and
resolves between ticks: on 2026-07-16 a 17-minute KubeHpaMaxedOut
(authentik) and last evening's four-alert home-assistant incident were
invisible to the monitor (owner noticed the blips; monitor did not).
The fetch step now also queries Prometheus's ALERTS history over a
35-minute window (~2 ticks + jitter). The dedup step synthesizes
pseudo-alerts for series not currently firing at all — checked against
the PRE-filter snapshot so live-but-null-routed alerts don't reappear
as blips — with a stable sha256-derived fingerprint so one noise
verdict suppresses a flappy alert permanently. Blips then flow through
the same title/noise dedup and triage as live alerts; the prompt
explains their semantics (self-resolved; judge the underlying cause;
null-routed per the repo route config = noise).
Verified against live data: currently-firing KubeCPUOvercommit
excluded, injected KubeHpaMaxedOut blip synthesized correctly;
sha256sum confirmed present in ci-claude; lookback query tested via
the service proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.
- issue-fix prompt: runtime claims must be attributed or framed as
hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
symptom, say so explicitly — the mismatch is often the real clue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Closes gitea issue #197. The update-audit agent fetches untrusted
third-party release pages via WebFetch/WebSearch and its Write output
is posted verbatim to a public Gitea issue, so injected content in a
fetched page could try to redirect the agent's behavior. Harden the
tool surface (verified empirically against the pinned Claude Code CLI):
- Drop WebSearch: it cannot be domain-scoped like WebFetch, and the
prompt already gives explicit source URLs or a well-known registry
for everything it needs to look up.
- Add ci/agents/update-audit/settings.json (loaded via --settings) that
denies Read on /proc, /root, /home, /etc, and the repo's own .git/ —
closing the credential-file and env-var-dump read paths a
prompt-injected instruction could otherwise try. Verified this deny
mechanism actually blocks reads, unlike Write path-scoping, which
was tested and found NOT enforced under --permission-mode acceptEdits
(documented in the pipeline comment; a full WebFetch domain allowlist
was also considered and rejected, since Helm chart lookups legitimately
hit whichever domain each HelmRelease's own trusted source URL points
at).
- prompt.md now tells the agent fetched content is untrusted data, not
instructions, and to never echo it verbatim into the report.
Not fixed here: GITEA_TOKEN is still the broad registry/release-writing
token (no write:issue-only token exists yet) — that requires an
operator to mint one; tracked via a comment on #197 and a code comment
at the secret binding.
rebuild-k8s.yml was the one-shot PVE->K8s migration/rebuild pipeline; it
is no longer needed and still pointed at the retired Garage S3 on the
now-off nas-1. Remove it (DR restore is via Velero + flux bootstrap).
The live cluster-access path is kept and migrated off Garage onto Hetzner
Object Storage (renner-uno-k8s-secrets, fsn1.your-objectstorage.com,
region fsn1, hetzner_s3_* keypair):
- scripts/ci/fetch-kubeconfig.sh: Garage -> Hetzner; env vars renamed
GARAGE_SECRETS_* -> S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY.
- .woodpecker/e2e.yml, e2e-test.yml: pass hetzner_s3_access_key/secret_key
(the live cron e2e pipeline needs this — it was broken against dead nas-1).
- scripts/import-cluster-access.sh: Garage -> Hetzner endpoint/bucket/region.
NOTE: the workstation aws-cli "secrets" profile must now hold the Hetzner
Object Storage keypair.
Also: delete the orphaned test-garage-smoke play + pipeline, and fix the
two dangling rebuild-k8s.yml references (zfs-tuning play comment;
update-audit prompt Talos-pin file list).
Assumes cluster-access objects (kubeconfig.sops/talosconfig.sops) were
migrated into renner-uno-k8s-secrets; with rebuild-k8s gone, nothing
auto-refreshes them, which is fine for a stable cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Woodpecker cron pipeline that runs a read-only Claude Code agent to
audit the components NOT covered by existing automation, and posts the report
to a single persistent Gitea issue.
Scope (complements, does not duplicate, existing automation):
- Flux image-automation already bumps enrolled container tags.
- The daily security-scan agent already owns CVEs.
- This agent audits manually-pinned currency instead: Talos / Kubernetes /
Flux versions, Helm chart minor/major bumps held back by ~MAJOR.MINOR.0
constraints, images excluded from automation, and Ansible/OpenTofu drift.
The agent is read-only (allowedTools: Read,Glob,Grep,Write,WebFetch,WebSearch;
no Edit/Bash) so it can never mutate the repo. It writes a report to
/tmp/update-audit.md; the pipeline find-or-creates the "Infrastructure update
audit" Gitea issue and refreshes its body.
Cron uses Woodpecker's documented `cron:` name filter (rather than the injected
PIPELINE var the other pipelines use) so the schedule is CLI-creatable:
woodpecker repo cron add --repo gitea_admin/homelab \
--name update-audit --branch master --schedule '@weekly'
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Container image tag updates are Flux's job — the homelab cluster
already runs image-reflector-controller and image-automation-
controller (in flux-system/gotk-components.yaml). Setting up
ImageRepository / ImagePolicy / ImageUpdateAutomation resources
is the right home for "bump postgres:15.17 → 15.18 when a new
image is published"; doing it from a CVE-driven scan agent
duplicates that pipeline and competes with it.
Reverting:
- Agent prompt: drop the `## Image tag bumps` section; image
findings now go straight to the Skipped list with a "Flux
image-automation owns this" reason.
- Woodpecker yml: drop Bash(crane *) from the agent's tool
allowlist (no longer used) and the `pull: true` workaround
that existed only to pick up crane's mid-day rebuild.
- ci-claude Containerfile: remove the crane install. The image
is back to Node + Claude Code + git/jq/curl basics.
The agent is now scoped exclusively to IaC misconfigurations
(Trivy fs scan results). Image vulns surface in the log + summary
for visibility but no PR action.
Note: there are currently zero ImageRepository / ImagePolicy /
ImageUpdateAutomation resources in kubernetes/ — the controllers
are running idle. Wiring them up for the actual images is a
separate workstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed KS owns influxdb-seed-ssh Secret in the influxdb namespace, but
the namespace was previously created by apps/influxdb/namespace.yaml —
which dependsOn seed → seed never reconciles → circular.
Hoist the namespace into infrastructure/namespaces/namespaces.yaml
(consistent with all the other app namespaces) and drop it from the
apps/influxdb Kustomization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 856's agent run found the Trivy JSON files via the
pipeline's own jq summary (IaC=0, Image=645 findings) but the
agent itself reported "No Trivy scan files (/tmp/trivy-iac.json
...) were present" — it hallucinated the /tmp prefix and didn't
look in the workspace root.
Prompt now states explicitly that the JSONs are in the current
working directory (project root, not /tmp), and instructs the
agent's first action to be `Glob trivy-*.json` to enumerate what
exists. Adds a stop-condition for the genuinely-empty case so
the agent can't get into a "I'll assume they're somewhere else"
loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cron security-scan now ends in an autonomous fix loop:
iac-scan → trivy-iac.json
image-scans → trivy-image-N.json (per scanned image)
security-fix-agent (master only):
1. Gates on >=1 CRITICAL/HIGH finding across all JSON outputs.
2. Creates branch security-fix/<date>-<short-sha>.
3. Hands the workspace to Claude Code in headless --bare mode
(subscription-backed via CLAUDE_CODE_OAUTH_TOKEN, no API key)
with a narrow tool allowlist (Read/Edit/Write/Glob/Grep —
no Bash). The agent reads the JSON, applies surgical fixes
under ansible/, opentofu/, kubernetes/, ci/images/, and writes
a Markdown summary to /tmp/security-fix-summary.md.
4. If the agent produced a diff, the step commits as
"Security Fix Agent <security-bot@renner.uno>" and pushes to
Gitea, then opens a PR via the pulls API using a dedicated
security_bot_token (NOT gitea_admin) so the PR is visibly
machine-authored.
The step is gated to branch=master so it doesn't fire on its own
PR branches — those still trigger the scan workflow as a re-validation
gate, but skip the agent (no fork-bombing).
Trivy now writes JSON output (--format json -o), and findings appear
both in the JSON and as one-line summaries in the step log via jq.
The legacy --exit-code 1 was kept off so the workflow always reaches
the agent step on master; on PR branches the scans are informational
and a human reviews the agent's PR.
Required Woodpecker secrets (added by user, not by this commit):
- claude_code_oauth_token (claude setup-token output)
- security_bot_token (Gitea PAT for the security-bot user)
Required Gitea setup (also out of band):
- User "security-bot" with email security-bot@renner.uno
- Write collaborator on gitea_admin/homelab
Agent constraints live in ci/agents/security-fix/prompt.md so they're
diffable + reviewable separately from the pipeline plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
