The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both agent steps hold Gitea write credentials in their environment
(issue-fix holds a push token too), but neither passed --settings, so
their Read tool was unrestricted — unlike update-audit, which got a
Read sandbox for exactly this reason (gitea issue #197).
That left a complete exfil path: Read /proc/self/environ yields the
token, Write puts it in the agent's output file, and the pipeline posts
that output verbatim as a Gitea issue body / PR body on a PUBLIC repo.
Add the same deny-list used by update-audit (/proc, /root, /home, /etc,
.git) to both agents and wire --settings into each invocation. The repo
checkout, .ci-alert-monitor/ and /tmp/alert-fix/ stay readable, which is
all either agent needs.
This is defense-in-depth only. The root cause — CI pipelines holding an
admin-owned token at all — is tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.
- issue-fix prompt: runtime claims must be attributed or framed as
hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
symptom, say so explicitly — the mismatch is often the real clue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
claude-botandClaude Opus 5