Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
New secrets holding the Hetzner Object Storage keypair, ready for the
per-app backup cutover:
- seed/{paperless,authentik,immich}/hetzner-creds.sops.yaml
→ Secret/hetzner-s3-creds (ACCESS_KEY_ID, ACCESS_SECRET_KEY, REGION=fsn1)
- infrastructure/velero/hetzner-creds.sops.yaml
→ Secret/cloud-credentials-hetzner (cloud INI blob)
These are not referenced by any kustomization yet, so they're inert —
the existing Garage creds keep driving backups until each ObjectStore /
BSL is repointed (staged, paperless-first). Also gitignore
*.plaintext.yaml so the pre-encryption template can never be committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cron security-scan now ends in an autonomous fix loop:
iac-scan → trivy-iac.json
image-scans → trivy-image-N.json (per scanned image)
security-fix-agent (master only):
1. Gates on >=1 CRITICAL/HIGH finding across all JSON outputs.
2. Creates branch security-fix/<date>-<short-sha>.
3. Hands the workspace to Claude Code in headless --bare mode
(subscription-backed via CLAUDE_CODE_OAUTH_TOKEN, no API key)
with a narrow tool allowlist (Read/Edit/Write/Glob/Grep —
no Bash). The agent reads the JSON, applies surgical fixes
under ansible/, opentofu/, kubernetes/, ci/images/, and writes
a Markdown summary to /tmp/security-fix-summary.md.
4. If the agent produced a diff, the step commits as
"Security Fix Agent <security-bot@renner.uno>" and pushes to
Gitea, then opens a PR via the pulls API using a dedicated
security_bot_token (NOT gitea_admin) so the PR is visibly
machine-authored.
The step is gated to branch=master so it doesn't fire on its own
PR branches — those still trigger the scan workflow as a re-validation
gate, but skip the agent (no fork-bombing).
Trivy now writes JSON output (--format json -o), and findings appear
both in the JSON and as one-line summaries in the step log via jq.
The legacy --exit-code 1 was kept off so the workflow always reaches
the agent step on master; on PR branches the scans are informational
and a human reviews the agent's PR.
Required Woodpecker secrets (added by user, not by this commit):
- claude_code_oauth_token (claude setup-token output)
- security_bot_token (Gitea PAT for the security-bot user)
Required Gitea setup (also out of band):
- User "security-bot" with email security-bot@renner.uno
- Write collaborator on gitea_admin/homelab
Agent constraints live in ci/agents/security-fix/prompt.md so they're
diffable + reviewable separately from the pipeline plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Woodpecker steps use separate containers; home dirs don't persist.
Copy talosconfig/kubeconfig from ~ to access-staging/ in the bootstrap
step so the export step can read them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lessons from manual validation run:
- Add privileged PodSecurity labels to metallb-system, rook-ceph, ingress namespaces
- Add Cilium HelmRelease with Talos-specific settings (cgroup, kubePrism)
- Consolidate all namespace definitions in namespaces/
- Update cert-manager to use crds.enabled instead of installCRDs
- Enable all infrastructure components in main kustomization
- Add Talos secrets to .gitignore (controlplane.yaml, talosconfig)
- Update node patches with correct install disk (vda) and zone labels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
claude-botandClaude Fable 5
k.rennerandClaude Opus 4.8