Enables the single-click HA onboarding shipped in shelly-manager PR #41:
shelly-manager registers a provisioned Shelly into HA's native integration
by driving HA's config-flow + device-registry API.
- shelly-manager-llat ExternalSecret projects the full-admin HA LLAT from
OpenBao (apps/home-assistant/shelly-manager-llat, key 'token').
- HomeAssistant__BaseUrl = http://home-assistant (in-namespace svc :80 → nginx
sidecar → HA :8123; nginx forwards the WS upgrade, so both the REST config-
flow and the device-registry websocket work). Reachability confirmed: same-
namespace ingress is allowed by the default-deny-ingress CNP's home-assistant
rule (probed :80 → 401 = reachable).
- HomeAssistant__Token from the new secret.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner directive: PVE runs ONLY the three k8s VMs. The proxmox-dns trio
(lxc 118/218/318), rustfs trio (lxc 800/801/802), fortress (vm 102)
and paperless (vm 111) are decommissioned; guest definitions are being
destroyed on PVE out-of-band.
Removed:
- opentofu: lxc_118_proxmox_dns.tf, vm_102_fortress.tf,
vm_111_paperless.tf, backup_jobs.tf (the vzdump job covered ONLY
destroyed guests — next tofu apply removes the job from PVE; k8s
data protection is Velero/CNPG)
- ansible: roles/proxmox-dns, install-proxmox-dns{,-ha}.yml,
check-chaos-testing.yml (tested failover of decommissioned DNS-HA
and postgres LXCs), proxmox_dns_vault.yml, rustfs_vault.yml,
[proxmox_dns_ha] inventory group
- monitoring: dns-servers scrape job in both the k8s HelmRelease and
the legacy prometheus-server template — resolves TargetDown
(dns-servers)
Closes#259
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Woodpecker's k8s backend deletes each workspace PVC when a pipeline
ends, but ceph-filesystem has reclaimPolicy Retain, so every finished
pipeline left a Released PV pinning a CephFS subvolume — 191 of them
by 2026-07-16 (found in #262). New ceph-filesystem-ephemeral class
(identical parameters, reclaimPolicy Delete) used only by the CI
workspace volumes; production PVCs keep the undoable Retain default.
Existing Released woodpecker-ci PVs are swept out-of-band (they
predate this class and are unaffected by it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
- obs checks: run e2e-helper in the monitoring namespace — default is
not on monitoring's default-deny-ingress allow-list, so every
loki/prometheus/alertmanager check timed out (curl exit 28)
- node count: 4 -> 3 (k8s-1/2/3; there is no 4th node)
- CNPG exec targets: resolve the primary via
cnpg.io/cluster + cnpg.io/instanceRole=primary labels instead of
hardcoding pod names (pprl-pg-1 no longer exists; ak-pg-1 was one
failover away from the same break)
- no-failing-pods: exclude the woodpecker-ci namespace — the check
raced the e2e run's own runner pod (ContainerCreating, 4s old)
- purge_released_pvs: drop jq (not in ci-k8s image — the function has
been a silent no-op, 305 Released PVs currently pinned); jsonpath +
awk instead
Closes#262
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
VMM has no cloud-init and only makes blank disks, so replace the cloud-init VM
step with the real method: remastered netinst ISO auto-booting a preseed that
powers off at end (breaks the guest-reboot->CD reinstall loop). Debian 13, DHCP,
Default VM Network. Document the Netbird mesh-reachability caveat (CI/home not
peers) and the obg.renner.uno split-horizon DNS trap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- inventory: claude-ws at its current DHCP lease (10.3.40.78). Switch to the
stable Netbird mesh IP post-enrolment; the obg.renner.uno FQDN is
split-horizon and resolves to the public proxy from home, so it must not be
used as ansible_host.
- group_vars/claude_workstation/vault.yml: vault'd netbird_setup_key, scoped to
the group so install-netbird-client.yml only enrols this host.
- .woodpecker/deploy-netbird-client.yml: pipeline running
install-netbird-client.yml --limit claude_workstation (manual + master-path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reload_all does not re-read YAML-mode Lovelace dashboards (HA caches them until
restart or a lovelace/config force=true WS call), so merged dashboard changes
never appeared without a manual force-reload.
After a pull-merge that touches dashboards/ or ui-lovelace.yaml, config-sync now
also calls shell_command.reload_lovelace_yaml (defined in the HA config repo,
packages/lovelace_reload.yaml) which runs a websockets helper inside the HA
container to force the re-read. busybox can't speak WebSocket, hence the hand-off.
Needs a one-time HA restart to register the shell_command domain (not covered by
reload_all); until then reload_lovelace logs a warning and sync is otherwise fine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Turns on the app's new Auth:ReverseProxy mode (shelly-manager PR #40) so
the X-authentik-username header injected by the proxy outpost signs the
gate session straight into the app — removing the double login introduced
with the auth hardening. Trust is scoped to the pod CIDR because the
outpost pod address is dynamic; the namespace CNP remains the control
that keeps other pods off the app port. Inert until the image built from
shelly-manager PR #40 is promoted (unknown env keys are ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TargetDown carries no namespace label, so two simultaneous TargetDowns
(job=pve and job=dns-servers) collapsed into one '[alert] TargetDown
(cluster)' issue — the dns-servers root cause was never triaged
because the title dedup buried it under the pve issue. (Prometheus
shows 'context deadline exceeded' on all three dns-servers targets
since 2026-07-13: names resolve, the :12345 metrics endpoint doesn't
answer — cause to be triaged once the issue exists.)
Title key is now namespace // job // "cluster". Verified against the
live API: yields distinct '(pve)' and '(dns-servers)' titles.
Migration: retitle open issue #250 to '[alert] TargetDown (pve)' after
merge so the pve thread keeps deduping; the monitor then files a fresh
dns-servers issue for the fix agent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The fix agent's #252 assessment surfaced this: KubeCPUOvercommit is
deliberately routed to receiver 'null' in the Alertmanager config
('by design unsatisfiable on a small homelab cluster'), yet the
triage pipeline filed an issue for it — re-litigating a decision the
owner already made in the route config. The Alertmanager v2 alerts
API returns each alert's matched receivers, so drop alerts whose
receivers are all 'null' before triage. Filter verified against the
live API: KubeCPUOvercommit and Watchdog drop, the email-routed
alerts stay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
No pipeline passed --model, so runs floated on the ci-claude image's
CLI default — behavior could shift silently on the next image bump.
All four headless agents (alert-triage, issue-fix, security-fix,
update-audit) now pin claude-sonnet-5 explicitly; upgrade per-agent
by editing the flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Found on the first real use of the restricted identity: assessing the
VeleroBackupFailed issue (#251) needs backups.velero.io (latest
apps-daily run is PartiallyFailed — data-mover pod unschedulable on an
unbound PVC), and the ClusterRole had neither velero.io nor CNPG.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.
- issue-fix prompt: runtime claims must be attributed or framed as
hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
symptom, say so explicitly — the mismatch is often the real clue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The public-release auth work (shelly-manager PR #38) makes the app refuse
to start without ADMIN_PASSWORD; Flux promoted the new image and the pod
went CrashLoopBackOff. Adds the shelly-manager-auth ExternalSecret
(OpenBao: apps/home-assistant/shelly-manager-auth) and the env wiring,
repoints both probes to the new unauthenticated /health endpoint (closes
the long-standing probe gap — / would now 302 to /login), and updates the
stale 'app has no auth of its own' annotation: the macvlan bind pinning
stays as defense in depth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Der Configurator-Pod hing (UI nicht erreichbar, "No Room Selected" beim
Zonen-Editieren, Live-Tracking eingefroren): RSS lag bei 254Mi/256Mi und
~0,95 CPU-Kerne – klassisches Node/V8 GC-Thrashing am cgroup-Memory-Limit.
Der Prozess blieb knapp unter dem Limit, wurde also nie OOMKilled und kam
nie von selbst zurück; ein Neustart hat ihn befreit (frisch ~115Mi).
- limits.memory 256Mi → 512Mi (Headroom gegen GC-Thrash), requests.memory
64Mi → 128Mi (realistischere Baseline).
- Liveness/Readiness httpGet auf / (Port http). Bisher gab es keine Probe,
daher hat kubelet den hängenden Server nie neu gestartet. Jetzt schlägt
die Probe beim Einfrieren fehl und der Pod wird automatisch recycelt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Der Everything-Presence mmWave Zone-Configurator lief bereits im Cluster
(Deployment in ns home-assistant, UI :42069), war aber nur ClusterIP-intern
erreichbar. Er hält ein HA Long-Lived Token und kann die ESPHome-Zonen-Entities
aller Präsenzmelder umschreiben, hat aber keine eigene Auth — daher wird er
(wie vscode/shelly-manager/esphome) hinter ein Authentik Passkey-Gate gelegt,
nicht nackt exponiert.
Neu:
- Authentik-Blueprint zone-configurator-gate: Group "Zone Configurators"
(k.renner), ProxyProvider (passkey-stepup, internal_host …:42069,
external_host zone-configurator.home.renner.uno), Application, PolicyBinding,
dedizierter Outpost.
- ExternalName-Service ak-outpost-zone-configurator-gate + Traefik-Ingress
(cert-manager letsencrypt-cluster) im app-Verzeichnis.
- Cilium-Egress (authentik) + Ingress-Allow (home-assistant-ns) für
outpost -> zone-configurator:42069. Port 42069 ist eindeutig (wie esphome
:6052), daher kein Port-Overlap mit anderen Apps.
Traffic: browser -> traefik -> ak-outpost-zone-configurator-gate -> Outpost
(Passkey) -> zone-configurator.svc:42069.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The woodpecker-ci namespace enforces pod-security `privileged` (required for
the allow-listed docker/buildx plugin), Woodpecker runs WOODPECKER_OPEN=true,
and no admission controller exists. PodSecurity is all-or-nothing per
namespace, so the buildx exemption also un-gates hostPath/hostNetwork/hostPID/
hostIPC and privileged on every step pod a pipeline author can shape.
Reduce the blast radius with three defense-in-depth controls, none of which
weaken the actual build capability (this repo's builds use rootless kaniko;
only the external buildx plugin needs privileged, and it stays allow-listed):
- ValidatingAdmissionPolicy (GA/in-tree on k8s v1.35, no new controller):
denies host namespaces + hostPath for all step pods, and privileged for
every image except woodpeckerci/plugin-docker-buildx. Re-narrows the
PodSecurity exemption at the authoritative K8s admission layer. CEL verified
to compile via server-side dry-run.
- CiliumNetworkPolicy: ingress default-deny for step pods (house style,
egress left default-allow so builds keep reaching the internet + Gitea).
- ResourceQuota + LimitRange: bound pods/CPU/memory/PVCs against a
resource-exhaustion DoS; sized generously so current builds are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
security-bot
claude-botandClaude Fable 5
k.renner