Closes#340. Supersedes the #337 persistence approach.
Problem
crowdsec-agent DaemonSet CrashLoops (920+ restarts) whenever the LAPI pod restarts. The LAPI has no persistence (PVCs disabled in 629a303d to break the Ceph bootstrap circular-dependency), so its sqlite machine-registration DB is ephemeral. Agents register once in an init container and cache a password; a wiped LAPI no longer knows it → 401 "machine not found" → CrashLoop → DaemonSet never Ready → the crowdsec HelmRelease upgrade --wait times out → the whole infrastructure Kustomization (and every app depending on it) stays unhealthy. This has been the days-long outage.
Fix — mTLS agent auth (durable, self-healing, no persistence)
Enable chart mTLS (tls.enabled + tls.agent.tlsClientAuth, cert-manager self-signed CA). With client-cert auth the agent no longer runs cscli lapi register — it authenticates by presenting a cert whose OU (agent-ou) the LAPI trusts, and a freshly-restarted LAPI with an empty DB auto-recreates each machine record from the cert on connect. Self-heals across LAPI restarts with no PVC, no Ceph, no manual re-registration.
Verified via helm template against chart 0.24.0: register step gone, 4 Certificates emitted, zero PVCs, bouncer key preserved.
Also in this PR
Revert #337's lapi.persistentVolume.data.enabled: true → false. That attempt is what thrashed in #340: it reintroduces the Ceph circular-dependency 629a303d removed, and doesn't heal the agents (fresh volume = empty DB). mTLS makes the ephemeral DB harmless instead of persisting it.
Pin chart to 0.24.x. The floating >=0.12.0 silently upgraded to 0.24.0 (changed persistence defaults) — part of this incident.
Note in traefik/crowdsec-middleware.yaml that the (inert) ingress bouncer now needs https + CA trust before re-enabling.
Post-merge
The HelmRelease is wedged in a failed upgrade/rollback loop; may need a flux reconcile helmrelease crowdsec -n crowdsec nudge (or clearing a pending-rollback) after merge. Will verify agents go Ready and infrastructure recovers.
Closes #340. Supersedes the #337 persistence approach.
## Problem
crowdsec-agent DaemonSet CrashLoops (920+ restarts) whenever the LAPI pod restarts. The LAPI has **no persistence** (PVCs disabled in `629a303d` to break the Ceph bootstrap circular-dependency), so its sqlite machine-registration DB is ephemeral. Agents register **once** in an init container and cache a password; a wiped LAPI no longer knows it → `401 "machine not found"` → CrashLoop → DaemonSet never Ready → the crowdsec HelmRelease upgrade `--wait` times out → the whole `infrastructure` Kustomization (and every app depending on it) stays unhealthy. This has been the days-long outage.
## Fix — mTLS agent auth (durable, self-healing, no persistence)
Enable chart mTLS (`tls.enabled` + `tls.agent.tlsClientAuth`, cert-manager self-signed CA). With client-cert auth the agent **no longer runs `cscli lapi register`** — it authenticates by presenting a cert whose OU (`agent-ou`) the LAPI trusts, and a freshly-restarted LAPI with an **empty DB auto-recreates each machine record from the cert on connect**. Self-heals across LAPI restarts with **no PVC, no Ceph, no manual re-registration**.
Verified via `helm template` against chart 0.24.0: register step gone, 4 Certificates emitted, **zero PVCs**, bouncer key preserved.
## Also in this PR
- **Revert #337's `lapi.persistentVolume.data.enabled: true` → `false`.** That attempt is what thrashed in #340: it reintroduces the Ceph circular-dependency `629a303d` removed, and doesn't heal the agents (fresh volume = empty DB). mTLS makes the ephemeral DB harmless instead of persisting it.
- **Pin chart to `0.24.x`.** The floating `>=0.12.0` silently upgraded to 0.24.0 (changed persistence defaults) — part of this incident.
- **Note in `traefik/crowdsec-middleware.yaml`** that the (inert) ingress bouncer now needs `https` + CA trust before re-enabling.
## Post-merge
The HelmRelease is wedged in a failed upgrade/rollback loop; may need a `flux reconcile helmrelease crowdsec -n crowdsec` nudge (or clearing a pending-rollback) after merge. Will verify agents go Ready and `infrastructure` recovers.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The crowdsec-agent DaemonSet CrashLoops whenever the LAPI pod restarts.
Root cause: the LAPI has no persistence (PVCs disabled in 629a303d to
break the Ceph bootstrap circular-dependency), so its sqlite machine-
registration DB is ephemeral. Agents register once (init container) and
cache a password; a wiped LAPI no longer knows it -> 401 "machine not
found" -> CrashLoop -> DaemonSet never Ready -> the crowdsec HelmRelease
upgrade --wait times out -> the whole `infrastructure` Kustomization and
every app that depends on it stays unhealthy.
Enable chart mTLS (tls.enabled + tls.agent.tlsClientAuth, cert-manager
self-signed CA). With client-cert auth the agent no longer runs
`cscli lapi register`; the LAPI auto-recreates each machine record from
the presented cert on connect, so a wiped/restarted LAPI self-heals with
no PVC and no manual re-registration. Verified via `helm template`
against chart 0.24.0: register step gone, 4 certs emitted, zero PVCs.
Supersede the #337 persistence attempt (revert lapi.persistentVolume.
data.enabled to false): it reintroduced the Ceph circular-dependency and
did not heal the agents (fresh volume = empty DB). mTLS makes the
ephemeral DB harmless instead of trying to persist it.
Pin the chart to 0.24.x: the floating ">=0.12.0" silently upgraded to
0.24.0 (changed persistence defaults), part of this incident.
Note in traefik/crowdsec-middleware.yaml that the inert ingress bouncer
now needs https + CA trust before it can be re-enabled.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #340. Supersedes the #337 persistence approach.
Problem
crowdsec-agent DaemonSet CrashLoops (920+ restarts) whenever the LAPI pod restarts. The LAPI has no persistence (PVCs disabled in
629a303dto break the Ceph bootstrap circular-dependency), so its sqlite machine-registration DB is ephemeral. Agents register once in an init container and cache a password; a wiped LAPI no longer knows it →401 "machine not found"→ CrashLoop → DaemonSet never Ready → the crowdsec HelmRelease upgrade--waittimes out → the wholeinfrastructureKustomization (and every app depending on it) stays unhealthy. This has been the days-long outage.Fix — mTLS agent auth (durable, self-healing, no persistence)
Enable chart mTLS (
tls.enabled+tls.agent.tlsClientAuth, cert-manager self-signed CA). With client-cert auth the agent no longer runscscli lapi register— it authenticates by presenting a cert whose OU (agent-ou) the LAPI trusts, and a freshly-restarted LAPI with an empty DB auto-recreates each machine record from the cert on connect. Self-heals across LAPI restarts with no PVC, no Ceph, no manual re-registration.Verified via
helm templateagainst chart 0.24.0: register step gone, 4 Certificates emitted, zero PVCs, bouncer key preserved.Also in this PR
lapi.persistentVolume.data.enabled: true→false. That attempt is what thrashed in #340: it reintroduces the Ceph circular-dependency629a303dremoved, and doesn't heal the agents (fresh volume = empty DB). mTLS makes the ephemeral DB harmless instead of persisting it.0.24.x. The floating>=0.12.0silently upgraded to 0.24.0 (changed persistence defaults) — part of this incident.traefik/crowdsec-middleware.yamlthat the (inert) ingress bouncer now needshttps+ CA trust before re-enabling.Post-merge
The HelmRelease is wedged in a failed upgrade/rollback loop; may need a
flux reconcile helmrelease crowdsec -n crowdsecnudge (or clearing a pending-rollback) after merge. Will verify agents go Ready andinfrastructurerecovers.🤖 Generated with Claude Code