KubePodCrashLooping is firing for all 3 crowdsec-agent DaemonSet pods (restart counts in the 800s and climbing). The agents fatally exit on every start with authenticate watcher: API error: ent: machine not found because the LAPI's local machine/watcher database has no persistent backing (kubernetes/infrastructure/crowdsec/helmrelease.yaml set lapi.persistentVolume.data.enabled: false). When the LAPI pod was last replaced, that in-memory-only database was reset, invalidating every previously-registered agent. Agents cache their credentials in a Pod-scoped emptyDir that survives container restarts, so they keep retrying login with now-invalid credentials instead of re-registering — crowdsec's agent does not fall back to auto re-register on a failed login. The fix enables persistence for lapi.persistentVolume.data so the machine/bouncer registration DB survives future LAPI pod replacements.
Changes
kubernetes/infrastructure/crowdsec/helmrelease.yaml — set lapi.persistentVolume.data.enabled: true (was false), with a comment explaining why. Left lapi.persistentVolume.config.enabled: false untouched since the config volume isn't what caused this failure and the issue only called for the data volume.
Verification
Verified live this run via ./.ci-diag/kubectl (not just taking the issue's word for it):
get pods -n crowdsec -o wide: confirms current state — crowdsec-agent-m94ct and crowdsec-agent-wbhvj are CrashLoopBackOff (811/810 restarts), crowdsec-agent-zr2ch is oscillating Running/crash (823 restarts) — matches the issue's claim, still actively firing.
logs pod/crowdsec-agent-zr2ch (main container): reproduces the exact fatal error from the issue — max attempts reached for status code 401 / ent: machine not found.
logs pod/crowdsec-agent-zr2ch -c wait-for-lapi-and-register --timestamps: this pod's init container did successfully register ("Successfully registered to Local API (LAPI)") — but at 2026-06-27T11:02:58Z, i.e. Pod creation time 24 days ago. Init containers run once per Pod, not per container-restart, which is why the agent has been retrying the same now-invalid June 27 credentials for 24 days straight rather than ever re-registering. This confirms the issue's mechanism precisely.
get pvc -n crowdsec: no PVCs exist in the namespace today, confirming persistence really is currently disabled (not just a stale claim).
Read kubernetes/infrastructure/crowdsec/helmrelease.yaml directly: confirmed lapi.persistentVolume.data.enabled: false prior to this change.
After Flux reconciles (HelmRelease interval 30m) and Helm creates the PVC, the reviewer can confirm:
kubectl get pvc -n crowdsec shows a new PVC bound for LAPI data.
KubePodCrashLooping for crowdsec-agent stops firing after the LAPI pod's registration DB has persisted through at least one subsequent LAPI restart.
Notes
This PR does not fix the 3 already-crashing agent pods. Their cached credentials in the Pod-scoped emptyDir are permanently invalid (tied to a machine registration that no longer exists in the LAPI DB) and no repo change can force a re-registration — only recreating those Pods does. A human with cluster write access needs to run (or equivalent): kubectl delete pod -n crowdsec crowdsec-agent-wbhvj crowdsec-agent-zr2ch crowdsec-agent-m94ct so the DaemonSet recreates them with fresh emptyDirs and clean auto-registration against the current (soon-to-be-persistent) LAPI.
The issue's secondary hypothesis — that a crowdsec-bouncer-key rotation via Reloader triggered the LAPI pod replacement on 2026-07-18 — is plausible (the annotation exists in the helmrelease) but is about why the LAPI restarted, not whether it should survive restarts. It doesn't change the fix and I didn't attempt to verify it (would require Secret/job-history access outside the read-only diagnostics scope).
Left lapi.persistentVolume.config.enabled as false: the issue only established the data volume (machine/bouncer DB) as the root cause; enabling config persistence too wasn't verified as necessary and would be outside a surgical fix.
## What this PR does
`KubePodCrashLooping` is firing for all 3 `crowdsec-agent` DaemonSet pods (restart counts in the 800s and climbing). The agents fatally exit on every start with `authenticate watcher: API error: ent: machine not found` because the LAPI's local machine/watcher database has no persistent backing (`kubernetes/infrastructure/crowdsec/helmrelease.yaml` set `lapi.persistentVolume.data.enabled: false`). When the LAPI pod was last replaced, that in-memory-only database was reset, invalidating every previously-registered agent. Agents cache their credentials in a Pod-scoped `emptyDir` that survives container restarts, so they keep retrying login with now-invalid credentials instead of re-registering — crowdsec's agent does not fall back to auto re-register on a failed login. The fix enables persistence for `lapi.persistentVolume.data` so the machine/bouncer registration DB survives future LAPI pod replacements.
## Changes
- `kubernetes/infrastructure/crowdsec/helmrelease.yaml` — set `lapi.persistentVolume.data.enabled: true` (was `false`), with a comment explaining why. Left `lapi.persistentVolume.config.enabled: false` untouched since the config volume isn't what caused this failure and the issue only called for the data volume.
## Verification
Verified live this run via `./.ci-diag/kubectl` (not just taking the issue's word for it):
- `get pods -n crowdsec -o wide`: confirms current state — `crowdsec-agent-m94ct` and `crowdsec-agent-wbhvj` are `CrashLoopBackOff` (811/810 restarts), `crowdsec-agent-zr2ch` is oscillating `Running`/crash (823 restarts) — matches the issue's claim, still actively firing.
- `logs pod/crowdsec-agent-zr2ch` (main container): reproduces the exact fatal error from the issue — `max attempts reached for status code 401` / `ent: machine not found`.
- `logs pod/crowdsec-agent-zr2ch -c wait-for-lapi-and-register --timestamps`: this pod's init container *did* successfully register ("Successfully registered to Local API (LAPI)") — but at `2026-06-27T11:02:58Z`, i.e. Pod creation time 24 days ago. Init containers run once per Pod, not per container-restart, which is why the agent has been retrying the same now-invalid June 27 credentials for 24 days straight rather than ever re-registering. This confirms the issue's mechanism precisely.
- `get pvc -n crowdsec`: **no PVCs exist** in the namespace today, confirming persistence really is currently disabled (not just a stale claim).
- Read `kubernetes/infrastructure/crowdsec/helmrelease.yaml` directly: confirmed `lapi.persistentVolume.data.enabled: false` prior to this change.
After Flux reconciles (HelmRelease interval 30m) and Helm creates the PVC, the reviewer can confirm:
- `kubectl get pvc -n crowdsec` shows a new PVC bound for LAPI data.
- `KubePodCrashLooping` for crowdsec-agent stops firing after the LAPI pod's registration DB has persisted through at least one subsequent LAPI restart.
## Notes
- **This PR does not fix the 3 already-crashing agent pods.** Their cached credentials in the Pod-scoped `emptyDir` are permanently invalid (tied to a machine registration that no longer exists in the LAPI DB) and no repo change can force a re-registration — only recreating those Pods does. A human with cluster write access needs to run (or equivalent): `kubectl delete pod -n crowdsec crowdsec-agent-wbhvj crowdsec-agent-zr2ch crowdsec-agent-m94ct` so the DaemonSet recreates them with fresh `emptyDir`s and clean auto-registration against the current (soon-to-be-persistent) LAPI.
- The issue's secondary hypothesis — that a `crowdsec-bouncer-key` rotation via Reloader triggered the LAPI pod replacement on 2026-07-18 — is plausible (the annotation exists in the helmrelease) but is about *why* the LAPI restarted, not *whether* it should survive restarts. It doesn't change the fix and I didn't attempt to verify it (would require Secret/job-history access outside the read-only diagnostics scope).
- Left `lapi.persistentVolume.config.enabled` as `false`: the issue only established the *data* volume (machine/bouncer DB) as the root cause; enabling config persistence too wasn't verified as necessary and would be outside a surgical fix.
Closes #337
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 96b0d70767785b8fe657e5a3ae851a29045d053f.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR enables persistent storage for the crowdsec LAPI pod's machine/bouncer registration database, fixing a cascading failure where LAPI pod restarts wipe the registration DB while agents keep retrying login with now-invalid cached credentials. The change is minimal, well-motivated, and the author has thoroughly verified both the root cause and the correctness of the fix.
🟢 Nits
The comment at line 43–47 is clear and explains the "why" well. No changes needed.
The decision to leave lapi.persistentVolume.config.enabled: false is correct and well-justified (only data persistence was identified as necessary).
Tests
This is a Flux HelmRelease configuration change. Testing would require a live cluster and the ability to restart pods. The author conducted appropriate manual verification:
Used kubectl get pods, kubectl logs, and kubectl get pvc to confirm the issue and current state.
Verified that init containers run once per Pod (explaining why agents don't re-register on container restarts).
Confirmed no PVCs exist today, validating that persistence is currently disabled.
After Flux reconciles, the reviewer can monitor:
Creation of a new PVC in the crowdsec namespace for LAPI data.
The KubePodCrashLooping alert stopping once the LAPI pod's database survives at least one subsequent restart.
Note: The already-crashing agent pods (800+ restart counts) will need manual recreation (kubectl delete pod) to clear their invalid cached credentials; the PR correctly documents this limitation.
Recommendation
Looks reasonable — no blocking issues; the fix is sound and well-documented.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `96b0d70767785b8fe657e5a3ae851a29045d053f`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR enables persistent storage for the crowdsec LAPI pod's machine/bouncer registration database, fixing a cascading failure where LAPI pod restarts wipe the registration DB while agents keep retrying login with now-invalid cached credentials. The change is minimal, well-motivated, and the author has thoroughly verified both the root cause and the correctness of the fix.
## 🟢 Nits
- The comment at line 43–47 is clear and explains the "why" well. No changes needed.
- The decision to leave `lapi.persistentVolume.config.enabled: false` is correct and well-justified (only data persistence was identified as necessary).
## Tests
This is a Flux HelmRelease configuration change. Testing would require a live cluster and the ability to restart pods. The author conducted appropriate manual verification:
- Used `kubectl get pods`, `kubectl logs`, and `kubectl get pvc` to confirm the issue and current state.
- Verified that init containers run once per Pod (explaining why agents don't re-register on container restarts).
- Confirmed no PVCs exist today, validating that persistence is currently disabled.
After Flux reconciles, the reviewer can monitor:
1. Creation of a new PVC in the crowdsec namespace for LAPI data.
2. The `KubePodCrashLooping` alert stopping once the LAPI pod's database survives at least one subsequent restart.
Note: The already-crashing agent pods (800+ restart counts) will need manual recreation (`kubectl delete pod`) to clear their invalid cached credentials; the PR correctly documents this limitation.
## Recommendation
Looks reasonable — no blocking issues; the fix is sound and well-documented.
<!-- pr-review:sha=96b0d70767785b8fe657e5a3ae851a29045d053f -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
KubePodCrashLoopingis firing for all 3crowdsec-agentDaemonSet pods (restart counts in the 800s and climbing). The agents fatally exit on every start withauthenticate watcher: API error: ent: machine not foundbecause the LAPI's local machine/watcher database has no persistent backing (kubernetes/infrastructure/crowdsec/helmrelease.yamlsetlapi.persistentVolume.data.enabled: false). When the LAPI pod was last replaced, that in-memory-only database was reset, invalidating every previously-registered agent. Agents cache their credentials in a Pod-scopedemptyDirthat survives container restarts, so they keep retrying login with now-invalid credentials instead of re-registering — crowdsec's agent does not fall back to auto re-register on a failed login. The fix enables persistence forlapi.persistentVolume.dataso the machine/bouncer registration DB survives future LAPI pod replacements.Changes
kubernetes/infrastructure/crowdsec/helmrelease.yaml— setlapi.persistentVolume.data.enabled: true(wasfalse), with a comment explaining why. Leftlapi.persistentVolume.config.enabled: falseuntouched since the config volume isn't what caused this failure and the issue only called for the data volume.Verification
Verified live this run via
./.ci-diag/kubectl(not just taking the issue's word for it):get pods -n crowdsec -o wide: confirms current state —crowdsec-agent-m94ctandcrowdsec-agent-wbhvjareCrashLoopBackOff(811/810 restarts),crowdsec-agent-zr2chis oscillatingRunning/crash (823 restarts) — matches the issue's claim, still actively firing.logs pod/crowdsec-agent-zr2ch(main container): reproduces the exact fatal error from the issue —max attempts reached for status code 401/ent: machine not found.logs pod/crowdsec-agent-zr2ch -c wait-for-lapi-and-register --timestamps: this pod's init container did successfully register ("Successfully registered to Local API (LAPI)") — but at2026-06-27T11:02:58Z, i.e. Pod creation time 24 days ago. Init containers run once per Pod, not per container-restart, which is why the agent has been retrying the same now-invalid June 27 credentials for 24 days straight rather than ever re-registering. This confirms the issue's mechanism precisely.get pvc -n crowdsec: no PVCs exist in the namespace today, confirming persistence really is currently disabled (not just a stale claim).kubernetes/infrastructure/crowdsec/helmrelease.yamldirectly: confirmedlapi.persistentVolume.data.enabled: falseprior to this change.After Flux reconciles (HelmRelease interval 30m) and Helm creates the PVC, the reviewer can confirm:
kubectl get pvc -n crowdsecshows a new PVC bound for LAPI data.KubePodCrashLoopingfor crowdsec-agent stops firing after the LAPI pod's registration DB has persisted through at least one subsequent LAPI restart.Notes
emptyDirare permanently invalid (tied to a machine registration that no longer exists in the LAPI DB) and no repo change can force a re-registration — only recreating those Pods does. A human with cluster write access needs to run (or equivalent):kubectl delete pod -n crowdsec crowdsec-agent-wbhvj crowdsec-agent-zr2ch crowdsec-agent-m94ctso the DaemonSet recreates them with freshemptyDirs and clean auto-registration against the current (soon-to-be-persistent) LAPI.crowdsec-bouncer-keyrotation via Reloader triggered the LAPI pod replacement on 2026-07-18 — is plausible (the annotation exists in the helmrelease) but is about why the LAPI restarted, not whether it should survive restarts. It doesn't change the fix and I didn't attempt to verify it (would require Secret/job-history access outside the read-only diagnostics scope).lapi.persistentVolume.config.enabledasfalse: the issue only established the data volume (machine/bouncer DB) as the root cause; enabling config persistence too wasn't verified as necessary and would be outside a surgical fix.Closes #337
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit96b0d70767785b8fe657e5a3ae851a29045d053f.Summary
This PR enables persistent storage for the crowdsec LAPI pod's machine/bouncer registration database, fixing a cascading failure where LAPI pod restarts wipe the registration DB while agents keep retrying login with now-invalid cached credentials. The change is minimal, well-motivated, and the author has thoroughly verified both the root cause and the correctness of the fix.
🟢 Nits
lapi.persistentVolume.config.enabled: falseis correct and well-justified (only data persistence was identified as necessary).Tests
This is a Flux HelmRelease configuration change. Testing would require a live cluster and the ability to restart pods. The author conducted appropriate manual verification:
kubectl get pods,kubectl logs, andkubectl get pvcto confirm the issue and current state.After Flux reconciles, the reviewer can monitor:
KubePodCrashLoopingalert stopping once the LAPI pod's database survives at least one subsequent restart.Note: The already-crashing agent pods (800+ restart counts) will need manual recreation (
kubectl delete pod) to clear their invalid cached credentials; the PR correctly documents this limitation.Recommendation
Looks reasonable — no blocking issues; the fix is sound and well-documented.