fix: [alert] KubeDaemonSetRolloutStuck (crowdsec) (#325) #327

2026-07-21T11:43:51Z

security-bot commented

2026-07-21 11:43:51 +00:00

What this PR does

KubeDaemonSetRolloutStuck fired for the crowdsec-agent daemonset because all 3 of its pods were stuck in CrashLoopBackOff (700+ restarts each, 24 days old). The root cause: the CrowdSec LAPI's machine-registration database (crowdsec.db) lives only on the LAPI pod's ephemeral filesystem (lapi.persistentVolume.data.enabled: false in the HelmRelease). The LAPI Deployment uses strategy: Recreate, so when its pod was recreated on 2026-07-18 (release lastDeployed: 2026-07-18T21:27:48Z), the registration DB was wiped, invalidating every existing agent's credentials. The 3 daemonset agent pods, which registered against the old DB 24 days ago, can never re-register on their own — their wait-for-lapi-and-register init container only runs once at pod creation, so restarting the crashing main container 700+ times just replays the same stale, now-invalid credentials forever. Fix: enable persistence for the LAPI's data volume so the registration DB survives future pod recreations.

Changes

kubernetes/infrastructure/crowdsec/helmrelease.yaml — set lapi.persistentVolume.data.enabled: true (was false), with a comment explaining why. config.enabled was left false: it's unrelated to this failure (the repo already supplies custom parser/whitelist config via the chart's top-level config: values block, and enabling a second, chart-managed config PVC on top of that is unverified and out of scope for this fix). Storage class/size were left unspecified since the cluster's default StorageClass (ceph-block, confirmed via kubectl get storageclass) is what every other explicitly-pinned PVC in this repo (e.g. openbao) also uses.

Verification

Confirmed live this run, matching the issue's claims:

./.ci-diag/kubectl get pods -n crowdsec: all 3 crowdsec-agent-* pods in CrashLoopBackOff (717-729 restarts, 24d old); current crowdsec-lapi-58c6bdd6f7-rt9tc is Running, age 2d14h (i.e. recreated well after the agents last registered).
./.ci-diag/kubectl logs -n crowdsec crowdsec-agent-m94ct: fatal crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-m94ct): API error: ent: machine not found after a 401 from the LAPI — confirms the stale-registration theory, not a config/network issue.
Read kubernetes/infrastructure/crowdsec/helmrelease.yaml: verified lines 41-45 (pre-fix) had persistentVolume.data.enabled: false and strategy: Recreate behavior is consistent with the LAPI pod's observed recreation.
./.ci-diag/kubectl get storageclass: ceph-block is the cluster default, so the new PVC will bind without needing an explicit storageClassName.

This repo change only prevents recurrence — it does not fix the 3 pods that are already crash-looping with stale credentials, since a PVC added now doesn't retroactively give the LAPI's current (already-running, pre-fix) database anything different, and the 3 daemonset pods still need their init container to re-run. A human/operator with cluster write access needs to separately run (not done here — this session has read-only access):

kubectl delete pod -n crowdsec crowdsec-agent-zr2ch crowdsec-agent-wbhvj crowdsec-agent-m94ct

so the DaemonSet recreates them and their wait-for-lapi-and-register init container re-registers against the LAPI's current database. After Flux reconciles this PR, a new PVC (crowdsec-lapi-data or similar, chart-dependent naming) should appear in the crowdsec namespace bound to ceph-block, and KubeDaemonSetRolloutStuck/KubePodCrashLooping/KubePodNotReady for crowdsec-agent should clear once the pods are cycled per above.

Notes

The issue correctly identified both the root cause and the fix location; I verified both independently rather than taking them on faith (per #249/#250 lessons).
The issue's own text flags one open question: what exactly triggered the 2026-07-18T21:27:48Z LAPI recreation (Reloader's own logs show no matching restart entry in that window). That trigger remains unconfirmed and doesn't change the fix — enabling persistence removes the impact of any future recreation, regardless of cause.
Not done in this PR: deleting the 3 stuck pods (requires cluster write access this session doesn't have) and enabling config persistence (unverified interaction with the existing custom-parser config mechanism — left alone per the "don't break what you can't verify" rule).

Closes #325

## What this PR does `KubeDaemonSetRolloutStuck` fired for the `crowdsec-agent` daemonset because all 3 of its pods were stuck in `CrashLoopBackOff` (700+ restarts each, 24 days old). The root cause: the CrowdSec LAPI's machine-registration database (`crowdsec.db`) lives only on the LAPI pod's ephemeral filesystem (`lapi.persistentVolume.data.enabled: false` in the HelmRelease). The LAPI Deployment uses `strategy: Recreate`, so when its pod was recreated on 2026-07-18 (release `lastDeployed: 2026-07-18T21:27:48Z`), the registration DB was wiped, invalidating every existing agent's credentials. The 3 daemonset agent pods, which registered against the old DB 24 days ago, can never re-register on their own — their `wait-for-lapi-and-register` init container only runs once at pod creation, so restarting the crashing main container 700+ times just replays the same stale, now-invalid credentials forever. Fix: enable persistence for the LAPI's `data` volume so the registration DB survives future pod recreations. ## Changes - `kubernetes/infrastructure/crowdsec/helmrelease.yaml` — set `lapi.persistentVolume.data.enabled: true` (was `false`), with a comment explaining why. `config.enabled` was left `false`: it's unrelated to this failure (the repo already supplies custom parser/whitelist config via the chart's top-level `config:` values block, and enabling a second, chart-managed config PVC on top of that is unverified and out of scope for this fix). Storage class/size were left unspecified since the cluster's default StorageClass (`ceph-block`, confirmed via `kubectl get storageclass`) is what every other explicitly-pinned PVC in this repo (e.g. openbao) also uses. ## Verification Confirmed live this run, matching the issue's claims: - `./.ci-diag/kubectl get pods -n crowdsec`: all 3 `crowdsec-agent-*` pods in `CrashLoopBackOff` (717-729 restarts, 24d old); current `crowdsec-lapi-58c6bdd6f7-rt9tc` is `Running`, age 2d14h (i.e. recreated well after the agents last registered). - `./.ci-diag/kubectl logs -n crowdsec crowdsec-agent-m94ct`: fatal `crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-m94ct): API error: ent: machine not found` after a 401 from the LAPI — confirms the stale-registration theory, not a config/network issue. - Read `kubernetes/infrastructure/crowdsec/helmrelease.yaml`: verified lines 41-45 (pre-fix) had `persistentVolume.data.enabled: false` and `strategy: Recreate` behavior is consistent with the LAPI pod's observed recreation. - `./.ci-diag/kubectl get storageclass`: `ceph-block` is the cluster default, so the new PVC will bind without needing an explicit `storageClassName`. This repo change only prevents **recurrence** — it does not fix the 3 pods that are already crash-looping with stale credentials, since a PVC added now doesn't retroactively give the LAPI's *current* (already-running, pre-fix) database anything different, and the 3 daemonset pods still need their init container to re-run. A human/operator with cluster write access needs to separately run (not done here — this session has read-only access): ``` kubectl delete pod -n crowdsec crowdsec-agent-zr2ch crowdsec-agent-wbhvj crowdsec-agent-m94ct ``` so the DaemonSet recreates them and their `wait-for-lapi-and-register` init container re-registers against the LAPI's current database. After Flux reconciles this PR, a new PVC (`crowdsec-lapi-data` or similar, chart-dependent naming) should appear in the `crowdsec` namespace bound to `ceph-block`, and `KubeDaemonSetRolloutStuck`/`KubePodCrashLooping`/`KubePodNotReady` for `crowdsec-agent` should clear once the pods are cycled per above. ## Notes - The issue correctly identified both the root cause and the fix location; I verified both independently rather than taking them on faith (per #249/#250 lessons). - The issue's own text flags one open question: what exactly triggered the 2026-07-18T21:27:48Z LAPI recreation (Reloader's own logs show no matching restart entry in that window). That trigger remains unconfirmed and doesn't change the fix — enabling persistence removes the *impact* of any future recreation, regardless of cause. - Not done in this PR: deleting the 3 stuck pods (requires cluster write access this session doesn't have) and enabling `config` persistence (unverified interaction with the existing custom-parser config mechanism — left alone per the "don't break what you can't verify" rule). Closes #325

security-bot added 1 commit 2026-07-21 11:43:52 +00:00

fix: [alert] KubeDaemonSetRolloutStuck (crowdsec) (#325 ) 44ca1f1e51

gitea_admin referenced this pull request

2026-07-21 11:43:52 +00:00

[alert] KubeDaemonSetRolloutStuck (crowdsec) #325

k.renner commented

2026-07-21 14:35:21 +00:00

The more interesting question is: Why did the agent not re-register themselves?

claude-bot referenced this pull request

2026-07-21 14:56:08 +00:00

security(ci): retire the admin gitea_token from CI; give each pipeline its own bot identity #330

claude-bot referenced this pull request

2026-07-21 15:11:16 +00:00

security(ci): retire the admin gitea_token from CI entirely #333

review-bot reviewed

2026-07-21 15:27:40 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 44ca1f1e5198a940a3dc5f21d0545378d6d1d9c9.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes the KubeDaemonSetRolloutStuck alert for CrowdSec by enabling persistent storage for the LAPI's registration database. The fix is correct and follows established patterns in the codebase. No blocking issues found.

🔴 Blocking

None.

🟡 Suggestions

None. The PR is well-scoped, the problem is thoroughly documented, and the implementation is sound.

🟢 Nits

None.

Tests

Not applicable — this is a Kubernetes Helm configuration change, not code. The author verified the fix independently:

Confirmed pods are in CrashLoopBackOff with 700+ restarts and stale credentials (pod logs show "machine not found" error)
Verified the LAPI pod was recreated on 2026-07-18, wiping the ephemeral registration DB
Confirmed ceph-block is the cluster's default StorageClass (matching the pattern used by other PVCs like openbao)
Verified the chart version constraint (>=0.12.0) supports this feature

Note on remediation: The PR correctly acknowledges that enabling persistence only prevents recurrence — existing stuck pods with stale credentials must be manually deleted by a human/operator to trigger re-registration. This is the expected behavior and is documented in the PR body.

Recommendation

Looks reasonable — no blocking issues; the change is safe and addresses the root cause.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `44ca1f1e5198a940a3dc5f21d0545378d6d1d9c9`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes the `KubeDaemonSetRolloutStuck` alert for CrowdSec by enabling persistent storage for the LAPI's registration database. The fix is correct and follows established patterns in the codebase. No blocking issues found. ## 🔴 Blocking None. ## 🟡 Suggestions None. The PR is well-scoped, the problem is thoroughly documented, and the implementation is sound. ## 🟢 Nits None. ## Tests Not applicable — this is a Kubernetes Helm configuration change, not code. The author verified the fix independently: - Confirmed pods are in `CrashLoopBackOff` with 700+ restarts and stale credentials (pod logs show "machine not found" error) - Verified the LAPI pod was recreated on 2026-07-18, wiping the ephemeral registration DB - Confirmed `ceph-block` is the cluster's default StorageClass (matching the pattern used by other PVCs like openbao) - Verified the chart version constraint (`>=0.12.0`) supports this feature **Note on remediation**: The PR correctly acknowledges that enabling persistence only prevents recurrence — existing stuck pods with stale credentials must be manually deleted by a human/operator to trigger re-registration. This is the expected behavior and is documented in the PR body. ## Recommendation Looks reasonable — no blocking issues; the change is safe and addresses the root cause.

k.renner closed this pull request

2026-07-21 22:45:06 +00:00

claude-bot referenced this pull request

2026-07-21 22:51:48 +00:00

security(ci): retire the admin gitea_token from CI; give each pipeline its own bot identity #330

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#327