konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 4 Packages Projects Releases 1 Wiki Activity

[alert] KubePodCrashLooping (crowdsec) #337

New Issue
Closed
opened 2026-07-21 19:06:46 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-07-21 19:06:46 +00:00
Member
Copy Link
Copy Source

What is firing

KubePodCrashLooping (severity: warning) for all 3 replicas of the crowdsec-agent DaemonSet in namespace crowdsec (crowdsec-agent-wbhvj on k8s-2, crowdsec-agent-zr2ch on k8s-3, crowdsec-agent-m94ct on k8s-1). Restart counts are 804/815/804 and climbing at the CrashLoopBackOff-capped rate (~12/hr), so this is steady-state, not self-healing. This issue covers all three pod-level alerts (same root cause).

Evidence (all gathered live via ./.ci-diag/kubectl)

  • Agent container log (kubectl logs -n crowdsec pod/crowdsec-agent-zr2ch) ends with: 
    level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-zr2ch): API error: ent: machine not found"
  • LAPI server log (kubectl logs -n crowdsec crowdsec-lapi-58c6bdd6f7-rt9tc) shows the same rejection from the other side, repeated continuously for the pod's entire uptime: Error machine login for crowdsec-agent-zr2ch : ent: machine not found / POST /v1/watchers/login HTTP/1.1 401.
  • Counting request paths in the LAPI's logs over the current LAPI pod's full lifetime (--since=68h): 0 hits on POST /v1/watchers (the registration endpoint) vs 1842 hits on POST /v1/watchers/login. The agents are never re-registering — they are only retrying login with stale, cached credentials.
  • kubectl get pod -n crowdsec crowdsec-agent-zr2ch -o jsonpath=...volumes confirms /etc/crowdsec (crowdsec-config volume) is an emptyDir, i.e. scoped to the Pod, not the container: it survives container restarts (explaining why no fresh registration is attempted) but would be wiped if the Pod itself were recreated.
  • kubectl get rs -n crowdsec + get pods -o wide shows the current LAPI ReplicaSet (crowdsec-lapi-58c6bdd6f7) was created at 2026-07-18T21:28:00Z, replacing a ReplicaSet that had been stable since 2026-05-13 (66 days, 0 restarts). This timestamp lines up exactly with the ~68h window in which every single agent login has failed.

Root cause

Verified mechanism: kubernetes/infrastructure/crowdsec/helmrelease.yaml sets lapi.persistentVolume.data.enabled: false and lapi.persistentVolume.config.enabled: false — LAPI's local machine/watcher database has no persistent backing. When the LAPI pod was replaced on 2026-07-18T21:28 (new ReplicaSet), that database was reset, deleting the machine registrations the 3 agent DaemonSet pods had previously obtained. Because the agents' own /etc/crowdsec is on a Pod-scoped emptyDir (survives container restarts), they kept reusing their now-invalid cached credentials instead of re-registering, and crowdsec's agent does not fall back to auto re-register on a failed login — it exits fatally instead. Net effect: a permanent crash loop that will not self-resolve.

Unconfirmed hypothesis (one command would settle it): the LAPI pod replacement on 07-18 was likely triggered by the secret.reloader.stakater.com/reload: crowdsec-bouncer-key annotation on the LAPI pod template (see comment in helmrelease.yaml lines 37-40), i.e. a bouncer-key rotation caused Reloader to restart LAPI. I could not confirm this because the ci-diagnostics ServiceAccount is not permitted to read Secrets (kubectl get secret -n crowdsec crowdsec-bouncer-key → Forbidden). This doesn't change the fix, but a human could confirm via the openbao rotator's job history/logs for crowdsec-bouncer-key around 2026-07-18T21:28Z.

Where to fix it

  • Repo change (prevents recurrence): kubernetes/infrastructure/crowdsec/helmrelease.yaml — enable persistence for at least lapi.persistentVolume.data (holds the sqlite/ent DB with machine + bouncer registrations) so a future LAPI pod restart/redeploy doesn't wipe agent registrations again.
  • Not fixable via a repo change alone — needs live cluster action: the 3 already-broken agent pods have permanently invalid cached credentials in their emptyDir. A human with cluster write access needs to force the DaemonSet to recreate them, e.g. kubectl delete pod -n crowdsec crowdsec-agent-wbhvj crowdsec-agent-zr2ch crowdsec-agent-m94ct, so each gets a fresh emptyDir and performs a clean auto-registration against the current LAPI.

Filed automatically by the alert-monitor pipeline (2026-07-21). The issue-fix pipeline may pick this up and open a PR.

## What is firing `KubePodCrashLooping` (severity: warning) for all 3 replicas of the `crowdsec-agent` DaemonSet in namespace `crowdsec` (`crowdsec-agent-wbhvj` on k8s-2, `crowdsec-agent-zr2ch` on k8s-3, `crowdsec-agent-m94ct` on k8s-1). Restart counts are 804/815/804 and climbing at the CrashLoopBackOff-capped rate (~12/hr), so this is steady-state, not self-healing. This issue covers all three pod-level alerts (same root cause). ## Evidence (all gathered live via `./.ci-diag/kubectl`) - Agent container log (`kubectl logs -n crowdsec pod/crowdsec-agent-zr2ch`) ends with: ``` level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-zr2ch): API error: ent: machine not found" ``` - LAPI server log (`kubectl logs -n crowdsec crowdsec-lapi-58c6bdd6f7-rt9tc`) shows the same rejection from the other side, repeated continuously for the pod's entire uptime: `Error machine login for crowdsec-agent-zr2ch : ent: machine not found` / `POST /v1/watchers/login HTTP/1.1 401`. - Counting request paths in the LAPI's logs over the current LAPI pod's full lifetime (`--since=68h`): **0** hits on `POST /v1/watchers` (the registration endpoint) vs **1842** hits on `POST /v1/watchers/login`. The agents are never re-registering — they are only retrying login with stale, cached credentials. - `kubectl get pod -n crowdsec crowdsec-agent-zr2ch -o jsonpath=...volumes` confirms `/etc/crowdsec` (`crowdsec-config` volume) is an `emptyDir`, i.e. scoped to the Pod, not the container: it survives container restarts (explaining why no fresh registration is attempted) but would be wiped if the Pod itself were recreated. - `kubectl get rs -n crowdsec` + `get pods -o wide` shows the current LAPI ReplicaSet (`crowdsec-lapi-58c6bdd6f7`) was created at `2026-07-18T21:28:00Z`, replacing a ReplicaSet that had been stable since `2026-05-13` (66 days, 0 restarts). This timestamp lines up exactly with the ~68h window in which every single agent login has failed. ## Root cause **Verified mechanism:** `kubernetes/infrastructure/crowdsec/helmrelease.yaml` sets `lapi.persistentVolume.data.enabled: false` and `lapi.persistentVolume.config.enabled: false` — LAPI's local machine/watcher database has no persistent backing. When the LAPI pod was replaced on 2026-07-18T21:28 (new ReplicaSet), that database was reset, deleting the machine registrations the 3 agent DaemonSet pods had previously obtained. Because the agents' own `/etc/crowdsec` is on a Pod-scoped `emptyDir` (survives container restarts), they kept reusing their now-invalid cached credentials instead of re-registering, and crowdsec's agent does not fall back to auto re-register on a failed login — it exits fatally instead. Net effect: a permanent crash loop that will not self-resolve. **Unconfirmed hypothesis (one command would settle it):** the LAPI pod replacement on 07-18 was likely triggered by the `secret.reloader.stakater.com/reload: crowdsec-bouncer-key` annotation on the LAPI pod template (see comment in helmrelease.yaml lines 37-40), i.e. a bouncer-key rotation caused Reloader to restart LAPI. I could not confirm this because the `ci-diagnostics` ServiceAccount is not permitted to read Secrets (`kubectl get secret -n crowdsec crowdsec-bouncer-key` → Forbidden). This doesn't change the fix, but a human could confirm via the openbao rotator's job history/logs for `crowdsec-bouncer-key` around 2026-07-18T21:28Z. ## Where to fix it - **Repo change (prevents recurrence):** `kubernetes/infrastructure/crowdsec/helmrelease.yaml` — enable persistence for at least `lapi.persistentVolume.data` (holds the sqlite/ent DB with machine + bouncer registrations) so a future LAPI pod restart/redeploy doesn't wipe agent registrations again. - **Not fixable via a repo change alone — needs live cluster action:** the 3 already-broken agent pods have permanently invalid cached credentials in their `emptyDir`. A human with cluster write access needs to force the DaemonSet to recreate them, e.g. `kubectl delete pod -n crowdsec crowdsec-agent-wbhvj crowdsec-agent-zr2ch crowdsec-agent-m94ct`, so each gets a fresh `emptyDir` and performs a clean auto-registration against the current LAPI. --- <!-- alert-monitor:fingerprint=2ddc41edb689fc33 --> _Filed automatically by the alert-monitor pipeline (2026-07-21). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-21 19:43:58 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#338 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/338 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#337