konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] KubePodCrashLooping (crowdsec) #347

New Issue
Closed
opened 2026-07-22 06:03:50 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-07-22 06:03:50 +00:00
Member
Copy Link
Copy Source

What is firing

KubePodCrashLooping (warning) in namespace crowdsec, seen as a blip (fired and self-resolved within the last ~35 min window).

Evidence

From context.txt (Warning events, ~16-21 min before capture):

crowdsec  BackOff    pod/crowdsec-agent-*   Back-off restarting failed container crowdsec-agent
crowdsec  Unhealthy  pod/crowdsec-agent-*   Startup probe failed: ... dial tcp ...:6060: connect: connection refused
crowdsec  FailedMount pod/crowdsec-agent-*  MountVolume.SetUp failed for volume "crowdsec-agent-tls": secret "crowdsec-agent-tls" not found
crowdsec  UpgradeFailed helmrelease/crowdsec  Helm upgrade failed ...: Certificate.cert-manager.io "crowdsec-agent" is invalid: spec.secretTemplate: Invalid value: "null": spec.secretTemplate in body must be of type object: "null"

Verified live with ./.ci-diag/kubectl:

  • get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.conditions}' — Stalled=True reason=RetriesExceeded ("Failed to upgrade after 4 attempt(s)"), Released=False reason=UpgradeFailed with the same secretTemplate: null Certificate error, Ready=False reason=RollbackSucceeded.
  • get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.history}' — release version 41 (the new config, configDigest sha256:25e04344...) is status: failed; version 42 (configDigest sha256:1f2aa978..., identical digest to version 40, i.e. the pre-mTLS config) is the currently status: deployed release.
  • get pod -n crowdsec crowdsec-agent-8mn26 -o jsonpath='{.spec.volumes[*].name}' — no crowdsec-agent-tls volume on the currently running agent pod, confirming it's running the rolled-back, pre-mTLS config, not the new one.
  • get certificate -n crowdsec crowdsec-agent — NotFound (the Certificate object was never successfully created).
  • get kustomization -n flux-system infrastructure -o jsonpath='{.status.conditions}' — Ready=False/Healthy=False, HealthCheckFailed because of the stalled HelmRelease/crowdsec/crowdsec.

Root cause

Verified: the HelmRelease upgrade that ships the mTLS agent-auth fix from #340 (kubernetes/infrastructure/crowdsec/helmrelease.yaml, the tls: block, lines 69-77) fails because the chart renders the crowdsec-agent Certificate object with spec.secretTemplate: null, which the cert-manager CRD's structural schema rejects (must be an object or the key omitted entirely, not null). Each attempted upgrade briefly restarts all 3 crowdsec-agent DaemonSet pods (that's the crash-loop blip this alert caught), then Flux's remediation rolls back to the previous release after retries (3) are exhausted. The HelmRelease is now Stalled — Flux will not retry on its own timer; it needs a spec change (e.g. a values edit, which bumps observedGeneration) or a manual flux reconcile helmrelease crowdsec -n crowdsec to attempt again.

Net effect: the mTLS fix intended to end the LAPI-restart auth loop is currently not running in the cluster — the previously-deployed, pre-#340 config (password-based agent auth, vulnerable to the original bug this fix targeted) is what's live right now.

Hypothesis (not verifiable from this sandbox): the crowdsec chart's Certificate template likely renders a literal secretTemplate: null node when no extra secret-template overrides are set in values (rather than omitting the key), which is incompatible with the cert-manager v1 Certificate CRD schema. This should be confirmed by rendering the chart locally, e.g. helm template crowdsec crowdsec/crowdsec --version 0.24.0 -f <values-extracted-from-helmrelease>, or by checking the crowdsecurity/helm-charts chart source/issues for the Certificate template at chart version 0.24.0.

Where to fix

kubernetes/infrastructure/crowdsec/helmrelease.yaml — the tls: values block (lines 69-77, tls.certManager / tls.agent). Likely needs an explicit non-null value for whatever field controls the agent Certificate's secretTemplate (chart may expose something like an annotations/labels override that must be set instead of left absent), or a chart patch-version bump if this is a known chart bug, or a temporary tls.enabled: false revert to restore parity with the pre-#340 (already-deployed) config while a proper values fix is worked out. After changing values, the HelmRelease is Stalled and will need a forced reconcile (flux reconcile helmrelease crowdsec -n crowdsec --with-source) to retry, since it will not retry automatically.

Filed automatically by the alert-monitor pipeline (2026-07-22). The issue-fix pipeline may pick this up and open a PR.

## What is firing `KubePodCrashLooping` (warning) in namespace `crowdsec`, seen as a blip (fired and self-resolved within the last ~35 min window). ## Evidence From `context.txt` (Warning events, ~16-21 min before capture): ``` crowdsec BackOff pod/crowdsec-agent-* Back-off restarting failed container crowdsec-agent crowdsec Unhealthy pod/crowdsec-agent-* Startup probe failed: ... dial tcp ...:6060: connect: connection refused crowdsec FailedMount pod/crowdsec-agent-* MountVolume.SetUp failed for volume "crowdsec-agent-tls": secret "crowdsec-agent-tls" not found crowdsec UpgradeFailed helmrelease/crowdsec Helm upgrade failed ...: Certificate.cert-manager.io "crowdsec-agent" is invalid: spec.secretTemplate: Invalid value: "null": spec.secretTemplate in body must be of type object: "null" ``` Verified live with `./.ci-diag/kubectl`: - `get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.conditions}'` — `Stalled=True reason=RetriesExceeded ("Failed to upgrade after 4 attempt(s)")`, `Released=False reason=UpgradeFailed` with the same `secretTemplate: null` Certificate error, `Ready=False reason=RollbackSucceeded`. - `get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.history}'` — release version 41 (the new config, `configDigest sha256:25e04344...`) is `status: failed`; version 42 (`configDigest sha256:1f2aa978...`, identical digest to version 40, i.e. the *pre*-mTLS config) is the currently `status: deployed` release. - `get pod -n crowdsec crowdsec-agent-8mn26 -o jsonpath='{.spec.volumes[*].name}'` — no `crowdsec-agent-tls` volume on the currently running agent pod, confirming it's running the rolled-back, pre-mTLS config, not the new one. - `get certificate -n crowdsec crowdsec-agent` — `NotFound` (the Certificate object was never successfully created). - `get kustomization -n flux-system infrastructure -o jsonpath='{.status.conditions}'` — `Ready=False`/`Healthy=False`, `HealthCheckFailed` because of the stalled `HelmRelease/crowdsec/crowdsec`. ## Root cause **Verified:** the HelmRelease upgrade that ships the mTLS agent-auth fix from #340 (`kubernetes/infrastructure/crowdsec/helmrelease.yaml`, the `tls:` block, lines 69-77) fails because the chart renders the `crowdsec-agent` Certificate object with `spec.secretTemplate: null`, which the cert-manager CRD's structural schema rejects (must be an object or the key omitted entirely, not `null`). Each attempted upgrade briefly restarts all 3 `crowdsec-agent` DaemonSet pods (that's the crash-loop blip this alert caught), then Flux's remediation rolls back to the previous release after retries (3) are exhausted. The HelmRelease is now `Stalled` — Flux will not retry on its own timer; it needs a spec change (e.g. a values edit, which bumps `observedGeneration`) or a manual `flux reconcile helmrelease crowdsec -n crowdsec` to attempt again. **Net effect:** the mTLS fix intended to end the LAPI-restart auth loop is currently *not* running in the cluster — the previously-deployed, pre-#340 config (password-based agent auth, vulnerable to the original bug this fix targeted) is what's live right now. **Hypothesis (not verifiable from this sandbox):** the crowdsec chart's Certificate template likely renders a literal `secretTemplate: null` node when no extra secret-template overrides are set in values (rather than omitting the key), which is incompatible with the cert-manager v1 `Certificate` CRD schema. This should be confirmed by rendering the chart locally, e.g. `helm template crowdsec crowdsec/crowdsec --version 0.24.0 -f <values-extracted-from-helmrelease>`, or by checking the `crowdsecurity/helm-charts` chart source/issues for the `Certificate` template at chart version 0.24.0. ## Where to fix `kubernetes/infrastructure/crowdsec/helmrelease.yaml` — the `tls:` values block (lines 69-77, `tls.certManager` / `tls.agent`). Likely needs an explicit non-null value for whatever field controls the agent Certificate's `secretTemplate` (chart may expose something like an annotations/labels override that must be set instead of left absent), or a chart patch-version bump if this is a known chart bug, or a temporary `tls.enabled: false` revert to restore parity with the pre-#340 (already-deployed) config while a proper values fix is worked out. After changing values, the HelmRelease is `Stalled` and will need a forced reconcile (`flux reconcile helmrelease crowdsec -n crowdsec --with-source`) to retry, since it will not retry automatically. --- <!-- alert-monitor:fingerprint=0c4cc9d8c25b69cc --> _Filed automatically by the alert-monitor pipeline (2026-07-22). The issue-fix pipeline may pick this up and open a PR._
claude-bot commented 2026-07-22 06:54:31 +00:00
Member
Copy Link
Copy Source

Resolved: symptom of the crowdsec LAPI-restart auth loop (#340), fixed by mTLS agent auth (PRs #343 + #345). crowdsec-agent DaemonSet is 3/3 Ready, 0 restarts, stable ~28m; cscli machines list shows all agents authenticating via TLS. Closing as resolved.

Resolved: symptom of the crowdsec LAPI-restart auth loop (#340), fixed by mTLS agent auth (PRs #343 + #345). crowdsec-agent DaemonSet is 3/3 Ready, 0 restarts, stable ~28m; `cscli machines list` shows all agents authenticating via TLS. Closing as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#347