konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubePodNotReady (crowdsec) #346

New Issue
Closed
opened 2026-07-22 05:48:17 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-07-22 05:48:17 +00:00
Member
Copy Link
Copy Source

What fired

KubePodNotReady (blip, namespace crowdsec) — fired and auto-resolved within the ~35 min lookback. Live check shows the crowdsec-agent DaemonSet is currently healthy (3/3/3 Running, confirmed via kubectl get daemonset -n crowdsec crowdsec-agent), so the pod symptom is gone. The underlying cause is not: it is a recurring flap caused by a stuck HelmRelease, not a one-off blip.

Evidence

  • context.txt shows a dense burst (last ~55 min) of crowdsec namespace warnings: repeated UpgradeFailed, RollbackFailed, BackOff, FailedMount (secret "crowdsec-agent-tls" not found), and agent Unhealthy startup-probe failures, e.g.:
    • UpgradeFailed helmrelease/crowdsec: ... failed to create resource: Certificate.cert-manager.io "crowdsec-agent" is invalid: spec.secretTemplate: Invalid value: "null": spec.secretTemplate in body must be of type object: "null"
    • flux-system HealthCheckFailed kustomization/infrastructure: ... failed early due to stalled resources: [HelmRelease/crowdsec/crowdsec status: 'Failed']
  • Live (kubectl get helmrelease -n crowdsec crowdsec): Stalled=True reason=RetriesExceeded message="Failed to upgrade after 4 attempt(s)", Released=False reason=UpgradeFailed (same Certificate error), Ready=False reason=RollbackSucceeded (rolled back to crowdsec.v40).
  • Live events (kubectl get events -n crowdsec) confirm the most recent cycle: UpgradeFailed (Certificate error) at 2m46s ago, immediately followed by RollbackSucceeded at 2m11s ago — matching the DaemonSet's current healthy-but-rolled-back state.
  • context.txt's flux kustomization list shows ~20 kustomizations (apps, authentik, immich, monitoring, etc.) all blocked with dependency 'flux-system/infrastructure' is not ready, and infrastructure itself is not ready due to the crowdsec HelmRelease failure — this is a wider blast radius than just crowdsec.

Root cause

Verified live: the HelmRelease's install/upgrade remediation retries (retries: 3 in kubernetes/infrastructure/crowdsec/helmrelease.yaml) are exhausted (RetriesExceeded), so Flux has stopped auto-retrying and the release is running the old, pre-mTLS revision (v40) — not the desired state.

Every attempt fails at the same step: creating the crowdsec-agent Certificate.cert-manager.io object, rejected by the API server with spec.secretTemplate ... must be of type object: "null". This is the certificate the chart creates for agent mTLS client auth (tls.certManager.enabled: true, tls.agent.tlsClientAuth: true, issuerRef: {} in helmrelease.yaml lines 69-77) — the mTLS change from the just-merged #340/#343 fix. Our values never set a secretTemplate override, so this looks like the crowdsec chart (0.24.x) unconditionally rendering an explicit secretTemplate: null key in the Certificate manifest instead of omitting it, which a structural CRD schema without nullable: true rejects. Hypothesis, not fully confirmed — I don't have permission to read the certificates.cert-manager.io CRD schema or vendor/render the chart locally to see the exact template line; helm template crowdsec/crowdsec --version 0.24.x -f <these values> (or diffing against the chart's templates/certificate*.yaml on the crowdsec-helm-charts repo) would confirm it.

Where to fix

kubernetes/infrastructure/crowdsec/helmrelease.yaml (the tls: block, lines 69-77) — the mTLS config that was just merged for #340 needs a follow-up:

  • Try setting an explicit non-null secretTemplate value (even {}-with-content, e.g. annotations) under the relevant tls.* values key if the chart exposes one, to stop it emitting a bare null.
  • Or pin chart.spec.version to a specific 0.24.x patch known to not hit this (check the chart's changelog/issues for secretTemplate).
  • Not fixable purely by waiting: retries are exhausted, so this needs either a values/version change (which bumps the HelmRelease generation and triggers a fresh attempt) or a manual flux reconcile helmrelease crowdsec -n crowdsec --with-source / suspend-resume by a human with cluster write access, since this agent is read-only.

Filed automatically by the alert-monitor pipeline (2026-07-22). The issue-fix pipeline may pick this up and open a PR.

## What fired `KubePodNotReady` (blip, namespace `crowdsec`) — fired and auto-resolved within the ~35 min lookback. Live check shows the crowdsec-agent DaemonSet is currently healthy (`3/3/3 Running`, confirmed via `kubectl get daemonset -n crowdsec crowdsec-agent`), so the *pod* symptom is gone. The underlying cause is not: it is a recurring flap caused by a stuck HelmRelease, not a one-off blip. ## Evidence - `context.txt` shows a dense burst (last ~55 min) of `crowdsec` namespace warnings: repeated `UpgradeFailed`, `RollbackFailed`, `BackOff`, `FailedMount` (`secret "crowdsec-agent-tls" not found`), and agent `Unhealthy` startup-probe failures, e.g.: - `UpgradeFailed helmrelease/crowdsec: ... failed to create resource: Certificate.cert-manager.io "crowdsec-agent" is invalid: spec.secretTemplate: Invalid value: "null": spec.secretTemplate in body must be of type object: "null"` - `flux-system HealthCheckFailed kustomization/infrastructure: ... failed early due to stalled resources: [HelmRelease/crowdsec/crowdsec status: 'Failed']` - Live (`kubectl get helmrelease -n crowdsec crowdsec`): `Stalled=True reason=RetriesExceeded message="Failed to upgrade after 4 attempt(s)"`, `Released=False reason=UpgradeFailed` (same Certificate error), `Ready=False reason=RollbackSucceeded` (rolled back to `crowdsec.v40`). - Live events (`kubectl get events -n crowdsec`) confirm the most recent cycle: `UpgradeFailed` (Certificate error) at 2m46s ago, immediately followed by `RollbackSucceeded` at 2m11s ago — matching the DaemonSet's current healthy-but-rolled-back state. - `context.txt`'s flux kustomization list shows ~20 kustomizations (`apps`, `authentik`, `immich`, `monitoring`, etc.) all blocked with `dependency 'flux-system/infrastructure' is not ready`, and `infrastructure` itself is not ready due to the crowdsec HelmRelease failure — this is a wider blast radius than just crowdsec. ## Root cause Verified live: the HelmRelease's `install`/`upgrade` remediation retries (`retries: 3` in `kubernetes/infrastructure/crowdsec/helmrelease.yaml`) are exhausted (`RetriesExceeded`), so Flux has stopped auto-retrying and the release is running the old, pre-mTLS revision (`v40`) — not the desired state. Every attempt fails at the same step: creating the `crowdsec-agent` `Certificate.cert-manager.io` object, rejected by the API server with `spec.secretTemplate ... must be of type object: "null"`. This is the certificate the chart creates for agent mTLS client auth (`tls.certManager.enabled: true`, `tls.agent.tlsClientAuth: true`, `issuerRef: {}` in `helmrelease.yaml` lines 69-77) — the mTLS change from the just-merged #340/#343 fix. Our values never set a `secretTemplate` override, so this looks like the crowdsec chart (`0.24.x`) unconditionally rendering an explicit `secretTemplate: null` key in the Certificate manifest instead of omitting it, which a structural CRD schema without `nullable: true` rejects. **Hypothesis, not fully confirmed** — I don't have permission to read the `certificates.cert-manager.io` CRD schema or vendor/render the chart locally to see the exact template line; `helm template crowdsec/crowdsec --version 0.24.x -f <these values>` (or diffing against the chart's `templates/certificate*.yaml` on the crowdsec-helm-charts repo) would confirm it. ## Where to fix `kubernetes/infrastructure/crowdsec/helmrelease.yaml` (the `tls:` block, lines 69-77) — the mTLS config that was just merged for #340 needs a follow-up: - Try setting an explicit non-null `secretTemplate` value (even `{}`-with-content, e.g. annotations) under the relevant `tls.*` values key if the chart exposes one, to stop it emitting a bare `null`. - Or pin `chart.spec.version` to a specific 0.24.x patch known to not hit this (check the chart's changelog/issues for `secretTemplate`). - Not fixable purely by waiting: retries are exhausted, so this needs either a values/version change (which bumps the HelmRelease generation and triggers a fresh attempt) or a manual `flux reconcile helmrelease crowdsec -n crowdsec --with-source` / suspend-resume by a human with cluster write access, since this agent is read-only. --- <!-- alert-monitor:fingerprint=311df10cf9061609 --> _Filed automatically by the alert-monitor pipeline (2026-07-22). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-22 06:43:48 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change needed — already fixed

Why no edits: Issue #346 was filed at 2026-07-22T05:48:17Z based on a real
retry storm in the crowdsec HelmRelease (UpgradeFailed: the
crowdsec-agent Certificate's spec.secretTemplate rendered as null and
was rejected by the cert-manager webhook, retries exhausted, rolled back to
crowdsec.v40). That diagnosis was correct — but the fix for exactly this
root cause was merged 35 minutes later, at 06:23:56Z, as commit 72ec339
("fix(crowdsec): set tls secretTemplate so cert-manager accepts the certs
(#340)", PR #345), which is already in this branch's history and already
sets tls.certManager.secretTemplate.labels in
kubernetes/infrastructure/crowdsec/helmrelease.yaml (lines 69-84).

Live verification this run (via ./.ci-diag/kubectl, read-only):

  • kubectl get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.conditions}'
    → Ready=True Released=True reason=UpgradeSucceeded message="Helm upgrade succeeded for release crowdsec/crowdsec.v44 with chart crowdsec@0.24.0"
  • kubectl get kustomization -n flux-system infrastructure -o jsonpath='{.status.conditions}'
    → Ready=True reason=ReconciliationSucceeded message="Applied revision: master@sha1:72ec3396d23296b0711261a23db8af9772cd67d0" (the fix commit),
    Healthy=True reason=Succeeded
  • kubectl get certificate -n crowdsec → all four (crowdsec-agent,
    crowdsec-bouncer, crowdsec-ca, crowdsec-lapi) show READY=True,
    age ~18m (issued right after the fix commit reconciled)
  • kubectl get daemonset -n crowdsec crowdsec-agent → 3/3/3 desired/current/ready
  • kubectl get pods -n crowdsec → all agent and lapi pods 1/1 Running,
    0 restarts
  • kubectl get events -n crowdsec --sort-by=.lastTimestamp shows the full
    sequence: UpgradeFailed (secretTemplate null) → RollbackSucceeded
    (v36, then v38, then v40) repeating over ~60m, then — starting 18m ago,
    right after the fix landed — Certificate ... Issuing /
    CertificateIssued for all four certs, followed by
    UpgradeSucceeded to v43 then v44, with no further failures since.

The alert's underlying condition (stuck HelmRelease / blocked infrastructure
Kustomization / dependent-stack blast radius) has cleared on its own as a
direct result of the already-merged fix. No further edit to
helmrelease.yaml or anywhere else in the edit scope is warranted — doing so
now would be a redundant, unverified change layered on top of a fix that is
already confirmed working live.

Notes

  • The issue's "Where to fix" section correctly pointed at the tls: block
    in helmrelease.yaml, and its hypothesis about the chart unconditionally
    emitting secretTemplate: null matches what the existing code comment
    (lines 76-81) already documents as the chart 0.24.0 bug — this was
    independently re-derived by the triage agent from the same symptom, not a
    new finding.
  • If this alert fires again for crowdsec with the same secretTemplate
    error, that would mean the fix regressed (e.g. a chart version bump
    dropping the workaround) and would warrant re-opening — but no such
    regression is present in current live state.
  • No manual action is required; Flux and cert-manager have already
    self-healed the release.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment: no repo change needed — already fixed **Why no edits:** Issue #346 was filed at 2026-07-22T05:48:17Z based on a real retry storm in the `crowdsec` HelmRelease (`UpgradeFailed`: the `crowdsec-agent` Certificate's `spec.secretTemplate` rendered as `null` and was rejected by the cert-manager webhook, retries exhausted, rolled back to `crowdsec.v40`). That diagnosis was correct — but the fix for exactly this root cause was merged 35 minutes later, at 06:23:56Z, as commit `72ec339` ("fix(crowdsec): set tls secretTemplate so cert-manager accepts the certs (#340)", PR #345), which is already in this branch's history and already sets `tls.certManager.secretTemplate.labels` in `kubernetes/infrastructure/crowdsec/helmrelease.yaml` (lines 69-84). **Live verification this run** (via `./.ci-diag/kubectl`, read-only): - `kubectl get helmrelease -n crowdsec crowdsec -o jsonpath='{.status.conditions}'` → `Ready=True Released=True reason=UpgradeSucceeded message="Helm upgrade succeeded for release crowdsec/crowdsec.v44 with chart crowdsec@0.24.0"` - `kubectl get kustomization -n flux-system infrastructure -o jsonpath='{.status.conditions}'` → `Ready=True reason=ReconciliationSucceeded message="Applied revision: master@sha1:72ec3396d23296b0711261a23db8af9772cd67d0"` (the fix commit), `Healthy=True reason=Succeeded` - `kubectl get certificate -n crowdsec` → all four (`crowdsec-agent`, `crowdsec-bouncer`, `crowdsec-ca`, `crowdsec-lapi`) show `READY=True`, age ~18m (issued right after the fix commit reconciled) - `kubectl get daemonset -n crowdsec crowdsec-agent` → `3/3/3` desired/current/ready - `kubectl get pods -n crowdsec` → all agent and lapi pods `1/1 Running`, `0` restarts - `kubectl get events -n crowdsec --sort-by=.lastTimestamp` shows the full sequence: `UpgradeFailed` (secretTemplate null) → `RollbackSucceeded` (v36, then v38, then v40) repeating over ~60m, then — starting 18m ago, right after the fix landed — `Certificate ... Issuing` / `CertificateIssued` for all four certs, followed by `UpgradeSucceeded` to `v43` then `v44`, with no further failures since. The alert's underlying condition (stuck HelmRelease / blocked `infrastructure` Kustomization / dependent-stack blast radius) has cleared on its own as a direct result of the already-merged fix. No further edit to `helmrelease.yaml` or anywhere else in the edit scope is warranted — doing so now would be a redundant, unverified change layered on top of a fix that is already confirmed working live. ## Notes - The issue's "Where to fix" section correctly pointed at the `tls:` block in `helmrelease.yaml`, and its hypothesis about the chart unconditionally emitting `secretTemplate: null` matches what the existing code comment (lines 76-81) already documents as the chart 0.24.0 bug — this was independently re-derived by the triage agent from the same symptom, not a new finding. - If this alert fires again for `crowdsec` with the *same* `secretTemplate` error, that would mean the fix regressed (e.g. a chart version bump dropping the workaround) and would warrant re-opening — but no such regression is present in current live state. - No manual action is required; Flux and cert-manager have already self-healed the release. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-07-22 06:54:31 +00:00
Member
Copy Link
Copy Source

Resolved: symptom of the crowdsec LAPI-restart auth loop (#340), fixed by mTLS agent auth (PRs #343 + #345). crowdsec-agent DaemonSet is 3/3 Ready, 0 restarts, stable ~28m; cscli machines list shows all agents authenticating via TLS. Closing as resolved.

Resolved: symptom of the crowdsec LAPI-restart auth loop (#340), fixed by mTLS agent auth (PRs #343 + #345). crowdsec-agent DaemonSet is 3/3 Ready, 0 restarts, stable ~28m; `cscli machines list` shows all agents authenticating via TLS. Closing as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#346