konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] KubePodCrashLooping (apps) #424

New Issue
Closed
opened 2026-07-26 10:32:35 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-07-26 10:32:35 +00:00
Member
Copy Link
Copy Source

What is firing: KubePodCrashLooping (severity: warning) in namespace apps. Alertmanager recorded this as a blip (fired within the ~35 min lookback and already resolved by the time of the scan), but live diagnostics show the crash loop is still ongoing at the time of this triage.

Evidence

  • ./.ci-diag/kubectl get pods -n apps -o wide: external-dns-unifi-89f464b66-98jjw has 13 restarts over its 4h38m age, most recent 2m34s before the check.
  • ./.ci-diag/kubectl describe pod external-dns-unifi-89f464b66-98jjw -n apps: Warning BackOff 2m41s (x36 over 71m) kubelet Back-off restarting failed container external-dns.
  • context.txt independently caught the same thing: apps 118s Warning BackOff pod/external-dns-unifi-89f464b66-98jjw Back-off restarting failed container external-dns.
  • Previous container log (kubectl logs ... -c external-dns --previous): level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" — the main external-dns container fatally exits (log.Fatalf -> exit 1) when its local webhook sidecar doesn't answer in time.
  • Current container log (webhook sidecar, kashalls/external-dns-unifi-webhook) shows the sidecar itself repeatedly failing against the UniFi controller: GET .../static-dns/ returned 401: User not authenticated, returned 502, and failed to decode json: unexpected end of JSON input.

Root cause
Verified via the previous-container log and the helmrelease source: kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml already has a documented fix for a related failure mode — a comment (lines ~114-124) explains that the UDM's static-dns endpoint can be slow and that external-dns treats any RunOnce error as fatal, and sets --webhook-provider-read-timeout=60s to give headroom. That flag IS present on the running pod's args (confirmed via kubectl get pod ... -o jsonpath='{.spec.containers[?(@.name=="external-dns")].args}') and the pod was recreated 2026-07-26T05:52:46Z (current HelmRelease generation 8, UpgradeSucceeded).

Despite that fix being live, the pod is still crash-looping — 13 restarts in under 5 hours, ongoing. The new evidence (401 auth failures and 502s from https://udm.renner.uno, not just slow-but-successful reads) suggests the fix addressed one failure mode (slow-but-good responses) but not this one: the UniFi controller is intermittently rejecting/failing requests outright, which likely still exceeds the 60s webhook timeout or otherwise stalls the sidecar long enough for the main container to time out and fatally exit.

This is a hypothesis, not fully confirmed — I could not check UDM controller health directly (no route from this read-only tooling to the UDM admin API/logs). It would be settled by watching whether the 401s coincide with UDM-side session expiry/reboots, e.g. via ./.ci-diag/kubectl logs deploy/external-dns-unifi -n apps -c webhook --tail=200 around the next restart, cross-referenced with UDM controller/system logs (outside this repo's reach).

Where to fix it

  • kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml: the existing --webhook-provider-read-timeout=60s mitigation is insufficient for this new 401/502 failure pattern. Consider whether the UNIFI_API_KEY (from external-dns-unifi-secret, synced via kubernetes/infrastructure/external-dns-unifi/externalsecret.yaml from OpenBao key infrastructure/external-dns-unifi) is still valid — a stale/rotated API key would explain repeated 401s from the UDM. ExternalSecret status shows SecretSynced: True (last sync 38m ago), which only confirms OpenBao->K8s sync is healthy, NOT that the key itself is accepted by the UDM.
  • If the key is fine, this is likely NOT fixable purely in this repo: it points at UniFi controller (udm.renner.uno) instability/auth flakiness, which needs a human to check the UDM controller's own health/logs and possibly the external-dns-unifi-webhook image version (currently pinned to v0.4.1) for known issues with UniFi session auth under transient errors.

Filed automatically by the alert-monitor pipeline (2026-07-26). The issue-fix pipeline may pick this up and open a PR.

**What is firing**: `KubePodCrashLooping` (severity: warning) in namespace `apps`. Alertmanager recorded this as a blip (fired within the ~35 min lookback and already resolved by the time of the scan), but live diagnostics show the crash loop is still ongoing at the time of this triage. **Evidence** - `./.ci-diag/kubectl get pods -n apps -o wide`: `external-dns-unifi-89f464b66-98jjw` has `13` restarts over its `4h38m` age, most recent `2m34s` before the check. - `./.ci-diag/kubectl describe pod external-dns-unifi-89f464b66-98jjw -n apps`: `Warning BackOff 2m41s (x36 over 71m) kubelet Back-off restarting failed container external-dns`. - `context.txt` independently caught the same thing: `apps 118s Warning BackOff pod/external-dns-unifi-89f464b66-98jjw Back-off restarting failed container external-dns`. - Previous container log (`kubectl logs ... -c external-dns --previous`): `level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"` — the main `external-dns` container fatally exits (log.Fatalf -> exit 1) when its local webhook sidecar doesn't answer in time. - Current container log (webhook sidecar, `kashalls/external-dns-unifi-webhook`) shows the sidecar itself repeatedly failing against the UniFi controller: `GET .../static-dns/ returned 401: User not authenticated`, `returned 502`, and `failed to decode json: unexpected end of JSON input`. **Root cause** Verified via the previous-container log and the helmrelease source: `kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml` already has a documented fix for a related failure mode — a comment (lines ~114-124) explains that the UDM's static-dns endpoint can be slow and that external-dns treats any RunOnce error as fatal, and sets `--webhook-provider-read-timeout=60s` to give headroom. That flag IS present on the running pod's args (confirmed via `kubectl get pod ... -o jsonpath='{.spec.containers[?(@.name=="external-dns")].args}'`) and the pod was recreated 2026-07-26T05:52:46Z (current HelmRelease generation 8, `UpgradeSucceeded`). Despite that fix being live, the pod is still crash-looping — 13 restarts in under 5 hours, ongoing. The new evidence (401 auth failures and 502s from `https://udm.renner.uno`, not just slow-but-successful reads) suggests the fix addressed one failure mode (slow-but-good responses) but not this one: the UniFi controller is intermittently rejecting/failing requests outright, which likely still exceeds the 60s webhook timeout or otherwise stalls the sidecar long enough for the main container to time out and fatally exit. This is a hypothesis, not fully confirmed — I could not check UDM controller health directly (no route from this read-only tooling to the UDM admin API/logs). It would be settled by watching whether the 401s coincide with UDM-side session expiry/reboots, e.g. via `./.ci-diag/kubectl logs deploy/external-dns-unifi -n apps -c webhook --tail=200` around the next restart, cross-referenced with UDM controller/system logs (outside this repo's reach). **Where to fix it** - `kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml`: the existing `--webhook-provider-read-timeout=60s` mitigation is insufficient for this new 401/502 failure pattern. Consider whether the `UNIFI_API_KEY` (from `external-dns-unifi-secret`, synced via `kubernetes/infrastructure/external-dns-unifi/externalsecret.yaml` from OpenBao key `infrastructure/external-dns-unifi`) is still valid — a stale/rotated API key would explain repeated 401s from the UDM. ExternalSecret status shows `SecretSynced: True` (last sync 38m ago), which only confirms OpenBao->K8s sync is healthy, NOT that the key itself is accepted by the UDM. - If the key is fine, this is likely NOT fixable purely in this repo: it points at UniFi controller (`udm.renner.uno`) instability/auth flakiness, which needs a human to check the UDM controller's own health/logs and possibly the `external-dns-unifi-webhook` image version (currently pinned to `v0.4.1`) for known issues with UniFi session auth under transient errors. --- <!-- alert-monitor:fingerprint=2b67dff384958744 --> _Filed automatically by the alert-monitor pipeline (2026-07-26). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-26 11:46:48 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#425 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/425 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#424