konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] KubePodCrashLooping (home-assistant) #438

New Issue
Closed
opened 2026-07-29 02:07:29 +00:00 by alert-monitor-bot · 3 comments
alert-monitor-bot commented 2026-07-29 02:07:29 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: KubePodCrashLooping (severity warning), namespace home-assistant, job kube-state-metrics
  • Workload: pod/zigbee2mqtt-7467986d94-fjrkw, container zigbee2mqtt, reason CrashLoopBackOff
  • Firing: 2026-07-29 01:44:57 UTC (single sample, then resolved — delivered as a blip)
  • Status now: pod is Running and functional again, but it has restarted 33 times, most recently ~01:58 UTC (kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt → 1/1 Running 33 (2m36s ago) 4d20h).

Evidence

The restart counter shows this is a new burst, not a chronic condition. kube_pod_container_status_restarts_total{namespace="home-assistant",container="zigbee2mqtt"} over the last 24 h (Prometheus query_range via the apiserver proxy) sat flat at 21 for the entire day, then:

01:30 UTC -> 24
02:00 UTC -> 33

12 restarts in ~30 minutes, starting exactly at 01:30 UTC.

context.txt corroborates:

home-assistant  25m  Warning  BackOff    pod/zigbee2mqtt-...  Back-off restarting failed container zigbee2mqtt
home-assistant  37s  Warning  Unhealthy  pod/zigbee2mqtt-...  Startup probe failed: dial tcp 10.244.1.44:8080: connection refused

The crash reason is unambiguous in the previous container's log (kubectl logs -n home-assistant zigbee2mqtt-7467986d94-fjrkw --previous --tail=40; timestamps are container-local Europe/Berlin = UTC+2):

[2026-07-29 03:58:17] info:  zh:zstack:znp: Opening TCP socket with slzb-zigbee.renner.uno:6638
[2026-07-29 03:58:22] error: zh:zstack:znp: Socket error Error: getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno
[2026-07-29 03:58:22] error: z2m: Failed to start zigbee-herdsman
[2026-07-29 03:58:22] error: z2m: Exiting...

EAI_AGAIN is a DNS timeout, not NXDOMAIN — the name was not resolvable at that instant. Five seconds after the socket attempt, Z2M calls process.exit.

The pod has since recovered on its own; current logs show normal MQTT publishing (kubectl logs ... --tail=12, entries through 04:04 local / 02:04 UTC).

Root cause

Two distinct causes, one of which is fixable here.

1. Trigger (external, verified): cluster DNS could not resolve slzb-zigbee.renner.uno during the incident window. CoreDNS on both replicas is logging upstream timeouts against 169.254.116.108 (the Talos host-DNS listener, hostDNS/forwardKubeDNSToHost in kubernetes/talos/patches/cluster.yaml:16), which forwards to the UDM:

[ERROR] plugin/errors: 2 api.getbring.com. A: read udp 10.244.2.17:47586->169.254.116.108:53: i/o timeout

The restart burst (01:30 UTC onward) lines up with the WAN/DNS incident tracked in [alert] NetbirdPublicEdgeWanDown (cluster) — file that issue for the uplink itself.

Caveat, stated as hypothesis: slzb-zigbee.renner.uno is a LAN name, resolved via the renner.uno stanza (CoreDNS → k8s-gateway 10.96.0.11 → fallthrough forward . 10.2.40.1 fd00::1), not via the .:53 upstream where the logged timeouts appear. I did not capture a CoreDNS log line for slzb-zigbee.renner.uno specifically, so "the same UDM failure broke this name too" is inference from timing, not direct proof. It is a consistent inference — the UDM is the terminus of both paths — but to settle it, capture during a burst: kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200 filtered for slzb-zigbee, or kubectl logs -n kube-system -l app.kubernetes.io/name=k8s-gateway --tail=100.

2. Amplifier (in-repo, verified by reading the manifests): a 5-second DNS blip destroys the Zigbee stack. kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:49 configures the coordinator by hostname:

serial:
  port: tcp://slzb-zigbee.renner.uno:6638
  adapter: zstack

Zigbee2MQTT does not retry adapter startup — it exits, Kubernetes restarts it, and with CrashLoopBackOff the backoff grows while the whole Zigbee network is offline. Given that issue #393 already documents the UDM's DNS as intermittently and repeatedly unavailable (kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67), depending on live DNS for a LAN device at process start is a standing fragility, not a one-off.

Where to fix it

Repo paths, most valuable first:

  • kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:49 — replace the hostname in serial.port with the SLZB-06's IP literal (or add a hostAliases entry in the Deployment) so coordinator startup does not depend on DNS at all. The device is on the IoT VLAN with a static/reserved lease; its IP is not recorded in this repo, so a human must supply it.

    ⚠️ Important: that ConfigMap is seed-only. Its own header (lines 7–12) states the init container copies it to /app/data/configuration.yaml only if that file does not exist, after which Z2M owns the file. Editing the ConfigMap alone will not change the running instance — the on-PVC configuration.yaml must also be edited (via the Z2M frontend or directly). A fix that touches only the ConfigMap will look merged and change nothing.

  • kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml:113 — the startupProbe (periodSeconds: 5, failureThreshold: 24) is generous enough, and is not the cause here; the container exits before the probe matters. Optionally add a dnsConfig with a longer timeout/higher attempts as defence in depth, but this is secondary to removing the DNS dependency.

  • No change is warranted to kubernetes/infrastructure/k8s-gateway/helmrelease.yaml or kubernetes/infrastructure/coredns-config/configmap.yaml — their cache / serve_stale 1h immediate mitigations for issue #393 are already in place and behaved as designed; they cannot help when the upstream is unreachable for longer than the cached entry's usefulness.

Why this is filed rather than suppressed as a symptom: the trigger is shared with the WAN issue, but the fix is different and lives in this repo, and KubePodCrashLooping (home-assistant) is a broad fingerprint — suppressing it would permanently silence any future crashloop in the home-assistant namespace, including a genuinely broken Zigbee coordinator.

Filed automatically by the alert-monitor pipeline (2026-07-29). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `KubePodCrashLooping` (severity `warning`), namespace `home-assistant`, job `kube-state-metrics` - **Workload:** `pod/zigbee2mqtt-7467986d94-fjrkw`, container `zigbee2mqtt`, reason `CrashLoopBackOff` - **Firing:** 2026-07-29 **01:44:57 UTC** (single sample, then resolved — delivered as a blip) - **Status now:** pod is `Running` and functional again, but it has restarted **33 times**, most recently **~01:58 UTC** (`kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt` → `1/1 Running 33 (2m36s ago) 4d20h`). ## Evidence The restart counter shows this is a new burst, not a chronic condition. `kube_pod_container_status_restarts_total{namespace="home-assistant",container="zigbee2mqtt"}` over the last 24 h (Prometheus `query_range` via the apiserver proxy) sat flat at **21** for the entire day, then: ``` 01:30 UTC -> 24 02:00 UTC -> 33 ``` **12 restarts in ~30 minutes**, starting exactly at 01:30 UTC. `context.txt` corroborates: ``` home-assistant 25m Warning BackOff pod/zigbee2mqtt-... Back-off restarting failed container zigbee2mqtt home-assistant 37s Warning Unhealthy pod/zigbee2mqtt-... Startup probe failed: dial tcp 10.244.1.44:8080: connection refused ``` The crash reason is unambiguous in the previous container's log (`kubectl logs -n home-assistant zigbee2mqtt-7467986d94-fjrkw --previous --tail=40`; timestamps are container-local Europe/Berlin = UTC+2): ``` [2026-07-29 03:58:17] info: zh:zstack:znp: Opening TCP socket with slzb-zigbee.renner.uno:6638 [2026-07-29 03:58:22] error: zh:zstack:znp: Socket error Error: getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno [2026-07-29 03:58:22] error: z2m: Failed to start zigbee-herdsman [2026-07-29 03:58:22] error: z2m: Exiting... ``` `EAI_AGAIN` is a DNS **timeout**, not `NXDOMAIN` — the name was not resolvable at that instant. Five seconds after the socket attempt, Z2M calls `process.exit`. The pod has since recovered on its own; current logs show normal MQTT publishing (`kubectl logs ... --tail=12`, entries through 04:04 local / 02:04 UTC). ## Root cause Two distinct causes, one of which is fixable here. **1. Trigger (external, verified):** cluster DNS could not resolve `slzb-zigbee.renner.uno` during the incident window. CoreDNS on both replicas is logging upstream timeouts against `169.254.116.108` (the Talos host-DNS listener, `hostDNS`/`forwardKubeDNSToHost` in `kubernetes/talos/patches/cluster.yaml:16`), which forwards to the UDM: ``` [ERROR] plugin/errors: 2 api.getbring.com. A: read udp 10.244.2.17:47586->169.254.116.108:53: i/o timeout ``` The restart burst (01:30 UTC onward) lines up with the WAN/DNS incident tracked in **`[alert] NetbirdPublicEdgeWanDown (cluster)`** — file that issue for the uplink itself. *Caveat, stated as hypothesis:* `slzb-zigbee.renner.uno` is a LAN name, resolved via the `renner.uno` stanza (CoreDNS → k8s-gateway `10.96.0.11` → fallthrough `forward . 10.2.40.1 fd00::1`), **not** via the `.:53` upstream where the logged timeouts appear. I did not capture a CoreDNS log line for `slzb-zigbee.renner.uno` specifically, so "the same UDM failure broke this name too" is inference from timing, not direct proof. It is a consistent inference — the UDM is the terminus of both paths — but to settle it, capture during a burst: `kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200` filtered for `slzb-zigbee`, or `kubectl logs -n kube-system -l app.kubernetes.io/name=k8s-gateway --tail=100`. **2. Amplifier (in-repo, verified by reading the manifests):** a **5-second DNS blip destroys the Zigbee stack.** `kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:49` configures the coordinator by hostname: ```yaml serial: port: tcp://slzb-zigbee.renner.uno:6638 adapter: zstack ``` Zigbee2MQTT does not retry adapter startup — it exits, Kubernetes restarts it, and with `CrashLoopBackOff` the backoff grows while the whole Zigbee network is offline. Given that issue #393 already documents the UDM's DNS as *intermittently and repeatedly* unavailable (`kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67`), depending on live DNS for a LAN device at process start is a standing fragility, not a one-off. ## Where to fix it Repo paths, most valuable first: - **`kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:49`** — replace the hostname in `serial.port` with the SLZB-06's IP literal (or add a `hostAliases` entry in the Deployment) so coordinator startup does not depend on DNS at all. The device is on the IoT VLAN with a static/reserved lease; its IP is not recorded in this repo, so a human must supply it. ⚠️ **Important:** that ConfigMap is **seed-only**. Its own header (lines 7–12) states the init container copies it to `/app/data/configuration.yaml` *only if that file does not exist*, after which Z2M owns the file. Editing the ConfigMap alone will **not** change the running instance — the on-PVC `configuration.yaml` must also be edited (via the Z2M frontend or directly). A fix that touches only the ConfigMap will look merged and change nothing. - **`kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml:113`** — the `startupProbe` (`periodSeconds: 5`, `failureThreshold: 24`) is generous enough, and is *not* the cause here; the container exits before the probe matters. Optionally add a `dnsConfig` with a longer `timeout`/higher `attempts` as defence in depth, but this is secondary to removing the DNS dependency. - No change is warranted to `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml` or `kubernetes/infrastructure/coredns-config/configmap.yaml` — their `cache` / `serve_stale 1h immediate` mitigations for issue #393 are already in place and behaved as designed; they cannot help when the upstream is unreachable for longer than the cached entry's usefulness. **Why this is filed rather than suppressed as a symptom:** the trigger is shared with the WAN issue, but the fix is different and lives in this repo, and `KubePodCrashLooping (home-assistant)` is a broad fingerprint — suppressing it would permanently silence *any* future crashloop in the home-assistant namespace, including a genuinely broken Zigbee coordinator. --- <!-- alert-monitor:fingerprint=a25507816c87c745 --> _Filed automatically by the alert-monitor pipeline (2026-07-29). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-29 03:46:34 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change made (issue #438, KubePodCrashLooping / home-assistant)

The alert's root cause is now confirmed with direct evidence — the issue's
own stated hypothesis was correct, and I settled the part it flagged as
unproven. But the one repo change that would actually fix it requires a value
this repository does not record and that I could not obtain from any
read-only source. Guessing it would take the entire Zigbee network down
permanently, so I made no edits. The recommended manual action is below.

What I verified live this run

1. The pod recovered and has stayed recovered.

./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt
→ zigbee2mqtt-7467986d94-fjrkw 1/1 Running 33 (103m ago) 4d22h

Container status (-o jsonpath on containerStatuses[0]):

  • last termination — exitCode: 1, reason: Error, finishedAt: 2026-07-29T01:58:22Z
  • current container — startedAt: 2026-07-29T01:59:43Z

No restart since 01:59:43 UTC (≈1 h 45 m stable as of 03:44 UTC). The restart
counter is 33, matching the issue.

2. The DNS failure for this exact name is proven, not inferred.

The issue explicitly flagged this as its weakest link: "I did not capture a
CoreDNS log line for slzb-zigbee.renner.uno specifically… inference from
timing, not direct proof." Loki has it. Query:

./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/loki:3100/proxy/loki/api/v1/query_range?query=%7Bnamespace%3D%22kube-system%22%7D%20%7C%3D%20%22slzb%22&start=1785288300000000000&end=1785291000000000000&limit=30&direction=backward"

→ k8s-gateway (both replicas), during the burst window:

[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: dial tcp 10.2.40.1:53: connect: connection refused
[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: dial tcp [fd00::1]:53: connect: connection refused
[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: read udp 10.244.1.159:43296->10.2.40.1:53: read: connection refused
[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. AAAA: dial tcp [fd00::1]:53: i/o timeout

Timestamps of the failures: 01:31:36, 01:34:35, 01:51:47, 01:57:20 and
01:58:17.277 UTC.

3. The causal chain is exact — 9 milliseconds. Same query against the Z2M
pod ({namespace="home-assistant",pod=~"zigbee2mqtt.*"} |= "6638"):

01:58:17.268 UTC  zh:zstack:znp: Opening TCP socket with slzb-zigbee.renner.uno:6638
01:58:17.277 UTC  [k8s-gateway] slzb-zigbee.renner.uno. A: dial tcp 10.2.40.1:53: connect: connection refused
01:58:22    UTC  error: Socket error getaddrinfo EAI_AGAIN → z2m: Exiting...

12 Opening TCP socket attempts between 01:28:38 and 01:59:44 UTC — the
"12 restarts in ~30 minutes" the issue measured from kube_pod_container_status_restarts_total.

Correction to the issue's caveat: the failing path is exactly the one it
hypothesised — renner.uno:53 → k8s-gateway (10.96.0.11) → fallthrough
forward . 10.2.40.1 fd00::1 → UDM. It is not the .:53 /
169.254.116.108 upstream where the api.getbring.com timeouts appear.
Different path, same terminus. The failure mode (connection refused in
bursts on both 10.2.40.1 and fd00::1) is the dnsmasq behaviour already
documented at kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:69
for issue #393.

4. The renner.uno path is healthy again; the WAN path is not. The same
Loki query over 02:10 → 03:44 UTC returns zero results — no slzb
resolution failures since the burst ended. But
./.ci-diag/kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 still
shows current upstream failures on the internet path:

[ERROR] plugin/errors: 2 api.getbring.com. A: read udp 10.244.2.17:47586->169.254.116.108:53: i/o timeout
[ERROR] plugin/errors: 2 fsn1.your-objectstorage.com. A: read udp 10.244.2.17:44575->169.254.116.108:53: i/o timeout

So the underlying DNS instability is ongoing; it simply is not currently
hitting the renner.uno stanza. A recurrence of this crashloop should be
expected until the coordinator's DNS dependency is removed.

5. No config drift. ./.ci-diag/kubectl get deployment zigbee2mqtt -n home-assistant -o jsonpath=...
→ image=ghcr.io/koenkk/zigbee2mqtt:2.12.1, hostAliases= (empty),
dnsConfig= (empty), dnsPolicy=ClusterFirst, availableReplicas=1. The
live Deployment matches kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml.

Why no repo change

The fix needs the SLZB-06's IP address, and it is not obtainable. I
searched for it and came up empty:

  • Repo-wide grep for slzb/SLZB → only two hits, both the hostname:
    kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:50 and
    kubernetes/apps/home-assistant/zigbee2mqtt-migration-runbook.md:13.
  • Loki, all Z2M logs, filter 6638 → Z2M logs only
    Opening TCP socket with slzb-zigbee.renner.uno:6638; it never logs the
    resolved address.
  • The device is not in the cluster, so no read-only kubectl call reaches it.

Hardcoding a guessed IP into a hostAliases entry would not blip the Zigbee
network — it would break it silently and permanently, which is strictly
worse than the intermittent 5-second failure being fixed. That is not a
trade I will make on an unverifiable value.

The issue's top-ranked fix is a guaranteed no-op. It lists
zigbee2mqtt-config.yaml:49 as "most valuable first" while itself warning the
ConfigMap is seed-only. The warning is correct and I confirmed the mechanism
in the manifest: zigbee2mqtt-deployment.yaml:56 guards the copy with
if [ ! -f /app/data/configuration.yaml ], and the PVC was seeded at the
2026-07-23 cutover. A PR touching only that ConfigMap would merge, reconcile
cleanly, and change nothing about the running instance — a cosmetic change,
which the fix rules forbid.

dnsConfig tuning was considered and rejected. It would force a
strategy: Recreate restart of a currently-healthy live Zigbee stack for an
unverified benefit: the Z2M image is Node-on-Alpine (musl), and musl's
resolver has historically ignored options timeout:/attempts:, so the
change may well have no effect at all. The issue itself ranks this as
secondary. Not worth bouncing the Zigbee network to find out.

No change to the DNS mitigations. k8s-gateway/helmrelease.yaml:93 and
coredns-config/configmap.yaml:30 already carry serve_stale 1h immediate
from issue #393 and are out of scope here. See the open question in Notes.

The trigger is external. The UDM refusing DNS on 10.2.40.1/fd00::1
is not fixable in this repo and belongs to the NetbirdPublicEdgeWanDown
issue and #393.

Recommended manual action

Step 1 — get the IP. Read the SLZB-06's address off the UniFi client list
(IoT VLAN, reserved lease) or the device's own web UI. Confirm the reservation
is static before pinning it.

Step 2 — pin it via hostAliases, not the ConfigMap. This is the better
of the two options the issue offered, and worth stating plainly: hostAliases
writes /etc/hosts in the pod, which getaddrinfo consults before DNS.
The on-PVC configuration.yaml keeps its existing
port: tcp://slzb-zigbee.renner.uno:6638 and resolves locally with zero DNS
traffic — so no PVC edit is needed at all, which sidesteps the seed-only
problem entirely rather than working around it.

In kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml, under
spec.template.spec (alongside enableServiceLinks, line 27):

      # The SLZB-06 coordinator is pinned to /etc/hosts so herdsman's adapter
      # connect never touches DNS: Z2M exits (no retry) if the name fails to
      # resolve, and the UDM refuses DNS in bursts (issue #393, #438 —
      # 12 restarts in 30 min on 2026-07-29). Keep in sync with the device's
      # static lease on the IoT VLAN.
      hostAliases:
        - ip: "<SLZB-06 IP>"
          hostnames:
            - slzb-zigbee.renner.uno

Also update zigbee2mqtt-config.yaml's comment block so a future PVC reseed
does not silently reintroduce the dependency. Total change ≈ 8 lines.

Step 3 — verify after Flux reconciles. strategy: Recreate means the pod
is deleted and recreated, so the Zigbee network drops for the restart window
— do this at a quiet time.

./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt
./.ci-diag/kubectl logs -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt --tail=30

Expect 1/1 Running, restart count reset to 0, and
Opening TCP socket with slzb-zigbee.renner.uno:6638 followed by a successful
herdsman start rather than EAI_AGAIN. KubePodCrashLooping for this
workload should not recur on the next UDM DNS blip.

Notes

  • What the issue got right: the root cause, the crash mechanism (Z2M does
    not retry adapter startup — it calls process.exit), the seed-only
    ConfigMap warning, and the judgement not to suppress the alert (agreed —
    KubePodCrashLooping (home-assistant) is a namespace-wide fingerprint).
    Its explicitly-flagged uncertainty is now resolved in its favour.
  • What it got wrong: the ranking. It puts the ConfigMap edit first and
    hostAliases in parentheses, when hostAliases is the only one of the two
    that works without a manual PVC edit.
  • Open question worth a follow-up, not a change here: both
    coredns-config/configmap.yaml:30 and k8s-gateway/helmrelease.yaml:93
    carry serve_stale 1h immediate, yet Z2M still got EAI_AGAIN. Z2M had
    been querying this name every ~90 s throughout the burst, so a cached entry
    should have existed. I am not claiming the mitigation is broken — with
    serve_stale immediate, CoreDNS serves the stale answer and refreshes in
    the background, so those [ERROR] lines could be failed background
    refreshes rather than client-visible SERVFAILs. But the client demonstrably
    did not get a usable answer at 01:58:17, so something did not hold. Likely
    candidates: the AAAA leg failing independently of A (musl returns
    EAI_AGAIN if either leg errors), or a cold cache on the k8s-gateway
    replica that received the query. Settling it needs a capture during a live
    burst, and it does not change the recommendation above — pinning the host
    removes the dependency on any of this working.
  • opentofu/dns_renner_uno.tf is not the answer. It manages
    unifi_dns_record resources, i.e. records served by the UDM — the very
    resolver that was refusing connections. Adding slzb-zigbee there would not
    have prevented this.
  • Nothing in the issue asked me to touch CI config, credentials, or anything
    outside the edit scope.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

# Assessment: no repo change made (issue #438, `KubePodCrashLooping` / home-assistant) The alert's root cause is now **confirmed with direct evidence** — the issue's own stated hypothesis was correct, and I settled the part it flagged as unproven. But the one repo change that would actually fix it requires a value this repository does not record and that I could not obtain from any read-only source. Guessing it would take the entire Zigbee network down permanently, so I made no edits. The recommended manual action is below. ## What I verified live this run **1. The pod recovered and has stayed recovered.** `./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt` → `zigbee2mqtt-7467986d94-fjrkw 1/1 Running 33 (103m ago) 4d22h` Container status (`-o jsonpath` on `containerStatuses[0]`): - last termination — `exitCode: 1`, `reason: Error`, `finishedAt: 2026-07-29T01:58:22Z` - current container — `startedAt: 2026-07-29T01:59:43Z` No restart since 01:59:43 UTC (≈1 h 45 m stable as of 03:44 UTC). The restart counter is 33, matching the issue. **2. The DNS failure for this exact name is proven, not inferred.** The issue explicitly flagged this as its weakest link: *"I did not capture a CoreDNS log line for `slzb-zigbee.renner.uno` specifically… inference from timing, not direct proof."* Loki has it. Query: ``` ./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/loki:3100/proxy/loki/api/v1/query_range?query=%7Bnamespace%3D%22kube-system%22%7D%20%7C%3D%20%22slzb%22&start=1785288300000000000&end=1785291000000000000&limit=30&direction=backward" ``` → `k8s-gateway` (both replicas), during the burst window: ``` [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: dial tcp 10.2.40.1:53: connect: connection refused [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: dial tcp [fd00::1]:53: connect: connection refused [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: read udp 10.244.1.159:43296->10.2.40.1:53: read: connection refused [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. AAAA: dial tcp [fd00::1]:53: i/o timeout ``` Timestamps of the failures: 01:31:36, 01:34:35, 01:51:47, 01:57:20 and **01:58:17.277 UTC**. **3. The causal chain is exact — 9 milliseconds.** Same query against the Z2M pod (`{namespace="home-assistant",pod=~"zigbee2mqtt.*"} |= "6638"`): ``` 01:58:17.268 UTC zh:zstack:znp: Opening TCP socket with slzb-zigbee.renner.uno:6638 01:58:17.277 UTC [k8s-gateway] slzb-zigbee.renner.uno. A: dial tcp 10.2.40.1:53: connect: connection refused 01:58:22 UTC error: Socket error getaddrinfo EAI_AGAIN → z2m: Exiting... ``` 12 `Opening TCP socket` attempts between 01:28:38 and 01:59:44 UTC — the "12 restarts in ~30 minutes" the issue measured from `kube_pod_container_status_restarts_total`. **Correction to the issue's caveat:** the failing path is exactly the one it hypothesised — `renner.uno:53` → k8s-gateway (`10.96.0.11`) → fallthrough `forward . 10.2.40.1 fd00::1` → **UDM**. It is *not* the `.:53` / `169.254.116.108` upstream where the `api.getbring.com` timeouts appear. Different path, same terminus. The failure mode (`connection refused` in bursts on both `10.2.40.1` and `fd00::1`) is the dnsmasq behaviour already documented at `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:69` for issue #393. **4. The renner.uno path is healthy again; the WAN path is not.** The same Loki query over 02:10 → 03:44 UTC returns **zero** results — no `slzb` resolution failures since the burst ended. But `./.ci-diag/kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100` still shows current upstream failures on the *internet* path: ``` [ERROR] plugin/errors: 2 api.getbring.com. A: read udp 10.244.2.17:47586->169.254.116.108:53: i/o timeout [ERROR] plugin/errors: 2 fsn1.your-objectstorage.com. A: read udp 10.244.2.17:44575->169.254.116.108:53: i/o timeout ``` So the underlying DNS instability is **ongoing**; it simply is not currently hitting the `renner.uno` stanza. A recurrence of this crashloop should be expected until the coordinator's DNS dependency is removed. **5. No config drift.** `./.ci-diag/kubectl get deployment zigbee2mqtt -n home-assistant -o jsonpath=...` → `image=ghcr.io/koenkk/zigbee2mqtt:2.12.1`, `hostAliases=` (empty), `dnsConfig=` (empty), `dnsPolicy=ClusterFirst`, `availableReplicas=1`. The live Deployment matches `kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml`. ## Why no repo change **The fix needs the SLZB-06's IP address, and it is not obtainable.** I searched for it and came up empty: - Repo-wide grep for `slzb`/`SLZB` → only two hits, both the **hostname**: `kubernetes/apps/home-assistant/zigbee2mqtt-config.yaml:50` and `kubernetes/apps/home-assistant/zigbee2mqtt-migration-runbook.md:13`. - Loki, all Z2M logs, filter `6638` → Z2M logs only `Opening TCP socket with slzb-zigbee.renner.uno:6638`; it never logs the resolved address. - The device is not in the cluster, so no read-only `kubectl` call reaches it. Hardcoding a guessed IP into a `hostAliases` entry would not blip the Zigbee network — it would break it **silently and permanently**, which is strictly worse than the intermittent 5-second failure being fixed. That is not a trade I will make on an unverifiable value. **The issue's top-ranked fix is a guaranteed no-op.** It lists `zigbee2mqtt-config.yaml:49` as "most valuable first" while itself warning the ConfigMap is seed-only. The warning is correct and I confirmed the mechanism in the manifest: `zigbee2mqtt-deployment.yaml:56` guards the copy with `if [ ! -f /app/data/configuration.yaml ]`, and the PVC was seeded at the 2026-07-23 cutover. A PR touching only that ConfigMap would merge, reconcile cleanly, and change nothing about the running instance — a cosmetic change, which the fix rules forbid. **`dnsConfig` tuning was considered and rejected.** It would force a `strategy: Recreate` restart of a currently-healthy live Zigbee stack for an unverified benefit: the Z2M image is Node-on-Alpine (musl), and musl's resolver has historically ignored `options timeout:`/`attempts:`, so the change may well have no effect at all. The issue itself ranks this as secondary. Not worth bouncing the Zigbee network to find out. **No change to the DNS mitigations.** `k8s-gateway/helmrelease.yaml:93` and `coredns-config/configmap.yaml:30` already carry `serve_stale 1h immediate` from issue #393 and are out of scope here. See the open question in Notes. **The trigger is external.** The UDM refusing DNS on `10.2.40.1`/`fd00::1` is not fixable in this repo and belongs to the `NetbirdPublicEdgeWanDown` issue and #393. ## Recommended manual action **Step 1 — get the IP.** Read the SLZB-06's address off the UniFi client list (IoT VLAN, reserved lease) or the device's own web UI. Confirm the reservation is static before pinning it. **Step 2 — pin it via `hostAliases`, not the ConfigMap.** This is the better of the two options the issue offered, and worth stating plainly: `hostAliases` writes `/etc/hosts` in the pod, which `getaddrinfo` consults **before** DNS. The on-PVC `configuration.yaml` keeps its existing `port: tcp://slzb-zigbee.renner.uno:6638` and resolves locally with zero DNS traffic — **so no PVC edit is needed at all**, which sidesteps the seed-only problem entirely rather than working around it. In `kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml`, under `spec.template.spec` (alongside `enableServiceLinks`, line 27): ```yaml # The SLZB-06 coordinator is pinned to /etc/hosts so herdsman's adapter # connect never touches DNS: Z2M exits (no retry) if the name fails to # resolve, and the UDM refuses DNS in bursts (issue #393, #438 — # 12 restarts in 30 min on 2026-07-29). Keep in sync with the device's # static lease on the IoT VLAN. hostAliases: - ip: "<SLZB-06 IP>" hostnames: - slzb-zigbee.renner.uno ``` Also update `zigbee2mqtt-config.yaml`'s comment block so a future PVC reseed does not silently reintroduce the dependency. Total change ≈ 8 lines. **Step 3 — verify after Flux reconciles.** `strategy: Recreate` means the pod is deleted and recreated, so the Zigbee network drops for the restart window — do this at a quiet time. ``` ./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt ./.ci-diag/kubectl logs -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt --tail=30 ``` Expect `1/1 Running`, restart count reset to `0`, and `Opening TCP socket with slzb-zigbee.renner.uno:6638` followed by a successful herdsman start rather than `EAI_AGAIN`. `KubePodCrashLooping` for this workload should not recur on the next UDM DNS blip. ## Notes - **What the issue got right:** the root cause, the crash mechanism (Z2M does not retry adapter startup — it calls `process.exit`), the seed-only ConfigMap warning, and the judgement not to suppress the alert (agreed — `KubePodCrashLooping (home-assistant)` is a namespace-wide fingerprint). Its explicitly-flagged uncertainty is now resolved in its favour. - **What it got wrong:** the ranking. It puts the ConfigMap edit first and `hostAliases` in parentheses, when `hostAliases` is the only one of the two that works without a manual PVC edit. - **Open question worth a follow-up, not a change here:** both `coredns-config/configmap.yaml:30` and `k8s-gateway/helmrelease.yaml:93` carry `serve_stale 1h immediate`, yet Z2M still got `EAI_AGAIN`. Z2M had been querying this name every ~90 s throughout the burst, so a cached entry should have existed. I am **not** claiming the mitigation is broken — with `serve_stale immediate`, CoreDNS serves the stale answer and refreshes in the background, so those `[ERROR]` lines could be failed background refreshes rather than client-visible SERVFAILs. But the client demonstrably did not get a usable answer at 01:58:17, so something did not hold. Likely candidates: the AAAA leg failing independently of A (musl returns `EAI_AGAIN` if either leg errors), or a cold cache on the k8s-gateway replica that received the query. Settling it needs a capture during a live burst, and it does not change the recommendation above — pinning the host removes the dependency on any of this working. - **`opentofu/dns_renner_uno.tf` is not the answer.** It manages `unifi_dns_record` resources, i.e. records served *by the UDM* — the very resolver that was refusing connections. Adding `slzb-zigbee` there would not have prevented this. - Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit scope. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-08-16 21:16:29 +00:00
Member
Copy Link
Copy Source

Quiet now, but the recommended fix should not be applied as written.

Current state (2026-08-16): zigbee2mqtt-84c586688b-rw7ff is 2/2 Running. The container has 14 restarts and its last termination was 2026-08-09 11:44:25Z — seven days ago. Loki shows zero CoreDNS i/o timeout lines against 169.254.116.108 in the last 24h. The trigger condition is not currently present.

Why I am not implementing the headline fix. The issue proposes replacing the hostname in serial.port with the SLZB-06's IP literal, or adding hostAliases. Both pin a device identity to an address, which is against standing policy here: addresses are lease state, not identity, and the fix would rot silently the next time the device's lease changes — turning a self-healing 5-minute DNS blip into a permanent, silent outage of the whole Zigbee network. Worse failure mode than the one being fixed.

It also would not work as specified. The issue itself notes zigbee2mqtt-config.yaml is seed-only — the init container copies it only if /app/data/configuration.yaml is absent. So editing that ConfigMap changes nothing on the running instance, and making it take effect means hand-editing the PVC, which is exactly the untracked drift this repo avoids. The proposed change is both against policy and inert.

What actually fixes this. The trigger is cluster DNS failing to resolve a LAN name, which is the subject of #393 and #436 (and the WAN side, #437). This issue is a symptom of those. Z2M not retrying adapter startup is upstream behaviour and not fixable here.

The one in-repo mitigation that keeps the hostname is a dnsConfig on the Deployment with more attempts and a shorter per-attempt timeout, so a brief blip is ridden out rather than fatal. I have not added it: there is no dnsConfig anywhere in kubernetes/apps/ today, so it would be a new pattern introduced speculatively against a trigger that has not fired in a week. Worth doing deliberately if the DNS work does not land, not as a reflex.

Leaving this open and dependent on #393 / #436. If DNS stops blipping, this stops crashing.

Quiet now, but the recommended fix should not be applied as written. **Current state (2026-08-16):** `zigbee2mqtt-84c586688b-rw7ff` is 2/2 Running. The container has 14 restarts and its last termination was **2026-08-09 11:44:25Z — seven days ago**. Loki shows **zero** CoreDNS `i/o timeout` lines against `169.254.116.108` in the last 24h. The trigger condition is not currently present. **Why I am not implementing the headline fix.** The issue proposes replacing the hostname in `serial.port` with the SLZB-06's IP literal, or adding `hostAliases`. Both pin a device identity to an address, which is against standing policy here: addresses are lease state, not identity, and the fix would rot silently the next time the device's lease changes — turning a self-healing 5-minute DNS blip into a permanent, silent outage of the whole Zigbee network. Worse failure mode than the one being fixed. It also would not work as specified. The issue itself notes `zigbee2mqtt-config.yaml` is **seed-only** — the init container copies it only if `/app/data/configuration.yaml` is absent. So editing that ConfigMap changes nothing on the running instance, and making it take effect means hand-editing the PVC, which is exactly the untracked drift this repo avoids. The proposed change is both against policy and inert. **What actually fixes this.** The trigger is cluster DNS failing to resolve a LAN name, which is the subject of #393 and #436 (and the WAN side, #437). This issue is a *symptom* of those. Z2M not retrying adapter startup is upstream behaviour and not fixable here. The one in-repo mitigation that keeps the hostname is a `dnsConfig` on the Deployment with more attempts and a shorter per-attempt timeout, so a brief blip is ridden out rather than fatal. I have not added it: there is no `dnsConfig` anywhere in `kubernetes/apps/` today, so it would be a new pattern introduced speculatively against a trigger that has not fired in a week. Worth doing deliberately if the DNS work does not land, not as a reflex. Leaving this open and dependent on #393 / #436. If DNS stops blipping, this stops crashing.
claude-bot commented 2026-08-16 21:20:02 +00:00
Member
Copy Link
Copy Source

Closing per owner decision.

State at close: zigbee2mqtt 2/2 Running, last container termination 2026-08-09 11:44:25Z, no CoreDNS upstream timeouts in the preceding 24h.

The DNS fragility behind it is tracked in #393 and #436. If Z2M starts crashlooping on EAI_AGAIN again, alert-monitor files a fresh issue with current data.

Closing per owner decision. State at close: `zigbee2mqtt` 2/2 Running, last container termination 2026-08-09 11:44:25Z, no CoreDNS upstream timeouts in the preceding 24h. The DNS fragility behind it is tracked in #393 and #436. If Z2M starts crashlooping on `EAI_AGAIN` again, alert-monitor files a fresh issue with current data.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#438