konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests Packages Projects Releases 1 Wiki Activity

[alert] NetbirdPublicEdgeWanDown (cluster) #437

New Issue
Closed
opened 2026-07-29 02:07:29 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-07-29 02:07:29 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: NetbirdPublicEdgeWanDown (severity critical, cluster-scoped, no namespace)
  • Rule: max(probe_success{job="blackbox-edge-wan"}) == 0 for: 5m — defined in kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:28
  • Firing window: 2026-07-29 01:33:57 → 01:36:57 UTC (alert resolved, hence delivered as a blip). Underlying probe failure began 01:28:57 UTC, i.e. ~8 minutes with zero reachable edge members.
  • Status now: the alert is not active, but the underlying condition is still recurring — see below.

Evidence

ALERTS history (kubectl get --raw .../prometheus:9090/proxy/api/v1/query_range on ALERTS{alertname=~"NetbirdPublicEdge.*"}) confirms four consecutive firing samples at 01:33:57–01:36:57 UTC.

probe_success{job="blackbox-edge-wan"} over the same range (same query API) shows all three IP-pinned members going to 0 simultaneously and recovering simultaneously:

member 01:28:57 01:36:57 01:57:57 02:00:57
https://45.157.177.26 (proxy-2 v4) 0 0 0 0
https://[2a03:4000:28:f6e:...:9a29] 0 1 0 1
https://[2a03:4000:4b:c1e:...:8f85] 0 1 0 0

At 1-minute resolution the flapping is ongoing: all three members were simultaneously 0 again for roughly 01:57–01:58 UTC and again 02:04–02:07 UTC, recovering in between. These bursts are ~3–4 min long, just under the rule's for: 5m, which is why the alert only latched once.

The decisive datapoint: querying probe_success for every job over the incident window shows the failures are not specific to the NetBird edge:

  • Failed together in the same windows: all 3 blackbox-edge-wan members, http://proxy-{1,2}.renner.uno:9000/health, proxy-{1,2}.renner.uno:33080 (TCP relays), https://netbird.renner.uno, and https://fsn1.your-objectstorage.com — Hetzner object storage, which has nothing to do with NetBird or netcup.
  • Never dropped below 1: every target served from inside the cluster — home.renner.uno, gitea, immich, auth, dash, alerts, prometheus.renner.uno, and the two *.monitoring.svc.cluster.local:*/-/healthy probes.

So: 100% of probes that leave the LAN for the public internet failed; 100% of probes that stay inside the cluster succeeded.

DNS is ruled out as the mechanism for these particular probes — probe_dns_lookup_time_seconds{job="blackbox-edge-wan"} is ~2.8 µs for all three targets (they are IP literals, no resolution performed). These are pure IP-path failures.

Corroborating symptoms from the same minutes, in context.txt:

  • ~20 ReadOperationFailed events on flux-system ImageRepositories: dial tcp: lookup ghcr.io on [fd10:96::a]:53: server misbehaving / i/o timeout.
  • home-assistant BackOff on zigbee2mqtt (filed separately, see [alert] KubePodCrashLooping (home-assistant)).

And live CoreDNS logs on both replicas (kubectl logs -n kube-system coredns-7859998f6-{dhf28,xxfr6} --tail=25) are full of upstream timeouts, still occurring at the time of writing:

[ERROR] plugin/errors: 2 fsn1.your-objectstorage.com. A: read udp 10.244.2.17:44575->169.254.116.108:53: i/o timeout
[ERROR] plugin/errors: 2 api.getbring.com. AAAA: read udp 10.244.0.106:47787->169.254.116.108:53: i/o timeout
[ERROR] plugin/errors: 2 mail-api.proton.me. AAAA: read udp 10.244.0.106:37161->169.254.116.108:53: i/o timeout

169.254.116.108 is the Talos host-DNS listener (hostDNS.enabled + forwardKubeDNSToHost: true, kubernetes/talos/patches/cluster.yaml:16), which forwards to the node's configured resolver — the UDM at 10.2.40.1 / fd00::1.

Root cause

Verified: the public edge itself is almost certainly not the failing component. The alert's own description tells a story about the netcup proxies ("Check the netcup proxies (netbird-proxy containers), the mesh path to the cluster, and cluster Traefik") — that generic story does not match the data. Three independent edge members on two different netcup hosts, across both IP families, plus a Hetzner VPS and Hetzner S3, cannot plausibly fail and recover in lockstep on 3-minute boundaries. A single shared component in front of all of them can.

Hypothesis (not yet directly verified): the shared component is the UDM (10.2.40.1 / fd00::1) in its dual role as default gateway and recursive resolver — either its WAN uplink or the device itself is flapping. This fits every observation: IP-literal egress fails (gateway), upstream recursion times out (resolver), and in-cluster traffic is untouched.

This is consistent with prior documented behaviour — kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67 records issue #393: "The UDM's dnsmasq refuses connections in bursts — 330 'connection refused' errors in the last 3000 log lines here, on BOTH 10.2.40.1 and fd00::1." The serve_stale/cache mitigations there address the DNS symptom only; they cannot help IP-level egress.

Commands that would settle it (none are runnable from the CI ServiceAccount — they need host or UDM access):

  • UDM UI/SSH: WAN uplink event log and ISP session state around 01:28–01:36 UTC and 01:57 / 02:04 UTC.
  • From a LAN host (not the cluster): mtr -rw 45.157.177.26 and dig @10.2.40.1 ghcr.io during a burst, to separate "UDM dead" from "UDM alive, WAN down".
  • If it turns out the UDM is fine, the next suspect is the Talos nodes' egress path (Cilium masquerading); cilium-dbg on a node would be the check.

Where to fix it

This is NOT fixable in this repository. No manifest change can restore a failing uplink. The repo files below are the right places to look for context, not the fix:

  • kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml — the alert definitions. Worth noting the current for: 5m means the ongoing 3–4 minute all-members outages are invisible; if this uplink instability persists, consider a companion rule with a shorter for: or an avg_over_time form so repeated sub-threshold flapping is still surfaced.
  • kubernetes/apps/monitoring-rules/probes-blackbox.yaml — probe target definitions (blackbox-edge-wan at line 72).
  • kubernetes/infrastructure/k8s-gateway/helmrelease.yaml and kubernetes/infrastructure/coredns-config/configmap.yaml — the existing issue #393 DNS mitigations, which are working as designed but only cover DNS.

What a human should do:

  1. Check the UDM's WAN/uplink health and ISP line status for 2026-07-29 01:28 UTC onward — the flapping was still active at 02:07 UTC.
  2. If the uplink is clean, escalate to the Talos/Cilium egress path.
  3. Consider whether an ICMP/TCP probe of the UDM itself (10.2.40.1) belongs in probes-blackbox.yaml, so "our own gateway is down" is directly observable instead of inferred from six downstream alerts. Today the only UDM check is a DNS-only Gatus check (kubernetes/apps/gatus/configmap.yaml:206), which would not have distinguished this.

Related

[alert] KubePodCrashLooping (home-assistant) was filed separately — its trigger is the DNS half of this same incident, but it also exposes an independently fixable brittleness in the zigbee2mqtt deployment, so it is not merely a duplicate of this issue.

Filed automatically by the alert-monitor pipeline (2026-07-29). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `NetbirdPublicEdgeWanDown` (severity `critical`, cluster-scoped, no namespace) - **Rule:** `max(probe_success{job="blackbox-edge-wan"}) == 0` `for: 5m` — defined in `kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:28` - **Firing window:** 2026-07-29 **01:33:57 → 01:36:57 UTC** (alert resolved, hence delivered as a blip). Underlying probe failure began **01:28:57 UTC**, i.e. ~8 minutes with *zero* reachable edge members. - **Status now:** the alert is not active, but the underlying condition **is still recurring** — see below. ## Evidence `ALERTS` history (`kubectl get --raw .../prometheus:9090/proxy/api/v1/query_range` on `ALERTS{alertname=~"NetbirdPublicEdge.*"}`) confirms four consecutive firing samples at 01:33:57–01:36:57 UTC. `probe_success{job="blackbox-edge-wan"}` over the same range (same query API) shows **all three** IP-pinned members going to 0 simultaneously and recovering simultaneously: | member | 01:28:57 | 01:36:57 | 01:57:57 | 02:00:57 | |---|---|---|---|---| | `https://45.157.177.26` (proxy-2 v4) | 0 | 0 | 0 | 0 | | `https://[2a03:4000:28:f6e:...:9a29]` | 0 | 1 | 0 | 1 | | `https://[2a03:4000:4b:c1e:...:8f85]` | 0 | 1 | 0 | 0 | At 1-minute resolution the flapping is **ongoing**: all three members were simultaneously 0 again for roughly 01:57–01:58 UTC and again 02:04–02:07 UTC, recovering in between. These bursts are ~3–4 min long, just under the rule's `for: 5m`, which is why the alert only latched once. **The decisive datapoint:** querying `probe_success` for *every* job over the incident window shows the failures are not specific to the NetBird edge: - Failed together in the same windows: all 3 `blackbox-edge-wan` members, `http://proxy-{1,2}.renner.uno:9000/health`, `proxy-{1,2}.renner.uno:33080` (TCP relays), `https://netbird.renner.uno`, **and `https://fsn1.your-objectstorage.com`** — Hetzner object storage, which has nothing to do with NetBird or netcup. - Never dropped below 1: every target served from inside the cluster — `home.renner.uno`, `gitea`, `immich`, `auth`, `dash`, `alerts`, `prometheus.renner.uno`, and the two `*.monitoring.svc.cluster.local:*/-/healthy` probes. So: **100% of probes that leave the LAN for the public internet failed; 100% of probes that stay inside the cluster succeeded.** DNS is ruled out as the mechanism for these particular probes — `probe_dns_lookup_time_seconds{job="blackbox-edge-wan"}` is ~2.8 µs for all three targets (they are IP literals, no resolution performed). These are pure IP-path failures. Corroborating symptoms from the same minutes, in `context.txt`: - ~20 `ReadOperationFailed` events on `flux-system` ImageRepositories: `dial tcp: lookup ghcr.io on [fd10:96::a]:53: server misbehaving` / `i/o timeout`. - `home-assistant` `BackOff` on `zigbee2mqtt` (filed separately, see `[alert] KubePodCrashLooping (home-assistant)`). And live CoreDNS logs on **both** replicas (`kubectl logs -n kube-system coredns-7859998f6-{dhf28,xxfr6} --tail=25`) are full of upstream timeouts, still occurring at the time of writing: ``` [ERROR] plugin/errors: 2 fsn1.your-objectstorage.com. A: read udp 10.244.2.17:44575->169.254.116.108:53: i/o timeout [ERROR] plugin/errors: 2 api.getbring.com. AAAA: read udp 10.244.0.106:47787->169.254.116.108:53: i/o timeout [ERROR] plugin/errors: 2 mail-api.proton.me. AAAA: read udp 10.244.0.106:37161->169.254.116.108:53: i/o timeout ``` `169.254.116.108` is the Talos host-DNS listener (`hostDNS.enabled` + `forwardKubeDNSToHost: true`, `kubernetes/talos/patches/cluster.yaml:16`), which forwards to the node's configured resolver — the UDM at `10.2.40.1` / `fd00::1`. ## Root cause **Verified:** the public edge itself is almost certainly *not* the failing component. The alert's own description tells a story about the netcup proxies ("Check the netcup proxies (netbird-proxy containers), the mesh path to the cluster, and cluster Traefik") — that generic story does not match the data. Three independent edge members on two different netcup hosts, across both IP families, plus a Hetzner VPS and Hetzner S3, cannot plausibly fail and recover in lockstep on 3-minute boundaries. A single shared component in front of all of them can. **Hypothesis (not yet directly verified):** the shared component is the **UDM (10.2.40.1 / fd00::1)** in its dual role as default gateway *and* recursive resolver — either its WAN uplink or the device itself is flapping. This fits every observation: IP-literal egress fails (gateway), upstream recursion times out (resolver), and in-cluster traffic is untouched. This is consistent with prior documented behaviour — `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67` records issue #393: *"The UDM's dnsmasq refuses connections in bursts — 330 'connection refused' errors in the last 3000 log lines here, on BOTH 10.2.40.1 and fd00::1."* The `serve_stale`/cache mitigations there address the DNS symptom only; they cannot help IP-level egress. **Commands that would settle it** (none are runnable from the CI ServiceAccount — they need host or UDM access): - UDM UI/SSH: WAN uplink event log and ISP session state around 01:28–01:36 UTC and 01:57 / 02:04 UTC. - From a LAN host (not the cluster): `mtr -rw 45.157.177.26` and `dig @10.2.40.1 ghcr.io` during a burst, to separate "UDM dead" from "UDM alive, WAN down". - If it turns out the UDM is fine, the next suspect is the Talos nodes' egress path (Cilium masquerading); `cilium-dbg` on a node would be the check. ## Where to fix it **This is NOT fixable in this repository.** No manifest change can restore a failing uplink. The repo files below are the *right places to look* for context, not the fix: - `kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml` — the alert definitions. Worth noting the current `for: 5m` means the ongoing 3–4 minute all-members outages are invisible; if this uplink instability persists, consider a companion rule with a shorter `for:` or an `avg_over_time` form so repeated sub-threshold flapping is still surfaced. - `kubernetes/apps/monitoring-rules/probes-blackbox.yaml` — probe target definitions (`blackbox-edge-wan` at line 72). - `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml` and `kubernetes/infrastructure/coredns-config/configmap.yaml` — the existing issue #393 DNS mitigations, which are working as designed but only cover DNS. **What a human should do:** 1. Check the UDM's WAN/uplink health and ISP line status for 2026-07-29 01:28 UTC onward — the flapping was still active at 02:07 UTC. 2. If the uplink is clean, escalate to the Talos/Cilium egress path. 3. Consider whether an ICMP/TCP probe of the UDM itself (`10.2.40.1`) belongs in `probes-blackbox.yaml`, so "our own gateway is down" is directly observable instead of inferred from six downstream alerts. Today the only UDM check is a DNS-only Gatus check (`kubernetes/apps/gatus/configmap.yaml:206`), which would not have distinguished this. ## Related `[alert] KubePodCrashLooping (home-assistant)` was filed separately — its trigger is the DNS half of this same incident, but it also exposes an independently fixable brittleness in the zigbee2mqtt deployment, so it is not merely a duplicate of this issue. --- <!-- alert-monitor:fingerprint=2d098a1215d57d5a --> _Filed automatically by the alert-monitor pipeline (2026-07-29). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-29 02:45:37 +00:00
Member
Copy Link
Copy Source

Assessment: not fixable in this repository — no code changes made

NetbirdPublicEdgeWanDown fired on a real condition, but the failing
component is outside this repo and outside the cluster: the LAN's single
egress path (UDM gateway at 10.2.40.1/fd00::1 and/or the ISP uplink
behind it) dropped in three bursts. I confirmed the issue's core evidence
live, tested and rejected one hypothesis the issue did not consider, and
found no manifest change that resolves the cause. Per the "don't force a
cosmetic repo change" rule, I made no edits.

What I verified live this run

All commands via ./.ci-diag/kubectl get --raw .../kube-prometheus-stack-prometheus:9090/proxy/...
unless noted. Times UTC, 2026-07-29.

1. The outage is real, and the verified windows differ slightly from the issue.
query_range on max(probe_success{job="blackbox-edge-wan"}), 01:00→02:40, step 60 — value 0 in exactly three windows:

window duration
01:29:00 – 01:36:00 8 min (this is the one that latched for: 5m)
01:52:00 – 01:54:00 3 min
01:57:00 – 02:00:00 4 min

Clean 1 from 02:01:00 through 02:41:00 (last sample) — ~40 minutes stable at time of writing.
The issue reported the later bursts as "01:57–01:58" and "02:04–02:07"; the
series shows 01:52–01:54 and 01:57–02:00, with 02:04–02:07 healthy. Use the
windows above when correlating against UDM/ISP logs.

2. The failure is "anything leaving the LAN", not the NetBird edge.
query_range on count(probe_success==0) by (job) over 01:25–02:05 — all
three blackbox-edge-wan members, both blackbox-tcp-relays, and exactly 4
blackbox-http targets go to 0 in lockstep. count_over_time identifies
those 4 as the only external ones:

https://fsn1.your-objectstorage.com          11 failing samples
https://netbird.renner.uno                   11
http://proxy-1.renner.uno:9000/health        12
http://proxy-2.renner.uno:9000/health        11

Hetzner Object Storage failing in lockstep with netcup-hosted edge members
on two IP families rules out the edge itself. No in-cluster target
(gitea, immich, auth, home, dash, the two *.monitoring.svc/-/healthy)
ever left 1. The issue's decisive datapoint holds.

3. The DNS half correlates to the minute.
sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m])), 01:00→02:40:
flat 0 everywhere except the two 5-minute buckets covering 01:30–01:40
(peak 3.89/s) and 01:55–02:05 (peak 1.31/s). Both CoreDNS pods
(coredns-7859998f6-dhf28 on k8s-3, coredns-7859998f6-xxfr6 on k8s-1) are
Ready with 0 restarts — so this is not a node-local or pod-local fault;
it is the shared upstream. IP-path egress and recursive DNS failing in the
same minutes points at the one box that is both default gateway and resolver.

4. Hypothesis I tested that the issue did not raise — uplink saturation by our own backups — is REJECTED.
Kopia maintenance runs against Hetzner S3 every ~35 min, which made
"we saturated our own uplink" a plausible repo-fixable cause (schedule or
bandwidth limit). It does not hold:

  • sum(rate(node_network_transmit_bytes_total{device=~"e.*"}[3m]))*8/1e6 over
    01:00–02:40 shows 25–55 Mbps throughout all three outage windows. The
    only high-throughput period (~1.2 Gbps, 01:03–01:12) is before the first
    outage and had zero probe failures.
  • Job timing is backwards: the outage began 01:29; the
    apps-default-kopia-maintain-job batch started 01:31:18. That job took
    63 s versus its usual ~5 s (kubectl get jobs -A, compare the 02:36:18
    run: 5 s) — it was a victim of the degraded path, not the cause. The
    01:52 and 01:57 bursts had no job running at all, and the 02:06 batch ran
    at normal speed with probes healthy.

Why no repo change

  • No manifest restores a flapping uplink or a flaky gateway. (Decision rule 6.)
  • I deliberately did not add a direct UDM/gateway probe, which the issue
    suggests. The live blackbox config
    (configmap/blackbox-exporter-prometheus-blackbox-exporter) defines only
    http_2xx, http_2xx_wan_edge, and tcp_connect — there is no icmp
    module    , so a ping probe needs a Helm values change plus NET_RAW on the
    exporter. The alternative, tcp_connect to 10.2.40.1:53, depends on the
    UDM actually listening on TCP 53, which I cannot verify read-only from
    the cluster; if it does not, the probe fails permanently and manufactures a
    false alert. Unverifiable change → no edit (decision rule 7).
  • I did not add a shorter-for: companion rule. The sub-5-minute bursts are
    a symptom of the same off-cluster fault; a second rule would add pages
    without adding diagnostic power, and the existing rule is not wrong.
  • I did not touch container image tags, thresholds, or inhibitions.

What a human should do

  1. Check the UDM WAN/uplink and ISP session state for 2026-07-29
    01:29–01:36, 01:52–01:54, and 01:57–02:00 UTC.     Three ~3–8 min drops
    inside 30 minutes, then 40 min clean. UDM UI → Internet event log, or SSH
    and read the WAN interface/PPPoE session log.
  2. If it recurs, discriminate "UDM dead" from "UDM alive, WAN down" from a
    LAN host that is not a cluster node, during a burst:
    • mtr -rw 45.157.177.26 — first hop failing ⇒ the UDM; later hops ⇒ ISP.
    • dig @10.2.40.1 ghcr.io — connection refused reproduces the known
      dnsmasq fault from issue #393 (referenced at
      kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67); a clean
      answer while mtr fails means the device is fine and the uplink is not.
  3. If the uplink is clean, escalate to the Talos/Cilium egress path
    (cilium-dbg on a node) — though note both CoreDNS replicas on different
    nodes degraded identically, which argues against a per-node fault.
  4. Observability gap worth a follow-up decision (not taken here): there is
    currently no Prometheus signal for the gateway itself. I confirmed
    Gatus is not scraped — /api/v1/label/__name__/values?match[]={__name__=~"gatus.*"}
    returns an empty list — so the UDM DNS check at
    kubernetes/apps/gatus/configmap.yaml:206 produces no metrics and no
    alerts. A UDM/WAN fault therefore surfaces only as six unrelated critical
    alerts. Closing this means either enabling an icmp blackbox module
    (Helm values + NET_RAW) or confirming a reachable TCP port on the UDM
    first. Both need a decision plus a live check I cannot perform.
  5. Optional runbook fix: the NetbirdPublicEdgeWanDown description at
    kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:39
    sends the operator to "the netcup proxies, the mesh path, and cluster
    Traefik" — which is right for a true edge outage but misdirected the
    triage of this incident. Adding "if fsn1.your-objectstorage.com and the
    relays failed in the same window, suspect local egress/DNS, not the edge"
    would help. I left it alone because it does not address the root cause and
    the wording change is an owner judgement call.

Notes

  • The issue's file pointers are accurate: the rule is at
    prometheusrule-netbird-availability.yaml:28, the probe job at
    probes-blackbox.yaml:72.
  • The issue states the Gatus UDM check "would not have distinguished this."
    Partly inaccurate — a DNS check against 10.2.40.1 would very likely have
    gone red during the bursts. The real problem is that it feeds no
    metrics/alerting at all (verified above), so nobody would have seen it.
  • The issue contained no "Verified owner guidance" section, so nothing here
    was authorised beyond the standard surgical scope.
  • Nothing in the issue asked me to touch CI config, credentials, or anything
    outside the edit allowlist.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

# Assessment: not fixable in this repository — no code changes made `NetbirdPublicEdgeWanDown` fired on a real condition, but the failing component is outside this repo and outside the cluster: the LAN's single egress path (UDM gateway at `10.2.40.1`/`fd00::1` and/or the ISP uplink behind it) dropped in three bursts. I confirmed the issue's core evidence live, tested and **rejected** one hypothesis the issue did not consider, and found no manifest change that resolves the cause. Per the "don't force a cosmetic repo change" rule, I made no edits. ## What I verified live this run All commands via `./.ci-diag/kubectl get --raw .../kube-prometheus-stack-prometheus:9090/proxy/...` unless noted. Times UTC, 2026-07-29. **1. The outage is real, and the verified windows differ slightly from the issue.** `query_range` on `max(probe_success{job="blackbox-edge-wan"})`, 01:00→02:40, step 60 — value `0` in exactly three windows: | window | duration | |---|---| | 01:29:00 – 01:36:00 | 8 min (this is the one that latched `for: 5m`) | | 01:52:00 – 01:54:00 | 3 min | | 01:57:00 – 02:00:00 | 4 min | Clean `1` from 02:01:00 through 02:41:00 (last sample) — **~40 minutes stable at time of writing.** The issue reported the later bursts as "01:57–01:58" and "02:04–02:07"; the series shows 01:52–01:54 and 01:57–02:00, with 02:04–02:07 healthy. Use the windows above when correlating against UDM/ISP logs. **2. The failure is "anything leaving the LAN", not the NetBird edge.** `query_range` on `count(probe_success==0) by (job)` over 01:25–02:05 — all three `blackbox-edge-wan` members, both `blackbox-tcp-relays`, and exactly 4 `blackbox-http` targets go to 0 in lockstep. `count_over_time` identifies those 4 as the only external ones: ``` https://fsn1.your-objectstorage.com 11 failing samples https://netbird.renner.uno 11 http://proxy-1.renner.uno:9000/health 12 http://proxy-2.renner.uno:9000/health 11 ``` Hetzner Object Storage failing in lockstep with netcup-hosted edge members on two IP families rules out the edge itself. No in-cluster target (`gitea`, `immich`, `auth`, `home`, `dash`, the two `*.monitoring.svc/-/healthy`) ever left `1`. The issue's decisive datapoint holds. **3. The DNS half correlates to the minute.** `sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))`, 01:00→02:40: flat `0` everywhere except the two 5-minute buckets covering 01:30–01:40 (peak 3.89/s) and 01:55–02:05 (peak 1.31/s). Both CoreDNS pods (`coredns-7859998f6-dhf28` on k8s-3, `coredns-7859998f6-xxfr6` on k8s-1) are `Ready` with **0 restarts** — so this is not a node-local or pod-local fault; it is the shared upstream. IP-path egress and recursive DNS failing in the same minutes points at the one box that is both default gateway and resolver. **4. Hypothesis I tested that the issue did not raise — uplink saturation by our own backups — is REJECTED.** Kopia maintenance runs against Hetzner S3 every ~35 min, which made "we saturated our own uplink" a plausible repo-fixable cause (schedule or bandwidth limit). It does not hold: - `sum(rate(node_network_transmit_bytes_total{device=~"e.*"}[3m]))*8/1e6` over 01:00–02:40 shows **25–55 Mbps throughout all three outage windows**. The only high-throughput period (~1.2 Gbps, 01:03–01:12) is *before* the first outage and had zero probe failures. - Job timing is backwards: the outage began 01:29; the `apps-default-kopia-maintain-job` batch started 01:31:18. That job took **63 s** versus its usual ~5 s (`kubectl get jobs -A`, compare the 02:36:18 run: 5 s) — it was a *victim* of the degraded path, not the cause. The 01:52 and 01:57 bursts had no job running at all, and the 02:06 batch ran at normal speed with probes healthy. ## Why no repo change - No manifest restores a flapping uplink or a flaky gateway. (Decision rule 6.) - **I deliberately did not add a direct UDM/gateway probe**, which the issue suggests. The live blackbox config (`configmap/blackbox-exporter-prometheus-blackbox-exporter`) defines only `http_2xx`, `http_2xx_wan_edge`, and `tcp_connect` — there is **no `icmp` module**, so a ping probe needs a Helm values change plus `NET_RAW` on the exporter. The alternative, `tcp_connect` to `10.2.40.1:53`, depends on the UDM actually listening on **TCP** 53, which I cannot verify read-only from the cluster; if it does not, the probe fails permanently and manufactures a false alert. Unverifiable change → no edit (decision rule 7). - I did not add a shorter-`for:` companion rule. The sub-5-minute bursts are a symptom of the same off-cluster fault; a second rule would add pages without adding diagnostic power, and the existing rule is not wrong. - I did not touch container image tags, thresholds, or inhibitions. ## What a human should do 1. **Check the UDM WAN/uplink and ISP session state for 2026-07-29 01:29–01:36, 01:52–01:54, and 01:57–02:00 UTC.** Three ~3–8 min drops inside 30 minutes, then 40 min clean. UDM UI → Internet event log, or SSH and read the WAN interface/PPPoE session log. 2. **If it recurs, discriminate "UDM dead" from "UDM alive, WAN down"** from a LAN host that is *not* a cluster node, during a burst: - `mtr -rw 45.157.177.26` — first hop failing ⇒ the UDM; later hops ⇒ ISP. - `dig @10.2.40.1 ghcr.io` — `connection refused` reproduces the known dnsmasq fault from issue #393 (referenced at `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:67`); a clean answer while `mtr` fails means the device is fine and the uplink is not. 3. **If the uplink is clean**, escalate to the Talos/Cilium egress path (`cilium-dbg` on a node) — though note both CoreDNS replicas on *different* nodes degraded identically, which argues against a per-node fault. 4. **Observability gap worth a follow-up decision (not taken here):** there is currently **no Prometheus signal for the gateway itself**. I confirmed Gatus is not scraped — `/api/v1/label/__name__/values?match[]={__name__=~"gatus.*"}` returns an empty list — so the UDM DNS check at `kubernetes/apps/gatus/configmap.yaml:206` produces no metrics and no alerts. A UDM/WAN fault therefore surfaces only as six unrelated critical alerts. Closing this means either enabling an `icmp` blackbox module (Helm values + `NET_RAW`) or confirming a reachable TCP port on the UDM first. Both need a decision plus a live check I cannot perform. 5. **Optional runbook fix:** the `NetbirdPublicEdgeWanDown` description at `kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:39` sends the operator to "the netcup proxies, the mesh path, and cluster Traefik" — which is right for a true edge outage but misdirected the triage of this incident. Adding "if `fsn1.your-objectstorage.com` and the relays failed in the same window, suspect local egress/DNS, not the edge" would help. I left it alone because it does not address the root cause and the wording change is an owner judgement call. ## Notes - The issue's file pointers are accurate: the rule is at `prometheusrule-netbird-availability.yaml:28`, the probe job at `probes-blackbox.yaml:72`. - The issue states the Gatus UDM check "would not have distinguished this." Partly inaccurate — a DNS check against `10.2.40.1` would very likely have gone red during the bursts. The real problem is that it feeds no metrics/alerting at all (verified above), so nobody would have seen it. - The issue contained no "Verified owner guidance" section, so nothing here was authorised beyond the standard surgical scope. - Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit allowlist. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-08-16 21:17:23 +00:00
Member
Copy Link
Copy Source

Condition has largely cleared, and what remains is not repo-fixable.

Live state, 2026-08-16:

  • probe_success{job="blackbox-edge-wan"} = 1 for all three members (both IPv6 edges and the v4 proxy).
  • ALERTS{alertname="NetbirdPublicEdgeWanDown",alertstate="firing"} — no firing samples in 7 days.
  • Total downtime per member over 7 days, at 1-minute resolution: 6, 6 and 4 minutes. That is ~99.94%, against an incident window that had all three flapping in 3–4 minute bursts every few minutes.

The residual few minutes still go down together across both address families and two netcup hosts, which keeps supporting the issue's conclusion: a single shared component in front of all of them, i.e. the UDM as combined gateway and resolver. Nothing about that changed — it just stopped flapping at incident rate.

Why I am closing rather than fixing. The analysis already establishes the failing component is the UDM's WAN uplink or the device itself. There is no repo change that addresses that; the serve_stale/cache mitigations in k8s-gateway help the DNS symptom and cannot help IP-level egress. Chasing it further means UDM event logs and ISP session state, which is device-side work, not GitOps work.

Closing on current evidence. The alert-monitor dedups on open issues with this exact title, so if the edge drops for 5+ minutes again a fresh issue is filed automatically with new data — which is more useful than keeping a stale one open against a condition that has not reproduced.

Noting the link for whoever picks up the DNS thread: this shares a root cause with #438 (zigbee2mqtt crashlooping on EAI_AGAIN) and with the UDM dnsmasq bursts recorded in #393. The DNS-side work belongs there.

Condition has largely cleared, and what remains is not repo-fixable. **Live state, 2026-08-16:** - `probe_success{job="blackbox-edge-wan"}` = **1 for all three** members (both IPv6 edges and the v4 proxy). - `ALERTS{alertname="NetbirdPublicEdgeWanDown",alertstate="firing"}` — **no firing samples in 7 days**. - Total downtime per member over 7 days, at 1-minute resolution: **6, 6 and 4 minutes**. That is ~99.94%, against an incident window that had all three flapping in 3–4 minute bursts every few minutes. The residual few minutes still go down *together* across both address families and two netcup hosts, which keeps supporting the issue's conclusion: a single shared component in front of all of them, i.e. the UDM as combined gateway and resolver. Nothing about that changed — it just stopped flapping at incident rate. **Why I am closing rather than fixing.** The analysis already establishes the failing component is the UDM's WAN uplink or the device itself. There is no repo change that addresses that; the `serve_stale`/cache mitigations in `k8s-gateway` help the DNS symptom and cannot help IP-level egress. Chasing it further means UDM event logs and ISP session state, which is device-side work, not GitOps work. Closing on current evidence. The alert-monitor dedups on *open* issues with this exact title, so if the edge drops for 5+ minutes again a fresh issue is filed automatically with new data — which is more useful than keeping a stale one open against a condition that has not reproduced. Noting the link for whoever picks up the DNS thread: this shares a root cause with #438 (zigbee2mqtt crashlooping on `EAI_AGAIN`) and with the UDM dnsmasq bursts recorded in #393. The DNS-side work belongs there.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#437