feat(monitoring): WAN-path probes for the NetBird edge, controller, relays #415
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#415
Reference in New Issue
Block a user
Closes SPOF-assessment open item: external monitoring. The 2026-07-24 controller outage ran ~20 h with no signal because every existing probe of the public names resolves in-cluster (CoreDNS/k8s_gateway) or gets hairpinned — nothing measured the WAN path.
How the WAN vantage works without new infrastructure
The Server/Ingress VLANs are deliberately absent from the UniFi hairpin (
opentofu/unifi_nat_hairpin.tf), so probes from the in-cluster blackbox-exporter aimed at IP literals dodge both the split-horizon DNS and the hairpin, genuinely leave via the WAN, and come back through the netcup edge. Verified live from a cluster pod: the v4 edge probe returns 200 end-to-end. (The v6 probe currently fails TLS against proxy-1 — that turned out to be a real regression, fixed separately in #414; expectNetbirdPublicEdgeMemberDownto fire for proxy-1 until #414 is deployed, which nicely proves the alert works.)Changes
http_2xx_wan_edge(helm map-merge keeps stock modules): SNI + Host pinned tohome.renner.unofor IP-literal targets.blackbox-edge-wan:https://45.157.177.26(proxy-2 v4) andhttps://[2a03:4000:28:f6e:…9a29](proxy-1, v6-only member).https://netbird.renner.uno(controller — Hetzner, not hairpinned, so this detects a controller/DC outage), and relay:9000/healthon both netcup members (endpoint verified on 0.73.2).blackbox-tcp-relays: tcp_connect toproxy-{1,2}.renner.uno:33080.netbird-availability:NetbirdControllerUnreachable(critical, 10 m),NetbirdPublicEdgeWanDown(both members, critical, 5 m),NetbirdPublicEdgeMemberDown(warning, 15 m),NetbirdRelayMemberDown(warning, 15 m — redundancy loss, clients fail over).Validated: kustomize build passes, PrometheusRule server-side dry-run passes, WAN path proven from a pod as above.
🤖 Generated with Claude Code
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit8b1480f429855f86ece1bc792de4d4e9f53cb3f3.Summary
This PR adds WAN-path blackbox probes and alerting for the NetBird public edge, controller, and relays, closing a real blind spot (the 2026-07-24 controller outage went undetected because existing probes resolve in-cluster). The mechanism — IP-literal targets to dodge split-horizon DNS and the UniFi hairpin — is sound and matches how the hairpin is actually configured (
opentofu/unifi_nat_hairpin.tf). However, cross-checking the new v6 edge-probe target against that same Terraform file (which the PR's own comments cite as the source of truth to stay in sync with) turns up a member-identity mix-up that leaves one edge member effectively unmonitored and mislabels alerts for the other.🔴 Blocking
kubernetes/apps/monitoring-rules/probes-blackbox.yaml:88-89andprometheusrule-netbird-availability.yaml:39,52— the "proxy-1" v6 WAN probe actually targets proxy-2, and proxy-1's real address is never probed.opentofu/unifi_nat_hairpin.tf:40-47(the authoritative endpoint list, also mirrored inkubernetes/infrastructure/network-policies/ingress-isolation.yaml:63-65) defines:"proxy-1 v6"→2a03:4000:4b:c1e:4474:43ff:fedf:8f85"proxy-2 v6"→2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29The new
blackbox-edge-wanprobe targetshttps://[2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29]— that's proxy-2's address per the Terraform source — but labels it# proxy-1 v6 (v6-only member). TheNetbirdPublicEdgeMemberDowndescription repeats the same claim ("proxy-1 is the v6-only member"). Two consequences:2a03:4000:4b:c1e:4474:43ff:fedf:8f85) is never probed at all, so the "one target per edge member" goal stated in the comment atprobes-blackbox.yaml:60-67isn't met — if proxy-1 genuinely goes fully dark, no WAN probe observes it, andNetbirdPublicEdgeWanDown/NetbirdPublicEdgeMemberDownwon't fire for that failure.Fix: probe
2a03:4000:4b:c1e:4474:43ff:fedf:8f85(proxy-1's real v6 address) instead of/in addition to the current target, and correct the comment/annotations to matchunifi_nat_hairpin.tf's naming.🟡 Suggestions
prometheusrule-netbird-availability.yaml:20—NetbirdControllerUnreachable's expression,probe_success{instance="https://netbird.renner.uno"} == 0, has nojoblabel selector, unlike every other rule in this file (job="blackbox-edge-wan",job="blackbox-tcp-relays",job="blackbox-http"). It's safe today because that literal target string is unique across all Probes in the repo, but it's an implicit assumption — addingjob="blackbox-http"would make it consistent with the rest of the file and robust against a future Probe reusing the same target string.Tests
No automated test suite applies to this class of change (GitOps manifests). The PR body states
kustomize buildand a PrometheusRule server-side dry-run were run, and the WAN path was verified live from a cluster pod for the v4 target — reasonable validation for this repo's conventions. That said, the manual verification apparently didn't catch the proxy-1/proxy-2 address mix-up above, since the "verified live" claim in the PR body is specifically about the v4 probe, not the mislabeled v6 one.Recommendation
Request changes — 1 blocking issue.
Review follow-up (
e3bc0091) — the blocking finding is resolved, but the identity is the other way around: the probe's original attribution was correct;unifi_nat_hairpin.tf's label keys are the swapped ones. Evidence:dig proxy-1.renner.uno→ AAAA-only2a03:…:28:f6e…;dig proxy-2.renner.uno→45.157.177.26+2a03:…:4b:c1e…; the live host answering as proxy-1 is the v6-only one (nameserver-less v6 dhcpcd lease — the #414 bug), matching the netbird-proxy role's long-standing 'proxy-1 is v6-only' doc. The tf keys attribute the two v6 addresses to the opposite members; functionally harmless there (both addresses get identical hairpin rules), and renaming tf map keys churns the NAT-rule resources, so the relabel is tracked as a separate follow-up rather than smuggled into this PR.That said, the review exposed two real gaps, both fixed: proxy-2's v6 path was unprobed — added as a third target (verified live from a cluster pod, returns 200), so coverage is now genuinely one probe per member/family; and
NetbirdControllerUnreachablenow pinsjob="blackbox-http"like the other rules. Comments in the Probe now cite DNS as the member truth and flag the tf mislabel explicitly.🤖 Automated review — model
claude-sonnet-5(complexity: normal), commite3bc00916dc1d4dec142014cb80caba716b735c9.Summary
Adds WAN-vantage blackbox probes (controller, both edge members by IP-literal, relay health/TCP) plus a
netbird-availabilityPrometheusRule with four alerts, to close the blind spot that let the 2026-07-24 controller outage run ~20h unnoticed. I traced the WAN-vantage mechanism againstopentofu/unifi_nat_hairpin.tf(Server/Ingress VLANs are indeed absent from the hairpin, confirming the design rationale) and cross-checked every new PromQL expression against the label/job names the new Probe CRs actually emit. The change is careful, well-commented, and consistent with existing conventions in this file. No blocking issues found.🟡 Suggestions
kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:61— the regexinstance=~"http://proxy-[12].renner.uno:9000/health"has an unescaped.beforerenner, so it matches any character there, not just a literal dot. Harmless in practice (no other label value coincidentally fits the pattern) but worth\.for correctness/consistency with how the rest of the repo tends to be precise about these things.NetbirdPublicEdgeWanDown(all-members-down, critical, 5m) andNetbirdPublicEdgeMemberDown(single-member-down, warning, 15m) can both fire at once when the whole edge is down (the member-down expression doesn't exclude the all-down case). That's a reasonable and common alerting pattern, but if alert noise during a full outage is undesirable, consider anunless on() ALERTS{alertname="NetbirdPublicEdgeWanDown"}style exclusion or just note in the description that both firing together is expected.opentofu/unifi_nat_hairpin.tflabel/member mismatch called out in theblackbox-edge-wancomment (probes-blackbox.yaml:86-93) is real — I confirmed the address swap directly in that file ("proxy-1 v6"there =2a03:4000:4b:c1e:..., which this PR's comment attributes to proxy-2, and vice versa). Since it's already flagged as "tracked for a separate relabel," no action needed here, just confirming the claim checks out rather than taking it on faith.🟢 Nits
helmrelease-blackbox-exporter.yaml:45-53andprobes-blackbox.yamlboth carry near-duplicate explanations of why IP-literal probing dodges the hairpin/split-horizon DNS. Not wrong, just a little repetitive across the two files — a minor readability tax for future readers of the diff, not the file.Tests
No automated tests exist for this repo's kustomize/Helm layer (consistent with the rest of
monitoring-rules/), so this is validated structurally rather than by test suite. I was not able to independently runkustomize buildor a PrometheusRule dry-run in this environment (nokustomize/kubectlbinary available), so I could not re-verify the PR body's claim that both pass — I instead hand-verified: YAML structure/indentation of all three touched/added files by reading them in full, thatkustomization.yamlcorrectly lists the newprometheusrule-netbird-availability.yamlin alphabetical order, that every new alert'sjob=/instance=label selector matches thejobNameand target strings of the corresponding new/edited Probe, and that the Helm module addition follows the deep-merge pattern the adjacent comment claims (config.modules.<name>alongside stock modules, matching the standardprometheus-blackbox-exporterchart's config schema). If this hasn't already been confirmed in this PR's own CI, runningkustomize build kubernetes/apps/monitoring-rulesbefore merge is still the cheapest way to catch anything I couldn't execute here.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Follow-up (
2bd54ef3): dots escaped in the relay-health instance regex. On the double-fire of EdgeWanDown + EdgeMemberDown during a full outage: accepted as-is — two pages for 'the whole edge is down' is tolerable and the common pattern, and inhibition rules live in Alertmanager config if it ever gets annoying. Thanks for independently confirming the unifi_nat_hairpin.tf label swap — relabel PR to follow separately.🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit2bd54ef3781f43b781767d2ae73d4c8d5d3635c7.Summary
Adds WAN-vantage monitoring for the NetBird edge/controller/relays: a new blackbox module (
http_2xx_wan_edge) with SNI/Host pinning, two newProbeCRs (blackbox-edge-wanfor IP-literal edge checks,blackbox-tcp-relaysfor relay TCP reachability), two new targets on the existingblackbox-httpprobe, and a newPrometheusRulewith four alerts. The change is well-reasoned and self-documenting (comments explain why IP-literal targets dodge the hairpin/split-horizon DNS, and flag a pre-existing, unrelated label mismatch inopentofu/unifi_nat_hairpin.tfrather than silently "fixing" it). No blocking issues found.🟡 Suggestions
kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:33(NetbirdPublicEdgeWanDown) and theNetbirdControllerUnreachable/NetbirdRelayMemberDownalerts all depend onblackbox-exporteritself staying up and being scraped. There's no existing "exporter/scrape down" meta-alert in this repo (checkedmonitoring-rules/*.yaml), so ifblackbox-exportercrashes, all of these — including the new critical ones meant to close the 2026-07-24 blind spot — go silently quiet rather than firing. This is a pre-existing gap shared with the olderblackbox-httpprobes, not introduced by this PR, but since the whole point of this PR is closing a "no signal for 20h" gap, it's worth a follow-upup{job=~"blackbox.*"} == 0-style alert.kubernetes/apps/monitoring-rules/prometheusrule-netbird-availability.yaml:59-61(NetbirdRelayMemberDown): theorcombines two different jobs/instance-label shapes (proxy-1.renner.uno:33080fromblackbox-tcp-relaysvshttp://proxy-1.renner.uno:9000/healthfromblackbox-http). That's fine functionally, but it means the same physical relay member can independently trigger this alert twice (once per arm) if both the TCP port and the health endpoint fail together, each with a differentinstancelabel in the notification. Not a bug, just worth knowing for alert-routing/dedup expectations.Tests
No automated tests exist for this kind of change in the repo (it's Kustomize/Helm YAML), which matches the project's existing pattern — validation here is
kustomize build, a PrometheusRule server-side dry-run, and manual verification from a cluster pod, all of which the PR description says were run. One caveat:kustomize buildvalidates YAML/CRD structure but does not render the Helm chart, so theconfig.modules.http_2xx_wan_edgedeep-merge intoprometheus-blackbox-exporter's values (kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml:34-53) isn't verified by that check — it relies on the chart's standardconfig.modulesconvention and Helm's default map-merge behavior, which is correct for this chart but wasn't confirmed by an actualhelm template/dry-run in the stated validation steps.I did not find other correctness issues: the new multi-document
probes-blackbox.yaml(---separators) is well-formed,kustomization.yaml's new entry keeps alphabetical ordering, the regex inNetbirdRelayMemberDownhas its dots properly escaped (already fixed in the latest commit,2bd54ef), IPv6 literals are correctly bracketed, and all newinstance=label matches follow the existing exact-string convention used elsewhere in this file (e.g.prometheusrule-hetzner-backup.yaml:25).Recommendation
Looks reasonable — no blocking issues; comments above are optional.