feat(monitoring): watch DNS publication — external-dns metrics plus resolution probes #865

2026-09-17T08:20:37Z

claude-bot commented

2026-09-17 08:20:37 +00:00

Der Ausfall vom 12.09. lief fünf Tage unbemerkt. Zwei Commits, zwei Ebenen: die Ursache an der Quelle, das Symptom beim Client.

Commit 1 — external-dns scrapen

Beide external-dns-Releases exponieren /metrics auf :7979, es gab keinen ServiceMonitor, und external_dns_* existierte in Prometheus schlicht nicht. Der Prozess, der jeden LAN-Namen des Clusters publiziert, war komplett uninstrumentiert.

external-dns-servicemonitor.yaml scraped beide Instanzen (jobLabel: app.kubernetes.io/instance hält sie auseinander). prometheusrule-external-dns.yaml bringt fünf Regeln:

Alert	Ausdruck
`ExternalDNSSourceEndpointsDropped`	`< 0.9 * max_over_time(...[24h])`, `for: 30m`
`ExternalDNSSourceErrors`	`increase(external_dns_source_errors_total[15m]) > 0`
`ExternalDNSRegistryErrors`	`increase(external_dns_registry_errors_total[15m]) > 0`
`ExternalDNSDown`	`up{job=~"external-dns.*"} == 0`
`ExternalDNSMetricsMissing`	`absent(external_dns_source_endpoints_total)`

Die erste ist die für diesen Fall: am 12.09. fiel source_endpoints_total von 48 auf 36, controller_verified_records von 24 auf 18 je Familie. Die 90-%-Schwelle ist am beobachteten Bestand gewählt — ein einzelner zurückgebauter Ingress sind 2 von ~48 und bleibt still, eine ganze Klasse von Namen nicht. Eine bewusste Massenlöschung feuert einmal und räumt sich nach 24 h selbst ab; das ist der Preis für ein Signal, das sonst erst spricht, wenn die Records schon weg sind.

Commit 2 — die Namen selbst proben

Zwei Messungen haben das Design bestimmt.

Der Probe fragt die UDM direkt, nicht den Cluster-Resolver. k8s-gateway cached diese Namen mit serve_stale 1h immediate — über den Pfad wäre ein gelöschter Record eine Stunde lang grün geblieben.

Der Rcode allein ist hier wertlos. Gemessen gegen 10.2.40.1:

does-not-exist-xyz.renner.uno            -> NOERROR, CNAME proxy.renner.uno
does-not-exist-xyz.paperless.renner.uno  -> keine Antwort

Der öffentliche *.renner.uno-Wildcard greift eine Label-Ebene tief — genau dort liegen ha, mqtt, ldap, syslog. Ihre Records zu löschen ändert am Rcode nichts, die UDM antwortet weiter NOERROR und zeigt auf den WAN-Proxy. Ein Probe, der nur valid_rcodes prüft, wäre bei vier von sechs Namen durch den ganzen Ausfall grün geblieben.

Die Module verlangen deshalb ein A im Answer und lehnen ein CNAME ab. Das trennt echten Record von Wildcard, ohne irgendeine Adresse einzubacken.

Ein Modul pro Name, weil der dns-Prober query_name aus dem Modul liest und als Target den Resolver nimmt — belegt in prober/dns.go und prober/handler.go, query_name ist nicht per Scrape überschreibbar. Beide Familien werden pro Name geprobt, min by (dns_name) lässt den Alert erst feuern wenn keine antwortet; die UDM lehnt DNS schubweise ab (siehe Kommentar in k8s-gateway zu Issue #393), deshalb for: 30m statt der 10m der HTTP-Probes.

Dazu LanDnsProbesAbsent als Totmann — ein fehlendes Modul liefert 400 und damit gar keine probe_*-Series, was von „kein Probe" nicht zu unterscheiden ist.

Verifiziert vorab

Erreichbarkeit aus monitoring: nur default-deny-ingress, Egress offen; nslookup ftp.paperless.renner.uno 10.2.40.1 aus einem Pod dort liefert 10.2.99.82 / fd99::82.
fd00::1 antwortet ebenfalls (dig ha.renner.uno @fd00::1 → 10.2.99.87).
kubectl kustomize rendert beide Verzeichnisse sauber, 9 Probe-Objekte (3 bestehende + 6 neue).
Alle neuen Objekte tragen release: kube-prometheus-stack, sonst greifen ruleSelector/probeSelector nicht.

Bekannter Trade-off

Die sechs Namen stehen jetzt an zwei Stellen — in der Service-Annotation und im Probe. Ein neuer LoadBalancer-Name wird nicht automatisch geprobt. Das ist bei einem Check gewollt (er behauptet unabhängig, was herauskommen soll), heißt aber: kommt ein Name dazu, gehört er hier ergänzt.

🤖 Generated with Claude Code

Der Ausfall vom 12.09. lief fünf Tage unbemerkt. Zwei Commits, zwei Ebenen: die Ursache an der Quelle, das Symptom beim Client. ## Commit 1 — external-dns scrapen Beide external-dns-Releases exponieren `/metrics` auf `:7979`, es gab keinen ServiceMonitor, und `external_dns_*` existierte in Prometheus schlicht nicht. Der Prozess, der jeden LAN-Namen des Clusters publiziert, war komplett uninstrumentiert. `external-dns-servicemonitor.yaml` scraped beide Instanzen (`jobLabel: app.kubernetes.io/instance` hält sie auseinander). `prometheusrule-external-dns.yaml` bringt fünf Regeln: | Alert | Ausdruck | |---|---| | `ExternalDNSSourceEndpointsDropped` | `< 0.9 * max_over_time(...[24h])`, `for: 30m` | | `ExternalDNSSourceErrors` | `increase(external_dns_source_errors_total[15m]) > 0` | | `ExternalDNSRegistryErrors` | `increase(external_dns_registry_errors_total[15m]) > 0` | | `ExternalDNSDown` | `up{job=~"external-dns.*"} == 0` | | `ExternalDNSMetricsMissing` | `absent(external_dns_source_endpoints_total)` | Die erste ist die für diesen Fall: am 12.09. fiel `source_endpoints_total` von 48 auf 36, `controller_verified_records` von 24 auf 18 je Familie. Die 90-%-Schwelle ist am beobachteten Bestand gewählt — ein einzelner zurückgebauter Ingress sind 2 von ~48 und bleibt still, eine ganze Klasse von Namen nicht. Eine bewusste Massenlöschung feuert einmal und räumt sich nach 24 h selbst ab; das ist der Preis für ein Signal, das sonst erst spricht, wenn die Records schon weg sind. ## Commit 2 — die Namen selbst proben Zwei Messungen haben das Design bestimmt. **Der Probe fragt die UDM direkt, nicht den Cluster-Resolver.** k8s-gateway cached diese Namen mit `serve_stale 1h immediate` — über den Pfad wäre ein gelöschter Record eine Stunde lang grün geblieben. **Der Rcode allein ist hier wertlos.** Gemessen gegen `10.2.40.1`: ``` does-not-exist-xyz.renner.uno -> NOERROR, CNAME proxy.renner.uno does-not-exist-xyz.paperless.renner.uno -> keine Antwort ``` Der öffentliche `*.renner.uno`-Wildcard greift eine Label-Ebene tief — genau dort liegen `ha`, `mqtt`, `ldap`, `syslog`. Ihre Records zu löschen ändert am Rcode nichts, die UDM antwortet weiter `NOERROR` und zeigt auf den WAN-Proxy. Ein Probe, der nur `valid_rcodes` prüft, wäre bei **vier von sechs Namen** durch den ganzen Ausfall grün geblieben. Die Module verlangen deshalb ein A im Answer und lehnen ein CNAME ab. Das trennt echten Record von Wildcard, ohne irgendeine Adresse einzubacken. Ein Modul pro Name, weil der dns-Prober `query_name` aus dem Modul liest und als Target den Resolver nimmt — belegt in `prober/dns.go` und `prober/handler.go`, `query_name` ist nicht per Scrape überschreibbar. Beide Familien werden pro Name geprobt, `min by (dns_name)` lässt den Alert erst feuern wenn keine antwortet; die UDM lehnt DNS schubweise ab (siehe Kommentar in k8s-gateway zu Issue #393), deshalb `for: 30m` statt der 10m der HTTP-Probes. Dazu `LanDnsProbesAbsent` als Totmann — ein fehlendes Modul liefert 400 und damit gar keine `probe_*`-Series, was von „kein Probe" nicht zu unterscheiden ist. ## Verifiziert vorab - Erreichbarkeit aus `monitoring`: nur default-deny-**ingress**, Egress offen; `nslookup ftp.paperless.renner.uno 10.2.40.1` aus einem Pod dort liefert `10.2.99.82` / `fd99::82`. - `fd00::1` antwortet ebenfalls (`dig ha.renner.uno @fd00::1` → `10.2.99.87`). - `kubectl kustomize` rendert beide Verzeichnisse sauber, 9 Probe-Objekte (3 bestehende + 6 neue). - Alle neuen Objekte tragen `release: kube-prometheus-stack`, sonst greifen `ruleSelector`/`probeSelector` nicht. ## Bekannter Trade-off Die sechs Namen stehen jetzt an zwei Stellen — in der Service-Annotation und im Probe. Ein neuer LoadBalancer-Name wird nicht automatisch geprobt. Das ist bei einem Check gewollt (er behauptet unabhängig, was herauskommen soll), heißt aber: kommt ein Name dazu, gehört er hier ergänzt. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 2 commits 2026-09-17 08:20:38 +00:00

feat(monitoring): scrape external-dns and alert when it stops publishing d35c06d21d

Neither external-dns release was scraped: both expose /metrics on :7979 and
no ServiceMonitor existed, so external_dns_* series were absent from
Prometheus entirely. The process that publishes every LAN name in the
cluster had no instrumentation at all.

That is how 2026-09-12 went unnoticed for five days. external-dns v0.22.0
changed the default annotation prefix, the service source went empty, and
policy=sync deleted six records. source_endpoints_total fell 48 -> 36 and
controller_verified_records 24 -> 18 per family; nothing was watching either.

ExternalDNSSourceEndpointsDropped is the rule for that shape: a sustained
drop below 90% of the 24h peak. The threshold is chosen against the observed
inventory — one retired Ingress is 2 endpoints of ~48 and stays quiet, a
whole class of names does not. A deliberate mass removal fires it once and
clears itself after 24h, which is the right trade for a signal that
otherwise only speaks when records are already gone.

The rest of the group covers the ways the drop rule itself can go blind:
source and registry errors, the instance being down, and an absent() dead
man mirroring BlackboxProbesAbsent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat(monitoring): probe that the LAN names still resolve

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

2131c7b5f5

external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.

Two findings shaped it.

The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.

And rcode alone is not a usable signal here. Measured against 10.2.40.1:

  does-not-exist-xyz.renner.uno            -> NOERROR, CNAME proxy.renner.uno
  does-not-exist-xyz.paperless.renner.uno  -> no answer

The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.

One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-17 08:24:51 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 2131c7b5f5be38806b174592ac5b29cb2b152423.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds two layers of DNS-publication monitoring: (1) a ServiceMonitor + PrometheusRule scraping external_dns_* metrics from both external-dns releases, and (2) six blackbox-exporter DNS Probes that query the UDM directly for the six LAN names external-dns publishes, plus a matching PrometheusRule. The change is purely additive (6 new files + 1 kustomization edit + append-only edit to helmrelease-blackbox-exporter.yaml), well-documented, and follows this repo's existing conventions closely (label usage matches probes-blackbox.yaml/rook-servicemonitors.yaml, release: kube-prometheus-stack correctly applied only where ruleSelector/probeSelector require it per helmrelease-prometheus.yaml:56-74, cross-namespace ServiceMonitor pattern matches authentik-outpost-servicemonitors.yaml/rook-servicemonitors.yaml). No blocking issues found.

🟡 Suggestions

kubernetes/apps/monitoring-rules/external-dns-servicemonitor.yaml:29 (port: http) — the metrics port name comes from the external-dns Helm chart's rendered Service, which isn't vendored in this repo, so I couldn't directly confirm the chart names it http for both the Cloudflare release (kubernetes/apps/external-dns/helmrelease.yaml) and the UniFi release (kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml, which only names its extra webhook port http-wh-metrics on 8080, implying the base container's port is the chart default). If the port name is ever wrong, the ServiceMonitor silently produces zero scrape targets — but this is not a blind spot: ExternalDNSMetricsMissing (absent(external_dns_source_endpoints_total), 15m) will catch it independently. Worth a quick post-merge check (kubectl get servicemonitor external-dns -n monitoring / confirm targets appear in Prometheus) rather than a pre-merge blocker.
kubernetes/apps/monitoring-rules/prometheusrule-dns-resolution.yaml:243 — LanDnsProbesAbsent (absent(probe_success{job="blackbox-dns"})) only fires when all six DNS probes stop reporting. If a single module name were to drift out of sync between probes-dns.yaml and the dns_lan_* module list in helmrelease-blackbox-exporter.yaml (they match today, verified), that one name would silently stop being probed with no alert — absent() still sees the other five series and never fires, and min by (dns_name) simply never groups a series for the missing name into existence. This is an inherent limitation of the dead-man's-switch pattern already used for probes-blackbox.yaml/BlackboxProbesAbsent, so it's consistent with existing repo conventions rather than a new gap — just worth keeping in mind if a name is ever added/renamed (as the PR body's own "known trade-off" section already flags for the Service-annotation/Probe duplication).

Tests

No automated tests exist for this repo's Kubernetes manifests beyond kubectl kustomize rendering (per the PR body, verified locally: 9 Probe objects render, both directories build cleanly) — consistent with how other monitoring additions in this repo are validated (no CI test suite for YAML found). The PR body documents manual verification of network reachability (nslookup from a pod in monitoring to 10.2.40.1 and fd00::1) and of the specific failure this is meant to catch (wildcard CNAME vs. missing record behavior). That's an appropriate level of testing for this kind of change; nothing further to add.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `2131c7b5f5be38806b174592ac5b29cb2b152423`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds two layers of DNS-publication monitoring: (1) a `ServiceMonitor` + `PrometheusRule` scraping `external_dns_*` metrics from both external-dns releases, and (2) six `blackbox-exporter` DNS `Probe`s that query the UDM directly for the six LAN names external-dns publishes, plus a matching `PrometheusRule`. The change is purely additive (6 new files + 1 kustomization edit + append-only edit to `helmrelease-blackbox-exporter.yaml`), well-documented, and follows this repo's existing conventions closely (label usage matches `probes-blackbox.yaml`/`rook-servicemonitors.yaml`, `release: kube-prometheus-stack` correctly applied only where `ruleSelector`/`probeSelector` require it per `helmrelease-prometheus.yaml:56-74`, cross-namespace `ServiceMonitor` pattern matches `authentik-outpost-servicemonitors.yaml`/`rook-servicemonitors.yaml`). No blocking issues found. ## 🟡 Suggestions - `kubernetes/apps/monitoring-rules/external-dns-servicemonitor.yaml:29` (`port: http`) — the metrics port name comes from the external-dns Helm chart's rendered `Service`, which isn't vendored in this repo, so I couldn't directly confirm the chart names it `http` for both the Cloudflare release (`kubernetes/apps/external-dns/helmrelease.yaml`) and the UniFi release (`kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml`, which only names its *extra* webhook port `http-wh-metrics` on 8080, implying the base container's port is the chart default). If the port name is ever wrong, the `ServiceMonitor` silently produces zero scrape targets — but this is not a blind spot: `ExternalDNSMetricsMissing` (`absent(external_dns_source_endpoints_total)`, 15m) will catch it independently. Worth a quick post-merge check (`kubectl get servicemonitor external-dns -n monitoring` / confirm targets appear in Prometheus) rather than a pre-merge blocker. - `kubernetes/apps/monitoring-rules/prometheusrule-dns-resolution.yaml:243` — `LanDnsProbesAbsent` (`absent(probe_success{job="blackbox-dns"})`) only fires when *all six* DNS probes stop reporting. If a single module name were to drift out of sync between `probes-dns.yaml` and the `dns_lan_*` module list in `helmrelease-blackbox-exporter.yaml` (they match today, verified), that one name would silently stop being probed with no alert — `absent()` still sees the other five series and never fires, and `min by (dns_name)` simply never groups a series for the missing name into existence. This is an inherent limitation of the dead-man's-switch pattern already used for `probes-blackbox.yaml`/`BlackboxProbesAbsent`, so it's consistent with existing repo conventions rather than a new gap — just worth keeping in mind if a name is ever added/renamed (as the PR body's own "known trade-off" section already flags for the Service-annotation/Probe duplication). ## Tests No automated tests exist for this repo's Kubernetes manifests beyond `kubectl kustomize` rendering (per the PR body, verified locally: 9 Probe objects render, both directories build cleanly) — consistent with how other monitoring additions in this repo are validated (no CI test suite for YAML found). The PR body documents manual verification of network reachability (`nslookup` from a pod in `monitoring` to `10.2.40.1` and `fd00::1`) and of the specific failure this is meant to catch (wildcard CNAME vs. missing record behavior). That's an appropriate level of testing for this kind of change; nothing further to add. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 4ad47b3925 into master

2026-09-17 08:26:04 +00:00

k.renner referenced this issue from a commit

2026-09-17 08:26:05 +00:00

Merge pull request 'feat(monitoring): watch DNS publication — external-dns metrics plus resolution probes' (#865) from feat/dns-monitoring into master

claude-bot referenced this issue from a commit

2026-09-17 08:29:59 +00:00

fix(external-dns): let Prometheus reach the metrics port

claude-bot referenced this pull request

2026-09-17 08:30:08 +00:00

fix(external-dns): let Prometheus reach the metrics port #866

Sign in to join this conversation.