konairius/homelab
10
0
Fork 0
Code Issues 10 Pull Requests 11 Packages Projects Releases 1 Wiki Activity

feat(monitoring): watch DNS publication — external-dns metrics plus resolution probes #865

Merged
k.renner merged 2 commits from feat/dns-monitoring into master 2026-09-17 08:26:04 +00:00
Conversation 1 Commits 2 Files Changed 6
+382
2 Commits
Author SHA1 Message Date
claude-botandClaude Opus 5 2131c7b5f5 feat(monitoring): probe that the LAN names still resolve
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
ci/woodpecker/pr/pr-review Pipeline was successful
Details 
external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.

Two findings shaped it.

The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.

And rcode alone is not a usable signal here. Measured against 10.2.40.1:

  does-not-exist-xyz.renner.uno            -> NOERROR, CNAME proxy.renner.uno
  does-not-exist-xyz.paperless.renner.uno  -> no answer

The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.

One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 10:20:04 +02:00
claude-botandClaude Opus 5 d35c06d21d feat(monitoring): scrape external-dns and alert when it stops publishing 
Neither external-dns release was scraped: both expose /metrics on :7979 and
no ServiceMonitor existed, so external_dns_* series were absent from
Prometheus entirely. The process that publishes every LAN name in the
cluster had no instrumentation at all.

That is how 2026-09-12 went unnoticed for five days. external-dns v0.22.0
changed the default annotation prefix, the service source went empty, and
policy=sync deleted six records. source_endpoints_total fell 48 -> 36 and
controller_verified_records 24 -> 18 per family; nothing was watching either.

ExternalDNSSourceEndpointsDropped is the rule for that shape: a sustained
drop below 90% of the 24h peak. The threshold is chosen against the observed
inventory — one retired Ingress is 2 endpoints of ~48 and stays quiet, a
whole class of names does not. A deliberate mass removal fires it once and
clears itself after 24h, which is the right trade for a signal that
otherwise only speaks when records are already gone.

The rest of the group covers the ways the drop rule itself can go blind:
source and registry errors, the instance being down, and an absent() dead
man mirroring BlackboxProbesAbsent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-17 10:15:18 +02:00