konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

iot.renner.uno answers inconsistently: pinned device A records intermittently shadowed by the proxy wildcard CNAME #393

New Issue
Closed
opened 2026-07-24 19:30:39 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-07-24 19:30:39 +00:00
Member
Copy Link
Copy Source

Observed from the workstation on 2026-07-24, same resolver path, ~90 minutes apart:

  • ~14:10 CEST: shellyplusi4-90380c3808b8.iot.renner.uno (and 4 sibling device names) returned pinned private A records (10.2.30.x); devices were directly reachable via those answers (HTTP 200 on /shelly).
  • ~15:45 CEST: the SAME names returned only the wildcard CNAME to proxy.renner.uno (45.157.177.26, 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29, 2a03:4000:4b:c1e:4474:43ff:fedf:8f85) — no device A records at all.

Impact: any client resolving devices through the suffix zone (including shelly-manager once homelab#391 lands) intermittently gets the public proxy instead of the device. This is the wildcard-shadow failure mode the zone design (pinned per-device records) was meant to eliminate — it is still manifesting, and not only for AAAA: the pinned A records themselves are disappearing intermittently. Possible directions: DHCP-lease-driven record registration expiring between leases, split-horizon/resolver cache differences, or the pinning job not covering all devices continuously.

Related: shelly-manager#74 makes the app resilient to wildcard answers (identity-based proxy exclusion), but the zone should still serve pinned device records consistently.

Observed from the workstation on 2026-07-24, same resolver path, ~90 minutes apart: - ~14:10 CEST: shellyplusi4-90380c3808b8.iot.renner.uno (and 4 sibling device names) returned pinned private A records (10.2.30.x); devices were directly reachable via those answers (HTTP 200 on /shelly). - ~15:45 CEST: the SAME names returned only the wildcard CNAME to proxy.renner.uno (45.157.177.26, 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29, 2a03:4000:4b:c1e:4474:43ff:fedf:8f85) — no device A records at all. Impact: any client resolving devices through the suffix zone (including shelly-manager once homelab#391 lands) intermittently gets the public proxy instead of the device. This is the wildcard-shadow failure mode the zone design (pinned per-device records) was meant to eliminate — it is still manifesting, and not only for AAAA: the pinned A records themselves are disappearing intermittently. Possible directions: DHCP-lease-driven record registration expiring between leases, split-horizon/resolver cache differences, or the pinning job not covering all devices continuously. Related: shelly-manager#74 makes the app resilient to wildcard answers (identity-based proxy exclusion), but the zone should still serve pinned device records consistently.
claude-bot commented 2026-07-27 19:45:26 +00:00
Author
Member
Copy Link
Copy Source

Root cause confirmed during the 2026-07-27 issue round. Fix in PR #435.

Correction to this issue's premise: there is no pinning job. The per-device records are ephemeral DHCP-lease records in the UDM's dnsmasq — dig @10.2.50.1 shellyplusi4-90380c3808b8.iot.renner.uno returns aa with TTL 5, whereas the OpenTofu static records are TTL 300. Different origin. external-dns-unifi is also ruled out: despite running --policy=sync with --domain-filter=renner.uno, it writes the UDM's static-DNS list rather than the lease table, and --registry=txt confines deletions to records carrying its own TXT.

Three things combine:

  1. The public *.renner.uno CNAME matches at every depth, so renner.uno never returns NXDOMAIN — verified: definitely-not-real-xyz123.iot.renner.uno → proxy.renner.uno → 45.157.177.26. A local failure resolves to the WAN proxy instead of failing honestly.
  2. TTL asymmetry: correct answer TTL 5, wildcard answer TTL 300 — 60x stickier. One 5-second failure pins a wrong answer for 5 minutes, which is exactly the ~90-minute-apart observation reported here.
  3. The UDM's dnsmasq refuses connections in bursts — 330 connection refused errors in the last 3000 k8s-gateway log lines, on both 10.2.40.1 and fd00::1.

Same root cause as the zigbee2mqtt crash-loop (17 restarts): z2m exits on getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno with no retry, and that exact name appears in the failure log.

PR #435 raises the cache floor to 60s and adds serve_stale 1h immediate so a UDM blip stops becoming a lookup failure the wildcard can capture. Validated against the real coredns:v1.13.2 and k8s_gateway:1.7.0 images before merging, given the prior template-plugin outage (7a09bad5) — and template is not reintroduced.

Split out to #436: AAAA for these names is deterministically shadowed (8/8 queries) because dnsmasq holds an A from the lease but no AAAA, so AAAA forwards upstream every time. serve_stale does not help that; it needs a NODATA response, which means the same plugin that caused the earlier outage — so it gets its own tightly-scoped change.

Keeping this issue open until #435 merges and the intermittent symptom is confirmed gone.

Root cause **confirmed** during the 2026-07-27 issue round. Fix in **PR #435**. **Correction to this issue's premise:** there is no pinning job. The per-device records are ephemeral **DHCP-lease records in the UDM's dnsmasq** — `dig @10.2.50.1 shellyplusi4-90380c3808b8.iot.renner.uno` returns `aa` with **TTL 5**, whereas the OpenTofu static records are TTL 300. Different origin. `external-dns-unifi` is also ruled out: despite running `--policy=sync` with `--domain-filter=renner.uno`, it writes the UDM's *static-DNS* list rather than the lease table, and `--registry=txt` confines deletions to records carrying its own TXT. **Three things combine:** 1. The public `*.renner.uno` CNAME matches at **every depth**, so `renner.uno` never returns NXDOMAIN — verified: `definitely-not-real-xyz123.iot.renner.uno` → `proxy.renner.uno` → `45.157.177.26`. A local failure resolves to the WAN proxy instead of failing honestly. 2. **TTL asymmetry**: correct answer TTL **5**, wildcard answer TTL **300** — 60x stickier. One 5-second failure pins a wrong answer for 5 minutes, which is exactly the ~90-minute-apart observation reported here. 3. The UDM's dnsmasq **refuses connections in bursts** — 330 `connection refused` errors in the last 3000 k8s-gateway log lines, on both `10.2.40.1` and `fd00::1`. **Same root cause as the zigbee2mqtt crash-loop** (17 restarts): z2m exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry, and that exact name appears in the failure log. PR #435 raises the cache floor to 60s and adds `serve_stale 1h immediate` so a UDM blip stops becoming a lookup failure the wildcard can capture. Validated against the real `coredns:v1.13.2` and `k8s_gateway:1.7.0` images before merging, given the prior `template`-plugin outage (`7a09bad5`) — and `template` is **not** reintroduced. **Split out to #436:** AAAA for these names is *deterministically* shadowed (8/8 queries) because dnsmasq holds an A from the lease but no AAAA, so AAAA forwards upstream every time. `serve_stale` does not help that; it needs a NODATA response, which means the same plugin that caused the earlier outage — so it gets its own tightly-scoped change. Keeping this issue open until #435 merges and the intermittent symptom is confirmed gone.
claude-bot commented 2026-08-16 21:22:49 +00:00
Author
Member
Copy Link
Copy Source

Closing per owner decision. The intermittent half this issue tracks was mitigated and the mitigation is live.

In the Corefile today, on both the renner.uno:53 and .:53 blocks, citing this issue by number:

cache 300 {
    success 9984 300 60
    denial  9984 300 30
    serve_stale 1h immediate
}

That is PR #435's fix in place: the 60s TTL floor stops dnsmasq's TTL-5 lease answers being evicted faster than the TTL-300 wildcard answer that shadows them, and serve_stale 1h immediate keeps a previously-correct answer alive through a UDM blip instead of falling through to the wildcard.

Supporting evidence that the burst condition has calmed: zero CoreDNS i/o timeout lines against the Talos host-DNS listener 169.254.116.108 in the last 24h of Loki. The "330 connection refused errors in the last 3000 log lines" recorded in this issue is not the current picture.

The deterministic AAAA half was split into #436, which is also closed today — its reproduction (shellyplusi4-...iot.renner.uno AAAA) now correctly returns NODATA with a private A record alongside.

One residual is recorded on #436 rather than lost: slzb-zigbee.renner.uno, a bare renner.uno name, still resolves to the proxy.renner.uno wildcard from the LAN resolver in both address families. It causes no impact — in-cluster resolution reaches the real device, as a live zigbee2mqtt proves — but it is the loose end if this class of bug resurfaces.

Closing per owner decision. The intermittent half this issue tracks was mitigated and the mitigation is live. **In the Corefile today**, on both the `renner.uno:53` and `.:53` blocks, citing this issue by number: ``` cache 300 { success 9984 300 60 denial 9984 300 30 serve_stale 1h immediate } ``` That is PR #435's fix in place: the 60s TTL floor stops dnsmasq's TTL-5 lease answers being evicted faster than the TTL-300 wildcard answer that shadows them, and `serve_stale 1h immediate` keeps a previously-correct answer alive through a UDM blip instead of falling through to the wildcard. **Supporting evidence that the burst condition has calmed:** zero CoreDNS `i/o timeout` lines against the Talos host-DNS listener `169.254.116.108` in the last 24h of Loki. The "330 connection refused errors in the last 3000 log lines" recorded in this issue is not the current picture. The deterministic AAAA half was split into #436, which is also closed today — its reproduction (`shellyplusi4-...iot.renner.uno` AAAA) now correctly returns NODATA with a private A record alongside. One residual is recorded on #436 rather than lost: `slzb-zigbee.renner.uno`, a bare `renner.uno` name, still resolves to the `proxy.renner.uno` wildcard from the LAN resolver in both address families. It causes no impact — in-cluster resolution reaches the real device, as a live zigbee2mqtt proves — but it is the loose end if this class of bug resurfaces.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#393