fix(dns): cache floor + serve_stale so UDM blips stop resolving to the WAN proxy #435

Merged

k.renner merged 1 commits from claude-bot/dns-serve-stale into master 2026-07-27 20:19:00 +00:00

Conversation 1 Commits 1 Files Changed 2

+41 -2

claude-bot commented 2026-07-27 19:44:42 +00:00

Member

Copy Link

Copy Source

Fixes the confirmed root cause of #393 — and the same cause explains the zigbee2mqtt crash-loop.

⚠️ First: the issue's premise is wrong. There is no "pinning job". The per-device records are ephemeral DHCP-lease records in the UDM's dnsmasq, not IaC — dig @10.2.50.1 shellyplusi4-....iot.renner.uno returns aa with TTL 5, whereas the OpenTofu static records are TTL 300. Different origin entirely.

external-dns-unifi is also not the culprit, despite running --policy=sync with --domain-filter=renner.uno. It writes the UDM's static-DNS list, not the lease table, and --registry=txt confines its deletions to records carrying its own k8s-homelab TXT — which these never had.

Root cause: three things combine

1. The public wildcard matches at every depth, so renner.uno never returns NXDOMAIN.

dig +short @1.1.1.1 definitely-not-real-xyz123.iot.renner.uno
  proxy.renner.uno.
  45.157.177.26

A name that fails locally doesn't fail — it resolves to the WAN proxy.

2. TTL asymmetry makes the wrong answer stick 60× longer.

	TTL
correct answer (dnsmasq lease record)	5
wrong answer (public wildcard CNAME)	300

One 5-second local failure pins a wrong answer for 5 minutes. That is precisely the "correct at 14:10, wildcard-only at 15:45" report in the issue.

3. The UDM's dnsmasq refuses connections in bursts — 330 connection refused errors in the last 3000 k8s-gateway log lines, on both 10.2.40.1 and fd00::1:

[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. AAAA: read udp ...->10.2.40.1:53: read: connection refused
[ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A:    read udp ...->[fd00::1]:53: read: connection refused

The fix

• success ... 60 — clamps the 5s TTL up to a 60s floor
• serve_stale 1h immediate — keeps answering from cache while the upstream is refusing connections

So a UDM blip stops becoming a lookup failure that the wildcard can capture.

This also fixes the z2m crash-loop. z2m exits on getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno with no retry — and that exact name is in the failure log above.

Validation — done thoroughly, because this is cluster-critical

CoreDNS changes have bitten here before: the template plugin broke cluster DNS in 7a09bad5. This PR does not reintroduce template.

Check	Result
Repo's full Corefile under real coredns:v1.13.2	parses and starts (only the expected out-of-cluster kubernetes-plugin error)
Repo's k8s-gateway values via helm template	rc=0, cache line renders intact
Rendered Corefile under real k8s_gateway:1.7.0 (CoreDNS 1.14.2)	starts cleanly

Two constraints I hit and encoded in comments so they aren't rediscovered:

• The cache parameters must stay on ONE line. A multi-line value breaks the chart's ConfigMap render with could not find expected ':'. I hit this — the originally-proposed multi-line form does not work.
• The forward block is deliberately untouched. Its options (policy, max_fails, health_check) require newlines and cannot be expressed through the chart's single-line parameters; CoreDNS rejects the one-line form with health_check: unknown option }.

Known not fixed

AAAA for lease-backed names is deterministically shadowed — dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and hits the wildcard every time (8/8 in-cluster queries). IPv6-preferring clients get the public proxy consistently, not intermittently. serve_stale does not help here.

That needs a NODATA response for those names, which means the template plugin — the exact thing that broke DNS before. Filing separately so it can be scoped tightly and tested on its own.

Also not established: why the UDM's dnsmasq drops off (rate-limiting vs. reload vs. lease churn). That needs UDM-side logs.

Refs #393

🤖 Generated with Claude Code

Fixes the confirmed root cause of #393 — and the same cause explains the zigbee2mqtt crash-loop. > ⚠️ First: **the issue's premise is wrong.** There is no "pinning job". The per-device records are ephemeral **DHCP-lease records in the UDM's dnsmasq**, not IaC — `dig @10.2.50.1 shellyplusi4-....iot.renner.uno` returns `aa` with **TTL 5**, whereas the OpenTofu static records are TTL 300. Different origin entirely. > > `external-dns-unifi` is also **not** the culprit, despite running `--policy=sync` with `--domain-filter=renner.uno`. It writes the UDM's *static-DNS* list, not the lease table, and `--registry=txt` confines its deletions to records carrying its own `k8s-homelab` TXT — which these never had. ## Root cause: three things combine **1. The public wildcard matches at every depth, so `renner.uno` never returns NXDOMAIN.** ``` dig +short @1.1.1.1 definitely-not-real-xyz123.iot.renner.uno proxy.renner.uno. 45.157.177.26 ``` A name that fails locally doesn't fail — it resolves to the WAN proxy. **2. TTL asymmetry makes the wrong answer stick 60× longer.** | | TTL | |---|---| | correct answer (dnsmasq lease record) | **5** | | wrong answer (public wildcard CNAME) | **300** | One 5-second local failure pins a wrong answer for 5 minutes. That is precisely the *"correct at 14:10, wildcard-only at 15:45"* report in the issue. **3. The UDM's dnsmasq refuses connections in bursts** — **330** `connection refused` errors in the last 3000 k8s-gateway log lines, on **both** `10.2.40.1` and `fd00::1`: ``` [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. AAAA: read udp ...->10.2.40.1:53: read: connection refused [ERROR] plugin/errors: 2 slzb-zigbee.renner.uno. A: read udp ...->[fd00::1]:53: read: connection refused ``` ## The fix - `success ... 60` — clamps the 5s TTL up to a **60s floor** - `serve_stale 1h immediate` — keeps answering from cache while the upstream is refusing connections So a UDM blip stops becoming a lookup failure that the wildcard can capture. **This also fixes the z2m crash-loop.** z2m exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry — and that exact name is in the failure log above. ## Validation — done thoroughly, because this is cluster-critical CoreDNS changes have bitten here before: the `template` plugin broke cluster DNS in `7a09bad5`. **This PR does not reintroduce `template`.** | Check | Result | |---|---| | Repo's full Corefile under real `coredns:v1.13.2` | parses and starts (only the expected out-of-cluster kubernetes-plugin error) | | Repo's k8s-gateway values via `helm template` | `rc=0`, cache line renders intact | | Rendered Corefile under real `k8s_gateway:1.7.0` (CoreDNS 1.14.2) | starts cleanly | Two constraints I hit and encoded in comments so they aren't rediscovered: - **The cache parameters must stay on ONE line.** A multi-line value breaks the chart's ConfigMap render with `could not find expected ':'`. I hit this — the originally-proposed multi-line form does not work. - **The `forward` block is deliberately untouched.** Its options (`policy`, `max_fails`, `health_check`) require newlines and cannot be expressed through the chart's single-line `parameters`; CoreDNS rejects the one-line form with `health_check: unknown option }`. ## Known *not* fixed **AAAA for lease-backed names is deterministically shadowed** — dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and hits the wildcard *every* time (8/8 in-cluster queries). IPv6-preferring clients get the public proxy consistently, not intermittently. `serve_stale` does not help here. That needs a NODATA response for those names, which means the `template` plugin — the exact thing that broke DNS before. Filing separately so it can be scoped tightly and tested on its own. Also not established: *why* the UDM's dnsmasq drops off (rate-limiting vs. reload vs. lease churn). That needs UDM-side logs. Refs #393 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-27 19:44:42 +00:00

fix(dns): cache floor + serve_stale so UDM blips stop resolving to the WAN proxy ... adbdd53a38

Root cause of #393, confirmed. Three things combine:

1. The public renner.uno zone has a *.renner.uno CNAME to proxy.renner.uno
   that matches at EVERY depth, so renner.uno never returns NXDOMAIN:
     definitely-not-real-xyz123.iot.renner.uno -> proxy.renner.uno -> 45.157.177.26
   Any name that fails locally resolves publicly to the WAN proxy instead of
   failing honestly.

2. TTL asymmetry. The correct answer is a dnsmasq DHCP-lease record with
   TTL 5 (verified: `dig @10.2.50.1 shellyplusi4-....iot.renner.uno` returns
   `aa` + TTL 5). The wrong wildcard answer has TTL 300 — 60x more durable.
   So one 5-second failure pins a wrong answer for 5 minutes. That is exactly
   the "correct at 14:10, wildcard-only at 15:45" report in the issue.

3. The UDM's dnsmasq refuses connections in bursts — 330 "connection refused"
   errors in the last 3000 k8s-gateway log lines, on BOTH 10.2.40.1 and
   fd00::1.

Note the issue's premise was wrong: there is no "pinning job". The device
records are ephemeral DHCP-lease records in dnsmasq, not IaC. external-dns-unifi
is also NOT the culprit despite running --policy=sync: it writes the UDM's
static-DNS list, not the lease table, and --registry=txt confines deletion to
records carrying its own TXT.

Fix: raise the cache floor and serve through upstream outages.
  success ... 60      clamps the 5s TTL up to a 60s floor
  serve_stale 1h immediate  keeps answering while the upstream refuses

This also fixes the zigbee2mqtt crash-loop, which has the same cause — z2m
exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry, and the
k8s-gateway log shows that exact name failing against both UDM addresses.

Validated before committing, because CoreDNS is cluster-critical and the
`template` plugin previously broke cluster DNS here (7a09bad5):
- The repo's full Corefile runs under the real coredns:v1.13.2 image (parses;
  only the expected out-of-cluster kubernetes-plugin error).
- The repo's k8s-gateway values render via `helm template` (rc=0) and the
  resulting Corefile starts under the real k8s_gateway:1.7.0 image
  (CoreDNS 1.14.2).
- The cache parameters MUST stay on one line: a multi-line value breaks the
  chart's ConfigMap render with "could not find expected ':'". Confirmed by
  hitting it.
- The forward block is deliberately untouched — its options require newlines
  and cannot be expressed through the chart's single-line parameters.
- The `template` plugin is NOT reintroduced.

Known not fixed: AAAA for lease-backed names is deterministically shadowed
(dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and
hits the wildcard every time). That needs a NODATA response for those names
and is a separate, tightly-scoped change — filing separately.

Refs #393

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot referenced this pull request 2026-07-27 19:45:09 +00:00

AAAA for lease-backed iot.renner.uno names is deterministically shadowed by the public wildcard #436

claude-bot referenced this pull request 2026-07-27 19:45:26 +00:00

iot.renner.uno answers inconsistently: pinned device A records intermittently shadowed by the proxy wildcard CNAME #393

review-bot reviewed 2026-07-27 19:49:48 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit adbdd53a3861e122aa80a8ad8ecfacd0dfcc23a2.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR raises the CoreDNS cache TTL floor to 60s and enables serve_stale 1h immediate for the renner.uno zone, mirrored identically in both the main CoreDNS Corefile (kubernetes/infrastructure/coredns-config/configmap.yaml) and the k8s-gateway HelmRelease's extraZonePlugins (kubernetes/infrastructure/k8s-gateway/helmrelease.yaml). The diff is config-only, well-commented, and both files are kept in sync. The cache plugin syntax and YAML embedding both check out. One change — the negative-cache (denial) TTL ceiling — isn't explained by the stated rationale and is worth confirming before merge; no blocking issues found.

🟡 Suggestions

• kubernetes/infrastructure/coredns-config/configmap.yaml:24 and the mirrored kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:65 — denial 9984 300 30 raises the negative-cache (NXDOMAIN/NODATA) TTL ceiling from the old shared cache 30 cap of 30s up to 300s, a 10x increase. This isn't mentioned anywhere in the PR's rationale, which is entirely about the success-response floor (60s) and serve_stale. Under the success/denial CAPACITY [TTL] [MINTTL] syntax, the third argument is a floor and the second a ceiling, so this block now allows a "does not exist (yet)" answer to be cached for up to 5 minutes — the same class of problem this PR is fixing for positive answers ("TTL asymmetry ... pins a wrong answer for minutes"), just for negative ones. Concretely: a device that gets a new DHCP lease shortly after a query for its name returned NXDOMAIN would now take up to 5 minutes (vs. 30s before) to become resolvable, worse than before this change. Worth confirming this ceiling was chosen deliberately (e.g. to reduce repeat-querying a flaky UDM for names that consistently don't exist) rather than just mirrored from the success block's 300/60 pair without considering denial's different failure mode. I could not verify this against live CoreDNS behavior in this sandbox (no network/cluster access) — the parameter-order reading above is based on the documented cache plugin syntax; worth double-checking against coredns/plugin/cache/README.md for the exact semantics before relying on this analysis, though the author's own PR description states helm template and real-container validation were already done for the block as a whole.
• kubernetes/infrastructure/coredns-config/configmap.yaml:22 — the top-level cache 300 positional TTL becomes redundant once both success and denial sub-blocks specify their own TTL/MINTTL explicitly; per the plugin's docs the explicit blocks take precedence. Harmless, but slightly misleading to a future reader who might assume the 300 here does something on its own. Same applies to the mirrored HelmRelease parameters string.

Tests

This is a GitOps config repo; there's no CI step here that lints or renders the Corefile/HelmRelease values (checked .woodpecker/*.yml — none run helm template, kustomize build, or a YAML/Corefile linter for this path), so validation is inherently manual/Flux-reconcile-time, consistent with how this repo already operates. The PR body states manual validation was done against a real coredns:v1.13.2 container and a real k8s_gateway:1.7.0 container (CoreDNS 1.14.2) plus helm template; I could not independently reproduce that in this review sandbox (no container/cluster access), but the two config blocks are syntactically consistent with each other and with the existing file's conventions, so I have no reason to doubt the claim. No automated regression test guards against a future edit reintroducing the template plugin or breaking the single-line parameters constraint — both risks are called out in comments instead, which is a reasonable mitigation for a repo of this kind.

Recommendation

Looks reasonable — no blocking issues; the denial-TTL-ceiling point above is worth a quick confirmation but doesn't need to hold up the merge.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `adbdd53a3861e122aa80a8ad8ecfacd0dfcc23a2`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR raises the CoreDNS `cache` TTL floor to 60s and enables `serve_stale 1h immediate` for the `renner.uno` zone, mirrored identically in both the main CoreDNS `Corefile` (`kubernetes/infrastructure/coredns-config/configmap.yaml`) and the k8s-gateway HelmRelease's `extraZonePlugins` (`kubernetes/infrastructure/k8s-gateway/helmrelease.yaml`). The diff is config-only, well-commented, and both files are kept in sync. The `cache` plugin syntax and YAML embedding both check out. One change — the negative-cache (`denial`) TTL ceiling — isn't explained by the stated rationale and is worth confirming before merge; no blocking issues found. ## 🟡 Suggestions - **`kubernetes/infrastructure/coredns-config/configmap.yaml:24`** and the mirrored **`kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:65`** — `denial 9984 300 30` raises the negative-cache (NXDOMAIN/NODATA) TTL ceiling from the old shared `cache 30` cap of 30s up to **300s**, a 10x increase. This isn't mentioned anywhere in the PR's rationale, which is entirely about the *success*-response floor (60s) and `serve_stale`. Under the `success/denial CAPACITY [TTL] [MINTTL]` syntax, the third argument is a floor and the second a ceiling, so this block now allows a "does not exist (yet)" answer to be cached for up to 5 minutes — the same class of problem this PR is fixing for positive answers ("TTL asymmetry ... pins a wrong answer for minutes"), just for negative ones. Concretely: a device that gets a new DHCP lease shortly after a query for its name returned NXDOMAIN would now take up to 5 minutes (vs. 30s before) to become resolvable, worse than before this change. Worth confirming this ceiling was chosen deliberately (e.g. to reduce repeat-querying a flaky UDM for names that consistently don't exist) rather than just mirrored from the `success` block's `300`/`60` pair without considering denial's different failure mode. I could not verify this against live CoreDNS behavior in this sandbox (no network/cluster access) — the parameter-order reading above is based on the documented `cache` plugin syntax; worth double-checking against `coredns/plugin/cache/README.md` for the exact semantics before relying on this analysis, though the author's own PR description states `helm template` and real-container validation were already done for the block as a whole. - **`kubernetes/infrastructure/coredns-config/configmap.yaml:22`** — the top-level `cache 300` positional TTL becomes redundant once both `success` and `denial` sub-blocks specify their own TTL/MINTTL explicitly; per the plugin's docs the explicit blocks take precedence. Harmless, but slightly misleading to a future reader who might assume the `300` here does something on its own. Same applies to the mirrored HelmRelease parameters string. ## Tests This is a GitOps config repo; there's no CI step here that lints or renders the `Corefile`/HelmRelease values (checked `.woodpecker/*.yml` — none run `helm template`, `kustomize build`, or a YAML/Corefile linter for this path), so validation is inherently manual/Flux-reconcile-time, consistent with how this repo already operates. The PR body states manual validation was done against a real `coredns:v1.13.2` container and a real `k8s_gateway:1.7.0` container (CoreDNS 1.14.2) plus `helm template`; I could not independently reproduce that in this review sandbox (no container/cluster access), but the two config blocks are syntactically consistent with each other and with the existing file's conventions, so I have no reason to doubt the claim. No automated regression test guards against a future edit reintroducing the `template` plugin or breaking the single-line `parameters` constraint — both risks are called out in comments instead, which is a reasonable mitigation for a repo of this kind. ## Recommendation Looks reasonable — no blocking issues; the denial-TTL-ceiling point above is worth a quick confirmation but doesn't need to hold up the merge. <!-- pr-review:sha=adbdd53a3861e122aa80a8ad8ecfacd0dfcc23a2 -->

k.renner merged commit a67eee0e48 into master 2026-07-27 20:19:00 +00:00

k.renner referenced this issue from a commit 2026-07-27 20:19:01 +00:00

Merge pull request 'fix(dns): cache floor + serve_stale so UDM blips stop resolving to the WAN proxy' (#435) from claude-bot/dns-serve-stale into master

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#435