From adbdd53a3861e122aa80a8ad8ecfacd0dfcc23a2 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Mon, 27 Jul 2026 21:43:57 +0200 Subject: [PATCH] fix(dns): cache floor + serve_stale so UDM blips stop resolving to the WAN proxy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of #393, confirmed. Three things combine: 1. The public renner.uno zone has a *.renner.uno CNAME to proxy.renner.uno that matches at EVERY depth, so renner.uno never returns NXDOMAIN: definitely-not-real-xyz123.iot.renner.uno -> proxy.renner.uno -> 45.157.177.26 Any name that fails locally resolves publicly to the WAN proxy instead of failing honestly. 2. TTL asymmetry. The correct answer is a dnsmasq DHCP-lease record with TTL 5 (verified: `dig @10.2.50.1 shellyplusi4-....iot.renner.uno` returns `aa` + TTL 5). The wrong wildcard answer has TTL 300 — 60x more durable. So one 5-second failure pins a wrong answer for 5 minutes. That is exactly the "correct at 14:10, wildcard-only at 15:45" report in the issue. 3. The UDM's dnsmasq refuses connections in bursts — 330 "connection refused" errors in the last 3000 k8s-gateway log lines, on BOTH 10.2.40.1 and fd00::1. Note the issue's premise was wrong: there is no "pinning job". The device records are ephemeral DHCP-lease records in dnsmasq, not IaC. external-dns-unifi is also NOT the culprit despite running --policy=sync: it writes the UDM's static-DNS list, not the lease table, and --registry=txt confines deletion to records carrying its own TXT. Fix: raise the cache floor and serve through upstream outages. success ... 60 clamps the 5s TTL up to a 60s floor serve_stale 1h immediate keeps answering while the upstream refuses This also fixes the zigbee2mqtt crash-loop, which has the same cause — z2m exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry, and the k8s-gateway log shows that exact name failing against both UDM addresses. Validated before committing, because CoreDNS is cluster-critical and the `template` plugin previously broke cluster DNS here (7a09bad5): - The repo's full Corefile runs under the real coredns:v1.13.2 image (parses; only the expected out-of-cluster kubernetes-plugin error). - The repo's k8s-gateway values render via `helm template` (rc=0) and the resulting Corefile starts under the real k8s_gateway:1.7.0 image (CoreDNS 1.14.2). - The cache parameters MUST stay on one line: a multi-line value breaks the chart's ConfigMap render with "could not find expected ':'". Confirmed by hitting it. - The forward block is deliberately untouched — its options require newlines and cannot be expressed through the chart's single-line parameters. - The `template` plugin is NOT reintroduced. Known not fixed: AAAA for lease-backed names is deterministically shadowed (dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and hits the wildcard every time). That needs a NODATA response for those names and is a separate, tightly-scoped change — filing separately. Refs #393 Co-Authored-By: Claude Opus 5 (1M context) --- .../coredns-config/configmap.yaml | 16 ++++++++++- .../k8s-gateway/helmrelease.yaml | 27 ++++++++++++++++++- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/kubernetes/infrastructure/coredns-config/configmap.yaml b/kubernetes/infrastructure/coredns-config/configmap.yaml index d51bab9e..93d60c6e 100644 --- a/kubernetes/infrastructure/coredns-config/configmap.yaml +++ b/kubernetes/infrastructure/coredns-config/configmap.yaml @@ -15,9 +15,23 @@ data: # (The old Proxmox DNS VIP 10.2.40.53 is DECOMMISSIONED.) 10.96.0.11 is a # pinned ClusterIP of the k8s-gateway Service — avoids DNS-name # resolution during CoreDNS startup. + # Cache tuning mirrors k8s-gateway's (issue #393): the UDM's dnsmasq + # refuses connections in bursts, and because the public renner.uno zone + # has a *.renner.uno wildcard CNAME that matches at every depth, a local + # failure never returns NXDOMAIN — it resolves to the WAN proxy instead. + # Correct answers carry TTL 5 (dnsmasq lease records) while the wrong + # wildcard answer carries TTL 300, so one brief blip pins a wrong answer + # for minutes. `success ... 60` raises the 5s TTL to a 60s floor and + # `serve_stale 1h immediate` keeps serving through an upstream outage. + # NOTE: only cache/serve_stale — the `template` plugin is deliberately NOT + # reintroduced here (it broke cluster DNS in 7a09bad5). renner.uno:53 { errors - cache 30 + cache 300 { + success 9984 300 60 + denial 9984 300 30 + serve_stale 1h immediate + } forward . 10.96.0.11 loop reload diff --git a/kubernetes/infrastructure/k8s-gateway/helmrelease.yaml b/kubernetes/infrastructure/k8s-gateway/helmrelease.yaml index 657c78e0..75564d5c 100644 --- a/kubernetes/infrastructure/k8s-gateway/helmrelease.yaml +++ b/kubernetes/infrastructure/k8s-gateway/helmrelease.yaml @@ -64,8 +64,33 @@ spec: parameters: ":8080" - name: ready parameters: ":8181" + # Survive the UDM's intermittent DNS outages (issue #393). + # + # The UDM's dnsmasq refuses connections in bursts — 330 "connection + # refused" errors in the last 3000 log lines here, on BOTH 10.2.40.1 and + # fd00::1. That alone would only be a blip, but it combines with a + # wildcard trap: the public renner.uno zone has a *.renner.uno CNAME to + # proxy.renner.uno that matches at EVERY depth, so renner.uno never + # returns NXDOMAIN. Any name that fails locally resolves publicly to the + # WAN proxy instead of failing honestly. + # + # The TTLs make it stick: the correct answer is a dnsmasq lease record + # with TTL 5, the wrong wildcard answer has TTL 300 — 60x more durable. + # One 5-second UDM blip therefore pins a wrong answer for 5 minutes. + # + # `success ... 60` clamps that 5s TTL up to a 60s floor, and + # `serve_stale 1h immediate` keeps answering from cache while the + # upstream is refusing connections, so a blip stops becoming a lookup + # failure that the wildcard can capture. + # + # Written on ONE line deliberately: the chart interpolates `parameters` + # into its ConfigMap template, and a multi-line value breaks the YAML + # render ("could not find expected ':'"). Verified with `helm template` + # plus a real `k8s_gateway:1.7.0` container (CoreDNS 1.14.2) before + # merging. For the same reason the forward block is left untouched — + # its options need newlines and cannot be expressed here. - name: cache - parameters: "30" + parameters: "300 { success 9984 300 60 denial 9984 300 30 serve_stale 1h immediate }" - name: loop - name: reload - name: forward -- 2.54.0