From 4c6c23377dd63eb403d4f31bf025e7d93cd09842 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 27 Aug 2026 07:51:51 +0200 Subject: [PATCH] fix(talos): stop the VLAN parents autoconfiguring on the fallback network MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all while IPv4 from the same pod worked normally. enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0 is unused. Both sit on the untagged fallback network, where by design nothing is supposed to run — and both were autoconfiguring from its router advertisements: an fd00::/64 ULA each, plus an IPv6 default route each. That left four IPv6 default routes per node at identical priority 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally routable source address. With nothing to break the tie, outbound IPv6 mostly left through an interface whose only usable source was a ULA. This is the same argument the file already makes for ens19, where host addresses on a link that exists only as a macvlan parent are called "never needed and actively harmful". It was simply never extended to the physical ports. How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had been firing for 30 hours, reading as though the proxies were down. They were not — every failing target is an IPv6 literal and the blackbox exporter is a pod, so the probes were failing on cluster egress. The proxies answered IPv6 in 13 ms from a workstation on another VLAN throughout. Applied to k8s-1 first and verified before rolling to the other two, because the existing precedent disables v6 on a VLAN *child* while this targets VLAN *parents*, and mgmt0 carries the address Ceph and Cilium use: mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged fd00:: addresses 2 -> 0 IPv6 default routes 4 -> 2 (mgmt0, lo) failing blackbox probes 5 -> 0 Ceph 217 PGs active+clean, no OSD restart All three nodes are already patched live; this puts it in git so the next apply-config does not undo it. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/talos/patches/k8s-1.yaml | 23 +++++++++++++++++++++++ kubernetes/talos/patches/k8s-2.yaml | 9 +++++++++ kubernetes/talos/patches/k8s-3.yaml | 9 +++++++++ 3 files changed, 41 insertions(+) diff --git a/kubernetes/talos/patches/k8s-1.yaml b/kubernetes/talos/patches/k8s-1.yaml index e9dbb4a6..8c74ab05 100644 --- a/kubernetes/talos/patches/k8s-1.yaml +++ b/kubernetes/talos/patches/k8s-1.yaml @@ -20,6 +20,29 @@ machine: # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23). # This disables v6 entirely on that link, including link-local. net.ipv6.conf.ens19.disable_ipv6: "1" + # Same argument, applied to the physical ports rather than a VLAN child. + # + # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and + # enp87s0 is unused. Both sit on the untagged fallback network, where by + # design nothing is supposed to run — and both were autoconfiguring + # themselves from its router advertisements: an fd00::/64 ULA each, plus + # an IPv6 default route each. + # + # The result was four IPv6 default routes per node at identical priority + # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a + # globally routable source address. With nothing to break the tie, + # outbound IPv6 mostly left through an interface whose only usable source + # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked + # normally. It surfaced as two NetBird probes reporting the proxies down + # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from + # outside. + # + # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN + # parent leaves the VLAN child alone. mgmt0 kept its global address and + # its fd40:: address, Ceph did not notice, and every failing probe + # recovered on the next scrape. + net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1" + net.ipv6.conf.enp87s0.disable_ipv6: "1" install: # Three NVMe drives are ordinary block devices to Talos here: the Kingston # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the diff --git a/kubernetes/talos/patches/k8s-2.yaml b/kubernetes/talos/patches/k8s-2.yaml index 9a2e076d..8c00c4de 100644 --- a/kubernetes/talos/patches/k8s-2.yaml +++ b/kubernetes/talos/patches/k8s-2.yaml @@ -9,6 +9,15 @@ machine: # macvlan parent for the v30iot pod attachments; host addresses on that L2 # make node replies bypass the UDM and break every UDM-DNATed flow. net.ipv6.conf.ens19.disable_ipv6: "1" + # Same argument for the physical ports: enp2s0f0np0 is only the trunk + # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the + # untagged fallback network where nothing is supposed to run. Both were + # taking an fd00::/64 and an IPv6 default route from its router + # advertisements, leaving four equal-priority default routes per node of + # which only mgmt0 has a globally routable source. See k8s-1.yaml for the + # full reasoning and how it was verified. + net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1" + net.ipv6.conf.enp87s0.disable_ipv6: "1" install: # Three NVMe drives are visible on bare metal. Select the installer target # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever diff --git a/kubernetes/talos/patches/k8s-3.yaml b/kubernetes/talos/patches/k8s-3.yaml index 08e3bf4a..4053ebea 100644 --- a/kubernetes/talos/patches/k8s-3.yaml +++ b/kubernetes/talos/patches/k8s-3.yaml @@ -8,6 +8,15 @@ machine: sysctls: # v30iot NIC carries no host addressing — see k8s-1.yaml. net.ipv6.conf.ens19.disable_ipv6: "1" + # Same argument for the physical ports: enp2s0f0np0 is only the trunk + # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the + # untagged fallback network where nothing is supposed to run. Both were + # taking an fd00::/64 and an IPv6 default route from its router + # advertisements, leaving four equal-priority default routes per node of + # which only mgmt0 has a globally routable source. See k8s-1.yaml for the + # full reasoning and how it was verified. + net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1" + net.ipv6.conf.enp87s0.disable_ipv6: "1" install: # Three NVMe drives are visible on bare metal. Select the installer target # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever -- 2.54.0