fix(talos): stop the VLAN parents autoconfiguring on the fallback network #653

2026-08-27T05:52:17Z

claude-bot commented

2026-08-27 05:52:17 +00:00

Fixes #652. The cluster had no IPv6 egress — pods could not reach the IPv6 internet at all, while IPv4 from the same pod worked normally.

enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0 is unused. Both sit on the untagged fallback network, where by design nothing is supposed to run — and both were autoconfiguring from its router advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.

That left four IPv6 default routes per node at identical priority 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally routable source address. With nothing to break the tie, outbound IPv6 mostly left through an interface whose only usable source was a ULA.

This is the same argument the file already makes for ens19, where host addresses on a link that exists only as a macvlan parent are called "never needed and actively harmful". It was simply never extended to the physical ports.

How it surfaced. NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had been firing for 30 hours, reading as though the proxies were down. They were not — every failing target is an IPv6 literal and the blackbox exporter is a pod, so the probes were failing on cluster egress. The proxies answered IPv6 in 13 ms from a workstation on another VLAN throughout.

Applied to k8s-1 first and verified before rolling to the other two, because the existing precedent disables v6 on a VLAN child while this targets VLAN parents, and mgmt0 carries the address Ceph and Cilium use:

	result
`mgmt0` global + `fd40::`	unchanged
`fd00::` addresses	2 → 0
IPv6 default routes	4 → 2 (`mgmt0`, `lo`)
failing blackbox probes	5 → 0
Ceph	217 PGs `active+clean`, no OSD restart

All three nodes are already patched live; this puts it in git so the next apply-config does not undo it.

Unresolved and deliberately left alone: lo still carries an IPv6 default route. It is harmless — loopback is never chosen for external traffic — but it is odd and nobody put it there on purpose.

Fixes #652. The cluster had no IPv6 egress — pods could not reach the IPv6 internet at all, while IPv4 from the same pod worked normally. `enp2s0f0np0` is nothing but the trunk parent for `mgmt0` and `ens19`, and `enp87s0` is unused. Both sit on the untagged fallback network, where by design nothing is supposed to run — and both were autoconfiguring from its router advertisements: an `fd00::/64` ULA each, plus an IPv6 default route each. That left **four IPv6 default routes per node at identical priority 1024** — `mgmt0`, `enp2s0f0np0`, `enp87s0` and `lo` — of which only `mgmt0` has a globally routable source address. With nothing to break the tie, outbound IPv6 mostly left through an interface whose only usable source was a ULA. This is the same argument the file already makes for `ens19`, where host addresses on a link that exists only as a macvlan parent are called *"never needed and actively harmful"*. It was simply never extended to the physical ports. **How it surfaced.** `NetbirdRelayMemberDown` and `NetbirdPublicEdgeMemberDown` had been firing for 30 hours, reading as though the proxies were down. They were not — every failing target is an IPv6 literal and the blackbox exporter is a pod, so the probes were failing on cluster egress. The proxies answered IPv6 in 13 ms from a workstation on another VLAN throughout. **Applied to k8s-1 first and verified before rolling to the other two**, because the existing precedent disables v6 on a VLAN *child* while this targets VLAN *parents*, and `mgmt0` carries the address Ceph and Cilium use: | | result | |---|---| | `mgmt0` global + `fd40::` | **unchanged** | | `fd00::` addresses | 2 → **0** | | IPv6 default routes | 4 → **2** (`mgmt0`, `lo`) | | failing blackbox probes | 5 → **0** | | Ceph | 217 PGs `active+clean`, no OSD restart | All three nodes are already patched live; this puts it in git so the next `apply-config` does not undo it. Unresolved and deliberately left alone: `lo` still carries an IPv6 default route. It is harmless — loopback is never chosen for external traffic — but it is odd and nobody put it there on purpose.

claude-bot added 1 commit 2026-08-27 05:52:17 +00:00

fix(talos): stop the VLAN parents autoconfiguring on the fallback network

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

4c6c23377d

The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.

enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.

That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.

This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.

How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.

Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:

  mgmt0 kept 2003:d6:9718:3b02:… and fd40::…    unchanged
  fd00:: addresses                              2 -> 0
  IPv6 default routes                           4 -> 2 (mgmt0, lo)
  failing blackbox probes                       5 -> 0
  Ceph                                          217 PGs active+clean, no OSD restart

All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-27 05:55:11 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 4c6c23377dd63eb403d4f31bf025e7d93cd09842.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Extends the existing "no host IPv6 on link-only interfaces" pattern (already applied to the ens19 VLAN child) to the two physical ports it was never applied to: enp2s0f0np0 (the VLAN trunk parent for mgmt0/ens19) and enp87s0 (unused). Purely additive — three sysctl lines per node, no other config touched, changes already verified live on all three nodes per the PR body. No blocking issues found.

🟡 Suggestions

The PR body and per-file comments assert enp87s0 is "unused," but the repo doesn't otherwise define what it's connected to. Corroborating evidence for the interface actually existing does show up in kubernetes/infrastructure/cilium/helmrelease.yaml:105 (cilium-dbg status --verbose lists enp87s0 as a device with an address) — consistent with the RA-autoconfigured fd00::/64 this PR removes. One side effect worth a one-line callout in that file: once IPv6 is disabled here, enp87s0 will lose its only address and Cilium will presumably stop attaching to it, making that comment's device list stale. Not a functional problem, just worth a follow-up note so a future reader isn't confused by the mismatch.
kubernetes/RUNBOOK-baremetal-conversion.md:285-287 documents jupiter's onboard I226 as renumbering from enp89s0 to enp90s0 after a third NVMe was added — a different name than the enp87s0 this PR disables. These are presumably two distinct physical ports (MS-01 has two I226 ports), and the PR body's live-verified route count (4 before, matching only mgmt0, enp2s0f0np0, enp87s0, lo) implies the second I226 port isn't picking up any RA at all (likely unplugged), so this isn't a gap in the fix — but it's not stated explicitly anywhere. Worth a one-line note (here or in the RUNBOOK) confirming the second I226 port is unplugged/inactive, so a future reader doesn't have to re-derive that from the route count.

Tests

No automated test covers this: .woodpecker/ci-manifests-test.yml only runs kubectl kustomize over directories with a kustomization.yaml, and Talos machine-config patches under kubernetes/talos/patches/ aren't part of that tree, so there's no CI syntax/schema check on these three files. That's consistent with how the rest of this directory is handled (no existing tests either), and the PR body documents thorough live verification (route counts, address survival, Ceph PG status, blackbox probe recovery) on real hardware before rollout, which is the appropriate verification method for this kind of change.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `4c6c23377dd63eb403d4f31bf025e7d93cd09842`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Extends the existing "no host IPv6 on link-only interfaces" pattern (already applied to the `ens19` VLAN child) to the two physical ports it was never applied to: `enp2s0f0np0` (the VLAN trunk parent for `mgmt0`/`ens19`) and `enp87s0` (unused). Purely additive — three sysctl lines per node, no other config touched, changes already verified live on all three nodes per the PR body. No blocking issues found. ## 🟡 Suggestions - The PR body and per-file comments assert `enp87s0` is "unused," but the repo doesn't otherwise define what it's connected to. Corroborating evidence for the interface actually existing does show up in `kubernetes/infrastructure/cilium/helmrelease.yaml:105` (`cilium-dbg status --verbose` lists `enp87s0` as a device with an address) — consistent with the RA-autoconfigured `fd00::/64` this PR removes. One side effect worth a one-line callout in that file: once IPv6 is disabled here, `enp87s0` will lose its only address and Cilium will presumably stop attaching to it, making that comment's device list stale. Not a functional problem, just worth a follow-up note so a future reader isn't confused by the mismatch. - `kubernetes/RUNBOOK-baremetal-conversion.md:285-287` documents jupiter's onboard I226 as renumbering from `enp89s0` to `enp90s0` after a third NVMe was added — a different name than the `enp87s0` this PR disables. These are presumably two distinct physical ports (MS-01 has two I226 ports), and the PR body's live-verified route count (4 before, matching only `mgmt0`, `enp2s0f0np0`, `enp87s0`, `lo`) implies the second I226 port isn't picking up any RA at all (likely unplugged), so this isn't a gap in the fix — but it's not stated explicitly anywhere. Worth a one-line note (here or in the RUNBOOK) confirming the second I226 port is unplugged/inactive, so a future reader doesn't have to re-derive that from the route count. ## Tests No automated test covers this: `.woodpecker/ci-manifests-test.yml` only runs `kubectl kustomize` over directories with a `kustomization.yaml`, and Talos machine-config patches under `kubernetes/talos/patches/` aren't part of that tree, so there's no CI syntax/schema check on these three files. That's consistent with how the rest of this directory is handled (no existing tests either), and the PR body documents thorough live verification (route counts, address survival, Ceph PG status, blackbox probe recovery) on real hardware before rollout, which is the appropriate verification method for this kind of change. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 5fd0691ed2 into master

2026-08-27 06:34:52 +00:00

k.renner referenced this issue from a commit

2026-08-27 06:34:52 +00:00

Merge pull request 'fix(talos): stop the VLAN parents autoconfiguring on the fallback network' (#653) from no-ipv6-on-vlan-parents into master

claude-bot referenced this pull request

2026-08-27 07:26:53 +00:00

[alert] NetbirdPublicEdgeMemberDown (monitoring) #599

claude-bot referenced this pull request

2026-08-27 07:27:03 +00:00

[alert] NetbirdRelayMemberDown (monitoring) #603

Sign in to join this conversation.