konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 5 Packages Projects Releases 1 Wiki Activity

Cluster has no IPv6 egress: four equal-priority default routes, only one with a global address #652

New Issue
Closed
opened 2026-08-27 05:27:05 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-08-27 05:27:05 +00:00
Member
Copy Link
Copy Source

Pods cannot reach the IPv6 internet. IPv4 egress from the same pod works.

pod -> https://1.1.1.1                  301       (v4 fine)
pod -> https://[2606:4700:4700::1111]   timeout   (v6 dead)

This is what the two NetBird alerts actually are

NetbirdRelayMemberDown (#603) and NetbirdPublicEdgeMemberDown (#599) are firing and both read as though the proxies were down. They are not. Every failing probe target is an IPv6 literal, and the blackbox exporter is a pod, so the probes fail on cluster egress rather than on the far side.

Confirmed from outside the cluster — both proxies answer IPv6 in ~13 ms from a workstation on another VLAN:

2a03:4000:4b:c1e:4474:43ff:fedf:8f85   2 received, 0% loss, 13.8 ms
2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29   2 received, 0% loss, 13.2 ms

The one IPv4 target in the same probe group (45.157.177.26) was last up minutes ago, while every IPv6 target has been down for ~30 h. Same job, same exporter, split cleanly by address family.

What is actually wrong

Each node carries four IPv6 default routes at identical priority 1024, via the same gateway fe80::265a:4cff:fea6:fd3c:

interface global IPv6?
mgmt0 yes — 2003:d6:9718:3b02:…
enp2s0f0np0 no — ULA fd00::… only
enp87s0 no — ULA fd00::… only
lo no

Only mgmt0 has a globally routable source address. With four equal-priority defaults the kernel has no reason to prefer it, so outbound IPv6 mostly leaves through an interface whose only usable source is a ULA — unroutable on the internet.

Cilium is not the cause. It masquerades IPv6 (Masquerading: IPTables [IPv4: Enabled, IPv6: Enabled]) and its own interface list shows the same split: ULA on the physical ports, the global address only on mgmt0.

Identical on all three nodes: four IPv6 default routes each.

Why it probably started now

The node config requests IPv4 only — mgmt0 has a DHCPv4Config and nothing for IPv6 — so every IPv6 address and route here comes from router advertisements. RAs are arriving and are being accepted; that part is healthy:

Icmp6InRouterAdvertisements   1234
disable_ipv6 0   autoconf 1   accept_ra 2   accept_ra_defrtr 1

On the Proxmox VMs each network was a separate virtual NIC, so RAs were only ever seen where they belonged. On bare metal one physical port carries the untagged network plus the tagged VLANs, so the parent interface now receives RAs too and installs its own default route — competing with the VLAN interface that holds the actual global address.

That makes this most likely bare-metal-conversion fallout rather than a UniFi-side change, though it has not been proven against the RA contents.

Not yet established

  • Whether the untagged network on that port is supposed to carry IPv6 at all.
  • Whether the fix belongs on the node (stop accepting RAs on the parent and secondary interfaces, or give mgmt0 default route a better metric) or on the switch/UDM side.
  • Why lo has an IPv6 default route.

Impact

Low today, because nothing critical depends on outbound IPv6: IPv4 egress is unaffected and in-cluster IPv6 is unaffected. It costs the two NetBird probes, which are now a false signal about proxy health, and it means the cluster is silently IPv4-only outbound.

Pods cannot reach the IPv6 internet. IPv4 egress from the same pod works. ``` pod -> https://1.1.1.1 301 (v4 fine) pod -> https://[2606:4700:4700::1111] timeout (v6 dead) ``` ## This is what the two NetBird alerts actually are `NetbirdRelayMemberDown` (#603) and `NetbirdPublicEdgeMemberDown` (#599) are firing and both read as though the proxies were down. They are not. Every failing probe target is an IPv6 literal, and the blackbox exporter is a pod, so the probes fail on cluster egress rather than on the far side. Confirmed from outside the cluster — both proxies answer IPv6 in ~13 ms from a workstation on another VLAN: ``` 2a03:4000:4b:c1e:4474:43ff:fedf:8f85 2 received, 0% loss, 13.8 ms 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29 2 received, 0% loss, 13.2 ms ``` The one IPv4 target in the same probe group (45.157.177.26) was last up minutes ago, while every IPv6 target has been down for ~30 h. Same job, same exporter, split cleanly by address family. ## What is actually wrong Each node carries **four IPv6 default routes at identical priority 1024**, via the same gateway `fe80::265a:4cff:fea6:fd3c`: | interface | global IPv6? | |---|---| | `mgmt0` | **yes** — `2003:d6:9718:3b02:…` | | `enp2s0f0np0` | no — ULA `fd00::…` only | | `enp87s0` | no — ULA `fd00::…` only | | `lo` | no | Only `mgmt0` has a globally routable source address. With four equal-priority defaults the kernel has no reason to prefer it, so outbound IPv6 mostly leaves through an interface whose only usable source is a ULA — unroutable on the internet. Cilium is not the cause. It masquerades IPv6 (`Masquerading: IPTables [IPv4: Enabled, IPv6: Enabled]`) and its own interface list shows the same split: ULA on the physical ports, the global address only on `mgmt0`. Identical on all three nodes: four IPv6 default routes each. ## Why it probably started now The node config requests IPv4 only — `mgmt0` has a `DHCPv4Config` and nothing for IPv6 — so every IPv6 address and route here comes from router advertisements. RAs are arriving and are being accepted; that part is healthy: ``` Icmp6InRouterAdvertisements 1234 disable_ipv6 0 autoconf 1 accept_ra 2 accept_ra_defrtr 1 ``` On the Proxmox VMs each network was a separate virtual NIC, so RAs were only ever seen where they belonged. On bare metal one physical port carries the untagged network plus the tagged VLANs, so the parent interface now receives RAs too and installs its own default route — competing with the VLAN interface that holds the actual global address. That makes this most likely bare-metal-conversion fallout rather than a UniFi-side change, though it has not been proven against the RA contents. ## Not yet established - Whether the untagged network on that port is supposed to carry IPv6 at all. - Whether the fix belongs on the node (stop accepting RAs on the parent and secondary interfaces, or give `mgmt0` default route a better metric) or on the switch/UDM side. - Why `lo` has an IPv6 default route. ## Impact Low today, because nothing critical depends on outbound IPv6: IPv4 egress is unaffected and in-cluster IPv6 is unaffected. It costs the two NetBird probes, which are now a false signal about proxy health, and it means the cluster is silently IPv4-only outbound.
claude-bot commented 2026-08-27 05:33:07 +00:00
Author
Member
Copy Link
Copy Source

Owner input reframes this: the nodes should have no fd00:: addresses at all. The untagged network on that port is a fallback VLAN and nothing is supposed to run on it.

So the defect is not the route metrics — it is that the nodes are self-configuring on a segment they have no business being on, and the competing default routes are a symptom of that.

Exactly two interfaces are affected, and a third shows what correct looks like:

enp2s0f0np0   fd00::5a47:caff:fe74:7027/64   VLAN trunk parent
enp87s0       fd00::5a47:caff:fe74:7029/64   second NIC
enp2s0f1np1   fe80:: only                    Ceph SAN — no RAs there, so nothing was learned

Both affected interfaces are on the untagged fallback VLAN, took a ULA from its RAs, and each installed its own default route.

The house already has the pattern for this. k8s-{1,2,3}.yaml carry:

net.ipv6.conf.ens19.disable_ipv6: "1"

with the reasoning that host addresses on a link that only exists as a macvlan parent "were never needed and are actively harmful". The same argument applies to the physical ports — it was simply never extended to them.

Note the precedent disables IPv6 on a VLAN child, while the fix here would target VLAN parents. Whether disabling IPv6 on enp2s0f0np0 leaves mgmt0 (a VLAN on it, holding the global address Ceph and Cilium use) untouched needs verifying on one node before it goes to all three — Linux treats them as independent netdevs, but that is worth proving rather than assuming, given what depends on mgmt0.

accept_ra: 0 on the parents is the softer alternative: it stops them learning anything new without disabling the family outright.

Owner input reframes this: **the nodes should have no `fd00::` addresses at all.** The untagged network on that port is a fallback VLAN and nothing is supposed to run on it. So the defect is not the route metrics — it is that the nodes are self-configuring on a segment they have no business being on, and the competing default routes are a symptom of that. Exactly two interfaces are affected, and a third shows what correct looks like: ``` enp2s0f0np0 fd00::5a47:caff:fe74:7027/64 VLAN trunk parent enp87s0 fd00::5a47:caff:fe74:7029/64 second NIC enp2s0f1np1 fe80:: only Ceph SAN — no RAs there, so nothing was learned ``` Both affected interfaces are on the untagged fallback VLAN, took a ULA from its RAs, and each installed its own default route. **The house already has the pattern for this.** k8s-{1,2,3}.yaml carry: ```yaml net.ipv6.conf.ens19.disable_ipv6: "1" ``` with the reasoning that host addresses on a link that only exists as a macvlan parent "were never needed and are actively harmful". The same argument applies to the physical ports — it was simply never extended to them. Note the precedent disables IPv6 on a VLAN *child*, while the fix here would target VLAN *parents*. Whether disabling IPv6 on `enp2s0f0np0` leaves `mgmt0` (a VLAN on it, holding the global address Ceph and Cilium use) untouched needs verifying on one node before it goes to all three — Linux treats them as independent netdevs, but that is worth proving rather than assuming, given what depends on mgmt0. `accept_ra: 0` on the parents is the softer alternative: it stops them learning anything new without disabling the family outright.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#652