fix(pve): accept IPv6 RAs on forwarding nodes, converge vmbr0 MTU #313

2026-07-19T12:26:52Z

claude-bot commented

2026-07-19 12:26:52 +00:00

Adds an Ansible role for PVE node host networking, fixing two defects found by live read-only inspection of the cluster on 2026-07-19.

1. No IPv6 on any external interface (all three nodes)

/etc/frr/frr.conf enables ipv6 forwarding for the OSPFv3 Thunderbolt mesh that carries corosync, setting net.ipv6.conf.all.forwarding=1. The kernel default accept_ra=1 means "accept Router Advertisements only while NOT forwarding" — so since FRR came up, the nodes have silently discarded every RA the gateway sends on VLAN 40.

Result: every external interface had only a link-local fe80:: address. The sole non-link-local IPv6 anywhere on the nodes was the corosync mesh /128, which is static in frr.conf and never depended on RAs — which is exactly why the mesh was the only thing with IPv6.

The gateway is not at fault, verified before changing anything:

11:18:03  fe80::265a:4cff:fea6:fd3c > ff02::1: ICMP6, router advertisement
11:18:19  ... router advertisement
11:18:30  ... router advertisement      (5 in ~50s on vmbr1.40)

VLAN 40 carries both the Telekom GUA prefix 2003:d6:9717:a702::/64 and a ULA fd40::/64, and non-forwarding hosts on the LAN autoconfigure normally. The PVE nodes are the only hosts running FRR, which bounds the blast radius to exactly these three.

Fixed with a /etc/sysctl.d/ drop-in setting accept_ra=2 — the value a forwarding host needs to still autoconfigure its own uplink.

2. saturn's vmbr0 MTU asymmetry

saturn's vmbr0 sits at MTU 1500 while its own uplink enp2s0f0np0 and both peer nodes' bridges are at 9000 — saturn's stanza is simply missing the mtu line jupiter and neptune both have. Since the three bridges share one L2 domain, large frames between saturn's guests and the other nodes' guests blackhole rather than fragmenting.

Risk separation

The two fixes are not applied the same way:

Fix	Behaviour
`accept_ra` sysctl	Non-disruptive. Applies on every run, asserts the value took, then waits up to 60s for a global address and reports per node.
`vmbr0` MTU	Staged as a pending PVE network change. Applied only with `-e pve_node_network_apply_pending=true`, because applying it reloads networking and briefly interrupts the Talos k8s VM and its Ceph OSD.

The play uses serial: 1 so nodes converge one at a time.

Implementation note

The MTU task issues a pvesh get and passes the full bridge definition back on the set. A PUT on /nodes/<node>/network/<iface> replaces rather than merges — passing only --mtu would silently drop bridge_ports, bridge_vlan_aware and bridge_vids, detaching the uplink and flattening every SDN VNet.

How to run

Safe (converges IPv6, stages the MTU change without applying it):

ansible-playbook -i inventory/ plays/configure-pve-node-network.yml

Applying the staged MTU change, one node at a time:

ansible-playbook -i inventory/ plays/configure-pve-node-network.yml \
  --limit saturn -e pve_node_network_apply_pending=true

Status and caveats

Not yet applied. No run has been made against the cluster; --syntax-check only.
One unverified assumption: this relies on the kernel doing SLAAC from accept_ra=2 + autoconf=1 without an ifupdown inet6 stanza. jupiter has iface vmbr1.40 inet6 auto; saturn and neptune have no inet6 stanza at all. I believe the stanza is unnecessary but did not prove it. The role's wait-and-report task will show this per node — if jupiter gets an address and the other two don't, they need the stanza added.

Out of scope, found along the way

Neither is touched here; both deserve their own decision.

update-pve-rolling.yml targets inventory group proxmox_nodes, which is not defined anywhere (only pve_cluster exists), and still carries the comment # Replace with your inventory group name. That play is imported by plays/update.yml, which runs from the cron-driven .woodpecker/update-all.yml — so rolling PVE updates have been silently matching zero hosts. Fixing the group name would switch on scheduled dist-upgrades and reboots of the whole cluster that have never run, so it is left alone here. The guard on 'proxmox_nodes' not in group_names in roles/qemu-guest-agent is dead code for the same reason.
roles/keepalived-ntp does not exist on disk, so install-ntp-ha.yml fails today — yet the NTP VIP 10.2.40.4 is live on jupiter, presumably from a run when the role still existed. NTP HA is currently running unmanaged and would not survive a rebuild.

Wiki notes added/corrected for all of the above, including the corosync-over-FRR-mesh and Proxmox-SDN corrections this investigation turned up.

🤖 Generated with Claude Code

Adds an Ansible role for PVE node host networking, fixing two defects found by live read-only inspection of the cluster on 2026-07-19. ## 1. No IPv6 on any external interface (all three nodes) `/etc/frr/frr.conf` enables `ipv6 forwarding` for the OSPFv3 Thunderbolt mesh that carries corosync, setting `net.ipv6.conf.all.forwarding=1`. The kernel default `accept_ra=1` means *"accept Router Advertisements only while NOT forwarding"* — so since FRR came up, the nodes have silently discarded every RA the gateway sends on VLAN 40. Result: every external interface had only a link-local `fe80::` address. The sole non-link-local IPv6 anywhere on the nodes was the corosync mesh `/128`, which is static in `frr.conf` and never depended on RAs — which is exactly why the mesh was the only thing with IPv6. The gateway is **not** at fault, verified before changing anything: ``` 11:18:03 fe80::265a:4cff:fea6:fd3c > ff02::1: ICMP6, router advertisement 11:18:19 ... router advertisement 11:18:30 ... router advertisement (5 in ~50s on vmbr1.40) ``` VLAN 40 carries both the Telekom GUA prefix `2003:d6:9717:a702::/64` and a ULA `fd40::/64`, and non-forwarding hosts on the LAN autoconfigure normally. The PVE nodes are the only hosts running FRR, which bounds the blast radius to exactly these three. Fixed with a `/etc/sysctl.d/` drop-in setting `accept_ra=2` — the value a forwarding host needs to still autoconfigure its own uplink. ## 2. saturn's vmbr0 MTU asymmetry saturn's `vmbr0` sits at MTU 1500 while its own uplink `enp2s0f0np0` and both peer nodes' bridges are at 9000 — saturn's stanza is simply missing the `mtu` line jupiter and neptune both have. Since the three bridges share one L2 domain, large frames between saturn's guests and the other nodes' guests blackhole rather than fragmenting. ## Risk separation The two fixes are **not** applied the same way: | Fix | Behaviour | |---|---| | `accept_ra` sysctl | Non-disruptive. Applies on every run, asserts the value took, then waits up to 60s for a global address and reports per node. | | `vmbr0` MTU | Staged as a *pending* PVE network change. Applied only with `-e pve_node_network_apply_pending=true`, because applying it reloads networking and briefly interrupts the Talos k8s VM and its Ceph OSD. | The play uses `serial: 1` so nodes converge one at a time. ## Implementation note The MTU task issues a `pvesh get` and passes the **full** bridge definition back on the `set`. A PUT on `/nodes/<node>/network/<iface>` **replaces** rather than merges — passing only `--mtu` would silently drop `bridge_ports`, `bridge_vlan_aware` and `bridge_vids`, detaching the uplink and flattening every SDN VNet. ## How to run Safe (converges IPv6, stages the MTU change without applying it): ``` ansible-playbook -i inventory/ plays/configure-pve-node-network.yml ``` Applying the staged MTU change, one node at a time: ``` ansible-playbook -i inventory/ plays/configure-pve-node-network.yml \ --limit saturn -e pve_node_network_apply_pending=true ``` ## Status and caveats - **Not yet applied.** No run has been made against the cluster; `--syntax-check` only. - **One unverified assumption:** this relies on the kernel doing SLAAC from `accept_ra=2` + `autoconf=1` without an ifupdown `inet6` stanza. jupiter has `iface vmbr1.40 inet6 auto`; saturn and neptune have no `inet6` stanza at all. I believe the stanza is unnecessary but did not prove it. The role's wait-and-report task will show this per node — if jupiter gets an address and the other two don't, they need the stanza added. ## Out of scope, found along the way Neither is touched here; both deserve their own decision. 1. **`update-pve-rolling.yml` targets inventory group `proxmox_nodes`, which is not defined anywhere** (only `pve_cluster` exists), and still carries the comment `# Replace with your inventory group name`. That play is imported by `plays/update.yml`, which runs from the cron-driven `.woodpecker/update-all.yml` — so rolling PVE updates have been silently matching zero hosts. Fixing the group name would switch on scheduled dist-upgrades and reboots of the whole cluster that have never run, so it is left alone here. The guard on `'proxmox_nodes' not in group_names` in `roles/qemu-guest-agent` is dead code for the same reason. 2. **`roles/keepalived-ntp` does not exist on disk**, so `install-ntp-ha.yml` fails today — yet the NTP VIP `10.2.40.4` is live on jupiter, presumably from a run when the role still existed. NTP HA is currently running unmanaged and would not survive a rebuild. Wiki notes added/corrected for all of the above, including the corosync-over-FRR-mesh and Proxmox-SDN corrections this investigation turned up. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-19 12:26:52 +00:00

fix(pve): accept IPv6 RAs on forwarding nodes, converge vmbr0 MTU 2d6452650f

Live inspection of the PVE cluster on 2026-07-19 found two defects.

1. No IPv6 address on any external interface, on all three nodes.

   /etc/frr/frr.conf enables "ipv6 forwarding" for the OSPFv3 Thunderbolt
   mesh that carries corosync, which sets net.ipv6.conf.all.forwarding=1.
   The kernel default accept_ra=1 means "accept RAs only while NOT
   forwarding", so the nodes silently discarded the Router Advertisements
   the gateway sends on VLAN 40. Every external interface was left with
   only a link-local address; the sole non-link-local IPv6 anywhere on the
   nodes was the corosync mesh /128, which is static in frr.conf and never
   depended on RAs.

   Verified the gateway is not at fault: RAs arrive on vmbr1.40 from
   fe80::265a:4cff:fea6:fd3c every ~10-15s, carrying both the Telekom GUA
   prefix and a ULA, and non-forwarding hosts on the LAN autoconfigure
   normally. The PVE nodes are the only hosts running FRR.

   Fixed with a sysctl drop-in setting accept_ra=2, the value a forwarding
   host needs in order to still autoconfigure its own uplink.

2. saturn's vmbr0 is at MTU 1500 while its own uplink enp2s0f0np0 and both
   peer nodes' bridges are at 9000. Since the three bridges share one L2
   domain, large frames between saturn's guests and the other nodes' guests
   blackhole rather than fragmenting. saturn's vmbr0 stanza is simply
   missing the mtu line that jupiter and neptune both have.

The two fixes are separated by risk. The sysctl change is non-disruptive
and applies on every run. The MTU change is staged as a pending PVE network
change and applied only when pve_node_network_apply_pending is set, because
applying it reloads networking on the node and briefly interrupts the Talos
k8s VM and its Ceph OSD. The play uses serial: 1 so nodes converge one at a
time.

The MTU task reconstructs the full bridge definition from a pvesh GET before
issuing the PUT. A PUT on /nodes/<node>/network/<iface> replaces rather than
merges, so passing only --mtu would silently drop bridge_ports,
bridge_vlan_aware and bridge_vids, detaching the uplink and flattening every
SDN VNet.

Not applied yet — no run has been made against the cluster.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 71b86e8558 into master

2026-07-19 12:36:18 +00:00

k.renner referenced this issue from a commit

2026-07-19 12:36:18 +00:00

Merge pull request 'fix(pve): accept IPv6 RAs on forwarding nodes, converge vmbr0 MTU' (#313) from claude-bot/pve-node-network-ipv6-mtu into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#313