feat(talos): move the node IP and etcd to pod0 (#576)

Pod east-west has to leave the management VLAN, and the node IP is the
only thing that controls where it goes. Three tests established that,
rather than argument:

  - --ipv4-node / --ipv6-node, applied via CiliumNodeConfig: the agent
    logged receiving both and still published its mgmt0 address.
  - directRoutingDevice=pod0: the agent logged "Direct routing device
    detected direct-routing-device=pod0" and still installed
    10.244.2.0/24 via 10.2.40.138 dev mgmt0.
  - Both routing modes, both address families, always keyed to the
    peer's Kubernetes InternalIP.

etcd is pinned to the same segment. Left unset it advertises "the first
routable address of the node", which is not deterministic once the node
is re-addressed, so it has to be stated either way. pod0 is a closed 10G
L2 between the three nodes; the management VLAN is about to become a
single 1G NIC shared with everything else management does, which is the
wrong link for quorum traffic. listenSubnets defaults to this, so peer
and client listeners move together.

This does move the etcd peer URLs off 10.2.40.x, so unlike the node IP
it is not a no-op: one node at a time, waiting for a healthy quorum
between each.

The API VIP stays on mgmt0; it is a Layer2VIPConfig and does not follow
the node IP.

Apply one node at a time. Each node re-IPs, so per the note in
k8s-1.yaml (#635) kube-apiserver keeps advertising the old address until
its static pod restarts: after each node, check that
`kubectl get endpoints kubernetes -n default` agrees with
`kubectl get nodes -o wide` and reboot the node if it does not. The
cilium agent reads the node IP only at startup, so it needs recreating
on each node as well.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -432,8 +432,8 @@ is why it belongs here.
   among its devices but not `ens19`, because `ens19` is deliberately
   address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
   untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
-  on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
-  VLAN carrying pod traffic will need the same treatment.
+  on the shared parent `uplink`, still tagged. Any future address-less VLAN
+  carrying pod traffic will need the same treatment.
 
   The symptom does not look like a VLAN problem. A tc-layer drop reaches an
   AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space

kubernetes/apps/music-assistant/deployment.yaml

@@ -28,7 +28,7 @@ spec:
         # net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
         # worker, though it is no longer the same kind of device everywhere:
         # on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
-        # of enp2s0f0np0 rather than a NIC, which is why the datapath needs
+        # of uplink rather than a NIC, which is why the datapath needs
         # bpf.vlanBypass to carry it (issue #592).
         k8s.v1.cni.cncf.io/networks: music-assistant-iot
     spec:

kubernetes/infrastructure/post-crds/cilium-bgp.yaml

@@ -25,13 +25,13 @@ spec:
       stop: "fd99::ff"
 ---
 # Cilium BGPv2 control-plane config peering each node with the UDM over the
-# management LAN — never over v30iot, which is the untrusted IoT segment.
+# ingress VLAN 47 — never over v30iot, which is the untrusted IoT segment.
 # The source address comes from the kernel routing decision, so no explicit
 # localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
-#   - v4 session: peer 10.2.40.1, family IPv4 unicast only
-#   - v6 session: peer fd40::1, family IPv6 unicast only
+#   - v4 session: peer 10.2.47.1, family IPv4 unicast only
+#   - v6 session: peer fd02:47::1, family IPv6 unicast only
 # This avoids the "v6 NLRIs over v4 transport" path, which requires
 # bilateral extended-nexthop capability negotiation (RFC 5549/8950).
 # Cilium 1.19 BGPv2 advertises the capability only for the IPv4 Unicast
@@ -74,14 +74,14 @@ spec:
       peers:
         - name: udm-v6
           peerASN: 64512
-          # ULA — stable across ISP renumber events. UDM's br40 has
-          # fd40::1/64; nodes get fd40::<eui-64> via SLAAC on eth0.
-          peerAddress: fd40::1
+          # ULA — stable across ISP renumber events. UDM's br47 has
+          # fd02:47::1/64; nodes get fd02:47::<eui-64> via SLAAC on ingress0.
+          peerAddress: fd02:47::1
           peerConfigRef:
             name: bgp-peer-udm-v6
         - name: udm-v4
           peerASN: 64512
-          peerAddress: 10.2.40.1
+          peerAddress: 10.2.47.1
           peerConfigRef:
             name: bgp-peer-udm-v4
 ---

kubernetes/talos/patches/k8s-1.yaml

@@ -12,11 +12,28 @@ machine:
     # Left enabled they autoconfigure a ULA and an IPv6 default route from its
     # RAs, which ties with mgmt0's and costs the cluster IPv6 egress.
     # Disabling v6 on a VLAN parent does not affect the VLAN child.
-    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
-    net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    net.ipv6.conf.uplink.disable_ipv6: "1"
+    net.ipv6.conf.mgmt.disable_ipv6: "1"
+    # pod0 is a closed L2 fabric between the three nodes: statically
+    # addressed, no gateway, nothing to learn from an RA.
+    net.ipv6.conf.pod0.accept_ra: "0"
+    # ingress0 keeps RAs, at the default accept_ra of 2: an IPv6 default
+    # router is a link-local address learned only from the RA, and with
+    # net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
+    # entirely below 2. That is how mgmt0 has its IPv6 default today.
+    #
+    # The prefix is announced correctly, so SLAAC addresses this link the same
+    # way mgmt0 gets its fd40:: address: derived from the MAC, stable across
+    # reboots, nothing to allocate. Only IPv4 is declared here, and only
+    # because a second DHCP client is what broke the node's DNS record.
+    #
+    # Just the default route is suppressed, because mgmt0 owns it until the
+    # split flips it and two defaults at one metric tie. Dropping this line is
+    # what hands the IPv6 default over.
+    net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # That network also carries the mgmt VLAN's ARP. Answer ARP only on the
     # link that owns the address, or the UDM learns the node IP against
-    # enp87s0's MAC and delivers IPv4 to a NIC Cilium is not attached to (#793).
+    # mgmt's MAC and delivers IPv4 to a NIC Cilium is not attached to (#793).
     net.ipv4.conf.all.arp_ignore: "1"
     net.ipv4.conf.default.arp_ignore: "1"
     net.ipv4.conf.all.arp_announce: "2"
@@ -36,12 +53,15 @@ machine:
     # stated once, here, by the machine that knows.
     storage.renner.uno/ceph-osd: "true"
   udev:
-    # Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
-    # role name at boot, before the network stack opens links — a rule applied
-    # to a RUNNING node cannot rename (Resource busy) and takes effect at the
-    # next reboot.
+    # Renames every port this node uses to its role at boot, matched by
+    # burned-in MAC — kernel names derive from PCI topology and shift when
+    # hardware moves. Shared manifests then name roles and only this file
+    # knows the hardware. A rule applied to a RUNNING node cannot rename
+    # (Resource busy) and takes effect at the next reboot.
     rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:27", NAME="uplink"
       - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:29", NAME="mgmt"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -83,20 +103,17 @@ provisioning:
   grow: true
 ---
 # X710 port 0 carries the switch trunk and takes no addressing itself — its
-# untagged VLAN is a network this node has no business being on. Management
-# and v30iot ride it as tagged VLANs.
+# untagged VLAN is a network this node has no business being on. Management,
+# pod, ingress and v30iot all ride it as tagged VLANs.
 #
-# Links are named by predictable kernel name, which derives from PCI topology
-# and can still shift when hardware is added.
-#
-# LinkAliasConfig would pin these better, but aliasing the X710 ports by MAC
-# and using the aliases as VLAN parents left the node with no address on any
-# interface — the aliases were never created and every document referencing
-# them silently did nothing. Prove any alias-based revision with
-# `talosctl apply-config --mode=try` first; it rolls back on its own.
+# Named by the udev rule above, not by LinkAliasConfig. Aliases are valid VLAN
+# parents, but a selector matching zero links is skipped silently and Talos
+# disables default DHCP machine-wide as soon as any new-style network document
+# exists — so a silent miss leaves the node with no address at all. udev
+# renames the link itself, so a missed match is visible as a missing link.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f0np0
+name: uplink
 up: true
 ---
 # Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
@@ -105,7 +122,7 @@ apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
 # DHCP, and the lease is not reserved in this repo — reservations live on the
@@ -134,9 +151,51 @@ apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
+# Pod network, VLAN 48. Node-to-node only: the VXLAN overlay rides it once the
+# kubelet node IP moves here, and nothing on it is routed off the switch.
+#
+# Statically addressed, no gateway, no DHCP. A lease would make the cluster's
+# data path depend on the UDM being up in order to form, on a segment whose
+# traffic never reaches the UDM. Declared per node, the way ceph-san's segment
+# carries a declared range rather than a lease.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: pod0
+vlanID: 48
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.48.11/24
+  - address: fd02:48::11/64
+---
+# Ingress network, VLAN 47. World-to-pod: the BGP session, the next hop for
+# every advertised LoadBalancer VIP, and the masquerade source for pod egress.
+# Publicly routed, which is what separates it from pod0.
+#
+# Statically addressed, not leased. A second DHCP client on the node announced
+# the same hostname as mgmt0's and the UDM registered the node against this
+# segment instead, so mgmt0's `srv.renner.uno` record disappeared and the
+# lease's resolver displaced the management one. The UDM cannot register a
+# network's clients selectively, so the client goes rather than the record.
+# DHCP is off on VLAN 47 for the same reason.
+#
+# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
+# the segment. The default route moves here in the change that flips it.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ingress0
+vlanID: 47
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.47.11/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
+---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
 # section above renames the port to ceph-san at boot, so shared manifests name

kubernetes/talos/patches/k8s-2.yaml

@@ -10,8 +10,13 @@ machine:
     # Same for the physical ports on the untagged fallback network: left
     # enabled they take an IPv6 default route from its RAs and cost the
     # cluster IPv6 egress.
-    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
-    net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    net.ipv6.conf.uplink.disable_ipv6: "1"
+    net.ipv6.conf.mgmt.disable_ipv6: "1"
+    # pod0 is a closed segment with no router, so it ignores RAs outright.
+    # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
+    # and only the default route is held back — see k8s-1.yaml.
+    net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
     net.ipv4.conf.default.arp_ignore: "1"
@@ -29,9 +34,11 @@ machine:
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
   udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
+    # Renames every port to its role at boot — see k8s-1.yaml.
     rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:73", NAME="uplink"
       - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:75", NAME="mgmt"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -61,17 +68,14 @@ provisioning:
   grow: true
 ---
 # X710 port 0 carries the switch trunk and takes no addressing itself — the
-# port's untagged VLAN is not one this node belongs on. Management and v30iot
-# both ride it as tagged VLANs.
+# port's untagged VLAN is not one this node belongs on. Management, pod,
+# ingress and v30iot all ride it as tagged VLANs.
 #
-# Addressed by predictable kernel name, not by LinkAliasConfig. Aliases are
-# valid VLAN parents, but a selector matching zero links is skipped silently
-# and Talos disables default DHCP machine-wide as soon as any new-style network
-# document exists — so a silent miss leaves the node with no address on any
-# interface. See RUNBOOK-baremetal-conversion.md.
+# Named by the udev rule above, not by LinkAliasConfig — see k8s-1.yaml and
+# RUNBOOK-baremetal-conversion.md.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f0np0
+name: uplink
 up: true
 ---
 # Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
@@ -79,7 +83,7 @@ apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
 apiVersion: v1alpha1
@@ -99,9 +103,35 @@ apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
+# Pod network, VLAN 48. Node-to-node only, statically addressed, no gateway
+# and no DHCP — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: pod0
+vlanID: 48
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.48.12/24
+  - address: fd02:48::12/64
+---
+# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
+# addressed — see k8s-1.yaml for why it is not leased.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ingress0
+vlanID: 47
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.47.12/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
+---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.
 # Renamed to the role name by the udev rule above — see k8s-1.yaml.

kubernetes/talos/patches/k8s-3.yaml

@@ -6,8 +6,13 @@ machine:
     net.ipv6.conf.ens19.disable_ipv6: "1"
     # Same for the physical ports on the untagged fallback network — see
     # k8s-1.yaml.
-    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
-    net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    net.ipv6.conf.uplink.disable_ipv6: "1"
+    net.ipv6.conf.mgmt.disable_ipv6: "1"
+    # pod0 is a closed segment with no router, so it ignores RAs outright.
+    # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
+    # and only the default route is held back — see k8s-1.yaml.
+    net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
     net.ipv4.conf.default.arp_ignore: "1"
@@ -25,9 +30,11 @@ machine:
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
   udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
+    # Renames every port to its role at boot — see k8s-1.yaml.
     rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d3", NAME="uplink"
       - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d5", NAME="mgmt"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -58,14 +65,14 @@ provisioning:
 ---
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f0np0
+name: uplink
 up: true
 ---
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
 apiVersion: v1alpha1
@@ -83,9 +90,35 @@ apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: enp2s0f0np0
+parent: uplink
 up: true
 ---
+# Pod network, VLAN 48. Node-to-node only, statically addressed, no gateway
+# and no DHCP — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: pod0
+vlanID: 48
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.48.13/24
+  - address: fd02:48::13/64
+---
+# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
+# addressed — see k8s-1.yaml for why it is not leased.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ingress0
+vlanID: 47
+parent: uplink
+up: true
+addresses:
+  - address: 10.2.47.13/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
+---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
 # here on purpose; the link is handed whole to the OSD pod (host-device).
 apiVersion: v1alpha1

kubernetes/talos/patches/site-homelab.yaml

@@ -7,16 +7,32 @@
 # part of the config that could not be true anywhere but here.
 machine:
   kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
-    # kubelet picks the first non-loopback IP, which could land on the v30iot
-    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
-    # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
+    # Pins the node IP to pod0 (VLAN 48), which is what carries pod east-west.
+    # Cilium reaches a peer's pods at that peer's Kubernetes InternalIP and
+    # offers no way to point it elsewhere: --ipv4-node/--ipv6-node are accepted
+    # and then overwritten from the Node object, and directRoutingDevice only
+    # selects the NodePort/DSR device. So the node IP is the only control over
+    # which segment east-west uses, in both tunnel and native routing mode.
+    #
+    # Unpinned, kubelet takes the first non-loopback address, which could land
+    # on the v30iot NIC (ens19, 10.2.30.0/24).
+    #
+    # fd02:48::/64 is listed so the Node object carries an IPv6 InternalIP at
+    # all. Without one, Cilium detects the node's IPv6 itself and picks the
+    # SLAAC address from the dynamic provider prefix, then probes it until the
+    # agent restarts (#789).
     nodeIP:
       validSubnets:
-        - 10.2.40.0/24
-        - fd40::/64
+        - 10.2.48.0/24
+        - fd02:48::/64
+cluster:
+  etcd:
+    # Pins etcd to pod0 with the node IP. Unset, etcd advertises "the first
+    # routable address of the node", which is not deterministic across a
+    # re-address. listenSubnets defaults to this.
+    #
+    # pod0 is a closed 10G L2 between the three nodes. The management VLAN is
+    # about to become a single 1G NIC shared with everything else management
+    # does, which is the wrong link for quorum traffic.
+    advertisedSubnets:
+      - 10.2.48.0/24

unifi/bgp.conf

@@ -28,10 +28,19 @@ router bgp 64512
  ! advertise extended-nexthop for the IPv6 Unicast SAFI when v6 routes
  ! are carried over a v4 transport, so we run a native v6 session that
  ! carries v6 routes with v6 next-hops and skip the capability dance
- ! entirely. Dynamic neighbors — k8s nodes are SLAAC-addressed on mgmt
- ! LAN; the listen ranges cover the SLAAC suffix space.
+ ! entirely. Dynamic neighbors — nodes are SLAAC-addressed, so the listen
+ ! ranges cover the SLAAC suffix space rather than naming each peer.
+ !
+ ! Both segments are listed while the ingress split lands (#576). The
+ ! cluster peers from mgmt (40) today and moves to ingress (47), and a
+ ! listen range only accepts a session, it does not open one — so having
+ ! both costs nothing and means the move never has a window where the UDM
+ ! is not listening where the nodes are calling from. Drop the 40 ranges
+ ! once every session is established on 47.
  bgp listen range 10.2.40.0/24 peer-group K8S
  bgp listen range fd40::/64 peer-group K8S
+ bgp listen range 10.2.47.0/24 peer-group K8S
+ bgp listen range fd02:47::/64 peer-group K8S
  !
  address-family ipv4 unicast
   neighbor K8S activate