docs(network): traffic classes as built, node-to-node on VLAN 46 (#576)

NETWORK-traffic-classes.md described table 48 and rules 980/981/990/991,
which #857 replaced with one table (47) decided by longest prefix, and
listed management as not started. Rewritten to the live state: five
classes, table 47 contents owned by bird, node IP and etcd on node0, the
etcd leave-and-rejoin procedure, and the mixed-state hazard hit during
the node IP move. The bird DaemonSet comment still named table 48.

Closes #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kubernetes/NETWORK-traffic-classes.md

@@ -1,117 +1,99 @@
 # Traffic classes on the k8s nodes
 
-Issue #576. Three of the four classes are separated; management is not.
+Issue #576. Five classes, each on its own VLAN.
 
 ## Where each class runs
 
 | class | link | mechanism |
 |---|---|---|
 | Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
-| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
-| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
-| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
-| management | `mgmt0`, VLAN 40 | not started |
+| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
+| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
+| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
+| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
 
 Hard constraint, set by the owner: inter-node and pod traffic never traverses
-the UDM. Everything above stays on the 10G switch.
+the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
+statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
 
-## Why policy routing rather than moving the node IP
+## Pod traffic: one table
 
-Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
-and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
-`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
-Both were established by test (#837, #838), not by reading.
+Two rules, one per family, send every packet with a pod source to table 47
+(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
+table decides by longest prefix; no rule carves out exceptions.
 
-So the InternalIP decides where pod traffic goes — unless the lookup happens in
-a different table. The rules do that, and nothing about the node's identity
-moves. The routing decision runs before Cilium's masquerade hook, so a rule
-matching the pod source sees the pod address, and the masquerade then rewrites
-it to the egress link's.
+bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
 
-## Rule order
+- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
+  taken from the interfaces;
+- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
+  peer's `pod0` address;
+- the node's own pod CIDR via `cilium_host`, originated rather than read back
+  from the kernel (reading it back made BIRD depend on the shape of Cilium's
+  route, and KRT rejects the IPv4 one);
+- the egress default via the UDM on `ingress0`, from the node annotations
+  `net.renner.uno/egress-gw-v4/v6`.
 
-Lower priority runs first.
+`node0` has to be in the table: a pod reaching a node address — the
+`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
+address, or a hostNetwork pod — would otherwise take the default to the UDM.
 
-| prio | match | table |
-|---|---|---|
-| 980 | src pod, dst pod | 48 |
-| 981 | src pod, dst pod (v4) | 48 |
-| 990 | dst `fd00::/8` | main |
-| 991 | dst `10.0.0.0/8` | main |
-| 1000 | src pod | 47 |
-| 1001 | src pod (v4) | 47 |
+The routing decision runs before Cilium's masquerade hook, so the rule sees the
+pod address and the masquerade then rewrites it to the egress link's.
 
-980/981 match source **and** destination. Source alone would also catch the
-node's own traffic to a pod, which would then leave with a `pod0` source — an
-address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
-`world` rather than `host`.
-
-990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
-cluster-internal traffic out of table 47. Without them a pod's packet to a
-service matches the source rule, finds only a default route, and leaves for the
-UDM — measured, in-cluster DNS stopped resolving.
-
-Traffic to a pod on the same node falls through on its own: table 48 holds only
-the peers' CIDRs, so the lookup misses and 990/991 send it to main.
-
-## Where each table comes from
-
-- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
-  The IPv6 gateway is the UDM's configured address on the segment, not the
-  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
-  default does not land in main.
-- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
-  per peer pod CIDR with the peer's `pod0` address as next hop.
-
-## The bird mesh
-
-`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
-and installs what it learns into table 48. Cilium's routes in main are
-untouched, which is why the two coexist.
+Node-originated traffic does not match the rules and uses main, where Cilium
+routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
 
 No address is written into this repo twice. Each node publishes its `pod0`
-addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
-pod CIDRs come from the Node objects. An init container renders both from the
-API with `kubectl -o go-template-file`, excludes the node itself with a field
-selector, and ends with `bird -p` so BIRD's own parser gates startup.
+addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
+pod CIDRs come from the Node objects. bird's init container renders both from
+the API and ends with `bird -p`, so BIRD's own parser gates startup.
 
-The node's own CIDR is **originated**, not read back from the kernel. Reading
-it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
-rejected outright: its next hop sits inside the prefix it describes, which KRT
-refuses with `strange next-hop`. The IPv6 route is a device route and passed,
-so the failure was silent on one family only.
+## Node↔node
+
+`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
+VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
+is IPv4, and the apiserver's advertise address must match it.
+
+The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
+kubelet, scheduler, controller-manager and Cilium go through KubePrism
+(`localhost:7445`), which balances over every address of every control-plane
+member; pods use the ClusterIP.
+
+Talos never updates an existing etcd member's peer URLs
+(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
+stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
+reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
+and is promoted. EPHEMERAL does not need a wipe.
+
+## Moving node IPs: the mixed state
+
+While some nodes have their InternalIP on the new network and some on the old,
+host→remote-pod traffic leaves with a node address that is not an InternalIP:
+the kernel picks the source from the route to the peer's InternalIP. Cilium
+classifies that as `world`, and every ingress policy that admits only
+`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
+lose the OSDs on the other side; RBD mounts hang.
+
+Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
+admitting the node subnets of both networks, both families, with
+`enableDefaultDeny: false` so it restricts nothing else, and delete it once
+every node has moved. Keep all OSD nodes schedulable while one is drained:
+draining two takes Ceph below its data availability.
 
 ## Failure behaviour
 
-Benign at every layer. An empty or absent table 48 makes 980/981 fall through
-to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
-holds if the DaemonSet cannot start, which is the case during a cold rebuild
-until Gitea serves its image.
-
-## Management, the remaining class
-
-Not started. The levers differ per service and none of them is the node IP:
-
-| service | can leave the node IP? | lever |
-|---|---|---|
-| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
-| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
-| Talos API | n/a | `apid` answers on every node address; a firewall question |
-| kubelet | no | it *is* the node IP |
-
-Two approaches were rejected by the owner and are not to be proposed again:
-moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
-cluster VLAN while moving etcd and the API VIP to a new management VLAN.
-Replacing the CNI (#839, Antrea) was shelved.
-
-Note that the reasoning which produced that list also concluded pod↔pod was
-blocked, which policy routing disproved. Treat it as a record of decisions, not
-as an exhaustive analysis.
+If bird stops, table 47 empties, the rules find nothing and fall through to
+main: pod traffic returns to the node networks, unseparated rather than
+misrouted. The same holds during a cold rebuild until Gitea serves bird's
+image. The default route lives in table 47, not in the Talos config, so a
+default cannot outlive the specific routes and send pod traffic to the UDM.
 
 ## Sources
 
-Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
-`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
-`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
-`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
-#855; closed unmerged: #830, #838, #839.
+Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
+protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
+`machine.nodeAnnotations`, configuration patching (lists are appended,
+`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
+siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
+#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.

kubernetes/infrastructure/bird/daemonset.yaml

@@ -48,8 +48,8 @@ spec:
         tier: node
         app: bird
     spec:
-      # Carries pod east-west once the routing rules land. Without it table 48
-      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
+      # Owns table 47, where all pod traffic is routed. Without it the table
+      # stays empty and the rules fall through to main, unseparated.
       priorityClassName: system-node-critical
       # Writes host routes and peers on host addresses.
       hostNetwork: true

kubernetes/talos/patches/cluster.yaml

@@ -58,6 +58,12 @@ machine:
       imageGCHighThresholdPercent: 70
       imageGCLowThresholdPercent: 60
 cluster:
+  # etcd peers on node0, VLAN 46 (#576). Pinned: Talos never updates an
+  # existing member's peer URLs, so changing this subnet means leave + rejoin
+  # per member (talosctl etcd leave, reboot).
+  etcd:
+    advertisedSubnets:
+      - 10.2.46.0/24
   # controlPlane.endpoint is NOT here: `talosctl gen config` takes it as its
   # second argument, and bootstrap.sh already passes https://k8s.renner.uno:6443
   # there. Setting it again in this patch restated the same value in a second

kubernetes/talos/patches/k8s-1.yaml

@@ -17,6 +17,8 @@ machine:
     # pod0 is a closed L2 fabric between the three nodes: statically
     # addressed, no gateway, nothing to learn from an RA.
     net.ipv6.conf.pod0.accept_ra: "0"
+    # node0 is closed the same way.
+    net.ipv6.conf.node0.accept_ra: "0"
     # ingress0 keeps RAs, at the default accept_ra of 2: an IPv6 default
     # router is a link-local address learned only from the RA, and with
     # net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
@@ -122,8 +124,8 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network.
+# Management, VLAN 40: world↔node. The API VIP, admin access to the nodes, and
+# node egress (DNS, NTP, image pulls). Node↔node traffic is on node0 (#576).
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -177,6 +179,19 @@ addresses:
   - address: 10.2.48.11/24
   - address: fd02:48::11/64
 ---
+# Node network, VLAN 46: traffic among the three nodes themselves (#576) — the
+# node IP, etcd, apiserver↔kubelet. Closed like pod0: statically addressed, no
+# gateway, never routed via the UDM.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::11/64
+  - address: 10.2.46.11/24
+---
 # Ingress network, VLAN 47. World-to-pod: the BGP session, the next hop for
 # every advertised LoadBalancer VIP, and the masquerade source for pod egress.
 # Publicly routed, which is what separates it from pod0.

kubernetes/talos/patches/k8s-2.yaml

@@ -16,6 +16,7 @@ machine:
     # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
     # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
@@ -82,7 +83,7 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
+# Management, VLAN 40: world↔node — see k8s-1.yaml.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -122,6 +123,17 @@ addresses:
   - address: 10.2.48.12/24
   - address: fd02:48::12/64
 ---
+# Node network, VLAN 46 — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::12/64
+  - address: 10.2.46.12/24
+---
 # Ingress network, VLAN 47. World-to-pod, publicly routed, statically
 # addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1

kubernetes/talos/patches/k8s-3.yaml

@@ -12,6 +12,7 @@ machine:
     # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
     # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
@@ -109,6 +110,17 @@ addresses:
   - address: 10.2.48.13/24
   - address: fd02:48::13/64
 ---
+# Node network, VLAN 46 — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::13/64
+  - address: 10.2.46.13/24
+---
 # Ingress network, VLAN 47. World-to-pod, publicly routed, statically
 # addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1

kubernetes/talos/patches/site-homelab.yaml

@@ -7,19 +7,15 @@
 # part of the config that could not be true anywhere but here.
 machine:
   kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
-    # kubelet picks the first non-loopback IP, which could land on the v30iot
-    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
-    # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
+    # The node IP lives on node0, VLAN 46: the closed node↔node network (#576).
+    # Both families are listed so the Node object carries an IPv6 InternalIP at
+    # all; without one Cilium detects the node's IPv6 itself and picks a SLAAC
+    # address from the dynamic provider prefix (#789). Talos picks one address
+    # per family, IPv4 first, whatever the order here.
     nodeIP:
       validSubnets:
-        - 10.2.40.0/24
-        - fd40::/64
+        - fd02:46::/64
+        - 10.2.46.0/24
   # The UDM on the ingress segment. Published here rather than as a route
   # because bird owns table 47 and originates the default from it; this is the
   # one place the address is stated.