homelab/kubernetes/NETWORK-traffic-classes.md

Traffic classes on the k8s nodes

Issue #576. Five classes, each on its own VLAN.

Where each class runs

class	link	mechanism
Ceph replication	ceph-san, VLAN 49	the whole link is handed to the OSD pod (Multus host-device)
pod↔pod	pod0, VLAN 48	pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via pod0
world↔pod	ingress0, VLAN 47	BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default
node↔node	node0, VLAN 46	the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic
world↔node	mgmt0, VLAN 40	API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls)

Hard constraint, set by the owner: inter-node and pod traffic never traverses the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are statically addressed, without a gateway; VLAN 46 is isolated in UniFi.

Pod traffic: one table

Two rules, one per family, send every packet with a pod source to table 47 (talos/patches/site-homelab.yaml, priorities 1000/1001, below main). The table decides by longest prefix; no rule carves out exceptions.

A reply from a LoadBalancer VIP is pod traffic as well, but it reaches the routing decision with the VIP as its source. Two more rules (1002/1003) send the LoadBalancer pool to table 47, so the reply leaves over ingress0, where the request arrived as the BGP next hop. Without them main sent it out mgmt0. The pool is a site fact: the site patch's lb-pool-site ConfigMap, which bgp-pool takes by Flux substitution, and the rules beside it.

bird (kubernetes/infrastructure/bird/) owns table 47 and fills it with:

• the node's attached networks — mgmt0, ingress0, pod0, node0 — taken from the interfaces;
• the peers' pod CIDRs, learned over an iBGP mesh on pod0, next hop the peer's pod0 address;
• the node's own pod CIDR via cilium_host, originated rather than read back from the kernel (reading it back made BIRD depend on the shape of Cilium's route, and KRT rejects the IPv4 one);
• the egress default via the UDM on ingress0, from the node annotations net.renner.uno/egress-gw-v4/v6.

node0 has to be in the table: a pod reaching a node address — the kubernetes ClusterIP, which socket LB rewrites to the apiserver's advertise address, or a hostNetwork pod — would otherwise take the default to the UDM.

The routing decision runs before Cilium's masquerade hook, so the rule sees the pod address and the masquerade then rewrites it to the egress link's.

Node-originated traffic does not match the rules and uses main, where Cilium routes each peer's pod CIDR via the peer's InternalIP — on node0.

No address is written into this repo twice. Each node publishes its pod0 addresses as nodeAnnotations beside the VLANConfig that assigns them; the pod CIDRs come from the Node objects. bird's init container renders both from the API and ends with bird -p, so BIRD's own parser gates startup.

Node↔node

kubelet.nodeIP.validSubnets and cluster.etcd.advertisedSubnets point at VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR is IPv4, and the apiserver's advertise address must match it.

The API VIP stays on mgmt0 for kubectl. No in-cluster client uses it: kubelet, scheduler, controller-manager and Cilium go through KubePrism (localhost:7445), which balances over every address of every control-plane member; pods use the ClusterIP.

Talos never updates an existing etcd member's peer URLs (siderolabs/talos#12720). Changing advertisedSubnets means, per member: stage the config, talosctl etcd forfeit-leadership, talosctl etcd leave, reboot. leave removes only /var/lib/etcd; the node rejoins as a learner and is promoted. EPHEMERAL does not need a wipe.

Moving node IPs: the mixed state

While some nodes have their InternalIP on the new network and some on the old, host→remote-pod traffic leaves with a node address that is not an InternalIP: the kernel picks the source from the route to the peer's InternalIP. Cilium classifies that as world, and every ingress policy that admits only host/remote-node (rook-ceph, monitoring, …) drops it. Kernel Ceph clients lose the OSDs on the other side; RBD mounts hang.

Bridge it for the duration with a temporary CiliumClusterwideNetworkPolicy admitting the node subnets of both networks, both families, with enableDefaultDeny: false so it restricts nothing else, and delete it once every node has moved. Keep all OSD nodes schedulable while one is drained: draining two takes Ceph below its data availability.

Failure behaviour

If bird stops, table 47 empties, the rules find nothing and fall through to main: pod traffic returns to the node networks, unseparated rather than misrouted. The same holds during a cold rebuild until Gitea serves bird's image. The default route lives in table 47, not in the Talos config, so a default cannot outlive the specific routes and send pod traffic to the UDM.

Sources

Cilium v1.20 pkg/datapath/linux/node.go; BIRD 2.17 user manual (kernel protocol, strict bind, direct); Talos v1.13 RoutingRuleConfig, machine.nodeAnnotations, configuration patching (lists are appended, $patch: delete), etcd LeaveCluster; KubePrism docs; siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854, #855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.