homelab/kubernetes/NETWORK-traffic-classes.md

Traffic classes on the k8s nodes

Issue #576. Three of the four classes are separated; management is not.

Where each class runs

class	link	mechanism
Ceph replication	ceph-san, VLAN 49	the whole link is handed to the OSD pod (Multus host-device)
world→pod	ingress0, VLAN 47	BGP session sourced there, so the advertised next hop is that address
pod→world	ingress0, VLAN 47	routing rule on the pod source prefix → table 47
pod↔pod	pod0, VLAN 48	routing rule on pod source and destination → table 48
management	mgmt0, VLAN 40	not started

Hard constraint, set by the owner: inter-node and pod traffic never traverses the UDM. Everything above stays on the 10G switch.

Why policy routing rather than moving the node IP

Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native and tunnel mode alike. --ipv4-node is overwritten from the Node object, and nodePort.directRoutingDevice selects the BPF NodePort device, not a next hop. Both were established by test (#837, #838), not by reading.

So the InternalIP decides where pod traffic goes — unless the lookup happens in a different table. The rules do that, and nothing about the node's identity moves. The routing decision runs before Cilium's masquerade hook, so a rule matching the pod source sees the pod address, and the masquerade then rewrites it to the egress link's.

Rule order

Lower priority runs first.

prio	match	table
980	src pod, dst pod	48
981	src pod, dst pod (v4)	48
990	dst fd00::/8	main
991	dst 10.0.0.0/8	main
1000	src pod	47
1001	src pod (v4)	47

980/981 match source and destination. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode.spec.addresses does not list, so Cilium classifies it as world rather than host.

990/991 are Cilium's ipv4/ipv6NativeRoutingCIDR, and they are what keeps cluster-internal traffic out of table 47. Without them a pod's packet to a service matches the source rule, finds only a default route, and leaves for the UDM — measured, in-cluster DNS stopped resolving.

Traffic to a pod on the same node falls through on its own: table 48 holds only the peers' CIDRs, so the lookup misses and 990/991 send it to main.

Where each table comes from

• 47 — two default routes declared per node in talos/patches/k8s-*.yaml. The IPv6 gateway is the UDM's configured address on the segment, not the RA-learned link-local; accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
• 48 — filled by the bird DaemonSet: an iBGP mesh over pod0, one route per peer pod CIDR with the peer's pod0 address as next hop.

The bird mesh

kubernetes/infrastructure/bird/. Each node advertises only its own pod CIDR and installs what it learns into table 48. Cilium's routes in main are untouched, which is why the two coexist.

No address is written into this repo twice. Each node publishes its pod0 addresses as nodeAnnotations, beside the VLANConfig that assigns them; the pod CIDRs come from the Node objects. An init container renders both from the API with kubectl -o go-template-file, excludes the node itself with a field selector, and ends with bird -p so BIRD's own parser gates startup.

The node's own CIDR is originated, not read back from the kernel. Reading it back made BIRD depend on the shape Cilium installs, and the IPv4 route is rejected outright: its next hop sits inside the prefix it describes, which KRT refuses with strange next-hop. The IPv6 route is a device route and passed, so the failure was silent on one family only.

Failure behaviour

Benign at every layer. An empty or absent table 48 makes 980/981 fall through to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same holds if the DaemonSet cannot start, which is the case during a cold rebuild until Gitea serves its image.

Management, the remaining class

Not started. The levers differ per service and none of them is the node IP:

service	can leave the node IP?	lever
etcd	yes	cluster.etcd.advertisedSubnets; needs a reboot per node
kube-apiserver endpoint	yes	it is a VIP — Layer2VIPConfig.link
Talos API	n/a	apid answers on every node address; a firewall question
kubelet	no	it is the node IP

Two approaches were rejected by the owner and are not to be proposed again: moving the node IP and etcd to pod0 (#838), and relabelling VLAN 40 as the cluster VLAN while moving etcd and the API VIP to a new management VLAN. Replacing the CNI (#839, Antrea) was shelved.

Note that the reasoning which produced that list also concluded pod↔pod was blocked, which policy routing disproved. Treat it as a record of decisions, not as an exhaustive analysis.

Sources

Cilium v1.20.1 pkg/datapath/linux/node.go, pkg/node/types/node.go, pkg/node/sync/local_node_sync.go; BIRD 2.17 user manual (kernel protocol, learn all, strict bind, direct); Talos v1.13 RoutingRuleConfig and machine.nodeAnnotations. PRs #831, #833, #835, #836, #837, #842, #852, #854, #855; closed unmerged: #830, #838, #839.