konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

docs(network): traffic classes as built, node-to-node on VLAN 46 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
ci/woodpecker/pr/pr-review Pipeline was successful
Details

Browse Source 
NETWORK-traffic-classes.md described table 48 and rules 980/981/990/991,
which #857 replaced with one table (47) decided by longest prefix, and
listed management as not started. Rewritten to the live state: five
classes, table 47 contents owned by bird, node IP and etcd on node0, the
etcd leave-and-rejoin procedure, and the mixed-state hazard hit during
the node IP move. The bird DaemonSet comment still named table 48.

Closes #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
claude-bot
2026-09-23 14:48:12 +02:00
co-authored by Claude Opus 5.5
parent 8a0ed135f8
commit 32a995a6bf
2 changed files with 75 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/NETWORK-traffic-classes.md
+73 -91
View File
@@ -1,117 +1,99 @@
# Traffic classes on the k8s nodes
Issue #576. Three of the four classes are separated; management is not.
Issue #576. Five classes, each on its own VLAN.
## Where each class runs
| class | link | mechanism |
|---|---|---|
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
| management | `mgmt0`, VLAN 40 | not started |
| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
Hard constraint, set by the owner: inter-node and pod traffic never traverses
the UDM. Everything above stays on the 10G switch.
the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
## Why policy routing rather than moving the node IP
## Pod traffic: one table
Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
Both were established by test (#837, #838), not by reading.
Two rules, one per family, send every packet with a pod source to table 47
(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
table decides by longest prefix; no rule carves out exceptions.
So the InternalIP decides where pod traffic goes — unless the lookup happens in
a different table. The rules do that, and nothing about the node's identity
moves. The routing decision runs before Cilium's masquerade hook, so a rule
matching the pod source sees the pod address, and the masquerade then rewrites
it to the egress link's.
bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
## Rule order
- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
taken from the interfaces;
- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
peer's `pod0` address;
- the node's own pod CIDR via `cilium_host`, originated rather than read back
from the kernel (reading it back made BIRD depend on the shape of Cilium's
route, and KRT rejects the IPv4 one);
- the egress default via the UDM on `ingress0`, from the node annotations
`net.renner.uno/egress-gw-v4/v6`.
Lower priority runs first.
`node0` has to be in the table: a pod reaching a node address — the
`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
address, or a hostNetwork pod — would otherwise take the default to the UDM.
| prio | match | table |
|---|---|---|
| 980 | src pod, dst pod | 48 |
| 981 | src pod, dst pod (v4) | 48 |
| 990 | dst `fd00::/8` | main |
| 991 | dst `10.0.0.0/8` | main |
| 1000 | src pod | 47 |
| 1001 | src pod (v4) | 47 |
The routing decision runs before Cilium's masquerade hook, so the rule sees the
pod address and the masquerade then rewrites it to the egress link's.
980/981 match source **and** destination. Source alone would also catch the
node's own traffic to a pod, which would then leave with a `pod0` source — an
address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
`world` rather than `host`.
990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
cluster-internal traffic out of table 47. Without them a pod's packet to a
service matches the source rule, finds only a default route, and leaves for the
UDM — measured, in-cluster DNS stopped resolving.
Traffic to a pod on the same node falls through on its own: table 48 holds only
the peers' CIDRs, so the lookup misses and 990/991 send it to main.
## Where each table comes from
- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
The IPv6 gateway is the UDM's configured address on the segment, not the
RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
default does not land in main.
- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
per peer pod CIDR with the peer's `pod0` address as next hop.
## The bird mesh
`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
and installs what it learns into table 48. Cilium's routes in main are
untouched, which is why the two coexist.
Node-originated traffic does not match the rules and uses main, where Cilium
routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
No address is written into this repo twice. Each node publishes its `pod0`
addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
pod CIDRs come from the Node objects. An init container renders both from the
API with `kubectl -o go-template-file`, excludes the node itself with a field
selector, and ends with `bird -p` so BIRD's own parser gates startup.
addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
pod CIDRs come from the Node objects. bird's init container renders both from
the API and ends with `bird -p`, so BIRD's own parser gates startup.
The node's own CIDR is **originated**, not read back from the kernel. Reading
it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
rejected outright: its next hop sits inside the prefix it describes, which KRT
refuses with `strange next-hop`. The IPv6 route is a device route and passed,
so the failure was silent on one family only.
## Node↔node
`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
is IPv4, and the apiserver's advertise address must match it.
The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
kubelet, scheduler, controller-manager and Cilium go through KubePrism
(`localhost:7445`), which balances over every address of every control-plane
member; pods use the ClusterIP.
Talos never updates an existing etcd member's peer URLs
(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
and is promoted. EPHEMERAL does not need a wipe.
## Moving node IPs: the mixed state
While some nodes have their InternalIP on the new network and some on the old,
host→remote-pod traffic leaves with a node address that is not an InternalIP:
the kernel picks the source from the route to the peer's InternalIP. Cilium
classifies that as `world`, and every ingress policy that admits only
`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
lose the OSDs on the other side; RBD mounts hang.
Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
admitting the node subnets of both networks, both families, with
`enableDefaultDeny: false` so it restricts nothing else, and delete it once
every node has moved. Keep all OSD nodes schedulable while one is drained:
draining two takes Ceph below its data availability.
## Failure behaviour
Benign at every layer. An empty or absent table 48 makes 980/981 fall through
to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
holds if the DaemonSet cannot start, which is the case during a cold rebuild
until Gitea serves its image.
## Management, the remaining class
Not started. The levers differ per service and none of them is the node IP:
| service | can leave the node IP? | lever |
|---|---|---|
| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
| Talos API | n/a | `apid` answers on every node address; a firewall question |
| kubelet | no | it *is* the node IP |
Two approaches were rejected by the owner and are not to be proposed again:
moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
cluster VLAN while moving etcd and the API VIP to a new management VLAN.
Replacing the CNI (#839, Antrea) was shelved.
Note that the reasoning which produced that list also concluded pod↔pod was
blocked, which policy routing disproved. Treat it as a record of decisions, not
as an exhaustive analysis.
If bird stops, table 47 empties, the rules find nothing and fall through to
main: pod traffic returns to the node networks, unseparated rather than
misrouted. The same holds during a cold rebuild until Gitea serves bird's
image. The default route lives in table 47, not in the Talos config, so a
default cannot outlive the specific routes and send pod traffic to the UDM.
## Sources
Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
#855; closed unmerged: #830, #838, #839.
Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
`machine.nodeAnnotations`, configuration patching (lists are appended,
`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.
kubernetes/infrastructure/bird/daemonset.yaml
+2 -2
View File
@@ -48,8 +48,8 @@ spec:
tier: node
app: bird
spec:
# Carries pod east-west once the routing rules land. Without it table 48
# stays empty and the rules fall through to main, i.e. back to VLAN 40.
# Owns table 47, where all pod traffic is routed. Without it the table
# stays empty and the rules fall through to main, unseparated.
priorityClassName: system-node-critical
# Writes host routes and peers on host addresses.
hostNetwork: true