claude-botandClaude Opus 5.5
3501586841
A reply from a Cilium LoadBalancer VIP reaches the routing decision with the VIP as its source, so the pod-source rules 1000/1001 missed it and main sent it out mgmt0 while the request came in over ingress0, the BGP next hop (k8s-3, 2026-11-02: replies from 10.2.99.90 to an IoT device seen only on mgmt0). Rules 1002/1003 send the pool to table 47. The pool becomes a site fact: each site patch carries lb-pool-site (flux-system), and bgp-pool takes its blocks from it by Flux substitution in infrastructure-crds. The CIDR blocks cover the same addresses as the start/stop ranges did (allowFirstLastIPs defaults to Yes). The qemu and hcloud sites carry the production pool, since Services pin their addresses from it. Refs #1547 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185Qb5YG28FfnVi1ujMEV5C
107 lines
5.5 KiB
Markdown
107 lines
5.5 KiB
Markdown
# Traffic classes on the k8s nodes
|
|
|
|
Issue #576. Five classes, each on its own VLAN.
|
|
|
|
## Where each class runs
|
|
|
|
| class | link | mechanism |
|
|
|---|---|---|
|
|
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
|
|
| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
|
|
| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
|
|
| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
|
|
| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
|
|
|
|
Hard constraint, set by the owner: inter-node and pod traffic never traverses
|
|
the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
|
|
statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
|
|
|
|
## Pod traffic: one table
|
|
|
|
Two rules, one per family, send every packet with a pod source to table 47
|
|
(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
|
|
table decides by longest prefix; no rule carves out exceptions.
|
|
|
|
A reply from a LoadBalancer VIP is pod traffic as well, but it reaches the
|
|
routing decision with the VIP as its source. Two more rules (1002/1003) send
|
|
the LoadBalancer pool to table 47, so the reply leaves over `ingress0`, where
|
|
the request arrived as the BGP next hop. Without them main sent it out
|
|
`mgmt0`. The pool is a site fact: the site patch's `lb-pool-site` ConfigMap,
|
|
which `bgp-pool` takes by Flux substitution, and the rules beside it.
|
|
|
|
bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
|
|
|
|
- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
|
|
taken from the interfaces;
|
|
- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
|
|
peer's `pod0` address;
|
|
- the node's own pod CIDR via `cilium_host`, originated rather than read back
|
|
from the kernel (reading it back made BIRD depend on the shape of Cilium's
|
|
route, and KRT rejects the IPv4 one);
|
|
- the egress default via the UDM on `ingress0`, from the node annotations
|
|
`net.renner.uno/egress-gw-v4/v6`.
|
|
|
|
`node0` has to be in the table: a pod reaching a node address — the
|
|
`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
|
|
address, or a hostNetwork pod — would otherwise take the default to the UDM.
|
|
|
|
The routing decision runs before Cilium's masquerade hook, so the rule sees the
|
|
pod address and the masquerade then rewrites it to the egress link's.
|
|
|
|
Node-originated traffic does not match the rules and uses main, where Cilium
|
|
routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
|
|
|
|
No address is written into this repo twice. Each node publishes its `pod0`
|
|
addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
|
|
pod CIDRs come from the Node objects. bird's init container renders both from
|
|
the API and ends with `bird -p`, so BIRD's own parser gates startup.
|
|
|
|
## Node↔node
|
|
|
|
`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
|
|
VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
|
|
is IPv4, and the apiserver's advertise address must match it.
|
|
|
|
The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
|
|
kubelet, scheduler, controller-manager and Cilium go through KubePrism
|
|
(`localhost:7445`), which balances over every address of every control-plane
|
|
member; pods use the ClusterIP.
|
|
|
|
Talos never updates an existing etcd member's peer URLs
|
|
(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
|
|
stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
|
|
reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
|
|
and is promoted. EPHEMERAL does not need a wipe.
|
|
|
|
## Moving node IPs: the mixed state
|
|
|
|
While some nodes have their InternalIP on the new network and some on the old,
|
|
host→remote-pod traffic leaves with a node address that is not an InternalIP:
|
|
the kernel picks the source from the route to the peer's InternalIP. Cilium
|
|
classifies that as `world`, and every ingress policy that admits only
|
|
`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
|
|
lose the OSDs on the other side; RBD mounts hang.
|
|
|
|
Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
|
|
admitting the node subnets of both networks, both families, with
|
|
`enableDefaultDeny: false` so it restricts nothing else, and delete it once
|
|
every node has moved. Keep all OSD nodes schedulable while one is drained:
|
|
draining two takes Ceph below its data availability.
|
|
|
|
## Failure behaviour
|
|
|
|
If bird stops, table 47 empties, the rules find nothing and fall through to
|
|
main: pod traffic returns to the node networks, unseparated rather than
|
|
misrouted. The same holds during a cold rebuild until Gitea serves bird's
|
|
image. The default route lives in table 47, not in the Talos config, so a
|
|
default cannot outlive the specific routes and send pod traffic to the UDM.
|
|
|
|
## Sources
|
|
|
|
Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
|
|
protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
|
|
`machine.nodeAnnotations`, configuration patching (lists are appended,
|
|
`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
|
|
siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
|
|
#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.
|