From f0701370db29d3398aaa00882ceb95ed5496260f Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 16 Sep 2026 11:18:57 +0200 Subject: [PATCH] docs: describe the traffic-class split as built (#576) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the proposal, which was wrong before it was written down. It presented A, B and C as the option space and concluded pod-to-pod was blocked; policy routing was available the whole time and is what moved pod egress and then pod-to-pod. The file recorded a blind spot and gave it the authority of a checked-in document. Three of four classes are separated now, so this describes the built design rather than proposing one, and the PROPOSAL- name goes with it: which link each class uses, the rule order and why 980/981 match source and destination, where each table comes from, how the bird mesh renders itself from the API without repeating an address, and why the node's own CIDR is originated rather than read back from the kernel. Management is the remaining class. Its levers are listed, along with the two approaches the owner rejected — with the caveat that the same reasoning which produced that list also called pod-to-pod impossible, so it is a record of decisions and not an analysis to trust. Failure behaviour is stated because it is what makes the design safe to roll one node at a time: an empty table 48 falls through to main and pod traffic returns to VLAN 40, unseparated rather than broken. bootstrap.sh's reference follows the rename; its point was always that the QEMU/Hetzner sites use two segments, not the homelab's split. Refs #576 Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/NETWORK-traffic-classes.md | 117 +++++++++++ .../PROPOSAL-traffic-class-separation.md | 194 ------------------ kubernetes/bootstrap.sh | 4 +- 3 files changed, 119 insertions(+), 196 deletions(-) create mode 100644 kubernetes/NETWORK-traffic-classes.md delete mode 100644 kubernetes/PROPOSAL-traffic-class-separation.md diff --git a/kubernetes/NETWORK-traffic-classes.md b/kubernetes/NETWORK-traffic-classes.md new file mode 100644 index 00000000..29808ca8 --- /dev/null +++ b/kubernetes/NETWORK-traffic-classes.md @@ -0,0 +1,117 @@ +# Traffic classes on the k8s nodes + +Issue #576. Three of the four classes are separated; management is not. + +## Where each class runs + +| class | link | mechanism | +|---|---|---| +| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) | +| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address | +| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 | +| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 | +| management | `mgmt0`, VLAN 40 | not started | + +Hard constraint, set by the owner: inter-node and pod traffic never traverses +the UDM. Everything above stays on the 10G switch. + +## Why policy routing rather than moving the node IP + +Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native +and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and +`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop. +Both were established by test (#837, #838), not by reading. + +So the InternalIP decides where pod traffic goes — unless the lookup happens in +a different table. The rules do that, and nothing about the node's identity +moves. The routing decision runs before Cilium's masquerade hook, so a rule +matching the pod source sees the pod address, and the masquerade then rewrites +it to the egress link's. + +## Rule order + +Lower priority runs first. + +| prio | match | table | +|---|---|---| +| 980 | src pod, dst pod | 48 | +| 981 | src pod, dst pod (v4) | 48 | +| 990 | dst `fd00::/8` | main | +| 991 | dst `10.0.0.0/8` | main | +| 1000 | src pod | 47 | +| 1001 | src pod (v4) | 47 | + +980/981 match source **and** destination. Source alone would also catch the +node's own traffic to a pod, which would then leave with a `pod0` source — an +address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as +`world` rather than `host`. + +990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps +cluster-internal traffic out of table 47. Without them a pod's packet to a +service matches the source rule, finds only a default route, and leaves for the +UDM — measured, in-cluster DNS stopped resolving. + +Traffic to a pod on the same node falls through on its own: table 48 holds only +the peers' CIDRs, so the lookup misses and 990/991 send it to main. + +## Where each table comes from + +- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`. + The IPv6 gateway is the UDM's configured address on the segment, not the + RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA + default does not land in main. +- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route + per peer pod CIDR with the peer's `pod0` address as next hop. + +## The bird mesh + +`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR +and installs what it learns into table 48. Cilium's routes in main are +untouched, which is why the two coexist. + +No address is written into this repo twice. Each node publishes its `pod0` +addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the +pod CIDRs come from the Node objects. An init container renders both from the +API with `kubectl -o go-template-file`, excludes the node itself with a field +selector, and ends with `bird -p` so BIRD's own parser gates startup. + +The node's own CIDR is **originated**, not read back from the kernel. Reading +it back made BIRD depend on the shape Cilium installs, and the IPv4 route is +rejected outright: its next hop sits inside the prefix it describes, which KRT +refuses with `strange next-hop`. The IPv6 route is a device route and passed, +so the failure was silent on one family only. + +## Failure behaviour + +Benign at every layer. An empty or absent table 48 makes 980/981 fall through +to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same +holds if the DaemonSet cannot start, which is the case during a cold rebuild +until Gitea serves its image. + +## Management, the remaining class + +Not started. The levers differ per service and none of them is the node IP: + +| service | can leave the node IP? | lever | +|---|---|---| +| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node | +| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` | +| Talos API | n/a | `apid` answers on every node address; a firewall question | +| kubelet | no | it *is* the node IP | + +Two approaches were rejected by the owner and are not to be proposed again: +moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the +cluster VLAN while moving etcd and the API VIP to a new management VLAN. +Replacing the CNI (#839, Antrea) was shelved. + +Note that the reasoning which produced that list also concluded pod↔pod was +blocked, which policy routing disproved. Treat it as a record of decisions, not +as an exhaustive analysis. + +## Sources + +Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`, +`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol, +`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and +`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854, +#855; closed unmerged: #830, #838, #839. diff --git a/kubernetes/PROPOSAL-traffic-class-separation.md b/kubernetes/PROPOSAL-traffic-class-separation.md deleted file mode 100644 index 3f2a7225..00000000 --- a/kubernetes/PROPOSAL-traffic-class-separation.md +++ /dev/null @@ -1,194 +0,0 @@ -# Proposal — separate the four traffic classes onto their own VLANs - -Status: draft. The question this started with is **answered** — from the Cilium -source and flag reference, not from a test — and the answer changes the shape -of the proposal rather than confirming it. - -Supersedes the framing in issue #576, which described this as "give node -management its own NIC, and Ceph public its own VLAN". That was two of four -classes and one of them turned out already solved. - -## The four classes - -| class | what it is | where it goes today | -|---|---|---| -| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** | -| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` | -| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` | -| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` | - -Three of four share one interface. That is not a design decision — until the -bare-metal conversion there was one NIC to work with. - -Ceph client I/O living on the pod network is correct rather than a leftover: -that traffic *is* cluster-internal. Ceph is done. - -## Principle - -**Each class gets its own VLAN. Which physical interface a VLAN lands on is a -separate, changeable decision.** - -Defining classes by VLAN means the separation survives recabling, port failures -and capacity changes, and it does not require one port per class. Defining them -by interface would freeze a capacity decision into the architecture. - -## The finding that shapes everything else - -The original open question was whether `nodePort.directRoutingDevice` moves the -next-hop of Cilium's direct routes, or only selects an interface. It is -answerable from the source, and the answer is: **only the interface.** - -`installDirectRoute` takes the next-hop as a parameter: - -```go -func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP, - skipUnreachable bool) (routeSpec *netlink.Route, err error) -``` - -and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns, -in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The -same `newIP4` is what tunnel mode uses as the VXLAN endpoint. - -`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like -an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and -`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes -cluster there is no path by which Cilium's node IP differs from the kubelet's -`InternalIP`. - -`--direct-routing-device` is documented as "Device name used to connect nodes in -direct routing mode" — a device, not an address. - -**So cluster-internal traffic is welded to the node IP, and the node IP is the -kubelet InternalIP.** Two consequences: - -- The class cannot be moved off whatever VLAN the node IP lives on. -- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN - from the node IP; native mode uses it as the direct-route next-hop. Native - routing buys nothing here, so the disruptive, non-rolling step this proposal - previously contained is simply gone. - -## Consequence: draw the line the other way - -If cluster-internal cannot leave the node IP, then separation has to come from -moving *management services* off it — and each one has to be checked -individually, because they do not share a mechanism. - -| management service | can it leave the node IP? | lever | -|---|---|---| -| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) | -| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field | -| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting | -| kubelet | **no** | it *is* the node IP, by definition | - -So four VLANs are reachable, but the boundary does not fall where the class -names suggest: **kubelet stays with cluster-internal.** That is the honest -result, and it is worth stating plainly rather than filing kubelet under -management and hoping. - -The remaining classes keep the levers established earlier: - -- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`. - For eBGP the session source is the next-hop, so pinning the session pins - where return traffic arrives. The session to the UDM has been running for - months. -- **cluster-external egress** — eBPF masquerading follows the output device, so - this is a default-route decision plus `enable-masquerade-to-route-source` - (today `false`). -- **Ceph replication** — Multus macvlan on VLAN 49, done. - -## The general escape hatch - -Ceph is separated because its OSDs got a second interface and stopped using the -pod network for replication. That is the *only* mechanism that fully detaches a -workload from the node IP, and it generalises: anything that must not ride the -node IP gets its own `NetworkAttachmentDefinition`. - -It is also expensive per workload — an attachment, an IPAM pool, a pod spec -change. Worth it for Ceph replication. Not worth it for east-west traffic in -general. - -## Recommended mapping - -Two ways to arrange this, and they differ only in which VLAN gets relabelled: - -**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP -and Talos API. Cleanest match to the class names, but it changes every node's -identity, which is the one address the whole cluster is administered through. - -**B (recommended) — leave the node IP on VLAN 40 and move management off it.** -VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP. -Identical end state, and no node ever changes its InternalIP. - -Under B: - -| VLAN | carries | -|---|---| -| 49 | Ceph replication | -| 40 (relabelled *cluster*) | pod east-west, kubelet | -| new *management* | etcd, kube-apiserver VIP, Talos API by convention | -| new *external* | LoadBalancer ingress, egress default route | - -The property the user asked for — management carries only management traffic — -holds under B. The label moves; the property is what matters. - -## Migration shape - -Every step is rolling and reversible. The routing-mode change is gone. - -1. **New VLANs** on the switch, trunked to the X710. Nothing uses them. -2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`. - Still unused; this only proves the L2 works. -3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time, - waiting for a healthy quorum between each. -4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for - `k8s.renner.uno` follows. -5. **cluster-external**: BGP `localAddress`, default route, masquerade flag. - Reversible by reverting the override; failure is visible immediately in - `cilium-dbg shell -- bgp/peers`. -6. Verify VLAN 40 by what is *absent* from it, not by what still works. - -Step 3 is the one that deserves care — etcd is the only step where a mistake -costs quorum rather than connectivity. - -## What this does not buy - -Management currently shares a 10 Gb link and is not contended. Nothing observed -today is slow because of this. The case for the split is that traffic classes -with different failure modes and security postures should not be -indistinguishable — not that something is broken. - -## Sources and measurements - -Source, Cilium v1.20.1 (the running version): - -- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`, - fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects - a next-hop that is not directly reachable. -- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`. -- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies - `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the - Kubernetes Node. -- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used - to connect nodes in direct routing mode". - -Talos: - -- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets` - defaults to it. kube-apiserver and Talos API listen addresses are not - covered there. -- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for - `10.2.40.60` with `link: mgmt0`. - -Measured on the cluster: - -- `apid` answered on a node address that is not the InternalIP, so the Talos - API is not bound to one address. -- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its - InternalIP had changed since the last session, which is why nothing here - names an address as fixed. -- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source` - unset and no direct-routing device configured. - -Not verified: whether the kube-apiserver's `--advertise-address` follows the -node IP independently of the VIP. It does not affect the recommendation, since -clients reach the API through the VIP either way. diff --git a/kubernetes/bootstrap.sh b/kubernetes/bootstrap.sh index 5f4b584a..ad3fccc6 100755 --- a/kubernetes/bootstrap.sh +++ b/kubernetes/bootstrap.sh @@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then # Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which # would not match the amd64 schematic this cluster installs. That leaves the # cx*/cpx*2 families in the German locations. - # Today's traffic-class split, not the target one in - # PROPOSAL-traffic-class-separation.md: the node address carries management, + # Two segments only, unlike the homelab's split in + # NETWORK-traffic-classes.md: here the node address carries management, # cluster-internal and external alike, and Ceph replication has its own. # # The order is load-bearing: it decides which kernel interface each segment -- 2.54.0