diff --git a/kubernetes/NETWORK-traffic-classes.md b/kubernetes/NETWORK-traffic-classes.md new file mode 100644 index 00000000..29808ca8 --- /dev/null +++ b/kubernetes/NETWORK-traffic-classes.md @@ -0,0 +1,117 @@ +# Traffic classes on the k8s nodes + +Issue #576. Three of the four classes are separated; management is not. + +## Where each class runs + +| class | link | mechanism | +|---|---|---| +| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) | +| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address | +| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 | +| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 | +| management | `mgmt0`, VLAN 40 | not started | + +Hard constraint, set by the owner: inter-node and pod traffic never traverses +the UDM. Everything above stays on the 10G switch. + +## Why policy routing rather than moving the node IP + +Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native +and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and +`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop. +Both were established by test (#837, #838), not by reading. + +So the InternalIP decides where pod traffic goes — unless the lookup happens in +a different table. The rules do that, and nothing about the node's identity +moves. The routing decision runs before Cilium's masquerade hook, so a rule +matching the pod source sees the pod address, and the masquerade then rewrites +it to the egress link's. + +## Rule order + +Lower priority runs first. + +| prio | match | table | +|---|---|---| +| 980 | src pod, dst pod | 48 | +| 981 | src pod, dst pod (v4) | 48 | +| 990 | dst `fd00::/8` | main | +| 991 | dst `10.0.0.0/8` | main | +| 1000 | src pod | 47 | +| 1001 | src pod (v4) | 47 | + +980/981 match source **and** destination. Source alone would also catch the +node's own traffic to a pod, which would then leave with a `pod0` source — an +address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as +`world` rather than `host`. + +990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps +cluster-internal traffic out of table 47. Without them a pod's packet to a +service matches the source rule, finds only a default route, and leaves for the +UDM — measured, in-cluster DNS stopped resolving. + +Traffic to a pod on the same node falls through on its own: table 48 holds only +the peers' CIDRs, so the lookup misses and 990/991 send it to main. + +## Where each table comes from + +- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`. + The IPv6 gateway is the UDM's configured address on the segment, not the + RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA + default does not land in main. +- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route + per peer pod CIDR with the peer's `pod0` address as next hop. + +## The bird mesh + +`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR +and installs what it learns into table 48. Cilium's routes in main are +untouched, which is why the two coexist. + +No address is written into this repo twice. Each node publishes its `pod0` +addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the +pod CIDRs come from the Node objects. An init container renders both from the +API with `kubectl -o go-template-file`, excludes the node itself with a field +selector, and ends with `bird -p` so BIRD's own parser gates startup. + +The node's own CIDR is **originated**, not read back from the kernel. Reading +it back made BIRD depend on the shape Cilium installs, and the IPv4 route is +rejected outright: its next hop sits inside the prefix it describes, which KRT +refuses with `strange next-hop`. The IPv6 route is a device route and passed, +so the failure was silent on one family only. + +## Failure behaviour + +Benign at every layer. An empty or absent table 48 makes 980/981 fall through +to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same +holds if the DaemonSet cannot start, which is the case during a cold rebuild +until Gitea serves its image. + +## Management, the remaining class + +Not started. The levers differ per service and none of them is the node IP: + +| service | can leave the node IP? | lever | +|---|---|---| +| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node | +| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` | +| Talos API | n/a | `apid` answers on every node address; a firewall question | +| kubelet | no | it *is* the node IP | + +Two approaches were rejected by the owner and are not to be proposed again: +moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the +cluster VLAN while moving etcd and the API VIP to a new management VLAN. +Replacing the CNI (#839, Antrea) was shelved. + +Note that the reasoning which produced that list also concluded pod↔pod was +blocked, which policy routing disproved. Treat it as a record of decisions, not +as an exhaustive analysis. + +## Sources + +Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`, +`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol, +`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and +`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854, +#855; closed unmerged: #830, #838, #839. diff --git a/kubernetes/PROPOSAL-traffic-class-separation.md b/kubernetes/PROPOSAL-traffic-class-separation.md deleted file mode 100644 index 3f2a7225..00000000 --- a/kubernetes/PROPOSAL-traffic-class-separation.md +++ /dev/null @@ -1,194 +0,0 @@ -# Proposal — separate the four traffic classes onto their own VLANs - -Status: draft. The question this started with is **answered** — from the Cilium -source and flag reference, not from a test — and the answer changes the shape -of the proposal rather than confirming it. - -Supersedes the framing in issue #576, which described this as "give node -management its own NIC, and Ceph public its own VLAN". That was two of four -classes and one of them turned out already solved. - -## The four classes - -| class | what it is | where it goes today | -|---|---|---| -| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** | -| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` | -| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` | -| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` | - -Three of four share one interface. That is not a design decision — until the -bare-metal conversion there was one NIC to work with. - -Ceph client I/O living on the pod network is correct rather than a leftover: -that traffic *is* cluster-internal. Ceph is done. - -## Principle - -**Each class gets its own VLAN. Which physical interface a VLAN lands on is a -separate, changeable decision.** - -Defining classes by VLAN means the separation survives recabling, port failures -and capacity changes, and it does not require one port per class. Defining them -by interface would freeze a capacity decision into the architecture. - -## The finding that shapes everything else - -The original open question was whether `nodePort.directRoutingDevice` moves the -next-hop of Cilium's direct routes, or only selects an interface. It is -answerable from the source, and the answer is: **only the interface.** - -`installDirectRoute` takes the next-hop as a parameter: - -```go -func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP, - skipUnreachable bool) (routeSpec *netlink.Route, err error) -``` - -and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns, -in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The -same `newIP4` is what tunnel mode uses as the VXLAN endpoint. - -`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like -an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and -`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes -cluster there is no path by which Cilium's node IP differs from the kubelet's -`InternalIP`. - -`--direct-routing-device` is documented as "Device name used to connect nodes in -direct routing mode" — a device, not an address. - -**So cluster-internal traffic is welded to the node IP, and the node IP is the -kubelet InternalIP.** Two consequences: - -- The class cannot be moved off whatever VLAN the node IP lives on. -- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN - from the node IP; native mode uses it as the direct-route next-hop. Native - routing buys nothing here, so the disruptive, non-rolling step this proposal - previously contained is simply gone. - -## Consequence: draw the line the other way - -If cluster-internal cannot leave the node IP, then separation has to come from -moving *management services* off it — and each one has to be checked -individually, because they do not share a mechanism. - -| management service | can it leave the node IP? | lever | -|---|---|---| -| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) | -| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field | -| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting | -| kubelet | **no** | it *is* the node IP, by definition | - -So four VLANs are reachable, but the boundary does not fall where the class -names suggest: **kubelet stays with cluster-internal.** That is the honest -result, and it is worth stating plainly rather than filing kubelet under -management and hoping. - -The remaining classes keep the levers established earlier: - -- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`. - For eBGP the session source is the next-hop, so pinning the session pins - where return traffic arrives. The session to the UDM has been running for - months. -- **cluster-external egress** — eBPF masquerading follows the output device, so - this is a default-route decision plus `enable-masquerade-to-route-source` - (today `false`). -- **Ceph replication** — Multus macvlan on VLAN 49, done. - -## The general escape hatch - -Ceph is separated because its OSDs got a second interface and stopped using the -pod network for replication. That is the *only* mechanism that fully detaches a -workload from the node IP, and it generalises: anything that must not ride the -node IP gets its own `NetworkAttachmentDefinition`. - -It is also expensive per workload — an attachment, an IPAM pool, a pod spec -change. Worth it for Ceph replication. Not worth it for east-west traffic in -general. - -## Recommended mapping - -Two ways to arrange this, and they differ only in which VLAN gets relabelled: - -**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP -and Talos API. Cleanest match to the class names, but it changes every node's -identity, which is the one address the whole cluster is administered through. - -**B (recommended) — leave the node IP on VLAN 40 and move management off it.** -VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP. -Identical end state, and no node ever changes its InternalIP. - -Under B: - -| VLAN | carries | -|---|---| -| 49 | Ceph replication | -| 40 (relabelled *cluster*) | pod east-west, kubelet | -| new *management* | etcd, kube-apiserver VIP, Talos API by convention | -| new *external* | LoadBalancer ingress, egress default route | - -The property the user asked for — management carries only management traffic — -holds under B. The label moves; the property is what matters. - -## Migration shape - -Every step is rolling and reversible. The routing-mode change is gone. - -1. **New VLANs** on the switch, trunked to the X710. Nothing uses them. -2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`. - Still unused; this only proves the L2 works. -3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time, - waiting for a healthy quorum between each. -4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for - `k8s.renner.uno` follows. -5. **cluster-external**: BGP `localAddress`, default route, masquerade flag. - Reversible by reverting the override; failure is visible immediately in - `cilium-dbg shell -- bgp/peers`. -6. Verify VLAN 40 by what is *absent* from it, not by what still works. - -Step 3 is the one that deserves care — etcd is the only step where a mistake -costs quorum rather than connectivity. - -## What this does not buy - -Management currently shares a 10 Gb link and is not contended. Nothing observed -today is slow because of this. The case for the split is that traffic classes -with different failure modes and security postures should not be -indistinguishable — not that something is broken. - -## Sources and measurements - -Source, Cilium v1.20.1 (the running version): - -- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`, - fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects - a next-hop that is not directly reachable. -- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`. -- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies - `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the - Kubernetes Node. -- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used - to connect nodes in direct routing mode". - -Talos: - -- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets` - defaults to it. kube-apiserver and Talos API listen addresses are not - covered there. -- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for - `10.2.40.60` with `link: mgmt0`. - -Measured on the cluster: - -- `apid` answered on a node address that is not the InternalIP, so the Talos - API is not bound to one address. -- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its - InternalIP had changed since the last session, which is why nothing here - names an address as fixed. -- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source` - unset and no direct-routing device configured. - -Not verified: whether the kube-apiserver's `--advertise-address` follows the -node IP independently of the VIP. It does not affect the recommendation, since -clients reach the API through the VIP either way. diff --git a/kubernetes/bootstrap.sh b/kubernetes/bootstrap.sh index 5f4b584a..ad3fccc6 100755 --- a/kubernetes/bootstrap.sh +++ b/kubernetes/bootstrap.sh @@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then # Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which # would not match the amd64 schematic this cluster installs. That leaves the # cx*/cpx*2 families in the German locations. - # Today's traffic-class split, not the target one in - # PROPOSAL-traffic-class-separation.md: the node address carries management, + # Two segments only, unlike the homelab's split in + # NETWORK-traffic-classes.md: here the node address carries management, # cluster-internal and external alike, and Ceph replication has its own. # # The order is load-bearing: it decides which kernel interface each segment