From 774907eaf18b54a10bb840488bdbea51de65b697 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Tue, 15 Sep 2026 13:45:33 +0200 Subject: [PATCH] docs(#576): record the rejected options in the traffic-class proposal The proposal still recommended option B (node IP stays on VLAN 40, etcd and the API VIP move to a new management VLAN), which the owner has rejected repeatedly, and it described the cluster as running in tunnel mode. Rewritten to the measured state of 2026-09-15: world-pod is on VLAN 47 since #835/#836, the cluster runs native routing since #837, pod-pod still follows the kubelet InternalIP on mgmt0. Options A (#838) and B are marked rejected, C (#839, Antrea) shelved. The owner's 2026-09-08 target and the no-UDM constraint are kept as the reference. Refs #576 Co-Authored-By: Claude Opus 5 (1M context) --- .../PROPOSAL-traffic-class-separation.md | 247 ++++++------------ 1 file changed, 78 insertions(+), 169 deletions(-) diff --git a/kubernetes/PROPOSAL-traffic-class-separation.md b/kubernetes/PROPOSAL-traffic-class-separation.md index 3f2a7225..d3be6ab0 100644 --- a/kubernetes/PROPOSAL-traffic-class-separation.md +++ b/kubernetes/PROPOSAL-traffic-class-separation.md @@ -1,194 +1,103 @@ # Proposal — separate the four traffic classes onto their own VLANs -Status: draft. The question this started with is **answered** — from the Cilium -source and flag reference, not from a test — and the answer changes the shape -of the proposal rather than confirming it. +Status: open, issue #576. Two of the four classes are separated. Pod east-west +cannot leave the node IP under Cilium. Both ways of moving it within Cilium +(options A and B below) are rejected by the owner, and replacing the CNI +(option C) is shelved. -Supersedes the framing in issue #576, which described this as "give node -management its own NIC, and Ceph public its own VLAN". That was two of four -classes and one of them turned out already solved. +## Target -## The four classes +Set by the owner on 2026-09-08: -| class | what it is | where it goes today | +| class | link | +|---|---| +| Ceph replication | X710 p1, VLAN 49 | +| pod-pod | X710 trunk, a VLAN of its own | +| world-pod | X710 trunk, a VLAN of its own | +| node management | 1 Gb onboard NIC | + +Hard constraint: inter-node and pod traffic never traverses the UDM. It stays +on the 10G switch. + +## Where the classes are + +Measured 2026-09-15. + +| class | today | state | |---|---|---| -| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** | -| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` | -| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` | -| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` | +| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done | +| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) | +| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open | +| pod-pod | native routing, ` via dev mgmt0`, VLAN 40 | blocked, see below | +| management | `mgmt0`, VLAN 40 | not started | -Three of four share one interface. That is not a design decision — until the -bare-metal conversion there was one NIC to work with. +`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831, +#833). `pod0` holds addresses and carries no traffic. -Ceph client I/O living on the pod network is correct rather than a leftover: -that traffic *is* cluster-internal. Ceph is done. +## Why pod-pod is blocked -## Principle +Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the +next-hop, in native mode, and as the VXLAN endpoint in tunnel mode: -**Each class gets its own VLAN. Which physical interface a VLAN lands on is a -separate, changeable decision.** +- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is + fed from `newNode.GetNodeIP(false)`. +- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`. +- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after + `initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the + Node object. +- `--direct-routing-device` selects a device for the BPF NodePort path, not a + next-hop. -Defining classes by VLAN means the separation survives recabling, port failures -and capacity changes, and it does not require one port per class. Defining them -by interface would freeze a capacity decision into the architecture. +Tested on the cluster, none of these moved the next-hop: -## The finding that shapes everything else +| attempt | result | +|---|---| +| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address | +| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` | +| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP | -The original open question was whether `nodePort.directRoutingDevice` moves the -next-hop of Cilium's direct routes, or only selects an interface. It is -answerable from the source, and the answer is: **only the interface.** +The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in +`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`. +Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The +cluster has been in native routing mode since #837. `directRoutingDevice: pod0` +has no effect on pod-pod routing. -`installDirectRoute` takes the next-hop as a parameter: +## Options for pod-pod -```go -func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP, - skipUnreachable bool) (routeSpec *netlink.Route, err error) -``` - -and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns, -in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The -same `newIP4` is what tunnel mode uses as the VXLAN endpoint. - -`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like -an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and -`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes -cluster there is no path by which Cilium's node IP differs from the kubelet's -`InternalIP`. - -`--direct-routing-device` is documented as "Device name used to connect nodes in -direct routing mode" — a device, not an address. - -**So cluster-internal traffic is welded to the node IP, and the node IP is the -kubelet InternalIP.** Two consequences: - -- The class cannot be moved off whatever VLAN the node IP lives on. -- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN - from the node IP; native mode uses it as the direct-route next-hop. Native - routing buys nothing here, so the disruptive, non-rolling step this proposal - previously contained is simply gone. - -## Consequence: draw the line the other way - -If cluster-internal cannot leave the node IP, then separation has to come from -moving *management services* off it — and each one has to be checked -individually, because they do not share a mechanism. - -| management service | can it leave the node IP? | lever | +| | option | status | |---|---|---| -| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) | -| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field | -| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting | -| kubelet | **no** | it *is* the node IP, by definition | +| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 | +| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again | +| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha | -So four VLANs are reachable, but the boundary does not fall where the class -names suggest: **kubelet stays with cluster-internal.** That is the honest -result, and it is worth stating plainly rather than filing kubelet under -management and hoping. - -The remaining classes keep the levers established earlier: - -- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`. - For eBGP the session source is the next-hop, so pinning the session pins - where return traffic arrives. The session to the UDM has been running for - months. -- **cluster-external egress** — eBPF masquerading follows the output device, so - this is a default-route decision plus `enable-masquerade-to-route-source` - (today `false`). -- **Ceph replication** — Multus macvlan on VLAN 49, done. - -## The general escape hatch - -Ceph is separated because its OSDs got a second interface and stopped using the -pod network for replication. That is the *only* mechanism that fully detaches a -workload from the node IP, and it generalises: anything that must not ride the -node IP gets its own `NetworkAttachmentDefinition`. - -It is also expensive per workload — an attachment, an IPAM pool, a pod spec -change. Worth it for Ceph replication. Not worth it for east-west traffic in +A per-workload Multus attachment detaches a single workload from the node IP, +the way Ceph replication is detached. It does not cover east-west traffic in general. -## Recommended mapping +## Levers per management service -Two ways to arrange this, and they differ only in which VLAN gets relabelled: +These hold regardless of which option is chosen: -**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP -and Talos API. Cleanest match to the class names, but it changes every node's -identity, which is the one address the whole cluster is administered through. +| service | can leave the node IP? | lever | +|---|---|---| +| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node | +| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field | +| Talos API | n/a | `apid` answers on every node address; a client and firewall choice | +| kubelet | no | it is the node IP | -**B (recommended) — leave the node IP on VLAN 40 and move management off it.** -VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP. -Identical end state, and no node ever changes its InternalIP. +## Remaining, independent of pod-pod -Under B: +- **pod→world egress**: default route and masquerade source onto `ingress0`, + via route metrics, `devices` and `enable-masquerade-to-route-source`. This + moves the nodes' own egress (image pulls, NTP) to VLAN 47 too. +- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be + removed once ingress on VLAN 47 is settled. -| VLAN | carries | -|---|---| -| 49 | Ceph replication | -| 40 (relabelled *cluster*) | pod east-west, kubelet | -| new *management* | etcd, kube-apiserver VIP, Talos API by convention | -| new *external* | LoadBalancer ingress, egress default route | +## Sources -The property the user asked for — management carries only management traffic — -holds under B. The label moves; the property is what matters. - -## Migration shape - -Every step is rolling and reversible. The routing-mode change is gone. - -1. **New VLANs** on the switch, trunked to the X710. Nothing uses them. -2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`. - Still unused; this only proves the L2 works. -3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time, - waiting for a healthy quorum between each. -4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for - `k8s.renner.uno` follows. -5. **cluster-external**: BGP `localAddress`, default route, masquerade flag. - Reversible by reverting the override; failure is visible immediately in - `cilium-dbg shell -- bgp/peers`. -6. Verify VLAN 40 by what is *absent* from it, not by what still works. - -Step 3 is the one that deserves care — etcd is the only step where a mistake -costs quorum rather than connectivity. - -## What this does not buy - -Management currently shares a 10 Gb link and is not contended. Nothing observed -today is slow because of this. The case for the split is that traffic classes -with different failure modes and security postures should not be -indistinguishable — not that something is broken. - -## Sources and measurements - -Source, Cilium v1.20.1 (the running version): - -- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`, - fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects - a next-hop that is not directly reachable. -- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`. -- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies - `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the - Kubernetes Node. -- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used - to connect nodes in direct routing mode". - -Talos: - -- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets` - defaults to it. kube-apiserver and Talos API listen addresses are not - covered there. -- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for - `10.2.40.60` with `link: mgmt0`. - -Measured on the cluster: - -- `apid` answered on a node address that is not the InternalIP, so the Talos - API is not bound to one address. -- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its - InternalIP had changed since the last session, which is why nothing here - names an address as fixed. -- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source` - unset and no direct-routing device configured. - -Not verified: whether the kube-apiserver's `--advertise-address` follows the -node IP independently of the VIP. It does not affect the recommendation, since -clients reach the API through the VIP either way. +- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for + `--direct-routing-device`. +- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`. +- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576. +- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`, + `CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2. -- 2.54.0