docs(#576): record the rejected options in the traffic-class proposal

The proposal still recommended option B (node IP stays on VLAN 40, etcd and the API VIP move to a new management VLAN), which the owner has rejected repeatedly, and it described the cluster as running in tunnel mode. Rewritten to the measured state of 2026-09-15: world-pod is on VLAN 47 since #835/#836, the cluster runs native routing since #837, pod-pod still follows the kubelet InternalIP on mgmt0. Options A (#838) and B are marked rejected, C (#839, Antrea) shelved. The owner's 2026-09-08 target and the no-UDM constraint are kept as the reference.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -1,194 +1,103 @@
 # Proposal — separate the four traffic classes onto their own VLANs
 
-Status: draft. The question this started with is **answered** — from the Cilium
+Status: open, issue #576. Two of the four classes are separated. Pod east-west
-source and flag reference, not from a test — and the answer changes the shape
+cannot leave the node IP under Cilium. Both ways of moving it within Cilium
-of the proposal rather than confirming it.
+(options A and B below) are rejected by the owner, and replacing the CNI
+(option C) is shelved.
 
-Supersedes the framing in issue #576, which described this as "give node
+## Target
-management its own NIC, and Ceph public its own VLAN". That was two of four
-classes and one of them turned out already solved.
 
-## The four classes
+Set by the owner on 2026-09-08:
 
-| class | what it is | where it goes today |
+| class | link |
+|---|---|
+| Ceph replication | X710 p1, VLAN 49 |
+| pod-pod | X710 trunk, a VLAN of its own |
+| world-pod | X710 trunk, a VLAN of its own |
+| node management | 1 Gb onboard NIC |
+
+Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
+on the 10G switch.
+
+## Where the classes are
+
+Measured 2026-09-15.
+
+| class | today | state |
 |---|---|---|
-| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
-| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
-| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
-| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
+| management | `mgmt0`, VLAN 40 | not started |
 
-Three of four share one interface. That is not a design decision — until the
+`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
-bare-metal conversion there was one NIC to work with.
+#833). `pod0` holds addresses and carries no traffic.
 
-Ceph client I/O living on the pod network is correct rather than a leftover:
+## Why pod-pod is blocked
-that traffic *is* cluster-internal. Ceph is done.
 
-## Principle
+Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
+next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
 
-**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
-separate, changeable decision.**
+  fed from `newNode.GetNodeIP(false)`.
+- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
+  `initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
+  Node object.
+- `--direct-routing-device` selects a device for the BPF NodePort path, not a
+  next-hop.
 
-Defining classes by VLAN means the separation survives recabling, port failures
+Tested on the cluster, none of these moved the next-hop:
-and capacity changes, and it does not require one port per class. Defining them
-by interface would freeze a capacity decision into the architecture.
 
-## The finding that shapes everything else
+| attempt | result |
+|---|---|
+| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
+| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
+| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
 
-The original open question was whether `nodePort.directRoutingDevice` moves the
+The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
-next-hop of Cilium's direct routes, or only selects an interface. It is
+`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
-answerable from the source, and the answer is: **only the interface.**
+Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
+cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
+has no effect on pod-pod routing.
 
-`installDirectRoute` takes the next-hop as a parameter:
+## Options for pod-pod
 
-```go
+| | option | status |
-func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
-    skipUnreachable bool) (routeSpec *netlink.Route, err error)
-```
-
-and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
-in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
-same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
-
-`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
-an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
-`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
-cluster there is no path by which Cilium's node IP differs from the kubelet's
-`InternalIP`.
-
-`--direct-routing-device` is documented as "Device name used to connect nodes in
-direct routing mode" — a device, not an address.
-
-**So cluster-internal traffic is welded to the node IP, and the node IP is the
-kubelet InternalIP.** Two consequences:
-
-- The class cannot be moved off whatever VLAN the node IP lives on.
-- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
-  from the node IP; native mode uses it as the direct-route next-hop. Native
-  routing buys nothing here, so the disruptive, non-rolling step this proposal
-  previously contained is simply gone.
-
-## Consequence: draw the line the other way
-
-If cluster-internal cannot leave the node IP, then separation has to come from
-moving *management services* off it — and each one has to be checked
-individually, because they do not share a mechanism.
-
-| management service | can it leave the node IP? | lever |
 |---|---|---|
-| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
-| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
-| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
-| kubelet | **no** | it *is* the node IP, by definition |
 
-So four VLANs are reachable, but the boundary does not fall where the class
+A per-workload Multus attachment detaches a single workload from the node IP,
-names suggest: **kubelet stays with cluster-internal.** That is the honest
+the way Ceph replication is detached. It does not cover east-west traffic in
-result, and it is worth stating plainly rather than filing kubelet under
-management and hoping.
-
-The remaining classes keep the levers established earlier:
-
-- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
-  For eBGP the session source is the next-hop, so pinning the session pins
-  where return traffic arrives. The session to the UDM has been running for
-  months.
-- **cluster-external egress** — eBPF masquerading follows the output device, so
-  this is a default-route decision plus `enable-masquerade-to-route-source`
-  (today `false`).
-- **Ceph replication** — Multus macvlan on VLAN 49, done.
-
-## The general escape hatch
-
-Ceph is separated because its OSDs got a second interface and stopped using the
-pod network for replication. That is the *only* mechanism that fully detaches a
-workload from the node IP, and it generalises: anything that must not ride the
-node IP gets its own `NetworkAttachmentDefinition`.
-
-It is also expensive per workload — an attachment, an IPAM pool, a pod spec
-change. Worth it for Ceph replication. Not worth it for east-west traffic in
 general.
 
-## Recommended mapping
+## Levers per management service
 
-Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+These hold regardless of which option is chosen:
 
-**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+| service | can leave the node IP? | lever |
-and Talos API. Cleanest match to the class names, but it changes every node's
+|---|---|---|
-identity, which is the one address the whole cluster is administered through.
+| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
+| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
+| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
+| kubelet | no | it is the node IP |
 
-**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+## Remaining, independent of pod-pod
-VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
-Identical end state, and no node ever changes its InternalIP.
 
-Under B:
+- **pod→world egress**: default route and masquerade source onto `ingress0`,
+  via route metrics, `devices` and `enable-masquerade-to-route-source`. This
+  moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
+- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
+  removed once ingress on VLAN 47 is settled.
 
-| VLAN | carries |
+## Sources
-|---|---|
-| 49 | Ceph replication |
-| 40 (relabelled *cluster*) | pod east-west, kubelet |
-| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
-| new *external* | LoadBalancer ingress, egress default route |
 
-The property the user asked for — management carries only management traffic —
+- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
-holds under B. The label moves; the property is what matters.
+  `--direct-routing-device`.
-
+- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
-## Migration shape
+- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
-
+- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
-Every step is rolling and reversible. The routing-mode change is gone.
+  `CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
-
-1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
-2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
-   Still unused; this only proves the L2 works.
-3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
-   waiting for a healthy quorum between each.
-4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
-   `k8s.renner.uno` follows.
-5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
-   Reversible by reverting the override; failure is visible immediately in
-   `cilium-dbg shell -- bgp/peers`.
-6. Verify VLAN 40 by what is *absent* from it, not by what still works.
-
-Step 3 is the one that deserves care — etcd is the only step where a mistake
-costs quorum rather than connectivity.
-
-## What this does not buy
-
-Management currently shares a 10 Gb link and is not contended. Nothing observed
-today is slow because of this. The case for the split is that traffic classes
-with different failure modes and security postures should not be
-indistinguishable — not that something is broken.
-
-## Sources and measurements
-
-Source, Cilium v1.20.1 (the running version):
-
-- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
-  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
-  a next-hop that is not directly reachable.
-- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
-- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
-  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
-  Kubernetes Node.
-- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
-  to connect nodes in direct routing mode".
-
-Talos:
-
-- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
-  defaults to it. kube-apiserver and Talos API listen addresses are not
-  covered there.
-- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
-  `10.2.40.60` with `link: mgmt0`.
-
-Measured on the cluster:
-
-- `apid` answered on a node address that is not the InternalIP, so the Talos
-  API is not bound to one address.
-- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
-  InternalIP had changed since the last session, which is why nothing here
-  names an address as fixed.
-- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
-  unset and no direct-routing device configured.
-
-Not verified: whether the kube-apiserver's `--advertise-address` follows the
-node IP independently of the VIP. It does not affect the recommendation, since
-clients reach the API through the VIP either way.