docs(network): traffic classes as built, node-to-node on VLAN 46 (#576)

NETWORK-traffic-classes.md described table 48 and rules 980/981/990/991,
which #857 replaced with one table (47) decided by longest prefix, and
listed management as not started. Rewritten to the live state: five
classes, table 47 contents owned by bird, node IP and etcd on node0, the
etcd leave-and-rejoin procedure, and the mixed-state hazard hit during
the node IP move. The bird DaemonSet comment still named table 48.

Closes #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kubernetes/NETWORK-traffic-classes.md

@@ -1,117 +1,99 @@
 # Traffic classes on the k8s nodes
 
-Issue #576. Three of the four classes are separated; management is not.
+Issue #576. Five classes, each on its own VLAN.
 
 ## Where each class runs
 
 | class | link | mechanism |
 |---|---|---|
 | Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
-| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
-| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
-| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
-| management | `mgmt0`, VLAN 40 | not started |
+| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
+| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
+| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
+| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
 
 Hard constraint, set by the owner: inter-node and pod traffic never traverses
-the UDM. Everything above stays on the 10G switch.
+the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
+statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
 
-## Why policy routing rather than moving the node IP
+## Pod traffic: one table
 
-Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
-and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
-`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
-Both were established by test (#837, #838), not by reading.
+Two rules, one per family, send every packet with a pod source to table 47
+(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
+table decides by longest prefix; no rule carves out exceptions.
 
-So the InternalIP decides where pod traffic goes — unless the lookup happens in
-a different table. The rules do that, and nothing about the node's identity
-moves. The routing decision runs before Cilium's masquerade hook, so a rule
-matching the pod source sees the pod address, and the masquerade then rewrites
-it to the egress link's.
+bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
 
-## Rule order
+- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
+  taken from the interfaces;
+- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
+  peer's `pod0` address;
+- the node's own pod CIDR via `cilium_host`, originated rather than read back
+  from the kernel (reading it back made BIRD depend on the shape of Cilium's
+  route, and KRT rejects the IPv4 one);
+- the egress default via the UDM on `ingress0`, from the node annotations
+  `net.renner.uno/egress-gw-v4/v6`.
 
-Lower priority runs first.
+`node0` has to be in the table: a pod reaching a node address — the
+`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
+address, or a hostNetwork pod — would otherwise take the default to the UDM.
 
-| prio | match | table |
-|---|---|---|
-| 980 | src pod, dst pod | 48 |
-| 981 | src pod, dst pod (v4) | 48 |
-| 990 | dst `fd00::/8` | main |
-| 991 | dst `10.0.0.0/8` | main |
-| 1000 | src pod | 47 |
-| 1001 | src pod (v4) | 47 |
+The routing decision runs before Cilium's masquerade hook, so the rule sees the
+pod address and the masquerade then rewrites it to the egress link's.
 
-980/981 match source **and** destination. Source alone would also catch the
-node's own traffic to a pod, which would then leave with a `pod0` source — an
-address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
-`world` rather than `host`.
-
-990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
-cluster-internal traffic out of table 47. Without them a pod's packet to a
-service matches the source rule, finds only a default route, and leaves for the
-UDM — measured, in-cluster DNS stopped resolving.
-
-Traffic to a pod on the same node falls through on its own: table 48 holds only
-the peers' CIDRs, so the lookup misses and 990/991 send it to main.
-
-## Where each table comes from
-
-- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
-  The IPv6 gateway is the UDM's configured address on the segment, not the
-  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
-  default does not land in main.
-- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
-  per peer pod CIDR with the peer's `pod0` address as next hop.
-
-## The bird mesh
-
-`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
-and installs what it learns into table 48. Cilium's routes in main are
-untouched, which is why the two coexist.
+Node-originated traffic does not match the rules and uses main, where Cilium
+routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
 
 No address is written into this repo twice. Each node publishes its `pod0`
-addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
-pod CIDRs come from the Node objects. An init container renders both from the
-API with `kubectl -o go-template-file`, excludes the node itself with a field
-selector, and ends with `bird -p` so BIRD's own parser gates startup.
+addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
+pod CIDRs come from the Node objects. bird's init container renders both from
+the API and ends with `bird -p`, so BIRD's own parser gates startup.
 
-The node's own CIDR is **originated**, not read back from the kernel. Reading
-it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
-rejected outright: its next hop sits inside the prefix it describes, which KRT
-refuses with `strange next-hop`. The IPv6 route is a device route and passed,
-so the failure was silent on one family only.
+## Node↔node
+
+`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
+VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
+is IPv4, and the apiserver's advertise address must match it.
+
+The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
+kubelet, scheduler, controller-manager and Cilium go through KubePrism
+(`localhost:7445`), which balances over every address of every control-plane
+member; pods use the ClusterIP.
+
+Talos never updates an existing etcd member's peer URLs
+(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
+stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
+reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
+and is promoted. EPHEMERAL does not need a wipe.
+
+## Moving node IPs: the mixed state
+
+While some nodes have their InternalIP on the new network and some on the old,
+host→remote-pod traffic leaves with a node address that is not an InternalIP:
+the kernel picks the source from the route to the peer's InternalIP. Cilium
+classifies that as `world`, and every ingress policy that admits only
+`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
+lose the OSDs on the other side; RBD mounts hang.
+
+Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
+admitting the node subnets of both networks, both families, with
+`enableDefaultDeny: false` so it restricts nothing else, and delete it once
+every node has moved. Keep all OSD nodes schedulable while one is drained:
+draining two takes Ceph below its data availability.
 
 ## Failure behaviour
 
-Benign at every layer. An empty or absent table 48 makes 980/981 fall through
-to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
-holds if the DaemonSet cannot start, which is the case during a cold rebuild
-until Gitea serves its image.
-
-## Management, the remaining class
-
-Not started. The levers differ per service and none of them is the node IP:
-
-| service | can leave the node IP? | lever |
-|---|---|---|
-| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
-| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
-| Talos API | n/a | `apid` answers on every node address; a firewall question |
-| kubelet | no | it *is* the node IP |
-
-Two approaches were rejected by the owner and are not to be proposed again:
-moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
-cluster VLAN while moving etcd and the API VIP to a new management VLAN.
-Replacing the CNI (#839, Antrea) was shelved.
-
-Note that the reasoning which produced that list also concluded pod↔pod was
-blocked, which policy routing disproved. Treat it as a record of decisions, not
-as an exhaustive analysis.
+If bird stops, table 47 empties, the rules find nothing and fall through to
+main: pod traffic returns to the node networks, unseparated rather than
+misrouted. The same holds during a cold rebuild until Gitea serves bird's
+image. The default route lives in table 47, not in the Talos config, so a
+default cannot outlive the specific routes and send pod traffic to the UDM.
 
 ## Sources
 
-Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
-`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
-`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
-`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
-#855; closed unmerged: #830, #838, #839.
+Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
+protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
+`machine.nodeAnnotations`, configuration patching (lists are appended,
+`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
+siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
+#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.

kubernetes/infrastructure/bird/bird.conf

@@ -23,7 +23,7 @@ protocol device { scan time 10; }
 # down. The usual advice is not to export these to the kernel because it
 # maintains them itself — true of main, not of a table the kernel never fills.
 protocol direct attached {
-  interface "mgmt0", "ingress0", "pod0";
+  interface "mgmt0", "ingress0", "pod0", "node0";
   ipv4 { table master4; };
   ipv6 { table master6; };
 }

kubernetes/infrastructure/bird/daemonset.yaml

@@ -48,8 +48,8 @@ spec:
         tier: node
         app: bird
     spec:
-      # Carries pod east-west once the routing rules land. Without it table 48
-      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
+      # Owns table 47, where all pod traffic is routed. Without it the table
+      # stays empty and the rules fall through to main, unseparated.
       priorityClassName: system-node-critical
       # Writes host routes and peers on host addresses.
       hostNetwork: true

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -118,14 +118,21 @@ spec:
       # traffic out to the VIP and back (which fails with L2 announcements).
       lbExternalClusterIP: true
     cni:
-      # Multus (kubernetes/infrastructure/multus) runs alongside Cilium and
-      # writes 00-multus.conf to /etc/cni/net.d so kubelet picks multus
-      # first; multus then delegates to Cilium's 05-cilium.conflist via
-      # clusterNetwork. The default `exclusive: true` makes cilium-cni's
-      # install step rename anything alphabetically before its own config
-      # to *.cilium_bak on every reconcile, which kept nuking multus.
-      # Setting it false lets multus's config coexist.
+      # Multus is the only writer of /etc/cni/net.d (#832): Cilium's conflist
+      # goes to /etc/cni/net.d/multus/, which the container runtime does not
+      # read, and Multus delegates to it via clusterNetwork. Without this a
+      # pod created after a reboot, before Multus rewrote 00-multus.conf, came
+      # up with Cilium alone. See kubernetes/infrastructure/multus.
+      confPath: /etc/cni/net.d/multus
+      # The default `exclusive: true` renames other configs in the conf dir to
+      # *.cilium_bak on every reconcile, which kept nuking multus.
       exclusive: false
+    # One agent at a time: the chart default of 2 restarts two of three
+    # nodes' datapath at once.
+    updateStrategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 1
     # Talos-specific settings
     securityContext:
       capabilities:

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -111,10 +111,18 @@ metadata:
   name: multus
   namespace: kube-system
 ---
+# Multus is the only writer of /etc/cni/net.d (#832). Cilium writes its conflist
+# to /etc/cni/net.d/multus/, a subdirectory the container runtime does not read
+# (libcni skips directories); Multus picks it up there via multusAutoconfigDir
+# and writes 00-multus.conf. Until that file exists the runtime has no network
+# config at all, so no pod can be created with Cilium alone after a reboot.
+#
+# The "-subdir" names date from the node-by-node rollout; renaming would roll
+# Multus again for nothing.
 kind: ConfigMap
 apiVersion: v1
 metadata:
-  name: multus-daemon-config
+  name: multus-daemon-config-subdir
   namespace: kube-system
   labels:
     tier: node
@@ -127,7 +135,7 @@ data:
         "logLevel": "verbose",
         "logToStderr": true,
         "cniConfigDir": "/host/etc/cni/net.d",
-        "multusAutoconfigDir": "/host/etc/cni/net.d",
+        "multusAutoconfigDir": "/host/etc/cni/net.d/multus",
         "multusConfigFile": "auto",
         "socketDir": "/host/run/multus/"
     }
@@ -135,16 +143,16 @@ data:
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
-  name: kube-multus-ds
+  name: kube-multus-ds-subdir
   namespace: kube-system
   labels:
     tier: node
     app: multus
-    name: multus
+    name: multus-subdir
 spec:
   selector:
     matchLabels:
-      name: multus
+      name: multus-subdir
   updateStrategy:
     type: RollingUpdate
   template:
@@ -152,7 +160,7 @@ spec:
       labels:
         tier: node
         app: multus
-        name: multus
+        name: multus-subdir
     spec:
       # Upstream ships this DaemonSet with no priority class, so it runs at 0
       # and can preempt nothing. Any event that makes the pod need a fresh
@@ -224,6 +232,10 @@ spec:
           volumeMounts:
             - name: cni
               mountPath: /host/etc/cni/net.d
+            # Exists before Multus starts, so it waits for Cilium's conflist
+            # instead of exiting on a missing directory.
+            - name: cni-cilium-conf
+              mountPath: /host/etc/cni/net.d/multus
             # multus-daemon expects that cnibin path must be identical between pod and container host.
             # e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
             # not to any other directory, like '/opt/bin' or '/usr/bin'.
@@ -279,6 +291,10 @@ spec:
         - name: cni
           hostPath:
             path: /etc/cni/net.d
+        - name: cni-cilium-conf
+          hostPath:
+            path: /etc/cni/net.d/multus
+            type: DirectoryOrCreate
         - name: cnibin
           hostPath:
             path: /opt/cni/bin
@@ -287,7 +303,7 @@ spec:
             path: /
         - name: multus-daemon-config
           configMap:
-            name: multus-daemon-config
+            name: multus-daemon-config-subdir
             items:
               - key: daemon-config.json
                 path: daemon-config.json

kubernetes/infrastructure/multus/whereabouts-reconciler.yaml

@@ -39,9 +39,11 @@ spec:
         name: whereabouts-reconciler
     spec:
       serviceAccountName: whereabouts
-      # Every node here is a control plane node.
+      # Every node here is a control plane node. Keyed: a keyless Exists would
+      # also tolerate node.kubernetes.io/unschedulable and ignore cordons.
       tolerations:
-        - operator: Exists
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
           effect: NoSchedule
       containers:
         - name: whereabouts-reconciler

kubernetes/talos/patches/cluster.yaml

@@ -58,6 +58,12 @@ machine:
       imageGCHighThresholdPercent: 70
       imageGCLowThresholdPercent: 60
 cluster:
+  # etcd peers on node0, VLAN 46 (#576). Pinned: Talos never updates an
+  # existing member's peer URLs, so changing this subnet means leave + rejoin
+  # per member (talosctl etcd leave, reboot).
+  etcd:
+    advertisedSubnets:
+      - 10.2.46.0/24
   # controlPlane.endpoint is NOT here: `talosctl gen config` takes it as its
   # second argument, and bootstrap.sh already passes https://k8s.renner.uno:6443
   # there. Setting it again in this patch restated the same value in a second

kubernetes/talos/patches/k8s-1.yaml

@@ -17,6 +17,8 @@ machine:
     # pod0 is a closed L2 fabric between the three nodes: statically
     # addressed, no gateway, nothing to learn from an RA.
     net.ipv6.conf.pod0.accept_ra: "0"
+    # node0 is closed the same way.
+    net.ipv6.conf.node0.accept_ra: "0"
     # ingress0 keeps RAs, at the default accept_ra of 2: an IPv6 default
     # router is a link-local address learned only from the RA, and with
     # net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
@@ -122,8 +124,8 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network.
+# Management, VLAN 40: world↔node. The API VIP, admin access to the nodes, and
+# node egress (DNS, NTP, image pulls). Node↔node traffic is on node0 (#576).
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -177,6 +179,19 @@ addresses:
   - address: 10.2.48.11/24
   - address: fd02:48::11/64
 ---
+# Node network, VLAN 46: traffic among the three nodes themselves (#576) — the
+# node IP, etcd, apiserver↔kubelet. Closed like pod0: statically addressed, no
+# gateway, never routed via the UDM.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::11/64
+  - address: 10.2.46.11/24
+---
 # Ingress network, VLAN 47. World-to-pod: the BGP session, the next hop for
 # every advertised LoadBalancer VIP, and the masquerade source for pod egress.
 # Publicly routed, which is what separates it from pod0.

kubernetes/talos/patches/k8s-2.yaml

@@ -16,6 +16,7 @@ machine:
     # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
     # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
@@ -82,7 +83,7 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
+# Management, VLAN 40: world↔node — see k8s-1.yaml.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -122,6 +123,17 @@ addresses:
   - address: 10.2.48.12/24
   - address: fd02:48::12/64
 ---
+# Node network, VLAN 46 — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::12/64
+  - address: 10.2.46.12/24
+---
 # Ingress network, VLAN 47. World-to-pod, publicly routed, statically
 # addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1

kubernetes/talos/patches/k8s-3.yaml

@@ -12,6 +12,7 @@ machine:
     # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
     # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
+    net.ipv6.conf.node0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
     net.ipv4.conf.all.arp_ignore: "1"
@@ -109,6 +110,17 @@ addresses:
   - address: 10.2.48.13/24
   - address: fd02:48::13/64
 ---
+# Node network, VLAN 46 — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: node0
+vlanID: 46
+parent: uplink
+up: true
+addresses:
+  - address: fd02:46::13/64
+  - address: 10.2.46.13/24
+---
 # Ingress network, VLAN 47. World-to-pod, publicly routed, statically
 # addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1

kubernetes/talos/patches/site-homelab.yaml

@@ -7,19 +7,15 @@
 # part of the config that could not be true anywhere but here.
 machine:
   kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
-    # kubelet picks the first non-loopback IP, which could land on the v30iot
-    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
-    # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
+    # The node IP lives on node0, VLAN 46: the closed node↔node network (#576).
+    # Both families are listed so the Node object carries an IPv6 InternalIP at
+    # all; without one Cilium detects the node's IPv6 itself and picks a SLAAC
+    # address from the dynamic provider prefix (#789). Talos picks one address
+    # per family, IPv4 first, whatever the order here.
     nodeIP:
       validSubnets:
-        - 10.2.40.0/24
-        - fd40::/64
+        - fd02:46::/64
+        - 10.2.46.0/24
   # The UDM on the ingress segment. Published here rather than as a route
   # because bird owns table 47 and originates the default from it; this is the
   # one place the address is stated.