|
|
|
@@ -1,99 +1,117 @@
|
|
|
|
|
# Traffic classes on the k8s nodes
|
|
|
|
|
|
|
|
|
|
Issue #576. Five classes, each on its own VLAN.
|
|
|
|
|
Issue #576. Three of the four classes are separated; management is not.
|
|
|
|
|
|
|
|
|
|
## Where each class runs
|
|
|
|
|
|
|
|
|
|
| class | link | mechanism |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
|
|
|
|
|
| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
|
|
|
|
|
| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
|
|
|
|
|
| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
|
|
|
|
|
| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
|
|
|
|
|
| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
|
|
|
|
|
| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
|
|
|
|
|
| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
|
|
|
|
|
| management | `mgmt0`, VLAN 40 | not started |
|
|
|
|
|
|
|
|
|
|
Hard constraint, set by the owner: inter-node and pod traffic never traverses
|
|
|
|
|
the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
|
|
|
|
|
statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
|
|
|
|
|
the UDM. Everything above stays on the 10G switch.
|
|
|
|
|
|
|
|
|
|
## Pod traffic: one table
|
|
|
|
|
## Why policy routing rather than moving the node IP
|
|
|
|
|
|
|
|
|
|
Two rules, one per family, send every packet with a pod source to table 47
|
|
|
|
|
(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
|
|
|
|
|
table decides by longest prefix; no rule carves out exceptions.
|
|
|
|
|
Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
|
|
|
|
|
and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
|
|
|
|
|
`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
|
|
|
|
|
Both were established by test (#837, #838), not by reading.
|
|
|
|
|
|
|
|
|
|
bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
|
|
|
|
|
So the InternalIP decides where pod traffic goes — unless the lookup happens in
|
|
|
|
|
a different table. The rules do that, and nothing about the node's identity
|
|
|
|
|
moves. The routing decision runs before Cilium's masquerade hook, so a rule
|
|
|
|
|
matching the pod source sees the pod address, and the masquerade then rewrites
|
|
|
|
|
it to the egress link's.
|
|
|
|
|
|
|
|
|
|
- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
|
|
|
|
|
taken from the interfaces;
|
|
|
|
|
- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
|
|
|
|
|
peer's `pod0` address;
|
|
|
|
|
- the node's own pod CIDR via `cilium_host`, originated rather than read back
|
|
|
|
|
from the kernel (reading it back made BIRD depend on the shape of Cilium's
|
|
|
|
|
route, and KRT rejects the IPv4 one);
|
|
|
|
|
- the egress default via the UDM on `ingress0`, from the node annotations
|
|
|
|
|
`net.renner.uno/egress-gw-v4/v6`.
|
|
|
|
|
## Rule order
|
|
|
|
|
|
|
|
|
|
`node0` has to be in the table: a pod reaching a node address — the
|
|
|
|
|
`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
|
|
|
|
|
address, or a hostNetwork pod — would otherwise take the default to the UDM.
|
|
|
|
|
Lower priority runs first.
|
|
|
|
|
|
|
|
|
|
The routing decision runs before Cilium's masquerade hook, so the rule sees the
|
|
|
|
|
pod address and the masquerade then rewrites it to the egress link's.
|
|
|
|
|
| prio | match | table |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| 980 | src pod, dst pod | 48 |
|
|
|
|
|
| 981 | src pod, dst pod (v4) | 48 |
|
|
|
|
|
| 990 | dst `fd00::/8` | main |
|
|
|
|
|
| 991 | dst `10.0.0.0/8` | main |
|
|
|
|
|
| 1000 | src pod | 47 |
|
|
|
|
|
| 1001 | src pod (v4) | 47 |
|
|
|
|
|
|
|
|
|
|
Node-originated traffic does not match the rules and uses main, where Cilium
|
|
|
|
|
routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
|
|
|
|
|
980/981 match source **and** destination. Source alone would also catch the
|
|
|
|
|
node's own traffic to a pod, which would then leave with a `pod0` source — an
|
|
|
|
|
address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
|
|
|
|
|
`world` rather than `host`.
|
|
|
|
|
|
|
|
|
|
990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
|
|
|
|
|
cluster-internal traffic out of table 47. Without them a pod's packet to a
|
|
|
|
|
service matches the source rule, finds only a default route, and leaves for the
|
|
|
|
|
UDM — measured, in-cluster DNS stopped resolving.
|
|
|
|
|
|
|
|
|
|
Traffic to a pod on the same node falls through on its own: table 48 holds only
|
|
|
|
|
the peers' CIDRs, so the lookup misses and 990/991 send it to main.
|
|
|
|
|
|
|
|
|
|
## Where each table comes from
|
|
|
|
|
|
|
|
|
|
- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
|
|
|
|
|
The IPv6 gateway is the UDM's configured address on the segment, not the
|
|
|
|
|
RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
|
|
|
|
|
default does not land in main.
|
|
|
|
|
- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
|
|
|
|
|
per peer pod CIDR with the peer's `pod0` address as next hop.
|
|
|
|
|
|
|
|
|
|
## The bird mesh
|
|
|
|
|
|
|
|
|
|
`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
|
|
|
|
|
and installs what it learns into table 48. Cilium's routes in main are
|
|
|
|
|
untouched, which is why the two coexist.
|
|
|
|
|
|
|
|
|
|
No address is written into this repo twice. Each node publishes its `pod0`
|
|
|
|
|
addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
|
|
|
|
|
pod CIDRs come from the Node objects. bird's init container renders both from
|
|
|
|
|
the API and ends with `bird -p`, so BIRD's own parser gates startup.
|
|
|
|
|
addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
|
|
|
|
|
pod CIDRs come from the Node objects. An init container renders both from the
|
|
|
|
|
API with `kubectl -o go-template-file`, excludes the node itself with a field
|
|
|
|
|
selector, and ends with `bird -p` so BIRD's own parser gates startup.
|
|
|
|
|
|
|
|
|
|
## Node↔node
|
|
|
|
|
|
|
|
|
|
`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
|
|
|
|
|
VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
|
|
|
|
|
is IPv4, and the apiserver's advertise address must match it.
|
|
|
|
|
|
|
|
|
|
The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
|
|
|
|
|
kubelet, scheduler, controller-manager and Cilium go through KubePrism
|
|
|
|
|
(`localhost:7445`), which balances over every address of every control-plane
|
|
|
|
|
member; pods use the ClusterIP.
|
|
|
|
|
|
|
|
|
|
Talos never updates an existing etcd member's peer URLs
|
|
|
|
|
(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
|
|
|
|
|
stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
|
|
|
|
|
reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
|
|
|
|
|
and is promoted. EPHEMERAL does not need a wipe.
|
|
|
|
|
|
|
|
|
|
## Moving node IPs: the mixed state
|
|
|
|
|
|
|
|
|
|
While some nodes have their InternalIP on the new network and some on the old,
|
|
|
|
|
host→remote-pod traffic leaves with a node address that is not an InternalIP:
|
|
|
|
|
the kernel picks the source from the route to the peer's InternalIP. Cilium
|
|
|
|
|
classifies that as `world`, and every ingress policy that admits only
|
|
|
|
|
`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
|
|
|
|
|
lose the OSDs on the other side; RBD mounts hang.
|
|
|
|
|
|
|
|
|
|
Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
|
|
|
|
|
admitting the node subnets of both networks, both families, with
|
|
|
|
|
`enableDefaultDeny: false` so it restricts nothing else, and delete it once
|
|
|
|
|
every node has moved. Keep all OSD nodes schedulable while one is drained:
|
|
|
|
|
draining two takes Ceph below its data availability.
|
|
|
|
|
The node's own CIDR is **originated**, not read back from the kernel. Reading
|
|
|
|
|
it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
|
|
|
|
|
rejected outright: its next hop sits inside the prefix it describes, which KRT
|
|
|
|
|
refuses with `strange next-hop`. The IPv6 route is a device route and passed,
|
|
|
|
|
so the failure was silent on one family only.
|
|
|
|
|
|
|
|
|
|
## Failure behaviour
|
|
|
|
|
|
|
|
|
|
If bird stops, table 47 empties, the rules find nothing and fall through to
|
|
|
|
|
main: pod traffic returns to the node networks, unseparated rather than
|
|
|
|
|
misrouted. The same holds during a cold rebuild until Gitea serves bird's
|
|
|
|
|
image. The default route lives in table 47, not in the Talos config, so a
|
|
|
|
|
default cannot outlive the specific routes and send pod traffic to the UDM.
|
|
|
|
|
Benign at every layer. An empty or absent table 48 makes 980/981 fall through
|
|
|
|
|
to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
|
|
|
|
|
holds if the DaemonSet cannot start, which is the case during a cold rebuild
|
|
|
|
|
until Gitea serves its image.
|
|
|
|
|
|
|
|
|
|
## Management, the remaining class
|
|
|
|
|
|
|
|
|
|
Not started. The levers differ per service and none of them is the node IP:
|
|
|
|
|
|
|
|
|
|
| service | can leave the node IP? | lever |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
|
|
|
|
|
| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
|
|
|
|
|
| Talos API | n/a | `apid` answers on every node address; a firewall question |
|
|
|
|
|
| kubelet | no | it *is* the node IP |
|
|
|
|
|
|
|
|
|
|
Two approaches were rejected by the owner and are not to be proposed again:
|
|
|
|
|
moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
|
|
|
|
|
cluster VLAN while moving etcd and the API VIP to a new management VLAN.
|
|
|
|
|
Replacing the CNI (#839, Antrea) was shelved.
|
|
|
|
|
|
|
|
|
|
Note that the reasoning which produced that list also concluded pod↔pod was
|
|
|
|
|
blocked, which policy routing disproved. Treat it as a record of decisions, not
|
|
|
|
|
as an exhaustive analysis.
|
|
|
|
|
|
|
|
|
|
## Sources
|
|
|
|
|
|
|
|
|
|
Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
|
|
|
|
|
protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
|
|
|
|
|
`machine.nodeAnnotations`, configuration patching (lists are appended,
|
|
|
|
|
`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
|
|
|
|
|
siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
|
|
|
|
|
#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.
|
|
|
|
|
Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
|
|
|
|
|
`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
|
|
|
|
|
`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
|
|
|
|
|
`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
|
|
|
|
|
#855; closed unmerged: #830, #838, #839.
|
|
|
|
|
issue-fix-bot