Compare commits

..
Author SHA1 Message Date
claude-botandClaude Fable 5.1 7a6cbc4e2e feat(talos): move the node IP and etcd to pod0 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pod east-west has to leave the management VLAN, and the node IP is the
only thing that controls where it goes. Three tests established that,
rather than argument:

  - --ipv4-node / --ipv6-node, applied via CiliumNodeConfig: the agent
    logged receiving both and still published its mgmt0 address.
  - directRoutingDevice=pod0: the agent logged "Direct routing device
    detected direct-routing-device=pod0" and still installed
    10.244.2.0/24 via 10.2.40.138 dev mgmt0.
  - Both routing modes, both address families, always keyed to the
    peer's Kubernetes InternalIP.

etcd is pinned to the same segment. Left unset it advertises "the first
routable address of the node", which is not deterministic once the node
is re-addressed, so it has to be stated either way. pod0 is a closed 10G
L2 between the three nodes; the management VLAN is about to become a
single 1G NIC shared with everything else management does, which is the
wrong link for quorum traffic. listenSubnets defaults to this, so peer
and client listeners move together.

This does move the etcd peer URLs off 10.2.40.x, so unlike the node IP
it is not a no-op: one node at a time, waiting for a healthy quorum
between each.

The API VIP stays on mgmt0; it is a Layer2VIPConfig and does not follow
the node IP.

Apply one node at a time. Each node re-IPs, so per the note in
k8s-1.yaml (#635) kube-apiserver keeps advertising the old address until
its static pod restarts: after each node, check that
`kubectl get endpoints kubernetes -n default` agrees with
`kubectl get nodes -o wide` and reboot the node if it does not. The
cilium agent reads the node IP only at startup, so it needs recreating
on each node as well.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:54:57 +02:00
12 changed files with 207 additions and 114 deletions
+169 -78
View File
@@ -1,103 +1,194 @@
# Proposal — separate the four traffic classes onto their own VLANs
Status: open, issue #576. Two of the four classes are separated. Pod east-west
cannot leave the node IP under Cilium. Both ways of moving it within Cilium
(options A and B below) are rejected by the owner, and replacing the CNI
(option C) is shelved.
Status: draft. The question this started with is **answered** — from the Cilium
source and flag reference, not from a test — and the answer changes the shape
of the proposal rather than confirming it.
## Target
Supersedes the framing in issue #576, which described this as "give node
management its own NIC, and Ceph public its own VLAN". That was two of four
classes and one of them turned out already solved.
Set by the owner on 2026-09-08:
## The four classes
| class | link |
|---|---|
| Ceph replication | X710 p1, VLAN 49 |
| pod-pod | X710 trunk, a VLAN of its own |
| world-pod | X710 trunk, a VLAN of its own |
| node management | 1 Gb onboard NIC |
Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
on the 10G switch.
## Where the classes are
Measured 2026-09-15.
| class | today | state |
| class | what it is | where it goes today |
|---|---|---|
| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
| management | `mgmt0`, VLAN 40 | not started |
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
#833). `pod0` holds addresses and carries no traffic.
Three of four share one interface. That is not a design decision — until the
bare-metal conversion there was one NIC to work with.
## Why pod-pod is blocked
Ceph client I/O living on the pod network is correct rather than a leftover:
that traffic *is* cluster-internal. Ceph is done.
Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
## Principle
- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
fed from `newNode.GetNodeIP(false)`.
- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
`initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
Node object.
- `--direct-routing-device` selects a device for the BPF NodePort path, not a
next-hop.
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
separate, changeable decision.**
Tested on the cluster, none of these moved the next-hop:
Defining classes by VLAN means the separation survives recabling, port failures
and capacity changes, and it does not require one port per class. Defining them
by interface would freeze a capacity decision into the architecture.
| attempt | result |
|---|---|
| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
## The finding that shapes everything else
The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
has no effect on pod-pod routing.
The original open question was whether `nodePort.directRoutingDevice` moves the
next-hop of Cilium's direct routes, or only selects an interface. It is
answerable from the source, and the answer is: **only the interface.**
## Options for pod-pod
`installDirectRoute` takes the next-hop as a parameter:
| | option | status |
```go
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
skipUnreachable bool) (routeSpec *netlink.Route, err error)
```
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
cluster there is no path by which Cilium's node IP differs from the kubelet's
`InternalIP`.
`--direct-routing-device` is documented as "Device name used to connect nodes in
direct routing mode" — a device, not an address.
**So cluster-internal traffic is welded to the node IP, and the node IP is the
kubelet InternalIP.** Two consequences:
- The class cannot be moved off whatever VLAN the node IP lives on.
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
from the node IP; native mode uses it as the direct-route next-hop. Native
routing buys nothing here, so the disruptive, non-rolling step this proposal
previously contained is simply gone.
## Consequence: draw the line the other way
If cluster-internal cannot leave the node IP, then separation has to come from
moving *management services* off it — and each one has to be checked
individually, because they do not share a mechanism.
| management service | can it leave the node IP? | lever |
|---|---|---|
| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
| kubelet | **no** | it *is* the node IP, by definition |
A per-workload Multus attachment detaches a single workload from the node IP,
the way Ceph replication is detached. It does not cover east-west traffic in
So four VLANs are reachable, but the boundary does not fall where the class
names suggest: **kubelet stays with cluster-internal.** That is the honest
result, and it is worth stating plainly rather than filing kubelet under
management and hoping.
The remaining classes keep the levers established earlier:
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
For eBGP the session source is the next-hop, so pinning the session pins
where return traffic arrives. The session to the UDM has been running for
months.
- **cluster-external egress** — eBPF masquerading follows the output device, so
this is a default-route decision plus `enable-masquerade-to-route-source`
(today `false`).
- **Ceph replication** — Multus macvlan on VLAN 49, done.
## The general escape hatch
Ceph is separated because its OSDs got a second interface and stopped using the
pod network for replication. That is the *only* mechanism that fully detaches a
workload from the node IP, and it generalises: anything that must not ride the
node IP gets its own `NetworkAttachmentDefinition`.
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
change. Worth it for Ceph replication. Not worth it for east-west traffic in
general.
## Levers per management service
## Recommended mapping
These hold regardless of which option is chosen:
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
| service | can leave the node IP? | lever |
|---|---|---|
| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
| kubelet | no | it is the node IP |
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
and Talos API. Cleanest match to the class names, but it changes every node's
identity, which is the one address the whole cluster is administered through.
## Remaining, independent of pod-pod
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
Identical end state, and no node ever changes its InternalIP.
- **pod→world egress**: default route and masquerade source onto `ingress0`,
via route metrics, `devices` and `enable-masquerade-to-route-source`. This
moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
removed once ingress on VLAN 47 is settled.
Under B:
## Sources
| VLAN | carries |
|---|---|
| 49 | Ceph replication |
| 40 (relabelled *cluster*) | pod east-west, kubelet |
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
| new *external* | LoadBalancer ingress, egress default route |
- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
`--direct-routing-device`.
- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
`CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
The property the user asked for — management carries only management traffic —
holds under B. The label moves; the property is what matters.
## Migration shape
Every step is rolling and reversible. The routing-mode change is gone.
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
Still unused; this only proves the L2 works.
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
waiting for a healthy quorum between each.
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
`k8s.renner.uno` follows.
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
Reversible by reverting the override; failure is visible immediately in
`cilium-dbg shell -- bgp/peers`.
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
Step 3 is the one that deserves care — etcd is the only step where a mistake
costs quorum rather than connectivity.
## What this does not buy
Management currently shares a 10 Gb link and is not contended. Nothing observed
today is slow because of this. The case for the split is that traffic classes
with different failure modes and security postures should not be
indistinguishable — not that something is broken.
## Sources and measurements
Source, Cilium v1.20.1 (the running version):
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
a next-hop that is not directly reachable.
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
Kubernetes Node.
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
to connect nodes in direct routing mode".
Talos:
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
defaults to it. kube-apiserver and Talos API listen addresses are not
covered there.
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
`10.2.40.60` with `link: mgmt0`.
Measured on the cluster:
- `apid` answered on a node address that is not the InternalIP, so the Talos
API is not bound to one address.
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
InternalIP had changed since the last session, which is why nothing here
names an address as fixed.
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
unset and no direct-routing device configured.
Not verified: whether the kube-apiserver's `--advertise-address` follows the
node IP independently of the VIP. It does not affect the recommendation, since
clients reach the API through the VIP either way.
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.9.2 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
+1 -1
View File
@@ -32,7 +32,7 @@ spec:
type: RuntimeDefault
containers:
- name: immich-machine-learning
image: ghcr.io/immich-app/immich-machine-learning:v3.2.1 # {"$imagepolicy": "flux-system:immich-machine-learning"}
image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
ports:
- name: http
containerPort: 3003
+1 -1
View File
@@ -87,7 +87,7 @@ spec:
mountPath: /ca
containers:
- name: immich-server
image: ghcr.io/immich-app/immich-server:v3.2.1 # {"$imagepolicy": "flux-system:immich-server"}
image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
ports:
- name: http
containerPort: 3001
@@ -55,7 +55,7 @@ spec:
# stdlib only; Flux keeps the tag in lockstep via the shared image
# policy) — no extra dependency.
- name: reconcile-publish-ip
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
volumeMounts:
- name: data
@@ -65,7 +65,7 @@ spec:
readOnly: true
containers:
- name: music-assistant
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
env:
- name: LOG_LEVEL
value: info
+1 -1
View File
@@ -22,7 +22,7 @@ spec:
type: RuntimeDefault
containers:
- name: gotenberg
image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
args:
- "gotenberg"
- "--chromium-disable-javascript=true"
+2 -2
View File
@@ -134,7 +134,7 @@ spec:
# is a device-bricking class of risk. Bumps are a reviewed, manual
# act — see kubernetes/infrastructure/image-automation/README.md.
- name: terminus
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
ports:
- name: http
containerPort: 2345
@@ -197,7 +197,7 @@ spec:
capabilities:
drop: ["ALL"]
- name: sidekiq
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
envFrom:
- configMapRef:
+1 -1
View File
@@ -18,7 +18,7 @@ spec:
spec:
containers:
- name: whisper
image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
args:
- --model=base-int8
- --language=de
+1 -1
View File
@@ -34,7 +34,7 @@ spec:
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
# track the same range/filter so they bump together on each release.
image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
env:
- name: WOODPECKER_SERVER
value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+1 -1
View File
@@ -40,7 +40,7 @@ spec:
# minor + patch auto-update within 3.x; only the 4.0 major is held for
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
# -bump hazard is gone). Stays in lockstep with the agent.
image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
env:
- name: WOODPECKER_HOST
value: https://ci.renner.uno
@@ -180,17 +180,3 @@ spec:
# the supernets here cover the LAN as well, which is the point.
ipv4NativeRoutingCIDR: 10.0.0.0/8
ipv6NativeRoutingCIDR: fd00::/8
# Pod east-west leaves the node on pod0 (VLAN 48) rather than the
# management VLAN. Native routing hands forwarding to the kernel;
# autoDirectNodeRoutes installs the remote PodCIDR routes, which is
# allowed because all three nodes share pod0's L2 segment.
#
# directRoutingDevice is explicit on purpose: with several addressed
# devices Cilium falls back to the one carrying the Kubernetes InternalIP,
# which is mgmt0.
#
# Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
routingMode: native
autoDirectNodeRoutes: true
nodePort:
directRoutingDevice: pod0
+27 -11
View File
@@ -7,16 +7,32 @@
# part of the config that could not be true anywhere but here.
machine:
kubelet:
# Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
# kubelet picks the first non-loopback IP, which could land on the v30iot
# NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
# must stay on mgmt LAN.
# fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
# an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
# node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
# provider prefix, which it then probes until the agent restarts (#789).
# The GUA is not a control-plane address: BGP peers on fd40::/64 too.
# Pins the node IP to pod0 (VLAN 48), which is what carries pod east-west.
# Cilium reaches a peer's pods at that peer's Kubernetes InternalIP and
# offers no way to point it elsewhere: --ipv4-node/--ipv6-node are accepted
# and then overwritten from the Node object, and directRoutingDevice only
# selects the NodePort/DSR device. So the node IP is the only control over
# which segment east-west uses, in both tunnel and native routing mode.
#
# Unpinned, kubelet takes the first non-loopback address, which could land
# on the v30iot NIC (ens19, 10.2.30.0/24).
#
# fd02:48::/64 is listed so the Node object carries an IPv6 InternalIP at
# all. Without one, Cilium detects the node's IPv6 itself and picks the
# SLAAC address from the dynamic provider prefix, then probes it until the
# agent restarts (#789).
nodeIP:
validSubnets:
- 10.2.40.0/24
- fd40::/64
- 10.2.48.0/24
- fd02:48::/64
cluster:
etcd:
# Pins etcd to pod0 with the node IP. Unset, etcd advertises "the first
# routable address of the node", which is not deterministic across a
# re-address. listenSubnets defaults to this.
#
# pod0 is a closed 10G L2 between the three nodes. The management VLAN is
# about to become a single 1G NIC shared with everything else management
# does, which is the wrong link for quorum traffic.
advertisedSubnets:
- 10.2.48.0/24