Compare commits
22
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
774907eaf1 | ||
 security-bot
|
717c5a5f8a | ||
|
security-bot
|
4fd3c21308 | ||
|
security-bot
|
5e79157e5e | ||
|
security-bot
|
4c36913fee | ||
|
security-bot
|
605690f9ff | ||
|
security-bot
|
fe6fa9ec4c | ||
|
security-bot
|
c954043f90 | ||
|
security-bot
|
31be9a115a | ||
|
security-bot
|
deda43ab85 | ||
|
security-bot
|
19a464b2bf | ||
|
security-bot
|
b29174dff2 | ||
|
security-bot
|
451e5002e9 | ||
 k.renner
|
d443f435aa | ||
|
claude-botandClaude Fable 5.1
|
63a3aebe84 | ||
|
k.renner
|
ee2d5d1d25 | ||
|
claude-botandClaude Fable 5.1
|
60b4286910 | ||
|
k.renner
|
cc52181352 | ||
|
claude-bot
|
56d55b6228 | ||
|
k.renner
|
6416f2f609 | ||
|
claude-bot
|
80c2def518 | ||
|
k.renner
|
0ca1753418 |
@@ -1,194 +1,103 @@
|
||||
# Proposal — separate the four traffic classes onto their own VLANs
|
||||
|
||||
Status: draft. The question this started with is **answered** — from the Cilium
|
||||
source and flag reference, not from a test — and the answer changes the shape
|
||||
of the proposal rather than confirming it.
|
||||
Status: open, issue #576. Two of the four classes are separated. Pod east-west
|
||||
cannot leave the node IP under Cilium. Both ways of moving it within Cilium
|
||||
(options A and B below) are rejected by the owner, and replacing the CNI
|
||||
(option C) is shelved.
|
||||
|
||||
Supersedes the framing in issue #576, which described this as "give node
|
||||
management its own NIC, and Ceph public its own VLAN". That was two of four
|
||||
classes and one of them turned out already solved.
|
||||
## Target
|
||||
|
||||
## The four classes
|
||||
Set by the owner on 2026-09-08:
|
||||
|
||||
| class | what it is | where it goes today |
|
||||
| class | link |
|
||||
|---|---|
|
||||
| Ceph replication | X710 p1, VLAN 49 |
|
||||
| pod-pod | X710 trunk, a VLAN of its own |
|
||||
| world-pod | X710 trunk, a VLAN of its own |
|
||||
| node management | 1 Gb onboard NIC |
|
||||
|
||||
Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
|
||||
on the 10G switch.
|
||||
|
||||
## Where the classes are
|
||||
|
||||
Measured 2026-09-15.
|
||||
|
||||
| class | today | state |
|
||||
|---|---|---|
|
||||
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
||||
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
||||
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
||||
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
||||
| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
|
||||
| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
|
||||
| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
|
||||
| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
|
||||
| management | `mgmt0`, VLAN 40 | not started |
|
||||
|
||||
Three of four share one interface. That is not a design decision — until the
|
||||
bare-metal conversion there was one NIC to work with.
|
||||
`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
|
||||
#833). `pod0` holds addresses and carries no traffic.
|
||||
|
||||
Ceph client I/O living on the pod network is correct rather than a leftover:
|
||||
that traffic *is* cluster-internal. Ceph is done.
|
||||
## Why pod-pod is blocked
|
||||
|
||||
## Principle
|
||||
Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
|
||||
next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
|
||||
|
||||
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
||||
separate, changeable decision.**
|
||||
- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
|
||||
fed from `newNode.GetNodeIP(false)`.
|
||||
- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
|
||||
`initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
|
||||
Node object.
|
||||
- `--direct-routing-device` selects a device for the BPF NodePort path, not a
|
||||
next-hop.
|
||||
|
||||
Defining classes by VLAN means the separation survives recabling, port failures
|
||||
and capacity changes, and it does not require one port per class. Defining them
|
||||
by interface would freeze a capacity decision into the architecture.
|
||||
Tested on the cluster, none of these moved the next-hop:
|
||||
|
||||
## The finding that shapes everything else
|
||||
| attempt | result |
|
||||
|---|---|
|
||||
| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
|
||||
| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
|
||||
| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
|
||||
|
||||
The original open question was whether `nodePort.directRoutingDevice` moves the
|
||||
next-hop of Cilium's direct routes, or only selects an interface. It is
|
||||
answerable from the source, and the answer is: **only the interface.**
|
||||
The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
|
||||
`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
|
||||
Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
|
||||
cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
|
||||
has no effect on pod-pod routing.
|
||||
|
||||
`installDirectRoute` takes the next-hop as a parameter:
|
||||
## Options for pod-pod
|
||||
|
||||
```go
|
||||
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
||||
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
||||
```
|
||||
|
||||
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
||||
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
||||
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
||||
|
||||
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
||||
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
||||
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
||||
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
||||
`InternalIP`.
|
||||
|
||||
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
||||
direct routing mode" — a device, not an address.
|
||||
|
||||
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
||||
kubelet InternalIP.** Two consequences:
|
||||
|
||||
- The class cannot be moved off whatever VLAN the node IP lives on.
|
||||
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
||||
from the node IP; native mode uses it as the direct-route next-hop. Native
|
||||
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
||||
previously contained is simply gone.
|
||||
|
||||
## Consequence: draw the line the other way
|
||||
|
||||
If cluster-internal cannot leave the node IP, then separation has to come from
|
||||
moving *management services* off it — and each one has to be checked
|
||||
individually, because they do not share a mechanism.
|
||||
|
||||
| management service | can it leave the node IP? | lever |
|
||||
| | option | status |
|
||||
|---|---|---|
|
||||
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
||||
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
||||
| kubelet | **no** | it *is* the node IP, by definition |
|
||||
| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
|
||||
| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
|
||||
| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
|
||||
|
||||
So four VLANs are reachable, but the boundary does not fall where the class
|
||||
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
||||
result, and it is worth stating plainly rather than filing kubelet under
|
||||
management and hoping.
|
||||
|
||||
The remaining classes keep the levers established earlier:
|
||||
|
||||
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
||||
For eBGP the session source is the next-hop, so pinning the session pins
|
||||
where return traffic arrives. The session to the UDM has been running for
|
||||
months.
|
||||
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
||||
this is a default-route decision plus `enable-masquerade-to-route-source`
|
||||
(today `false`).
|
||||
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
||||
|
||||
## The general escape hatch
|
||||
|
||||
Ceph is separated because its OSDs got a second interface and stopped using the
|
||||
pod network for replication. That is the *only* mechanism that fully detaches a
|
||||
workload from the node IP, and it generalises: anything that must not ride the
|
||||
node IP gets its own `NetworkAttachmentDefinition`.
|
||||
|
||||
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
||||
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
||||
A per-workload Multus attachment detaches a single workload from the node IP,
|
||||
the way Ceph replication is detached. It does not cover east-west traffic in
|
||||
general.
|
||||
|
||||
## Recommended mapping
|
||||
## Levers per management service
|
||||
|
||||
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
||||
These hold regardless of which option is chosen:
|
||||
|
||||
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
||||
and Talos API. Cleanest match to the class names, but it changes every node's
|
||||
identity, which is the one address the whole cluster is administered through.
|
||||
| service | can leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
|
||||
| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
|
||||
| kubelet | no | it is the node IP |
|
||||
|
||||
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
||||
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
||||
Identical end state, and no node ever changes its InternalIP.
|
||||
## Remaining, independent of pod-pod
|
||||
|
||||
Under B:
|
||||
- **pod→world egress**: default route and masquerade source onto `ingress0`,
|
||||
via route metrics, `devices` and `enable-masquerade-to-route-source`. This
|
||||
moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
|
||||
- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
|
||||
removed once ingress on VLAN 47 is settled.
|
||||
|
||||
| VLAN | carries |
|
||||
|---|---|
|
||||
| 49 | Ceph replication |
|
||||
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
||||
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
||||
| new *external* | LoadBalancer ingress, egress default route |
|
||||
## Sources
|
||||
|
||||
The property the user asked for — management carries only management traffic —
|
||||
holds under B. The label moves; the property is what matters.
|
||||
|
||||
## Migration shape
|
||||
|
||||
Every step is rolling and reversible. The routing-mode change is gone.
|
||||
|
||||
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
||||
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
||||
Still unused; this only proves the L2 works.
|
||||
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
||||
waiting for a healthy quorum between each.
|
||||
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
||||
`k8s.renner.uno` follows.
|
||||
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
||||
Reversible by reverting the override; failure is visible immediately in
|
||||
`cilium-dbg shell -- bgp/peers`.
|
||||
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
||||
|
||||
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
||||
costs quorum rather than connectivity.
|
||||
|
||||
## What this does not buy
|
||||
|
||||
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
||||
today is slow because of this. The case for the split is that traffic classes
|
||||
with different failure modes and security postures should not be
|
||||
indistinguishable — not that something is broken.
|
||||
|
||||
## Sources and measurements
|
||||
|
||||
Source, Cilium v1.20.1 (the running version):
|
||||
|
||||
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
||||
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
||||
a next-hop that is not directly reachable.
|
||||
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
||||
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
||||
Kubernetes Node.
|
||||
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
||||
to connect nodes in direct routing mode".
|
||||
|
||||
Talos:
|
||||
|
||||
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
||||
defaults to it. kube-apiserver and Talos API listen addresses are not
|
||||
covered there.
|
||||
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
||||
`10.2.40.60` with `link: mgmt0`.
|
||||
|
||||
Measured on the cluster:
|
||||
|
||||
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
||||
API is not bound to one address.
|
||||
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
||||
InternalIP had changed since the last session, which is why nothing here
|
||||
names an address as fixed.
|
||||
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
||||
unset and no direct-routing device configured.
|
||||
|
||||
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
||||
node IP independently of the VIP. It does not affect the recommendation, since
|
||||
clients reach the API through the VIP either way.
|
||||
- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
|
||||
`--direct-routing-device`.
|
||||
- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
|
||||
- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
|
||||
- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
|
||||
`CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
|
||||
|
||||
@@ -95,7 +95,7 @@ spec:
|
||||
memory: 32Mi
|
||||
containers:
|
||||
- name: home-assistant
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.2 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
env:
|
||||
- name: TZ
|
||||
value: Europe/Berlin
|
||||
|
||||
@@ -32,7 +32,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: immich-machine-learning
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.2.1 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3003
|
||||
|
||||
@@ -87,7 +87,7 @@ spec:
|
||||
mountPath: /ca
|
||||
containers:
|
||||
- name: immich-server
|
||||
image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
image: ghcr.io/immich-app/immich-server:v3.2.1 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3001
|
||||
|
||||
@@ -55,7 +55,7 @@ spec:
|
||||
# stdlib only; Flux keeps the tag in lockstep via the shared image
|
||||
# policy) — no extra dependency.
|
||||
- name: reconcile-publish-ip
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
@@ -65,7 +65,7 @@ spec:
|
||||
readOnly: true
|
||||
containers:
|
||||
- name: music-assistant
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
env:
|
||||
- name: LOG_LEVEL
|
||||
value: info
|
||||
|
||||
@@ -22,7 +22,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gotenberg
|
||||
image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
args:
|
||||
- "gotenberg"
|
||||
- "--chromium-disable-javascript=true"
|
||||
|
||||
@@ -134,7 +134,7 @@ spec:
|
||||
# is a device-bricking class of risk. Bumps are a reviewed, manual
|
||||
# act — see kubernetes/infrastructure/image-automation/README.md.
|
||||
- name: terminus
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 2345
|
||||
@@ -197,7 +197,7 @@ spec:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
- name: sidekiq
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
|
||||
@@ -18,7 +18,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: whisper
|
||||
image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
args:
|
||||
- --model=base-int8
|
||||
- --language=de
|
||||
|
||||
@@ -34,7 +34,7 @@ spec:
|
||||
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
|
||||
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
|
||||
# track the same range/filter so they bump together on each release.
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
env:
|
||||
- name: WOODPECKER_SERVER
|
||||
value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
|
||||
|
||||
@@ -40,7 +40,7 @@ spec:
|
||||
# minor + patch auto-update within 3.x; only the 4.0 major is held for
|
||||
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
|
||||
# -bump hazard is gone). Stays in lockstep with the agent.
|
||||
image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
env:
|
||||
- name: WOODPECKER_HOST
|
||||
value: https://ci.renner.uno
|
||||
|
||||
@@ -180,3 +180,17 @@ spec:
|
||||
# the supernets here cover the LAN as well, which is the point.
|
||||
ipv4NativeRoutingCIDR: 10.0.0.0/8
|
||||
ipv6NativeRoutingCIDR: fd00::/8
|
||||
# Pod east-west leaves the node on pod0 (VLAN 48) rather than the
|
||||
# management VLAN. Native routing hands forwarding to the kernel;
|
||||
# autoDirectNodeRoutes installs the remote PodCIDR routes, which is
|
||||
# allowed because all three nodes share pod0's L2 segment.
|
||||
#
|
||||
# directRoutingDevice is explicit on purpose: with several addressed
|
||||
# devices Cilium falls back to the one carrying the Kubernetes InternalIP,
|
||||
# which is mgmt0.
|
||||
#
|
||||
# Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
|
||||
routingMode: native
|
||||
autoDirectNodeRoutes: true
|
||||
nodePort:
|
||||
directRoutingDevice: pod0
|
||||
|
||||
@@ -25,13 +25,13 @@ spec:
|
||||
stop: "fd99::ff"
|
||||
---
|
||||
# Cilium BGPv2 control-plane config peering each node with the UDM over the
|
||||
# management LAN — never over v30iot, which is the untrusted IoT segment.
|
||||
# ingress VLAN 47 — never over v30iot, which is the untrusted IoT segment.
|
||||
# The source address comes from the kernel routing decision, so no explicit
|
||||
# localAddress override is needed.
|
||||
#
|
||||
# Two parallel sessions, one per address family:
|
||||
# - v4 session: peer 10.2.40.1, family IPv4 unicast only
|
||||
# - v6 session: peer fd40::1, family IPv6 unicast only
|
||||
# - v4 session: peer 10.2.47.1, family IPv4 unicast only
|
||||
# - v6 session: peer fd02:47::1, family IPv6 unicast only
|
||||
# This avoids the "v6 NLRIs over v4 transport" path, which requires
|
||||
# bilateral extended-nexthop capability negotiation (RFC 5549/8950).
|
||||
# Cilium 1.19 BGPv2 advertises the capability only for the IPv4 Unicast
|
||||
@@ -74,14 +74,14 @@ spec:
|
||||
peers:
|
||||
- name: udm-v6
|
||||
peerASN: 64512
|
||||
# ULA — stable across ISP renumber events. UDM's br40 has
|
||||
# fd40::1/64; nodes get fd40::<eui-64> via SLAAC on eth0.
|
||||
peerAddress: fd40::1
|
||||
# ULA — stable across ISP renumber events. UDM's br47 has
|
||||
# fd02:47::1/64; nodes get fd02:47::<eui-64> via SLAAC on ingress0.
|
||||
peerAddress: fd02:47::1
|
||||
peerConfigRef:
|
||||
name: bgp-peer-udm-v6
|
||||
- name: udm-v4
|
||||
peerASN: 64512
|
||||
peerAddress: 10.2.40.1
|
||||
peerAddress: 10.2.47.1
|
||||
peerConfigRef:
|
||||
name: bgp-peer-udm-v4
|
||||
---
|
||||
|
||||
@@ -17,9 +17,19 @@ machine:
|
||||
# pod0 is a closed L2 fabric between the three nodes: statically
|
||||
# addressed, no gateway, nothing to learn from an RA.
|
||||
net.ipv6.conf.pod0.accept_ra: "0"
|
||||
# ingress0 takes its prefix from the RA but not the default route — that
|
||||
# stays on mgmt0 until the split flips it, and two default routes at the
|
||||
# same metric tie.
|
||||
# ingress0 keeps RAs, at the default accept_ra of 2: an IPv6 default
|
||||
# router is a link-local address learned only from the RA, and with
|
||||
# net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
|
||||
# entirely below 2. That is how mgmt0 has its IPv6 default today.
|
||||
#
|
||||
# The prefix is announced correctly, so SLAAC addresses this link the same
|
||||
# way mgmt0 gets its fd40:: address: derived from the MAC, stable across
|
||||
# reboots, nothing to allocate. Only IPv4 is declared here, and only
|
||||
# because a second DHCP client is what broke the node's DNS record.
|
||||
#
|
||||
# Just the default route is suppressed, because mgmt0 owns it until the
|
||||
# split flips it and two defaults at one metric tie. Dropping this line is
|
||||
# what hands the IPv6 default over.
|
||||
net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
|
||||
# That network also carries the mgmt VLAN's ARP. Answer ARP only on the
|
||||
# link that owns the address, or the UDM learns the node IP against
|
||||
@@ -164,21 +174,27 @@ addresses:
|
||||
# Ingress network, VLAN 47. World-to-pod: the BGP session, the next hop for
|
||||
# every advertised LoadBalancer VIP, and the masquerade source for pod egress.
|
||||
# Publicly routed, which is what separates it from pod0.
|
||||
#
|
||||
# Statically addressed, not leased. A second DHCP client on the node announced
|
||||
# the same hostname as mgmt0's and the UDM registered the node against this
|
||||
# segment instead, so mgmt0's `srv.renner.uno` record disappeared and the
|
||||
# lease's resolver displaced the management one. The UDM cannot register a
|
||||
# network's clients selectively, so the client goes rather than the record.
|
||||
# DHCP is off on VLAN 47 for the same reason.
|
||||
#
|
||||
# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
|
||||
# the segment. The default route moves here in the change that flips it.
|
||||
apiVersion: v1alpha1
|
||||
kind: VLANConfig
|
||||
name: ingress0
|
||||
vlanID: 47
|
||||
parent: uplink
|
||||
up: true
|
||||
---
|
||||
# Metric loses to mgmt0's default 1024 deliberately: this pass only adds the
|
||||
# segment. The default route moves here in the change that flips it, which
|
||||
# needs no reboot and is provable with `--mode=try`. DHCPv4Config cannot
|
||||
# suppress a default route, only deprioritise it.
|
||||
apiVersion: v1alpha1
|
||||
kind: DHCPv4Config
|
||||
name: ingress0
|
||||
routeMetric: 2048
|
||||
addresses:
|
||||
- address: 10.2.47.11/24
|
||||
routes:
|
||||
- gateway: 10.2.47.1
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
|
||||
# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
|
||||
|
||||
@@ -12,8 +12,9 @@ machine:
|
||||
# cluster IPv6 egress.
|
||||
net.ipv6.conf.uplink.disable_ipv6: "1"
|
||||
net.ipv6.conf.mgmt.disable_ipv6: "1"
|
||||
# pod0 is statically addressed with no gateway; ingress0 takes its prefix
|
||||
# but not the RA default route — see k8s-1.yaml.
|
||||
# pod0 is a closed segment with no router, so it ignores RAs outright.
|
||||
# ingress0 keeps RAs: it learns its IPv6 router and its address from them,
|
||||
# and only the default route is held back — see k8s-1.yaml.
|
||||
net.ipv6.conf.pod0.accept_ra: "0"
|
||||
net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
|
||||
# Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
|
||||
@@ -117,18 +118,19 @@ addresses:
|
||||
- address: 10.2.48.12/24
|
||||
- address: fd02:48::12/64
|
||||
---
|
||||
# Ingress network, VLAN 47. World-to-pod, publicly routed — see k8s-1.yaml.
|
||||
# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
|
||||
# addressed — see k8s-1.yaml for why it is not leased.
|
||||
apiVersion: v1alpha1
|
||||
kind: VLANConfig
|
||||
name: ingress0
|
||||
vlanID: 47
|
||||
parent: uplink
|
||||
up: true
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: DHCPv4Config
|
||||
name: ingress0
|
||||
routeMetric: 2048
|
||||
addresses:
|
||||
- address: 10.2.47.12/24
|
||||
routes:
|
||||
- gateway: 10.2.47.1
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
|
||||
# wire. OSD-to-OSD replication only, not routed off its own segment.
|
||||
|
||||
@@ -8,8 +8,9 @@ machine:
|
||||
# k8s-1.yaml.
|
||||
net.ipv6.conf.uplink.disable_ipv6: "1"
|
||||
net.ipv6.conf.mgmt.disable_ipv6: "1"
|
||||
# pod0 is statically addressed with no gateway; ingress0 takes its prefix
|
||||
# but not the RA default route — see k8s-1.yaml.
|
||||
# pod0 is a closed segment with no router, so it ignores RAs outright.
|
||||
# ingress0 keeps RAs: it learns its IPv6 router and its address from them,
|
||||
# and only the default route is held back — see k8s-1.yaml.
|
||||
net.ipv6.conf.pod0.accept_ra: "0"
|
||||
net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
|
||||
# Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
|
||||
@@ -104,18 +105,19 @@ addresses:
|
||||
- address: 10.2.48.13/24
|
||||
- address: fd02:48::13/64
|
||||
---
|
||||
# Ingress network, VLAN 47. World-to-pod, publicly routed — see k8s-1.yaml.
|
||||
# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
|
||||
# addressed — see k8s-1.yaml for why it is not leased.
|
||||
apiVersion: v1alpha1
|
||||
kind: VLANConfig
|
||||
name: ingress0
|
||||
vlanID: 47
|
||||
parent: uplink
|
||||
up: true
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: DHCPv4Config
|
||||
name: ingress0
|
||||
routeMetric: 2048
|
||||
addresses:
|
||||
- address: 10.2.47.13/24
|
||||
routes:
|
||||
- gateway: 10.2.47.1
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
|
||||
# here on purpose; the link is handed whole to the OSD pod (host-device).
|
||||
|
||||
+11
-2
@@ -28,10 +28,19 @@ router bgp 64512
|
||||
! advertise extended-nexthop for the IPv6 Unicast SAFI when v6 routes
|
||||
! are carried over a v4 transport, so we run a native v6 session that
|
||||
! carries v6 routes with v6 next-hops and skip the capability dance
|
||||
! entirely. Dynamic neighbors — k8s nodes are SLAAC-addressed on mgmt
|
||||
! LAN; the listen ranges cover the SLAAC suffix space.
|
||||
! entirely. Dynamic neighbors — nodes are SLAAC-addressed, so the listen
|
||||
! ranges cover the SLAAC suffix space rather than naming each peer.
|
||||
!
|
||||
! Both segments are listed while the ingress split lands (#576). The
|
||||
! cluster peers from mgmt (40) today and moves to ingress (47), and a
|
||||
! listen range only accepts a session, it does not open one — so having
|
||||
! both costs nothing and means the move never has a window where the UDM
|
||||
! is not listening where the nodes are calling from. Drop the 40 ranges
|
||||
! once every session is established on 47.
|
||||
bgp listen range 10.2.40.0/24 peer-group K8S
|
||||
bgp listen range fd40::/64 peer-group K8S
|
||||
bgp listen range 10.2.47.0/24 peer-group K8S
|
||||
bgp listen range fd02:47::/64 peer-group K8S
|
||||
!
|
||||
address-family ipv4 unicast
|
||||
neighbor K8S activate
|
||||
|
||||
Reference in New Issue
Block a user