Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
deaf4b6a03 | ||
|
claude-botandClaude Fable 5.1
|
2818fb56b1 |
@@ -1,103 +1,194 @@
|
||||
# Proposal — separate the four traffic classes onto their own VLANs
|
||||
|
||||
Status: open, issue #576. Two of the four classes are separated. Pod east-west
|
||||
cannot leave the node IP under Cilium. Both ways of moving it within Cilium
|
||||
(options A and B below) are rejected by the owner, and replacing the CNI
|
||||
(option C) is shelved.
|
||||
Status: draft. The question this started with is **answered** — from the Cilium
|
||||
source and flag reference, not from a test — and the answer changes the shape
|
||||
of the proposal rather than confirming it.
|
||||
|
||||
## Target
|
||||
Supersedes the framing in issue #576, which described this as "give node
|
||||
management its own NIC, and Ceph public its own VLAN". That was two of four
|
||||
classes and one of them turned out already solved.
|
||||
|
||||
Set by the owner on 2026-09-08:
|
||||
## The four classes
|
||||
|
||||
| class | link |
|
||||
|---|---|
|
||||
| Ceph replication | X710 p1, VLAN 49 |
|
||||
| pod-pod | X710 trunk, a VLAN of its own |
|
||||
| world-pod | X710 trunk, a VLAN of its own |
|
||||
| node management | 1 Gb onboard NIC |
|
||||
|
||||
Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
|
||||
on the 10G switch.
|
||||
|
||||
## Where the classes are
|
||||
|
||||
Measured 2026-09-15.
|
||||
|
||||
| class | today | state |
|
||||
| class | what it is | where it goes today |
|
||||
|---|---|---|
|
||||
| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
|
||||
| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
|
||||
| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
|
||||
| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
|
||||
| management | `mgmt0`, VLAN 40 | not started |
|
||||
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
||||
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
||||
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
||||
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
||||
|
||||
`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
|
||||
#833). `pod0` holds addresses and carries no traffic.
|
||||
Three of four share one interface. That is not a design decision — until the
|
||||
bare-metal conversion there was one NIC to work with.
|
||||
|
||||
## Why pod-pod is blocked
|
||||
Ceph client I/O living on the pod network is correct rather than a leftover:
|
||||
that traffic *is* cluster-internal. Ceph is done.
|
||||
|
||||
Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
|
||||
next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
|
||||
## Principle
|
||||
|
||||
- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
|
||||
fed from `newNode.GetNodeIP(false)`.
|
||||
- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
|
||||
`initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
|
||||
Node object.
|
||||
- `--direct-routing-device` selects a device for the BPF NodePort path, not a
|
||||
next-hop.
|
||||
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
||||
separate, changeable decision.**
|
||||
|
||||
Tested on the cluster, none of these moved the next-hop:
|
||||
Defining classes by VLAN means the separation survives recabling, port failures
|
||||
and capacity changes, and it does not require one port per class. Defining them
|
||||
by interface would freeze a capacity decision into the architecture.
|
||||
|
||||
| attempt | result |
|
||||
|---|---|
|
||||
| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
|
||||
| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
|
||||
| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
|
||||
## The finding that shapes everything else
|
||||
|
||||
The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
|
||||
`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
|
||||
Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
|
||||
cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
|
||||
has no effect on pod-pod routing.
|
||||
The original open question was whether `nodePort.directRoutingDevice` moves the
|
||||
next-hop of Cilium's direct routes, or only selects an interface. It is
|
||||
answerable from the source, and the answer is: **only the interface.**
|
||||
|
||||
## Options for pod-pod
|
||||
`installDirectRoute` takes the next-hop as a parameter:
|
||||
|
||||
| | option | status |
|
||||
```go
|
||||
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
||||
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
||||
```
|
||||
|
||||
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
||||
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
||||
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
||||
|
||||
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
||||
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
||||
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
||||
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
||||
`InternalIP`.
|
||||
|
||||
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
||||
direct routing mode" — a device, not an address.
|
||||
|
||||
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
||||
kubelet InternalIP.** Two consequences:
|
||||
|
||||
- The class cannot be moved off whatever VLAN the node IP lives on.
|
||||
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
||||
from the node IP; native mode uses it as the direct-route next-hop. Native
|
||||
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
||||
previously contained is simply gone.
|
||||
|
||||
## Consequence: draw the line the other way
|
||||
|
||||
If cluster-internal cannot leave the node IP, then separation has to come from
|
||||
moving *management services* off it — and each one has to be checked
|
||||
individually, because they do not share a mechanism.
|
||||
|
||||
| management service | can it leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
|
||||
| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
|
||||
| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
|
||||
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
||||
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
||||
| kubelet | **no** | it *is* the node IP, by definition |
|
||||
|
||||
A per-workload Multus attachment detaches a single workload from the node IP,
|
||||
the way Ceph replication is detached. It does not cover east-west traffic in
|
||||
So four VLANs are reachable, but the boundary does not fall where the class
|
||||
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
||||
result, and it is worth stating plainly rather than filing kubelet under
|
||||
management and hoping.
|
||||
|
||||
The remaining classes keep the levers established earlier:
|
||||
|
||||
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
||||
For eBGP the session source is the next-hop, so pinning the session pins
|
||||
where return traffic arrives. The session to the UDM has been running for
|
||||
months.
|
||||
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
||||
this is a default-route decision plus `enable-masquerade-to-route-source`
|
||||
(today `false`).
|
||||
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
||||
|
||||
## The general escape hatch
|
||||
|
||||
Ceph is separated because its OSDs got a second interface and stopped using the
|
||||
pod network for replication. That is the *only* mechanism that fully detaches a
|
||||
workload from the node IP, and it generalises: anything that must not ride the
|
||||
node IP gets its own `NetworkAttachmentDefinition`.
|
||||
|
||||
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
||||
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
||||
general.
|
||||
|
||||
## Levers per management service
|
||||
## Recommended mapping
|
||||
|
||||
These hold regardless of which option is chosen:
|
||||
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
||||
|
||||
| service | can leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
|
||||
| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
|
||||
| kubelet | no | it is the node IP |
|
||||
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
||||
and Talos API. Cleanest match to the class names, but it changes every node's
|
||||
identity, which is the one address the whole cluster is administered through.
|
||||
|
||||
## Remaining, independent of pod-pod
|
||||
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
||||
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
||||
Identical end state, and no node ever changes its InternalIP.
|
||||
|
||||
- **pod→world egress**: default route and masquerade source onto `ingress0`,
|
||||
via route metrics, `devices` and `enable-masquerade-to-route-source`. This
|
||||
moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
|
||||
- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
|
||||
removed once ingress on VLAN 47 is settled.
|
||||
Under B:
|
||||
|
||||
## Sources
|
||||
| VLAN | carries |
|
||||
|---|---|
|
||||
| 49 | Ceph replication |
|
||||
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
||||
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
||||
| new *external* | LoadBalancer ingress, egress default route |
|
||||
|
||||
- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
|
||||
`--direct-routing-device`.
|
||||
- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
|
||||
- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
|
||||
- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
|
||||
`CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
|
||||
The property the user asked for — management carries only management traffic —
|
||||
holds under B. The label moves; the property is what matters.
|
||||
|
||||
## Migration shape
|
||||
|
||||
Every step is rolling and reversible. The routing-mode change is gone.
|
||||
|
||||
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
||||
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
||||
Still unused; this only proves the L2 works.
|
||||
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
||||
waiting for a healthy quorum between each.
|
||||
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
||||
`k8s.renner.uno` follows.
|
||||
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
||||
Reversible by reverting the override; failure is visible immediately in
|
||||
`cilium-dbg shell -- bgp/peers`.
|
||||
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
||||
|
||||
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
||||
costs quorum rather than connectivity.
|
||||
|
||||
## What this does not buy
|
||||
|
||||
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
||||
today is slow because of this. The case for the split is that traffic classes
|
||||
with different failure modes and security postures should not be
|
||||
indistinguishable — not that something is broken.
|
||||
|
||||
## Sources and measurements
|
||||
|
||||
Source, Cilium v1.20.1 (the running version):
|
||||
|
||||
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
||||
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
||||
a next-hop that is not directly reachable.
|
||||
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
||||
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
||||
Kubernetes Node.
|
||||
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
||||
to connect nodes in direct routing mode".
|
||||
|
||||
Talos:
|
||||
|
||||
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
||||
defaults to it. kube-apiserver and Talos API listen addresses are not
|
||||
covered there.
|
||||
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
||||
`10.2.40.60` with `link: mgmt0`.
|
||||
|
||||
Measured on the cluster:
|
||||
|
||||
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
||||
API is not bound to one address.
|
||||
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
||||
InternalIP had changed since the last session, which is why nothing here
|
||||
names an address as fixed.
|
||||
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
||||
unset and no direct-routing device configured.
|
||||
|
||||
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
||||
node IP independently of the VIP. It does not affect the recommendation, since
|
||||
clients reach the API through the VIP either way.
|
||||
|
||||
@@ -95,7 +95,7 @@ spec:
|
||||
memory: 32Mi
|
||||
containers:
|
||||
- name: home-assistant
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.2 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
env:
|
||||
- name: TZ
|
||||
value: Europe/Berlin
|
||||
|
||||
@@ -32,7 +32,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: immich-machine-learning
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.2.1 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3003
|
||||
|
||||
@@ -87,7 +87,7 @@ spec:
|
||||
mountPath: /ca
|
||||
containers:
|
||||
- name: immich-server
|
||||
image: ghcr.io/immich-app/immich-server:v3.2.1 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3001
|
||||
|
||||
@@ -55,7 +55,7 @@ spec:
|
||||
# stdlib only; Flux keeps the tag in lockstep via the shared image
|
||||
# policy) — no extra dependency.
|
||||
- name: reconcile-publish-ip
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
@@ -65,7 +65,7 @@ spec:
|
||||
readOnly: true
|
||||
containers:
|
||||
- name: music-assistant
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
env:
|
||||
- name: LOG_LEVEL
|
||||
value: info
|
||||
|
||||
@@ -22,7 +22,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gotenberg
|
||||
image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
args:
|
||||
- "gotenberg"
|
||||
- "--chromium-disable-javascript=true"
|
||||
|
||||
@@ -134,7 +134,7 @@ spec:
|
||||
# is a device-bricking class of risk. Bumps are a reviewed, manual
|
||||
# act — see kubernetes/infrastructure/image-automation/README.md.
|
||||
- name: terminus
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 2345
|
||||
@@ -197,7 +197,7 @@ spec:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
- name: sidekiq
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
|
||||
@@ -18,7 +18,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: whisper
|
||||
image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
args:
|
||||
- --model=base-int8
|
||||
- --language=de
|
||||
|
||||
@@ -34,7 +34,7 @@ spec:
|
||||
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
|
||||
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
|
||||
# track the same range/filter so they bump together on each release.
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
env:
|
||||
- name: WOODPECKER_SERVER
|
||||
value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
|
||||
|
||||
@@ -40,7 +40,7 @@ spec:
|
||||
# minor + patch auto-update within 3.x; only the 4.0 major is held for
|
||||
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
|
||||
# -bump hazard is gone). Stays in lockstep with the agent.
|
||||
image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
env:
|
||||
- name: WOODPECKER_HOST
|
||||
value: https://ci.renner.uno
|
||||
|
||||
@@ -1303,6 +1303,32 @@ EOF
|
||||
done
|
||||
ok
|
||||
|
||||
# CNI=antrea swaps the CNI for a throwaway evaluation cluster only — see
|
||||
# kubernetes/cni-eval/README.md. Unset, nothing below changes. Refused on
|
||||
# the production nodes: this installs a CNI Flux does not manage, so a real
|
||||
# cluster would come up with a network no HelmRelease owns.
|
||||
if [[ "${CNI:-cilium}" == "antrea" ]]; then
|
||||
[[ "$SITE" != "homelab" ]] \
|
||||
|| fail "CNI=antrea is for throwaway clusters (vmtest/hcloudtest), not the homelab site"
|
||||
step 0 "Installing Antrea (CNI) — evaluation override..."
|
||||
ANTREA_VALUES="$REPO_ROOT/kubernetes/cni-eval/antrea-values.yaml"
|
||||
[[ -f "$ANTREA_VALUES" ]] || fail "antrea values not found at $ANTREA_VALUES"
|
||||
# >=2.6 is a hard floor, not a preference: earlier agents fail to start
|
||||
# on Talos >=1.7 after its iptables-legacy to iptables-nft switch.
|
||||
helm repo add antrea https://charts.antrea.io >/dev/null
|
||||
helm repo update antrea >/dev/null
|
||||
helm upgrade --install antrea antrea/antrea \
|
||||
--version "${ANTREA_VERSION:->=2.6.0}" \
|
||||
--namespace kube-system \
|
||||
-f "$ANTREA_VALUES" \
|
||||
--wait --timeout 10m
|
||||
ok
|
||||
# Skips the Cilium install below without duplicating the node-Ready
|
||||
# wait that follows it.
|
||||
CNI_INSTALLED=1
|
||||
fi
|
||||
|
||||
if [[ "${CNI_INSTALLED:-0}" != "1" ]]; then
|
||||
step 0 "Installing Cilium (CNI)..."
|
||||
# Must run before the node-Ready wait below: with `cni: none` the kubelet
|
||||
# reports NotReady until a CNI is present, so that loop would time out.
|
||||
@@ -1331,6 +1357,7 @@ EOF
|
||||
--wait --timeout 10m
|
||||
rm -f "$CILIUM_VALUES"
|
||||
ok
|
||||
fi
|
||||
|
||||
step 0 "Waiting for K8s nodes..."
|
||||
expected=${#ALL_NODES[@]}
|
||||
@@ -1392,6 +1419,16 @@ EOF
|
||||
echo -e " $vm_node: partition and label ${GREEN}(OK)${NC}"
|
||||
done
|
||||
|
||||
# The CNI evaluation runs here, inside the same process: KUBECONFIG
|
||||
# points at the throwaway cluster only for the life of this phase, and
|
||||
# the VMs are destroyed the moment it ends.
|
||||
if [[ "${CNI:-cilium}" == "antrea" ]]; then
|
||||
step 0 "Running the Antrea BGP evaluation..."
|
||||
"$REPO_ROOT/kubernetes/cni-eval/run-bgp-test.sh" \
|
||||
|| fail "Antrea BGP evaluation failed — see above"
|
||||
ok
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}VM rebuild test passed.${NC} ${#ALL_NODES[@]} nodes Ready on the repo's own config."
|
||||
else
|
||||
mkdir -p "$ACCESS_STAGING"
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# CNI evaluation — Antrea
|
||||
|
||||
Throwaway-cluster harness for deciding whether Antrea can replace Cilium.
|
||||
Nothing here is reconciled by Flux and nothing here runs against the homelab.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Cilium ties pod east-west traffic to the kubelet's Kubernetes node IP and gives
|
||||
no way to separate them. Three attempts on the live cluster, 2026-09-08:
|
||||
|
||||
| Attempt | Result |
|
||||
|---|---|
|
||||
| `--ipv4-node`/`--ipv6-node` via `CiliumNodeConfig` | agent logged both flags at startup, still published its `mgmt0` address |
|
||||
| `directRoutingDevice: pod0` | agent logged `Direct routing device detected direct-routing-device=pod0`, still installed `10.244.2.0/24 via 10.2.40.138 dev mgmt0` |
|
||||
| tunnel and native mode, v4 and v6 | routes and tunnel endpoints always keyed to the peer's `InternalIP` |
|
||||
|
||||
That blocks issue #576: pod traffic cannot leave the management VLAN while the
|
||||
apiserver endpoints have to stay on it, because both follow the same address.
|
||||
|
||||
Antrea documents one setting covering both paths — *"The IP address used for
|
||||
tunneling **or** routing the traffic across Nodes is decided in the following
|
||||
order of preference: 1. TransportInterface 2. TransportInterfaceCIDRs 3. The
|
||||
Node IP."* Calico can do it too. Antrea fits the rest of the requirements
|
||||
better, with one exception: **its BGP is alpha**, and BGP is load-bearing here —
|
||||
every LoadBalancer VIP reaches the LAN through it.
|
||||
|
||||
So the question this harness answers is narrow: **does Antrea's alpha
|
||||
`BGPPolicy` actually hold a session with FRR and advertise what the homelab
|
||||
needs?** If it does not, the answer is Calico and this goes no further.
|
||||
|
||||
## Stage 1 — free, local
|
||||
|
||||
The QEMU provisioner gives one flat NIC per VM (`site-qemu.yaml`: *"One NIC, no
|
||||
VLANs"*), so this stage cannot test interface pinning. It tests everything else,
|
||||
at no cost:
|
||||
|
||||
Runs from the `feat/antrea-eval` branch, which is never merged:
|
||||
`bootstrap.sh` on master is the production rebuild path.
|
||||
|
||||
**In the cluster — the way to run it.** The VMs want ~40 GiB, and
|
||||
`kubernetes/vmtest-job.yaml` records that running this on the workstation
|
||||
OOM-killed the desktop session. The eval Job is that same Job with three values
|
||||
changed: the branch, `CNI=antrea`, and the clone source — the GitHub mirror is
|
||||
private and the stock Job clones it anonymously, so this one clones the
|
||||
anonymously-readable Gitea repo instead.
|
||||
|
||||
```bash
|
||||
yq -y '(select(.kind == "Job") | .spec.template.spec.containers[0].env[] | select(.name == "BRANCH") | .value) = "feat/antrea-eval"
|
||||
| (select(.kind == "Job") | .spec.template.spec.containers[0].env) += [{"name": "CNI", "value": "antrea"}]
|
||||
| (select(.kind == "Job") | .spec.template.spec.containers[0].args[0]) |= sub("https://github.com/konairius/Homelab.git"; "https://gitea.renner.uno/k.renner/homelab.git")' \
|
||||
kubernetes/vmtest-job.yaml | kubectl apply -f -
|
||||
|
||||
kubectl -n vmtest logs -f job/vmtest
|
||||
```
|
||||
|
||||
`-y` is python-yq; go-yq emits YAML without it. The Talos-only phase needs one
|
||||
secret, `ghcr-pull`; the log prints the exact command to stream it in. The BGP
|
||||
evaluation runs inside the Job once the nodes are Ready, so PASS/FAIL is in the
|
||||
log and the VMs are destroyed when it ends. The Job cannot reach this cluster:
|
||||
it mounts no service account token. Clean up with `kubectl delete namespace vmtest`.
|
||||
|
||||
**On a workstation — only with ~40 GiB to spare.** `vmtest` destroys its VMs on
|
||||
exit and exports its isolated `KUBECONFIG` only inside its own process, so hold
|
||||
them open and point the test at them explicitly:
|
||||
|
||||
```bash
|
||||
# terminal 1 — needs sudo; holds the VMs until stopped
|
||||
VMTEST_KEEP=1 CNI=antrea ./kubernetes/bootstrap.sh vmtest
|
||||
# it prints: talosctl --talosconfig /tmp/tmp.XXXXXX/talosconfig ...
|
||||
|
||||
# terminal 2 — the kubeconfig sits next to that talosconfig
|
||||
KUBECONFIG=/tmp/tmp.XXXXXX/kubeconfig ./kubernetes/cni-eval/run-bgp-test.sh
|
||||
|
||||
# teardown
|
||||
sudo talosctl cluster destroy --name bootstrap-vmtest --state /tmp/bootstrap-vmtest
|
||||
sudo rm -rf /tmp/bootstrap-vmtest
|
||||
```
|
||||
|
||||
`run-bgp-test.sh` refuses to run without an explicit `KUBECONFIG`, and refuses
|
||||
any cluster whose API is `k8s.renner.uno` or on 10.2.40.0/24, or whose nodes are
|
||||
named `k8s-1..3`.
|
||||
|
||||
What passing means:
|
||||
|
||||
- Antrea installs and runs on Talos at all (needs >= 2.6 — earlier agents fail
|
||||
to start on Talos >= 1.7 after its iptables-nft switch).
|
||||
- The cluster reaches Ready with `cluster.proxy.disabled: true`, i.e. AntreaProxy
|
||||
genuinely replaces kube-proxy, reaching the API through KubePrism.
|
||||
- `ServiceExternalIP` allocates a LoadBalancer IP.
|
||||
- `BGPPolicy` establishes with FRR from every node and advertises the pod CIDRs
|
||||
and the LoadBalancer IP — verified by reading FRR's table, not Antrea's.
|
||||
|
||||
`run-bgp-test.sh` deploys FRR as an ordinary pod standing in for the UDM: same
|
||||
ASNs as `unifi/bgp.conf` (64512/64513), dynamic neighbours via a listen range
|
||||
rather than named node addresses. Its peer address is read from the running pod
|
||||
at apply time rather than written into the manifest.
|
||||
|
||||
## Stage 2 — billed, only if stage 1 passes
|
||||
|
||||
Hetzner nodes have two interfaces and no shared L2, so this is the only place
|
||||
the two remaining requirements can be tested:
|
||||
|
||||
```bash
|
||||
CNI=antrea ./kubernetes/bootstrap.sh hcloudtest
|
||||
```
|
||||
|
||||
Uncomment `transportInterface` and `trafficEncapMode: hybrid` in
|
||||
`antrea-values.yaml` first, pointing the former at the private-network NIC while
|
||||
`site-hcloud.yaml` keeps the kubelet node IP where it is. Then confirm:
|
||||
|
||||
- pod east-west leaves on the nominated interface while the node IP does not
|
||||
move — the property Cilium cannot express;
|
||||
- `hybrid` encapsulates across a routed network, which is what the Hetzner
|
||||
cluster needs and the homelab does not.
|
||||
|
||||
`hcloudtest` bills by the hour on three cx43s and tears down from its EXIT trap.
|
||||
If a run is killed before the trap fires, `./kubernetes/bootstrap.sh hcloudclean`
|
||||
deletes by label.
|
||||
|
||||
## What this harness does not cover
|
||||
|
||||
- **Policy migration.** 59 `CiliumNetworkPolicy` objects across 34 files, using
|
||||
`fromEntities`/`toEntities` in 15, typed ICMP in 24, and `toFQDNs` in
|
||||
`trmnl-ns.yaml`. Antrea has equivalents including FQDN egress in OSS, but the
|
||||
port is real work and is not tested here.
|
||||
- **Multus.** Required either way — Antrea's native secondary networks are VLAN
|
||||
and SR-IOV only, and all six NADs are macvlan or host-device. Unchanged by a
|
||||
CNI swap.
|
||||
- **In-place migration.** There isn't one. Talos documents CNI choice as
|
||||
bootstrap-time only, and a Sidero maintainer's guidance is *"much easier to
|
||||
redeploy the cluster from scratch"*. Adopting Antrea means a cold rebuild.
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
kubectl delete namespace cni-eval
|
||||
kubectl delete bgppolicy cni-eval-peer
|
||||
kubectl delete externalippool cni-eval-lb
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Antrea Helm values for the CNI evaluation. Not reconciled by Flux; used only
|
||||
# by `CNI=antrea ./bootstrap.sh vmtest` and `hcloudtest`.
|
||||
|
||||
# Talos blocks CAP_SYS_MODULE for workloads and builds openvswitch into its
|
||||
# kernel, so the modprobe init step is skipped rather than granted.
|
||||
agent:
|
||||
dontLoadKernelModules: true
|
||||
installCNI:
|
||||
securityContext:
|
||||
capabilities: []
|
||||
|
||||
# talos/patches/cluster.yaml sets cluster.proxy.disabled, so there is no
|
||||
# kube-proxy: AntreaProxy has to serve every Service type, and the agent has to
|
||||
# reach the API server through KubePrism rather than a Service IP.
|
||||
antreaProxy:
|
||||
proxyAll: true
|
||||
kubeAPIServerOverride: "localhost:7445"
|
||||
|
||||
# BGPPolicy is alpha and off by default. ServiceExternalIP is the LoadBalancer
|
||||
# IPAM; on by default since 2.3, named here so the manifest states it.
|
||||
featureGates:
|
||||
BGPPolicy: true
|
||||
ServiceExternalIP: true
|
||||
|
||||
# Matches --service-cluster-ip-range in talos/patches/cluster.yaml.
|
||||
serviceCIDR: 10.96.0.0/12
|
||||
serviceCIDRv6: fd10:96::/112
|
||||
|
||||
# Stage 2 only (hcloudtest, two NICs). Pins pod east-west to a named interface
|
||||
# while the kubelet node IP stays where it is — the property Cilium cannot
|
||||
# express. Left unset here because the QEMU provisioner gives one flat NIC.
|
||||
# transportInterface: ""
|
||||
# trafficEncapMode: hybrid
|
||||
@@ -0,0 +1,48 @@
|
||||
# What the test asks Antrea to advertise. Applied after frr-peer.yaml.
|
||||
# __FRR_PEER_IP__ is substituted by run-bgp-test.sh from the running FRR pod —
|
||||
# not written here, because it is a lease, not an identity.
|
||||
---
|
||||
apiVersion: crd.antrea.io/v1beta1
|
||||
kind: ExternalIPPool
|
||||
metadata:
|
||||
name: cni-eval-lb
|
||||
spec:
|
||||
ipRanges:
|
||||
- start: 10.99.99.10
|
||||
end: 10.99.99.20
|
||||
nodeSelector: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: lb-probe
|
||||
namespace: cni-eval
|
||||
annotations:
|
||||
service.antrea.io/external-ip-pool: cni-eval-lb
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: frr
|
||||
ports:
|
||||
- port: 179
|
||||
targetPort: 179
|
||||
---
|
||||
apiVersion: crd.antrea.io/v1alpha1
|
||||
kind: BGPPolicy
|
||||
metadata:
|
||||
name: cni-eval-peer
|
||||
spec:
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/os: linux
|
||||
localASN: 64513
|
||||
listenPort: 179
|
||||
advertisements:
|
||||
pod: {}
|
||||
service:
|
||||
ipTypes:
|
||||
- LoadBalancerIP
|
||||
bgpPeers:
|
||||
- address: __FRR_PEER_IP__
|
||||
asn: 64512
|
||||
port: 179
|
||||
@@ -0,0 +1,106 @@
|
||||
# FRR standing in for the UDM, so Antrea's BGP can be tested without one.
|
||||
# Mirrors unifi/bgp.conf where it matters: same ASNs, dynamic neighbours via a
|
||||
# listen range rather than named node addresses.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cni-eval
|
||||
labels:
|
||||
# Talos enforces baseline, which rejects the NET_ADMIN and NET_RAW zebra
|
||||
# needs. Throwaway cluster only.
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: frr-config
|
||||
namespace: cni-eval
|
||||
data:
|
||||
daemons: |
|
||||
bgpd=yes
|
||||
zebra=yes
|
||||
vtysh_enable=yes
|
||||
bgpd_options="-A 127.0.0.1"
|
||||
zebra_options="-A 127.0.0.1 -s 90000000"
|
||||
frr.conf: |
|
||||
frr defaults traditional
|
||||
hostname frr-eval
|
||||
log stdout informational
|
||||
!
|
||||
router bgp 64512
|
||||
bgp router-id 172.31.255.1
|
||||
no bgp ebgp-requires-policy
|
||||
bgp listen range 0.0.0.0/0 peer-group K8S
|
||||
bgp listen range ::/0 peer-group K8S
|
||||
neighbor K8S peer-group
|
||||
neighbor K8S remote-as 64513
|
||||
neighbor K8S timers 30 90
|
||||
!
|
||||
address-family ipv4 unicast
|
||||
neighbor K8S activate
|
||||
neighbor K8S soft-reconfiguration inbound
|
||||
exit-address-family
|
||||
!
|
||||
address-family ipv6 unicast
|
||||
neighbor K8S activate
|
||||
neighbor K8S soft-reconfiguration inbound
|
||||
exit-address-family
|
||||
!
|
||||
line vty
|
||||
!
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: frr
|
||||
namespace: cni-eval
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: frr
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: frr
|
||||
spec:
|
||||
# /etc/frr must be writable for the daemons and vtysh; a ConfigMap mount
|
||||
# is not. Seeded from the image's own defaults plus the ConfigMap.
|
||||
initContainers:
|
||||
- name: seed-config
|
||||
image: quay.io/frrouting/frr:10.2.1
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- cp -a /etc/frr/. /frr-etc/ && cp /config/frr.conf /config/daemons /frr-etc/
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: etc
|
||||
mountPath: /frr-etc
|
||||
containers:
|
||||
- name: frr
|
||||
image: quay.io/frrouting/frr:10.2.1
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- NET_ADMIN
|
||||
- NET_BIND_SERVICE
|
||||
- NET_RAW
|
||||
ports:
|
||||
- containerPort: 179
|
||||
name: bgp
|
||||
volumeMounts:
|
||||
- name: etc
|
||||
mountPath: /etc/frr
|
||||
- name: run
|
||||
mountPath: /var/run/frr
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: frr-config
|
||||
- name: etc
|
||||
emptyDir: {}
|
||||
- name: run
|
||||
emptyDir: {}
|
||||
Executable
+118
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Verifies Antrea's alpha BGPPolicy against an FRR peer. Answers exactly one
|
||||
# question: does the session establish and carry the routes the homelab needs
|
||||
# (pod CIDRs and LoadBalancer IPs). Everything it asserts is read back from
|
||||
# FRR, not from Antrea's own view of what it advertised.
|
||||
#
|
||||
# kubernetes/cni-eval/run-bgp-test.sh
|
||||
#
|
||||
# Expects a cluster already built with CNI=antrea. Leaves the cni-eval
|
||||
# namespace behind for inspection; the cleanup line is printed at the end.
|
||||
|
||||
DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'; NC=$'\033[0m'
|
||||
fail() { echo "${RED}FAIL${NC} $*" >&2; exit 1; }
|
||||
pass() { echo "${GREEN}PASS${NC} $*"; }
|
||||
info() { echo "${YELLOW}==>${NC} $*"; }
|
||||
|
||||
# vmtest and hcloudtest export their isolated KUBECONFIG only inside their own
|
||||
# process. A shell that runs this afterwards gets the default kubeconfig, which
|
||||
# is production. Refuse unless pointed at a throwaway cluster explicitly.
|
||||
[[ -n "${KUBECONFIG:-}" ]] \
|
||||
|| fail "set KUBECONFIG to the throwaway cluster's kubeconfig — the default one is production"
|
||||
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
|
||||
case "$server" in
|
||||
*k8s.renner.uno*|*10.2.40.*) fail "KUBECONFIG points at production ($server) — refusing" ;;
|
||||
esac
|
||||
node_names=$(kubectl get nodes -o name)
|
||||
if grep -qx 'node/k8s-[123]' <<<"$node_names"; then
|
||||
fail "this cluster has production node names — refusing"
|
||||
fi
|
||||
pass "target is a throwaway cluster ($server)"
|
||||
|
||||
info "Checking Antrea is the running CNI..."
|
||||
kubectl -n kube-system get daemonset antrea-agent >/dev/null \
|
||||
|| fail "antrea-agent DaemonSet not found — was the cluster built with CNI=antrea?"
|
||||
|
||||
info "Confirming the BGPPolicy feature gate is on..."
|
||||
gates=$(kubectl -n kube-system get configmap antrea-config -o jsonpath='{.data.antrea-agent\.conf}')
|
||||
grep -q 'BGPPolicy: *true' <<<"$gates" \
|
||||
|| fail "BGPPolicy feature gate is not enabled in antrea-config"
|
||||
pass "BGPPolicy enabled"
|
||||
|
||||
info "Deploying the FRR peer..."
|
||||
kubectl apply -f "$DIR/frr-peer.yaml"
|
||||
kubectl -n cni-eval rollout status deployment/frr --timeout=180s \
|
||||
|| fail "FRR did not become ready"
|
||||
|
||||
# Read the address rather than assign one: the peer is a pod, and its address
|
||||
# is whatever the CNI under test gave it.
|
||||
frr_ip=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].status.podIP}')
|
||||
[[ -n "$frr_ip" ]] || fail "FRR pod has no IP"
|
||||
pass "FRR peer at $frr_ip"
|
||||
|
||||
info "Applying ExternalIPPool, LoadBalancer Service and BGPPolicy..."
|
||||
sed "s|__FRR_PEER_IP__|$frr_ip|" "$DIR/bgp-test.yaml" | kubectl apply -f -
|
||||
|
||||
info "Waiting for a LoadBalancer IP (ServiceExternalIP)..."
|
||||
lb=""
|
||||
for _ in $(seq 1 30); do
|
||||
lb=$(kubectl -n cni-eval get svc lb-probe -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
|
||||
[[ -n "$lb" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ -n "$lb" ]] || fail "no LoadBalancer IP assigned after 150s — ServiceExternalIP did not allocate"
|
||||
pass "LoadBalancer IP $lb"
|
||||
|
||||
frr_pod=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].metadata.name}')
|
||||
|
||||
info "Waiting for BGP sessions to establish..."
|
||||
nodes=$(kubectl get nodes -o name | wc -l)
|
||||
established=0
|
||||
for _ in $(seq 1 36); do
|
||||
# The text summary shows an established peer as a prefix count, not a state
|
||||
# name; the per-neighbour JSON carries bgpState explicitly.
|
||||
neighbors=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp neighbor json' || true)
|
||||
established=$(jq '[.[] | select(type == "object" and .bgpState == "Established")] | length' <<<"$neighbors" || echo 0)
|
||||
established=${established:-0}
|
||||
[[ "$established" -ge "$nodes" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp summary'
|
||||
[[ "$established" -ge "$nodes" ]] \
|
||||
|| fail "only $established of $nodes sessions established after 180s"
|
||||
pass "$established/$nodes sessions established"
|
||||
|
||||
info "Checking the advertised routes as FRR sees them..."
|
||||
kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp'
|
||||
routes=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp json')
|
||||
|
||||
# Exact prefix keys rather than substring matches.
|
||||
advertised() { jq -e --arg p "$1" '.routes | has($p)' <<<"$routes" >/dev/null; }
|
||||
|
||||
pod_cidrs=$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}')
|
||||
missing=0
|
||||
while read -r cidr; do
|
||||
[[ -z "$cidr" ]] && continue
|
||||
if advertised "$cidr"; then
|
||||
pass "pod CIDR $cidr advertised"
|
||||
else
|
||||
echo "${RED}FAIL${NC} pod CIDR $cidr NOT advertised"
|
||||
missing=1
|
||||
fi
|
||||
done <<<"$pod_cidrs"
|
||||
|
||||
if grep -qF "$lb" <<<"$routes"; then
|
||||
pass "LoadBalancer IP $lb advertised"
|
||||
else
|
||||
echo "${RED}FAIL${NC} LoadBalancer IP $lb NOT advertised"
|
||||
missing=1
|
||||
fi
|
||||
|
||||
[[ "$missing" -eq 0 ]] || fail "one or more expected prefixes were not advertised"
|
||||
|
||||
echo
|
||||
pass "Antrea BGPPolicy carried pod CIDRs and LoadBalancer IPs to FRR."
|
||||
echo "Clean up: kubectl delete namespace cni-eval && kubectl delete bgppolicy cni-eval-peer && kubectl delete externalippool cni-eval-lb"
|
||||
@@ -180,17 +180,3 @@ spec:
|
||||
# the supernets here cover the LAN as well, which is the point.
|
||||
ipv4NativeRoutingCIDR: 10.0.0.0/8
|
||||
ipv6NativeRoutingCIDR: fd00::/8
|
||||
# Pod east-west leaves the node on pod0 (VLAN 48) rather than the
|
||||
# management VLAN. Native routing hands forwarding to the kernel;
|
||||
# autoDirectNodeRoutes installs the remote PodCIDR routes, which is
|
||||
# allowed because all three nodes share pod0's L2 segment.
|
||||
#
|
||||
# directRoutingDevice is explicit on purpose: with several addressed
|
||||
# devices Cilium falls back to the one carrying the Kubernetes InternalIP,
|
||||
# which is mgmt0.
|
||||
#
|
||||
# Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
|
||||
routingMode: native
|
||||
autoDirectNodeRoutes: true
|
||||
nodePort:
|
||||
directRoutingDevice: pod0
|
||||
|
||||
Reference in New Issue
Block a user