Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
deaf4b6a03 | ||
|
claude-botandClaude Fable 5.1
|
2818fb56b1 |
@@ -376,29 +376,3 @@ steps:
|
||||
--dockerfile Containerfile \
|
||||
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
|
||||
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
|
||||
|
||||
# bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
|
||||
# artifact like heating-can; same registry so the audit tracks it.
|
||||
- name: build-bird
|
||||
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
|
||||
directory: ci/images/bird
|
||||
depends_on: []
|
||||
when:
|
||||
- event: deployment
|
||||
- event: push
|
||||
path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
|
||||
environment:
|
||||
GITEA_USER: gitea_admin
|
||||
GITEA_TOKEN:
|
||||
from_secret: ci_images_token
|
||||
commands:
|
||||
- mkdir -p /kaniko/.docker
|
||||
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
|
||||
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
|
||||
- TAG=$(date +%Y-%m-%d)
|
||||
- |
|
||||
/kaniko/executor \
|
||||
--context . \
|
||||
--dockerfile Containerfile \
|
||||
--destination gitea.renner.uno/gitea_admin/bird:$TAG \
|
||||
--destination gitea.renner.uno/gitea_admin/bird:latest
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
|
||||
|
||||
# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
|
||||
# step — the image is the binary plus its runtime.
|
||||
#
|
||||
# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
|
||||
# 3.22/community.
|
||||
#
|
||||
# Config is read from /etc/bird/bird.conf, not the package default
|
||||
# /etc/bird.conf, so the mount cannot shadow what the package ships.
|
||||
#
|
||||
# kubectl renders the init container's per-node config: go-template-file for
|
||||
# the templating, a field selector for excluding the node itself, in-cluster
|
||||
# config for the auth. From the distro, so there is no version literal to drift
|
||||
# and no download to trust. It is only ever used to read core/v1 Nodes.
|
||||
RUN apk add --no-cache bird2 kubectl
|
||||
|
||||
# Fails the build rather than the first pod if a binary or its path is wrong.
|
||||
RUN /usr/sbin/bird --version \
|
||||
&& kubectl version --client > /dev/null
|
||||
|
||||
ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]
|
||||
@@ -1,117 +0,0 @@
|
||||
# Traffic classes on the k8s nodes
|
||||
|
||||
Issue #576. Three of the four classes are separated; management is not.
|
||||
|
||||
## Where each class runs
|
||||
|
||||
| class | link | mechanism |
|
||||
|---|---|---|
|
||||
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
|
||||
| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
|
||||
| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
|
||||
| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
|
||||
| management | `mgmt0`, VLAN 40 | not started |
|
||||
|
||||
Hard constraint, set by the owner: inter-node and pod traffic never traverses
|
||||
the UDM. Everything above stays on the 10G switch.
|
||||
|
||||
## Why policy routing rather than moving the node IP
|
||||
|
||||
Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
|
||||
and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
|
||||
`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
|
||||
Both were established by test (#837, #838), not by reading.
|
||||
|
||||
So the InternalIP decides where pod traffic goes — unless the lookup happens in
|
||||
a different table. The rules do that, and nothing about the node's identity
|
||||
moves. The routing decision runs before Cilium's masquerade hook, so a rule
|
||||
matching the pod source sees the pod address, and the masquerade then rewrites
|
||||
it to the egress link's.
|
||||
|
||||
## Rule order
|
||||
|
||||
Lower priority runs first.
|
||||
|
||||
| prio | match | table |
|
||||
|---|---|---|
|
||||
| 980 | src pod, dst pod | 48 |
|
||||
| 981 | src pod, dst pod (v4) | 48 |
|
||||
| 990 | dst `fd00::/8` | main |
|
||||
| 991 | dst `10.0.0.0/8` | main |
|
||||
| 1000 | src pod | 47 |
|
||||
| 1001 | src pod (v4) | 47 |
|
||||
|
||||
980/981 match source **and** destination. Source alone would also catch the
|
||||
node's own traffic to a pod, which would then leave with a `pod0` source — an
|
||||
address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
|
||||
`world` rather than `host`.
|
||||
|
||||
990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
|
||||
cluster-internal traffic out of table 47. Without them a pod's packet to a
|
||||
service matches the source rule, finds only a default route, and leaves for the
|
||||
UDM — measured, in-cluster DNS stopped resolving.
|
||||
|
||||
Traffic to a pod on the same node falls through on its own: table 48 holds only
|
||||
the peers' CIDRs, so the lookup misses and 990/991 send it to main.
|
||||
|
||||
## Where each table comes from
|
||||
|
||||
- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
|
||||
The IPv6 gateway is the UDM's configured address on the segment, not the
|
||||
RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
|
||||
default does not land in main.
|
||||
- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
|
||||
per peer pod CIDR with the peer's `pod0` address as next hop.
|
||||
|
||||
## The bird mesh
|
||||
|
||||
`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
|
||||
and installs what it learns into table 48. Cilium's routes in main are
|
||||
untouched, which is why the two coexist.
|
||||
|
||||
No address is written into this repo twice. Each node publishes its `pod0`
|
||||
addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
|
||||
pod CIDRs come from the Node objects. An init container renders both from the
|
||||
API with `kubectl -o go-template-file`, excludes the node itself with a field
|
||||
selector, and ends with `bird -p` so BIRD's own parser gates startup.
|
||||
|
||||
The node's own CIDR is **originated**, not read back from the kernel. Reading
|
||||
it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
|
||||
rejected outright: its next hop sits inside the prefix it describes, which KRT
|
||||
refuses with `strange next-hop`. The IPv6 route is a device route and passed,
|
||||
so the failure was silent on one family only.
|
||||
|
||||
## Failure behaviour
|
||||
|
||||
Benign at every layer. An empty or absent table 48 makes 980/981 fall through
|
||||
to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
|
||||
holds if the DaemonSet cannot start, which is the case during a cold rebuild
|
||||
until Gitea serves its image.
|
||||
|
||||
## Management, the remaining class
|
||||
|
||||
Not started. The levers differ per service and none of them is the node IP:
|
||||
|
||||
| service | can leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
|
||||
| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
|
||||
| Talos API | n/a | `apid` answers on every node address; a firewall question |
|
||||
| kubelet | no | it *is* the node IP |
|
||||
|
||||
Two approaches were rejected by the owner and are not to be proposed again:
|
||||
moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
|
||||
cluster VLAN while moving etcd and the API VIP to a new management VLAN.
|
||||
Replacing the CNI (#839, Antrea) was shelved.
|
||||
|
||||
Note that the reasoning which produced that list also concluded pod↔pod was
|
||||
blocked, which policy routing disproved. Treat it as a record of decisions, not
|
||||
as an exhaustive analysis.
|
||||
|
||||
## Sources
|
||||
|
||||
Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
|
||||
`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
|
||||
`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
|
||||
`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
|
||||
#855; closed unmerged: #830, #838, #839.
|
||||
@@ -0,0 +1,194 @@
|
||||
# Proposal — separate the four traffic classes onto their own VLANs
|
||||
|
||||
Status: draft. The question this started with is **answered** — from the Cilium
|
||||
source and flag reference, not from a test — and the answer changes the shape
|
||||
of the proposal rather than confirming it.
|
||||
|
||||
Supersedes the framing in issue #576, which described this as "give node
|
||||
management its own NIC, and Ceph public its own VLAN". That was two of four
|
||||
classes and one of them turned out already solved.
|
||||
|
||||
## The four classes
|
||||
|
||||
| class | what it is | where it goes today |
|
||||
|---|---|---|
|
||||
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
||||
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
||||
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
||||
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
||||
|
||||
Three of four share one interface. That is not a design decision — until the
|
||||
bare-metal conversion there was one NIC to work with.
|
||||
|
||||
Ceph client I/O living on the pod network is correct rather than a leftover:
|
||||
that traffic *is* cluster-internal. Ceph is done.
|
||||
|
||||
## Principle
|
||||
|
||||
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
||||
separate, changeable decision.**
|
||||
|
||||
Defining classes by VLAN means the separation survives recabling, port failures
|
||||
and capacity changes, and it does not require one port per class. Defining them
|
||||
by interface would freeze a capacity decision into the architecture.
|
||||
|
||||
## The finding that shapes everything else
|
||||
|
||||
The original open question was whether `nodePort.directRoutingDevice` moves the
|
||||
next-hop of Cilium's direct routes, or only selects an interface. It is
|
||||
answerable from the source, and the answer is: **only the interface.**
|
||||
|
||||
`installDirectRoute` takes the next-hop as a parameter:
|
||||
|
||||
```go
|
||||
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
||||
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
||||
```
|
||||
|
||||
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
||||
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
||||
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
||||
|
||||
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
||||
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
||||
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
||||
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
||||
`InternalIP`.
|
||||
|
||||
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
||||
direct routing mode" — a device, not an address.
|
||||
|
||||
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
||||
kubelet InternalIP.** Two consequences:
|
||||
|
||||
- The class cannot be moved off whatever VLAN the node IP lives on.
|
||||
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
||||
from the node IP; native mode uses it as the direct-route next-hop. Native
|
||||
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
||||
previously contained is simply gone.
|
||||
|
||||
## Consequence: draw the line the other way
|
||||
|
||||
If cluster-internal cannot leave the node IP, then separation has to come from
|
||||
moving *management services* off it — and each one has to be checked
|
||||
individually, because they do not share a mechanism.
|
||||
|
||||
| management service | can it leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
||||
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
||||
| kubelet | **no** | it *is* the node IP, by definition |
|
||||
|
||||
So four VLANs are reachable, but the boundary does not fall where the class
|
||||
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
||||
result, and it is worth stating plainly rather than filing kubelet under
|
||||
management and hoping.
|
||||
|
||||
The remaining classes keep the levers established earlier:
|
||||
|
||||
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
||||
For eBGP the session source is the next-hop, so pinning the session pins
|
||||
where return traffic arrives. The session to the UDM has been running for
|
||||
months.
|
||||
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
||||
this is a default-route decision plus `enable-masquerade-to-route-source`
|
||||
(today `false`).
|
||||
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
||||
|
||||
## The general escape hatch
|
||||
|
||||
Ceph is separated because its OSDs got a second interface and stopped using the
|
||||
pod network for replication. That is the *only* mechanism that fully detaches a
|
||||
workload from the node IP, and it generalises: anything that must not ride the
|
||||
node IP gets its own `NetworkAttachmentDefinition`.
|
||||
|
||||
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
||||
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
||||
general.
|
||||
|
||||
## Recommended mapping
|
||||
|
||||
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
||||
|
||||
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
||||
and Talos API. Cleanest match to the class names, but it changes every node's
|
||||
identity, which is the one address the whole cluster is administered through.
|
||||
|
||||
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
||||
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
||||
Identical end state, and no node ever changes its InternalIP.
|
||||
|
||||
Under B:
|
||||
|
||||
| VLAN | carries |
|
||||
|---|---|
|
||||
| 49 | Ceph replication |
|
||||
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
||||
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
||||
| new *external* | LoadBalancer ingress, egress default route |
|
||||
|
||||
The property the user asked for — management carries only management traffic —
|
||||
holds under B. The label moves; the property is what matters.
|
||||
|
||||
## Migration shape
|
||||
|
||||
Every step is rolling and reversible. The routing-mode change is gone.
|
||||
|
||||
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
||||
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
||||
Still unused; this only proves the L2 works.
|
||||
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
||||
waiting for a healthy quorum between each.
|
||||
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
||||
`k8s.renner.uno` follows.
|
||||
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
||||
Reversible by reverting the override; failure is visible immediately in
|
||||
`cilium-dbg shell -- bgp/peers`.
|
||||
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
||||
|
||||
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
||||
costs quorum rather than connectivity.
|
||||
|
||||
## What this does not buy
|
||||
|
||||
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
||||
today is slow because of this. The case for the split is that traffic classes
|
||||
with different failure modes and security postures should not be
|
||||
indistinguishable — not that something is broken.
|
||||
|
||||
## Sources and measurements
|
||||
|
||||
Source, Cilium v1.20.1 (the running version):
|
||||
|
||||
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
||||
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
||||
a next-hop that is not directly reachable.
|
||||
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
||||
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
||||
Kubernetes Node.
|
||||
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
||||
to connect nodes in direct routing mode".
|
||||
|
||||
Talos:
|
||||
|
||||
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
||||
defaults to it. kube-apiserver and Talos API listen addresses are not
|
||||
covered there.
|
||||
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
||||
`10.2.40.60` with `link: mgmt0`.
|
||||
|
||||
Measured on the cluster:
|
||||
|
||||
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
||||
API is not bound to one address.
|
||||
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
||||
InternalIP had changed since the last session, which is why nothing here
|
||||
names an address as fixed.
|
||||
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
||||
unset and no direct-routing device configured.
|
||||
|
||||
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
||||
node IP independently of the VIP. It does not affect the recommendation, since
|
||||
clients reach the API through the VIP either way.
|
||||
@@ -95,7 +95,7 @@ spec:
|
||||
memory: 32Mi
|
||||
containers:
|
||||
- name: home-assistant
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.2 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
env:
|
||||
- name: TZ
|
||||
value: Europe/Berlin
|
||||
|
||||
@@ -32,7 +32,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: immich-machine-learning
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.2.2 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3003
|
||||
|
||||
@@ -87,7 +87,7 @@ spec:
|
||||
mountPath: /ca
|
||||
containers:
|
||||
- name: immich-server
|
||||
image: ghcr.io/immich-app/immich-server:v3.2.2 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3001
|
||||
|
||||
@@ -55,7 +55,7 @@ spec:
|
||||
# stdlib only; Flux keeps the tag in lockstep via the shared image
|
||||
# policy) — no extra dependency.
|
||||
- name: reconcile-publish-ip
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
@@ -65,7 +65,7 @@ spec:
|
||||
readOnly: true
|
||||
containers:
|
||||
- name: music-assistant
|
||||
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
|
||||
env:
|
||||
- name: LOG_LEVEL
|
||||
value: info
|
||||
|
||||
@@ -22,7 +22,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gotenberg
|
||||
image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
|
||||
args:
|
||||
- "gotenberg"
|
||||
- "--chromium-disable-javascript=true"
|
||||
|
||||
@@ -134,7 +134,7 @@ spec:
|
||||
# is a device-bricking class of risk. Bumps are a reviewed, manual
|
||||
# act — see kubernetes/infrastructure/image-automation/README.md.
|
||||
- name: terminus
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 2345
|
||||
@@ -197,7 +197,7 @@ spec:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
- name: sidekiq
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
|
||||
@@ -18,7 +18,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: whisper
|
||||
image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
|
||||
args:
|
||||
- --model=base-int8
|
||||
- --language=de
|
||||
|
||||
@@ -34,7 +34,7 @@ spec:
|
||||
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
|
||||
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
|
||||
# track the same range/filter so they bump together on each release.
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
|
||||
env:
|
||||
- name: WOODPECKER_SERVER
|
||||
value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
|
||||
|
||||
@@ -40,7 +40,7 @@ spec:
|
||||
# minor + patch auto-update within 3.x; only the 4.0 major is held for
|
||||
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
|
||||
# -bump hazard is gone). Stays in lockstep with the agent.
|
||||
image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
|
||||
env:
|
||||
- name: WOODPECKER_HOST
|
||||
value: https://ci.renner.uno
|
||||
|
||||
+39
-2
@@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
|
||||
# Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
|
||||
# would not match the amd64 schematic this cluster installs. That leaves the
|
||||
# cx*/cpx*2 families in the German locations.
|
||||
# Two segments only, unlike the homelab's split in
|
||||
# NETWORK-traffic-classes.md: here the node address carries management,
|
||||
# Today's traffic-class split, not the target one in
|
||||
# PROPOSAL-traffic-class-separation.md: the node address carries management,
|
||||
# cluster-internal and external alike, and Ceph replication has its own.
|
||||
#
|
||||
# The order is load-bearing: it decides which kernel interface each segment
|
||||
@@ -1303,6 +1303,32 @@ EOF
|
||||
done
|
||||
ok
|
||||
|
||||
# CNI=antrea swaps the CNI for a throwaway evaluation cluster only — see
|
||||
# kubernetes/cni-eval/README.md. Unset, nothing below changes. Refused on
|
||||
# the production nodes: this installs a CNI Flux does not manage, so a real
|
||||
# cluster would come up with a network no HelmRelease owns.
|
||||
if [[ "${CNI:-cilium}" == "antrea" ]]; then
|
||||
[[ "$SITE" != "homelab" ]] \
|
||||
|| fail "CNI=antrea is for throwaway clusters (vmtest/hcloudtest), not the homelab site"
|
||||
step 0 "Installing Antrea (CNI) — evaluation override..."
|
||||
ANTREA_VALUES="$REPO_ROOT/kubernetes/cni-eval/antrea-values.yaml"
|
||||
[[ -f "$ANTREA_VALUES" ]] || fail "antrea values not found at $ANTREA_VALUES"
|
||||
# >=2.6 is a hard floor, not a preference: earlier agents fail to start
|
||||
# on Talos >=1.7 after its iptables-legacy to iptables-nft switch.
|
||||
helm repo add antrea https://charts.antrea.io >/dev/null
|
||||
helm repo update antrea >/dev/null
|
||||
helm upgrade --install antrea antrea/antrea \
|
||||
--version "${ANTREA_VERSION:->=2.6.0}" \
|
||||
--namespace kube-system \
|
||||
-f "$ANTREA_VALUES" \
|
||||
--wait --timeout 10m
|
||||
ok
|
||||
# Skips the Cilium install below without duplicating the node-Ready
|
||||
# wait that follows it.
|
||||
CNI_INSTALLED=1
|
||||
fi
|
||||
|
||||
if [[ "${CNI_INSTALLED:-0}" != "1" ]]; then
|
||||
step 0 "Installing Cilium (CNI)..."
|
||||
# Must run before the node-Ready wait below: with `cni: none` the kubelet
|
||||
# reports NotReady until a CNI is present, so that loop would time out.
|
||||
@@ -1331,6 +1357,7 @@ EOF
|
||||
--wait --timeout 10m
|
||||
rm -f "$CILIUM_VALUES"
|
||||
ok
|
||||
fi
|
||||
|
||||
step 0 "Waiting for K8s nodes..."
|
||||
expected=${#ALL_NODES[@]}
|
||||
@@ -1392,6 +1419,16 @@ EOF
|
||||
echo -e " $vm_node: partition and label ${GREEN}(OK)${NC}"
|
||||
done
|
||||
|
||||
# The CNI evaluation runs here, inside the same process: KUBECONFIG
|
||||
# points at the throwaway cluster only for the life of this phase, and
|
||||
# the VMs are destroyed the moment it ends.
|
||||
if [[ "${CNI:-cilium}" == "antrea" ]]; then
|
||||
step 0 "Running the Antrea BGP evaluation..."
|
||||
"$REPO_ROOT/kubernetes/cni-eval/run-bgp-test.sh" \
|
||||
|| fail "Antrea BGP evaluation failed — see above"
|
||||
ok
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}VM rebuild test passed.${NC} ${#ALL_NODES[@]} nodes Ready on the repo's own config."
|
||||
else
|
||||
mkdir -p "$ACCESS_STAGING"
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# CNI evaluation — Antrea
|
||||
|
||||
Throwaway-cluster harness for deciding whether Antrea can replace Cilium.
|
||||
Nothing here is reconciled by Flux and nothing here runs against the homelab.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Cilium ties pod east-west traffic to the kubelet's Kubernetes node IP and gives
|
||||
no way to separate them. Three attempts on the live cluster, 2026-09-08:
|
||||
|
||||
| Attempt | Result |
|
||||
|---|---|
|
||||
| `--ipv4-node`/`--ipv6-node` via `CiliumNodeConfig` | agent logged both flags at startup, still published its `mgmt0` address |
|
||||
| `directRoutingDevice: pod0` | agent logged `Direct routing device detected direct-routing-device=pod0`, still installed `10.244.2.0/24 via 10.2.40.138 dev mgmt0` |
|
||||
| tunnel and native mode, v4 and v6 | routes and tunnel endpoints always keyed to the peer's `InternalIP` |
|
||||
|
||||
That blocks issue #576: pod traffic cannot leave the management VLAN while the
|
||||
apiserver endpoints have to stay on it, because both follow the same address.
|
||||
|
||||
Antrea documents one setting covering both paths — *"The IP address used for
|
||||
tunneling **or** routing the traffic across Nodes is decided in the following
|
||||
order of preference: 1. TransportInterface 2. TransportInterfaceCIDRs 3. The
|
||||
Node IP."* Calico can do it too. Antrea fits the rest of the requirements
|
||||
better, with one exception: **its BGP is alpha**, and BGP is load-bearing here —
|
||||
every LoadBalancer VIP reaches the LAN through it.
|
||||
|
||||
So the question this harness answers is narrow: **does Antrea's alpha
|
||||
`BGPPolicy` actually hold a session with FRR and advertise what the homelab
|
||||
needs?** If it does not, the answer is Calico and this goes no further.
|
||||
|
||||
## Stage 1 — free, local
|
||||
|
||||
The QEMU provisioner gives one flat NIC per VM (`site-qemu.yaml`: *"One NIC, no
|
||||
VLANs"*), so this stage cannot test interface pinning. It tests everything else,
|
||||
at no cost:
|
||||
|
||||
Runs from the `feat/antrea-eval` branch, which is never merged:
|
||||
`bootstrap.sh` on master is the production rebuild path.
|
||||
|
||||
**In the cluster — the way to run it.** The VMs want ~40 GiB, and
|
||||
`kubernetes/vmtest-job.yaml` records that running this on the workstation
|
||||
OOM-killed the desktop session. The eval Job is that same Job with three values
|
||||
changed: the branch, `CNI=antrea`, and the clone source — the GitHub mirror is
|
||||
private and the stock Job clones it anonymously, so this one clones the
|
||||
anonymously-readable Gitea repo instead.
|
||||
|
||||
```bash
|
||||
yq -y '(select(.kind == "Job") | .spec.template.spec.containers[0].env[] | select(.name == "BRANCH") | .value) = "feat/antrea-eval"
|
||||
| (select(.kind == "Job") | .spec.template.spec.containers[0].env) += [{"name": "CNI", "value": "antrea"}]
|
||||
| (select(.kind == "Job") | .spec.template.spec.containers[0].args[0]) |= sub("https://github.com/konairius/Homelab.git"; "https://gitea.renner.uno/k.renner/homelab.git")' \
|
||||
kubernetes/vmtest-job.yaml | kubectl apply -f -
|
||||
|
||||
kubectl -n vmtest logs -f job/vmtest
|
||||
```
|
||||
|
||||
`-y` is python-yq; go-yq emits YAML without it. The Talos-only phase needs one
|
||||
secret, `ghcr-pull`; the log prints the exact command to stream it in. The BGP
|
||||
evaluation runs inside the Job once the nodes are Ready, so PASS/FAIL is in the
|
||||
log and the VMs are destroyed when it ends. The Job cannot reach this cluster:
|
||||
it mounts no service account token. Clean up with `kubectl delete namespace vmtest`.
|
||||
|
||||
**On a workstation — only with ~40 GiB to spare.** `vmtest` destroys its VMs on
|
||||
exit and exports its isolated `KUBECONFIG` only inside its own process, so hold
|
||||
them open and point the test at them explicitly:
|
||||
|
||||
```bash
|
||||
# terminal 1 — needs sudo; holds the VMs until stopped
|
||||
VMTEST_KEEP=1 CNI=antrea ./kubernetes/bootstrap.sh vmtest
|
||||
# it prints: talosctl --talosconfig /tmp/tmp.XXXXXX/talosconfig ...
|
||||
|
||||
# terminal 2 — the kubeconfig sits next to that talosconfig
|
||||
KUBECONFIG=/tmp/tmp.XXXXXX/kubeconfig ./kubernetes/cni-eval/run-bgp-test.sh
|
||||
|
||||
# teardown
|
||||
sudo talosctl cluster destroy --name bootstrap-vmtest --state /tmp/bootstrap-vmtest
|
||||
sudo rm -rf /tmp/bootstrap-vmtest
|
||||
```
|
||||
|
||||
`run-bgp-test.sh` refuses to run without an explicit `KUBECONFIG`, and refuses
|
||||
any cluster whose API is `k8s.renner.uno` or on 10.2.40.0/24, or whose nodes are
|
||||
named `k8s-1..3`.
|
||||
|
||||
What passing means:
|
||||
|
||||
- Antrea installs and runs on Talos at all (needs >= 2.6 — earlier agents fail
|
||||
to start on Talos >= 1.7 after its iptables-nft switch).
|
||||
- The cluster reaches Ready with `cluster.proxy.disabled: true`, i.e. AntreaProxy
|
||||
genuinely replaces kube-proxy, reaching the API through KubePrism.
|
||||
- `ServiceExternalIP` allocates a LoadBalancer IP.
|
||||
- `BGPPolicy` establishes with FRR from every node and advertises the pod CIDRs
|
||||
and the LoadBalancer IP — verified by reading FRR's table, not Antrea's.
|
||||
|
||||
`run-bgp-test.sh` deploys FRR as an ordinary pod standing in for the UDM: same
|
||||
ASNs as `unifi/bgp.conf` (64512/64513), dynamic neighbours via a listen range
|
||||
rather than named node addresses. Its peer address is read from the running pod
|
||||
at apply time rather than written into the manifest.
|
||||
|
||||
## Stage 2 — billed, only if stage 1 passes
|
||||
|
||||
Hetzner nodes have two interfaces and no shared L2, so this is the only place
|
||||
the two remaining requirements can be tested:
|
||||
|
||||
```bash
|
||||
CNI=antrea ./kubernetes/bootstrap.sh hcloudtest
|
||||
```
|
||||
|
||||
Uncomment `transportInterface` and `trafficEncapMode: hybrid` in
|
||||
`antrea-values.yaml` first, pointing the former at the private-network NIC while
|
||||
`site-hcloud.yaml` keeps the kubelet node IP where it is. Then confirm:
|
||||
|
||||
- pod east-west leaves on the nominated interface while the node IP does not
|
||||
move — the property Cilium cannot express;
|
||||
- `hybrid` encapsulates across a routed network, which is what the Hetzner
|
||||
cluster needs and the homelab does not.
|
||||
|
||||
`hcloudtest` bills by the hour on three cx43s and tears down from its EXIT trap.
|
||||
If a run is killed before the trap fires, `./kubernetes/bootstrap.sh hcloudclean`
|
||||
deletes by label.
|
||||
|
||||
## What this harness does not cover
|
||||
|
||||
- **Policy migration.** 59 `CiliumNetworkPolicy` objects across 34 files, using
|
||||
`fromEntities`/`toEntities` in 15, typed ICMP in 24, and `toFQDNs` in
|
||||
`trmnl-ns.yaml`. Antrea has equivalents including FQDN egress in OSS, but the
|
||||
port is real work and is not tested here.
|
||||
- **Multus.** Required either way — Antrea's native secondary networks are VLAN
|
||||
and SR-IOV only, and all six NADs are macvlan or host-device. Unchanged by a
|
||||
CNI swap.
|
||||
- **In-place migration.** There isn't one. Talos documents CNI choice as
|
||||
bootstrap-time only, and a Sidero maintainer's guidance is *"much easier to
|
||||
redeploy the cluster from scratch"*. Adopting Antrea means a cold rebuild.
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
kubectl delete namespace cni-eval
|
||||
kubectl delete bgppolicy cni-eval-peer
|
||||
kubectl delete externalippool cni-eval-lb
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Antrea Helm values for the CNI evaluation. Not reconciled by Flux; used only
|
||||
# by `CNI=antrea ./bootstrap.sh vmtest` and `hcloudtest`.
|
||||
|
||||
# Talos blocks CAP_SYS_MODULE for workloads and builds openvswitch into its
|
||||
# kernel, so the modprobe init step is skipped rather than granted.
|
||||
agent:
|
||||
dontLoadKernelModules: true
|
||||
installCNI:
|
||||
securityContext:
|
||||
capabilities: []
|
||||
|
||||
# talos/patches/cluster.yaml sets cluster.proxy.disabled, so there is no
|
||||
# kube-proxy: AntreaProxy has to serve every Service type, and the agent has to
|
||||
# reach the API server through KubePrism rather than a Service IP.
|
||||
antreaProxy:
|
||||
proxyAll: true
|
||||
kubeAPIServerOverride: "localhost:7445"
|
||||
|
||||
# BGPPolicy is alpha and off by default. ServiceExternalIP is the LoadBalancer
|
||||
# IPAM; on by default since 2.3, named here so the manifest states it.
|
||||
featureGates:
|
||||
BGPPolicy: true
|
||||
ServiceExternalIP: true
|
||||
|
||||
# Matches --service-cluster-ip-range in talos/patches/cluster.yaml.
|
||||
serviceCIDR: 10.96.0.0/12
|
||||
serviceCIDRv6: fd10:96::/112
|
||||
|
||||
# Stage 2 only (hcloudtest, two NICs). Pins pod east-west to a named interface
|
||||
# while the kubelet node IP stays where it is — the property Cilium cannot
|
||||
# express. Left unset here because the QEMU provisioner gives one flat NIC.
|
||||
# transportInterface: ""
|
||||
# trafficEncapMode: hybrid
|
||||
@@ -0,0 +1,48 @@
|
||||
# What the test asks Antrea to advertise. Applied after frr-peer.yaml.
|
||||
# __FRR_PEER_IP__ is substituted by run-bgp-test.sh from the running FRR pod —
|
||||
# not written here, because it is a lease, not an identity.
|
||||
---
|
||||
apiVersion: crd.antrea.io/v1beta1
|
||||
kind: ExternalIPPool
|
||||
metadata:
|
||||
name: cni-eval-lb
|
||||
spec:
|
||||
ipRanges:
|
||||
- start: 10.99.99.10
|
||||
end: 10.99.99.20
|
||||
nodeSelector: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: lb-probe
|
||||
namespace: cni-eval
|
||||
annotations:
|
||||
service.antrea.io/external-ip-pool: cni-eval-lb
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: frr
|
||||
ports:
|
||||
- port: 179
|
||||
targetPort: 179
|
||||
---
|
||||
apiVersion: crd.antrea.io/v1alpha1
|
||||
kind: BGPPolicy
|
||||
metadata:
|
||||
name: cni-eval-peer
|
||||
spec:
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/os: linux
|
||||
localASN: 64513
|
||||
listenPort: 179
|
||||
advertisements:
|
||||
pod: {}
|
||||
service:
|
||||
ipTypes:
|
||||
- LoadBalancerIP
|
||||
bgpPeers:
|
||||
- address: __FRR_PEER_IP__
|
||||
asn: 64512
|
||||
port: 179
|
||||
@@ -0,0 +1,106 @@
|
||||
# FRR standing in for the UDM, so Antrea's BGP can be tested without one.
|
||||
# Mirrors unifi/bgp.conf where it matters: same ASNs, dynamic neighbours via a
|
||||
# listen range rather than named node addresses.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cni-eval
|
||||
labels:
|
||||
# Talos enforces baseline, which rejects the NET_ADMIN and NET_RAW zebra
|
||||
# needs. Throwaway cluster only.
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: frr-config
|
||||
namespace: cni-eval
|
||||
data:
|
||||
daemons: |
|
||||
bgpd=yes
|
||||
zebra=yes
|
||||
vtysh_enable=yes
|
||||
bgpd_options="-A 127.0.0.1"
|
||||
zebra_options="-A 127.0.0.1 -s 90000000"
|
||||
frr.conf: |
|
||||
frr defaults traditional
|
||||
hostname frr-eval
|
||||
log stdout informational
|
||||
!
|
||||
router bgp 64512
|
||||
bgp router-id 172.31.255.1
|
||||
no bgp ebgp-requires-policy
|
||||
bgp listen range 0.0.0.0/0 peer-group K8S
|
||||
bgp listen range ::/0 peer-group K8S
|
||||
neighbor K8S peer-group
|
||||
neighbor K8S remote-as 64513
|
||||
neighbor K8S timers 30 90
|
||||
!
|
||||
address-family ipv4 unicast
|
||||
neighbor K8S activate
|
||||
neighbor K8S soft-reconfiguration inbound
|
||||
exit-address-family
|
||||
!
|
||||
address-family ipv6 unicast
|
||||
neighbor K8S activate
|
||||
neighbor K8S soft-reconfiguration inbound
|
||||
exit-address-family
|
||||
!
|
||||
line vty
|
||||
!
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: frr
|
||||
namespace: cni-eval
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: frr
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: frr
|
||||
spec:
|
||||
# /etc/frr must be writable for the daemons and vtysh; a ConfigMap mount
|
||||
# is not. Seeded from the image's own defaults plus the ConfigMap.
|
||||
initContainers:
|
||||
- name: seed-config
|
||||
image: quay.io/frrouting/frr:10.2.1
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- cp -a /etc/frr/. /frr-etc/ && cp /config/frr.conf /config/daemons /frr-etc/
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: etc
|
||||
mountPath: /frr-etc
|
||||
containers:
|
||||
- name: frr
|
||||
image: quay.io/frrouting/frr:10.2.1
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- NET_ADMIN
|
||||
- NET_BIND_SERVICE
|
||||
- NET_RAW
|
||||
ports:
|
||||
- containerPort: 179
|
||||
name: bgp
|
||||
volumeMounts:
|
||||
- name: etc
|
||||
mountPath: /etc/frr
|
||||
- name: run
|
||||
mountPath: /var/run/frr
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: frr-config
|
||||
- name: etc
|
||||
emptyDir: {}
|
||||
- name: run
|
||||
emptyDir: {}
|
||||
Executable
+118
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Verifies Antrea's alpha BGPPolicy against an FRR peer. Answers exactly one
|
||||
# question: does the session establish and carry the routes the homelab needs
|
||||
# (pod CIDRs and LoadBalancer IPs). Everything it asserts is read back from
|
||||
# FRR, not from Antrea's own view of what it advertised.
|
||||
#
|
||||
# kubernetes/cni-eval/run-bgp-test.sh
|
||||
#
|
||||
# Expects a cluster already built with CNI=antrea. Leaves the cni-eval
|
||||
# namespace behind for inspection; the cleanup line is printed at the end.
|
||||
|
||||
DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'; NC=$'\033[0m'
|
||||
fail() { echo "${RED}FAIL${NC} $*" >&2; exit 1; }
|
||||
pass() { echo "${GREEN}PASS${NC} $*"; }
|
||||
info() { echo "${YELLOW}==>${NC} $*"; }
|
||||
|
||||
# vmtest and hcloudtest export their isolated KUBECONFIG only inside their own
|
||||
# process. A shell that runs this afterwards gets the default kubeconfig, which
|
||||
# is production. Refuse unless pointed at a throwaway cluster explicitly.
|
||||
[[ -n "${KUBECONFIG:-}" ]] \
|
||||
|| fail "set KUBECONFIG to the throwaway cluster's kubeconfig — the default one is production"
|
||||
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
|
||||
case "$server" in
|
||||
*k8s.renner.uno*|*10.2.40.*) fail "KUBECONFIG points at production ($server) — refusing" ;;
|
||||
esac
|
||||
node_names=$(kubectl get nodes -o name)
|
||||
if grep -qx 'node/k8s-[123]' <<<"$node_names"; then
|
||||
fail "this cluster has production node names — refusing"
|
||||
fi
|
||||
pass "target is a throwaway cluster ($server)"
|
||||
|
||||
info "Checking Antrea is the running CNI..."
|
||||
kubectl -n kube-system get daemonset antrea-agent >/dev/null \
|
||||
|| fail "antrea-agent DaemonSet not found — was the cluster built with CNI=antrea?"
|
||||
|
||||
info "Confirming the BGPPolicy feature gate is on..."
|
||||
gates=$(kubectl -n kube-system get configmap antrea-config -o jsonpath='{.data.antrea-agent\.conf}')
|
||||
grep -q 'BGPPolicy: *true' <<<"$gates" \
|
||||
|| fail "BGPPolicy feature gate is not enabled in antrea-config"
|
||||
pass "BGPPolicy enabled"
|
||||
|
||||
info "Deploying the FRR peer..."
|
||||
kubectl apply -f "$DIR/frr-peer.yaml"
|
||||
kubectl -n cni-eval rollout status deployment/frr --timeout=180s \
|
||||
|| fail "FRR did not become ready"
|
||||
|
||||
# Read the address rather than assign one: the peer is a pod, and its address
|
||||
# is whatever the CNI under test gave it.
|
||||
frr_ip=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].status.podIP}')
|
||||
[[ -n "$frr_ip" ]] || fail "FRR pod has no IP"
|
||||
pass "FRR peer at $frr_ip"
|
||||
|
||||
info "Applying ExternalIPPool, LoadBalancer Service and BGPPolicy..."
|
||||
sed "s|__FRR_PEER_IP__|$frr_ip|" "$DIR/bgp-test.yaml" | kubectl apply -f -
|
||||
|
||||
info "Waiting for a LoadBalancer IP (ServiceExternalIP)..."
|
||||
lb=""
|
||||
for _ in $(seq 1 30); do
|
||||
lb=$(kubectl -n cni-eval get svc lb-probe -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
|
||||
[[ -n "$lb" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ -n "$lb" ]] || fail "no LoadBalancer IP assigned after 150s — ServiceExternalIP did not allocate"
|
||||
pass "LoadBalancer IP $lb"
|
||||
|
||||
frr_pod=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].metadata.name}')
|
||||
|
||||
info "Waiting for BGP sessions to establish..."
|
||||
nodes=$(kubectl get nodes -o name | wc -l)
|
||||
established=0
|
||||
for _ in $(seq 1 36); do
|
||||
# The text summary shows an established peer as a prefix count, not a state
|
||||
# name; the per-neighbour JSON carries bgpState explicitly.
|
||||
neighbors=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp neighbor json' || true)
|
||||
established=$(jq '[.[] | select(type == "object" and .bgpState == "Established")] | length' <<<"$neighbors" || echo 0)
|
||||
established=${established:-0}
|
||||
[[ "$established" -ge "$nodes" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp summary'
|
||||
[[ "$established" -ge "$nodes" ]] \
|
||||
|| fail "only $established of $nodes sessions established after 180s"
|
||||
pass "$established/$nodes sessions established"
|
||||
|
||||
info "Checking the advertised routes as FRR sees them..."
|
||||
kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp'
|
||||
routes=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp json')
|
||||
|
||||
# Exact prefix keys rather than substring matches.
|
||||
advertised() { jq -e --arg p "$1" '.routes | has($p)' <<<"$routes" >/dev/null; }
|
||||
|
||||
pod_cidrs=$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}')
|
||||
missing=0
|
||||
while read -r cidr; do
|
||||
[[ -z "$cidr" ]] && continue
|
||||
if advertised "$cidr"; then
|
||||
pass "pod CIDR $cidr advertised"
|
||||
else
|
||||
echo "${RED}FAIL${NC} pod CIDR $cidr NOT advertised"
|
||||
missing=1
|
||||
fi
|
||||
done <<<"$pod_cidrs"
|
||||
|
||||
if grep -qF "$lb" <<<"$routes"; then
|
||||
pass "LoadBalancer IP $lb advertised"
|
||||
else
|
||||
echo "${RED}FAIL${NC} LoadBalancer IP $lb NOT advertised"
|
||||
missing=1
|
||||
fi
|
||||
|
||||
[[ "$missing" -eq 0 ]] || fail "one or more expected prefixes were not advertised"
|
||||
|
||||
echo
|
||||
pass "Antrea BGPPolicy carried pod CIDRs and LoadBalancer IPs to FRR."
|
||||
echo "Clean up: kubectl delete namespace cni-eval && kubectl delete bgppolicy cni-eval-peer && kubectl delete externalippool cni-eval-lb"
|
||||
@@ -1,67 +0,0 @@
|
||||
# Identical on every node and free of addresses. defines.conf and peers.conf
|
||||
# are rendered per node by the init container; the first is included above the
|
||||
# templates that reference it, the second below them.
|
||||
include "/etc/bird/defines.conf";
|
||||
|
||||
router id from "pod0";
|
||||
log stderr all;
|
||||
|
||||
protocol device { scan time 10; }
|
||||
|
||||
# This node's own pod CIDR, originated rather than read back out of the kernel.
|
||||
# Reading it back made BIRD's view depend on the shape of Cilium's route, and
|
||||
# the IPv4 one is rejected outright: its next hop sits inside the prefix it
|
||||
# describes, which KRT refuses with "strange next-hop". The prefix comes from
|
||||
# the Node object either way, so there is nothing to learn.
|
||||
#
|
||||
# blackhole because this route is only ever advertised. Nothing installs it:
|
||||
# the only kernel protocol exporting from these tables takes BGP routes.
|
||||
protocol static own4 {
|
||||
ipv4 { table master4; };
|
||||
route OWN_POD4 blackhole;
|
||||
}
|
||||
protocol static own6 {
|
||||
ipv6 { table master6; };
|
||||
route OWN_POD6 blackhole;
|
||||
}
|
||||
|
||||
# Only BGP-learned routes reach table 48 — not the static above, and not
|
||||
# anything else that may end up in the master tables.
|
||||
filter to_t48 {
|
||||
if source != RTS_BGP then reject;
|
||||
accept;
|
||||
}
|
||||
|
||||
protocol kernel k48v4 {
|
||||
kernel table 48;
|
||||
metric 0;
|
||||
ipv4 { table master4; import none; export filter to_t48; };
|
||||
}
|
||||
protocol kernel k48v6 {
|
||||
kernel table 48;
|
||||
metric 0;
|
||||
ipv6 { table master6; import none; export filter to_t48; };
|
||||
}
|
||||
|
||||
# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
|
||||
# from ingress0 and is unrelated.
|
||||
#
|
||||
# direct is explicit because iBGP defaults to multihop, which makes the gateway
|
||||
# recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
|
||||
# keeps the listener off mgmt0 and ingress0.
|
||||
template bgp pod4 {
|
||||
local as 64514;
|
||||
source address MY4;
|
||||
direct;
|
||||
strict bind;
|
||||
ipv4 { next hop self; import all; export all; };
|
||||
}
|
||||
template bgp pod6 {
|
||||
local as 64514;
|
||||
source address MY6;
|
||||
direct;
|
||||
strict bind;
|
||||
ipv6 { next hop self; import all; export all; };
|
||||
}
|
||||
|
||||
include "/etc/bird/peers.conf";
|
||||
@@ -1,115 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: bird
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: bird
|
||||
rules:
|
||||
# The peer list is read from the nodes' pod0 annotations, so no address is
|
||||
# written into this repo twice.
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["get", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: bird
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: bird
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: bird
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: bird
|
||||
namespace: kube-system
|
||||
labels:
|
||||
tier: node
|
||||
app: bird
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: bird
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
tier: node
|
||||
app: bird
|
||||
spec:
|
||||
# Carries pod east-west once the routing rules land. Without it table 48
|
||||
# stays empty and the rules fall through to main, i.e. back to VLAN 40.
|
||||
priorityClassName: system-node-critical
|
||||
# Writes host routes and peers on host addresses.
|
||||
hostNetwork: true
|
||||
serviceAccountName: bird
|
||||
# Every node here is a control plane node, so this is not optional.
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
initContainers:
|
||||
- name: render-config
|
||||
# By digest, not by the dated tag: the tag is rebuilt within the day
|
||||
# it names, and the cluster's image path keeps resolving it to the
|
||||
# first build of that day — imagePullPolicy: Always does not help,
|
||||
# because the tag itself resolves stale. Update this when the image
|
||||
# is rebuilt.
|
||||
image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
cp /config/bird.conf /etc/bird/bird.conf
|
||||
kubectl get node "$NODE_NAME" \
|
||||
-o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
|
||||
kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
|
||||
-o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
|
||||
# BIRD's own parser is the check: a node whose annotation is
|
||||
# missing renders as <no value> and fails here rather than
|
||||
# crashlooping the daemon or forming a mesh with a hole.
|
||||
bird -p -c /etc/bird/bird.conf
|
||||
env:
|
||||
- name: NODE_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.nodeName
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: etc-bird
|
||||
mountPath: /etc/bird
|
||||
containers:
|
||||
- name: bird
|
||||
image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
|
||||
resources:
|
||||
requests:
|
||||
cpu: "50m"
|
||||
memory: "32Mi"
|
||||
limits:
|
||||
cpu: "200m"
|
||||
memory: "128Mi"
|
||||
securityContext:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
# NET_ADMIN writes the routes; NET_BIND_SERVICE binds 179.
|
||||
add: ["NET_ADMIN", "NET_BIND_SERVICE"]
|
||||
volumeMounts:
|
||||
- name: etc-bird
|
||||
mountPath: /etc/bird
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: bird-config
|
||||
- name: etc-bird
|
||||
emptyDir: {}
|
||||
@@ -1,7 +0,0 @@
|
||||
define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
|
||||
define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
|
||||
{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
|
||||
IPv6 prefix into an ipv4 protocol, which fails the parse in the init
|
||||
container rather than passing silently. */}}
|
||||
define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
|
||||
define OWN_POD6 = {{ index .spec.podCIDRs 1 }};
|
||||
@@ -1,12 +0,0 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- daemonset.yaml
|
||||
configMapGenerator:
|
||||
# Hashed name, so editing any of these rolls the DaemonSet.
|
||||
- name: bird-config
|
||||
namespace: kube-system
|
||||
files:
|
||||
- bird.conf
|
||||
- defines.tmpl
|
||||
- peers.tmpl
|
||||
@@ -1,7 +0,0 @@
|
||||
{{- range $i, $n := .items -}}
|
||||
{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
|
||||
{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
|
||||
# {{ $n.metadata.name }}
|
||||
protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
|
||||
protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
|
||||
{{ end -}}
|
||||
@@ -180,17 +180,3 @@ spec:
|
||||
# the supernets here cover the LAN as well, which is the point.
|
||||
ipv4NativeRoutingCIDR: 10.0.0.0/8
|
||||
ipv6NativeRoutingCIDR: fd00::/8
|
||||
# Pod east-west leaves the node on pod0 (VLAN 48) rather than the
|
||||
# management VLAN. Native routing hands forwarding to the kernel;
|
||||
# autoDirectNodeRoutes installs the remote PodCIDR routes, which is
|
||||
# allowed because all three nodes share pod0's L2 segment.
|
||||
#
|
||||
# directRoutingDevice is explicit on purpose: with several addressed
|
||||
# devices Cilium falls back to the one carrying the Kubernetes InternalIP,
|
||||
# which is mgmt0.
|
||||
#
|
||||
# Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
|
||||
routingMode: native
|
||||
autoDirectNodeRoutes: true
|
||||
nodePort:
|
||||
directRoutingDevice: pod0
|
||||
|
||||
@@ -7,7 +7,6 @@ resources:
|
||||
- cluster-settings/
|
||||
- cilium/
|
||||
- multus/
|
||||
- bird/
|
||||
- cert-manager/
|
||||
- snapshot-controller/
|
||||
- metrics-server/
|
||||
|
||||
@@ -52,12 +52,6 @@ machine:
|
||||
# this label instead of listing hostnames, so which boxes carry storage is
|
||||
# stated once, here, by the machine that knows.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# Publishes this node's pod0 addresses, declared below in its VLANConfig, so
|
||||
# the bird DaemonSet reads its peers from the node list rather than from a
|
||||
# copy. Annotations, not labels: a label value cannot contain a colon.
|
||||
nodeAnnotations:
|
||||
net.renner.uno/pod0-v6: fd02:48::11
|
||||
net.renner.uno/pod0-v4: 10.2.48.11
|
||||
udev:
|
||||
# Renames every port this node uses to its role at boot, matched by
|
||||
# burned-in MAC — kernel names derive from PCI topology and shift when
|
||||
@@ -188,14 +182,8 @@ addresses:
|
||||
# network's clients selectively, so the client goes rather than the record.
|
||||
# DHCP is off on VLAN 47 for the same reason.
|
||||
#
|
||||
# Both defaults live in table 47, not in main, so only the pod-source rules in
|
||||
# site-homelab.yaml reach them. mgmt0 keeps main's default and with it every
|
||||
# reply the node sends from its own addresses.
|
||||
#
|
||||
# The v6 gateway is the UDM's configured ULA on this segment, the same address
|
||||
# BGP peers with. The RA-learned link-local would be the alternative and is not
|
||||
# a value to write down. accept_ra_defrtr stays 0 for that reason: the RA
|
||||
# default would land in main, where it does not belong.
|
||||
# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
|
||||
# the segment. The default route moves here in the change that flips it.
|
||||
apiVersion: v1alpha1
|
||||
kind: VLANConfig
|
||||
name: ingress0
|
||||
@@ -205,10 +193,8 @@ up: true
|
||||
addresses:
|
||||
- address: 10.2.47.11/24
|
||||
routes:
|
||||
- gateway: fd02:47::1
|
||||
table: "47"
|
||||
- gateway: 10.2.47.1
|
||||
table: "47"
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
|
||||
# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
|
||||
|
||||
@@ -33,10 +33,6 @@ machine:
|
||||
topology.kubernetes.io/zone: saturn
|
||||
# Holds a Ceph OSD drive — see k8s-1.yaml.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# pod0 addresses for the bird peer list — see k8s-1.yaml.
|
||||
nodeAnnotations:
|
||||
net.renner.uno/pod0-v6: fd02:48::12
|
||||
net.renner.uno/pod0-v4: 10.2.48.12
|
||||
udev:
|
||||
# Renames every port to its role at boot — see k8s-1.yaml.
|
||||
rules:
|
||||
@@ -133,10 +129,8 @@ up: true
|
||||
addresses:
|
||||
- address: 10.2.47.12/24
|
||||
routes:
|
||||
- gateway: fd02:47::1
|
||||
table: "47"
|
||||
- gateway: 10.2.47.1
|
||||
table: "47"
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
|
||||
# wire. OSD-to-OSD replication only, not routed off its own segment.
|
||||
|
||||
@@ -29,10 +29,6 @@ machine:
|
||||
topology.kubernetes.io/zone: neptune
|
||||
# Holds a Ceph OSD drive — see k8s-1.yaml.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# pod0 addresses for the bird peer list — see k8s-1.yaml.
|
||||
nodeAnnotations:
|
||||
net.renner.uno/pod0-v6: fd02:48::13
|
||||
net.renner.uno/pod0-v4: 10.2.48.13
|
||||
udev:
|
||||
# Renames every port to its role at boot — see k8s-1.yaml.
|
||||
rules:
|
||||
@@ -120,10 +116,8 @@ up: true
|
||||
addresses:
|
||||
- address: 10.2.47.13/24
|
||||
routes:
|
||||
- gateway: fd02:47::1
|
||||
table: "47"
|
||||
- gateway: 10.2.47.1
|
||||
table: "47"
|
||||
metric: 2048
|
||||
---
|
||||
# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
|
||||
# here on purpose; the link is handed whole to the OSD pod (host-device).
|
||||
|
||||
@@ -20,70 +20,3 @@ machine:
|
||||
validSubnets:
|
||||
- 10.2.40.0/24
|
||||
- fd40::/64
|
||||
---
|
||||
# Pod egress leaves over ingress0. Matching on the pod source prefix is what
|
||||
# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
|
||||
# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
|
||||
# source onto VLAN 47.
|
||||
#
|
||||
# The routing decision runs before Cilium's masquerade hook, so the rule sees
|
||||
# the pod address and the masquerade then rewrites it to ingress0's.
|
||||
#
|
||||
# Table 47 holds nothing but the two default routes, so the destination rules
|
||||
# below have to run first: without them a pod's packet to a service or to a pod
|
||||
# on another node also matches the source rule, finds only the default, and
|
||||
# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
|
||||
#
|
||||
# They send everything inside the cluster and the local networks back to main.
|
||||
# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
|
||||
# is not foreign traffic.
|
||||
#
|
||||
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
|
||||
# The two families are separate rule tables, so the priorities need not match;
|
||||
# the document name is the priority and must be unique.
|
||||
# Pod to pod between nodes goes over pod0, via table 48, which the bird mesh
|
||||
# fills with one route per peer pod CIDR and the peer's pod0 address as the
|
||||
# next hop. Ahead of 990/991 so it wins for that traffic, and matching source
|
||||
# AND destination so nothing else lands there: the node's own traffic to a pod
|
||||
# would otherwise leave with a pod0 source, which CiliumNode does not list, so
|
||||
# Cilium would classify it as world rather than host.
|
||||
#
|
||||
# Traffic to a pod on this node falls through — table 48 holds only the peers'
|
||||
# CIDRs, so the lookup misses and 990/991 send it to main and cilium_host.
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "980"
|
||||
src: fd10:244::/48
|
||||
dst: fd10:244::/48
|
||||
table: "48"
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "981"
|
||||
src: 10.244.0.0/16
|
||||
dst: 10.244.0.0/16
|
||||
table: "48"
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "990"
|
||||
dst: fd00::/8
|
||||
table: main
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "991"
|
||||
dst: 10.0.0.0/8
|
||||
table: main
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "1000"
|
||||
src: fd10:244::/48
|
||||
table: "47"
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: RoutingRuleConfig
|
||||
name: "1001"
|
||||
src: 10.244.0.0/16
|
||||
table: "47"
|
||||
|
||||
Reference in New Issue
Block a user