fix(cni-eval): make the Antrea test runnable, and unable to reach prod

Nothing here is merged: bootstrap.sh on master is the production rebuild
path. The test runs from this branch, in the vmtest Job.

Five defects in the first version, each found by checking it against the
real environment rather than by running it:

- run-bgp-test.sh used the default kubeconfig. vmtest exports its
  isolated KUBECONFIG only inside its own process, so a script started
  afterwards from another shell would have targeted production, with a
  single DaemonSet-existence check in the way. It now refuses without an
  explicit KUBECONFIG, refuses an API on k8s.renner.uno or 10.2.40.0/24,
  and refuses nodes named k8s-1..3.

- The procedure could not work: vmtest destroys its VMs on exit, so a
  separate test run afterwards finds no cluster. With CNI=antrea the test
  now runs inside the vmtest phase, while its KUBECONFIG is live.

- The session count read FRR's text summary for "Estab". FRR shows an
  established peer there as a prefix count, not a state name, so it would
  have reported 0 sessions against a working cluster. It now counts
  bgpState == "Established" in `show ip bgp neighbor json`, the shape
  FRR's own topotests assert.

- Route checks matched substrings of the text table. They now test exact
  prefix keys in `show ip bgp json` .routes, also the topotest shape.

- The FRR pod could not be admitted. Talos enforces PodSecurity baseline
  with only kube-system exempt (read off k8s-1's admission config), and
  baseline rejects NET_ADMIN/NET_RAW. The throwaway namespace is labelled
  privileged, and /etc/frr is seeded into a writable emptyDir by an
  initContainer rather than mounted read-only from the ConfigMap.

The README now runs this as the existing vmtest Job with three values
changed -- branch, CNI=antrea, and the clone source. The GitHub mirror the
stock Job clones is private and the Job has no credentials; the Gitea repo
is anonymously readable.

Verified before committing: antrea chart 2.6.0-2.7.0 exist, so >=2.6.0
resolves to 2.7.0; quay.io/frrouting/frr:10.2.1 exists for amd64; the
branch is on Gitea; the ci-vmtest image carries jq, kubectl and helm;
both scripts pass bash -n; the Job transform renders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/bootstrap.sh

@@ -1303,6 +1303,32 @@ EOF
     done
     ok
 
+    # CNI=antrea swaps the CNI for a throwaway evaluation cluster only — see
+    # kubernetes/cni-eval/README.md. Unset, nothing below changes. Refused on
+    # the production nodes: this installs a CNI Flux does not manage, so a real
+    # cluster would come up with a network no HelmRelease owns.
+    if [[ "${CNI:-cilium}" == "antrea" ]]; then
+        [[ "$SITE" != "homelab" ]] \
+            || fail "CNI=antrea is for throwaway clusters (vmtest/hcloudtest), not the homelab site"
+        step 0 "Installing Antrea (CNI) — evaluation override..."
+        ANTREA_VALUES="$REPO_ROOT/kubernetes/cni-eval/antrea-values.yaml"
+        [[ -f "$ANTREA_VALUES" ]] || fail "antrea values not found at $ANTREA_VALUES"
+        # >=2.6 is a hard floor, not a preference: earlier agents fail to start
+        # on Talos >=1.7 after its iptables-legacy to iptables-nft switch.
+        helm repo add antrea https://charts.antrea.io >/dev/null
+        helm repo update antrea >/dev/null
+        helm upgrade --install antrea antrea/antrea \
+            --version "${ANTREA_VERSION:->=2.6.0}" \
+            --namespace kube-system \
+            -f "$ANTREA_VALUES" \
+            --wait --timeout 10m
+        ok
+        # Skips the Cilium install below without duplicating the node-Ready
+        # wait that follows it.
+        CNI_INSTALLED=1
+    fi
+
+    if [[ "${CNI_INSTALLED:-0}" != "1" ]]; then
     step 0 "Installing Cilium (CNI)..."
     # Must run before the node-Ready wait below: with `cni: none` the kubelet
     # reports NotReady until a CNI is present, so that loop would time out.
@@ -1331,6 +1357,7 @@ EOF
         --wait --timeout 10m
     rm -f "$CILIUM_VALUES"
     ok
+    fi
 
     step 0 "Waiting for K8s nodes..."
     expected=${#ALL_NODES[@]}
@@ -1392,6 +1419,16 @@ EOF
             echo -e "  $vm_node: partition and label ${GREEN}(OK)${NC}"
         done
 
+        # The CNI evaluation runs here, inside the same process: KUBECONFIG
+        # points at the throwaway cluster only for the life of this phase, and
+        # the VMs are destroyed the moment it ends.
+        if [[ "${CNI:-cilium}" == "antrea" ]]; then
+            step 0 "Running the Antrea BGP evaluation..."
+            "$REPO_ROOT/kubernetes/cni-eval/run-bgp-test.sh" \
+                || fail "Antrea BGP evaluation failed — see above"
+            ok
+        fi
+
         echo -e "${GREEN}VM rebuild test passed.${NC} ${#ALL_NODES[@]} nodes Ready on the repo's own config."
     else
         mkdir -p "$ACCESS_STAGING"

kubernetes/cni-eval/README.md

@@ -0,0 +1,139 @@
+# CNI evaluation — Antrea
+
+Throwaway-cluster harness for deciding whether Antrea can replace Cilium.
+Nothing here is reconciled by Flux and nothing here runs against the homelab.
+
+## Why this exists
+
+Cilium ties pod east-west traffic to the kubelet's Kubernetes node IP and gives
+no way to separate them. Three attempts on the live cluster, 2026-09-08:
+
+| Attempt | Result |
+|---|---|
+| `--ipv4-node`/`--ipv6-node` via `CiliumNodeConfig` | agent logged both flags at startup, still published its `mgmt0` address |
+| `directRoutingDevice: pod0` | agent logged `Direct routing device detected direct-routing-device=pod0`, still installed `10.244.2.0/24 via 10.2.40.138 dev mgmt0` |
+| tunnel and native mode, v4 and v6 | routes and tunnel endpoints always keyed to the peer's `InternalIP` |
+
+That blocks issue #576: pod traffic cannot leave the management VLAN while the
+apiserver endpoints have to stay on it, because both follow the same address.
+
+Antrea documents one setting covering both paths — *"The IP address used for
+tunneling **or** routing the traffic across Nodes is decided in the following
+order of preference: 1. TransportInterface 2. TransportInterfaceCIDRs 3. The
+Node IP."* Calico can do it too. Antrea fits the rest of the requirements
+better, with one exception: **its BGP is alpha**, and BGP is load-bearing here —
+every LoadBalancer VIP reaches the LAN through it.
+
+So the question this harness answers is narrow: **does Antrea's alpha
+`BGPPolicy` actually hold a session with FRR and advertise what the homelab
+needs?** If it does not, the answer is Calico and this goes no further.
+
+## Stage 1 — free, local
+
+The QEMU provisioner gives one flat NIC per VM (`site-qemu.yaml`: *"One NIC, no
+VLANs"*), so this stage cannot test interface pinning. It tests everything else,
+at no cost:
+
+Runs from the `feat/antrea-eval` branch, which is never merged:
+`bootstrap.sh` on master is the production rebuild path.
+
+**In the cluster — the way to run it.** The VMs want ~40 GiB, and
+`kubernetes/vmtest-job.yaml` records that running this on the workstation
+OOM-killed the desktop session. The eval Job is that same Job with three values
+changed: the branch, `CNI=antrea`, and the clone source — the GitHub mirror is
+private and the stock Job clones it anonymously, so this one clones the
+anonymously-readable Gitea repo instead.
+
+```bash
+yq -y '(select(.kind == "Job") | .spec.template.spec.containers[0].env[] | select(.name == "BRANCH") | .value) = "feat/antrea-eval"
+  | (select(.kind == "Job") | .spec.template.spec.containers[0].env) += [{"name": "CNI", "value": "antrea"}]
+  | (select(.kind == "Job") | .spec.template.spec.containers[0].args[0]) |= sub("https://github.com/konairius/Homelab.git"; "https://gitea.renner.uno/k.renner/homelab.git")' \
+  kubernetes/vmtest-job.yaml | kubectl apply -f -
+
+kubectl -n vmtest logs -f job/vmtest
+```
+
+`-y` is python-yq; go-yq emits YAML without it. The Talos-only phase needs one
+secret, `ghcr-pull`; the log prints the exact command to stream it in. The BGP
+evaluation runs inside the Job once the nodes are Ready, so PASS/FAIL is in the
+log and the VMs are destroyed when it ends. The Job cannot reach this cluster:
+it mounts no service account token. Clean up with `kubectl delete namespace vmtest`.
+
+**On a workstation — only with ~40 GiB to spare.** `vmtest` destroys its VMs on
+exit and exports its isolated `KUBECONFIG` only inside its own process, so hold
+them open and point the test at them explicitly:
+
+```bash
+# terminal 1 — needs sudo; holds the VMs until stopped
+VMTEST_KEEP=1 CNI=antrea ./kubernetes/bootstrap.sh vmtest
+# it prints: talosctl --talosconfig /tmp/tmp.XXXXXX/talosconfig ...
+
+# terminal 2 — the kubeconfig sits next to that talosconfig
+KUBECONFIG=/tmp/tmp.XXXXXX/kubeconfig ./kubernetes/cni-eval/run-bgp-test.sh
+
+# teardown
+sudo talosctl cluster destroy --name bootstrap-vmtest --state /tmp/bootstrap-vmtest
+sudo rm -rf /tmp/bootstrap-vmtest
+```
+
+`run-bgp-test.sh` refuses to run without an explicit `KUBECONFIG`, and refuses
+any cluster whose API is `k8s.renner.uno` or on 10.2.40.0/24, or whose nodes are
+named `k8s-1..3`.
+
+What passing means:
+
+- Antrea installs and runs on Talos at all (needs >= 2.6 — earlier agents fail
+  to start on Talos >= 1.7 after its iptables-nft switch).
+- The cluster reaches Ready with `cluster.proxy.disabled: true`, i.e. AntreaProxy
+  genuinely replaces kube-proxy, reaching the API through KubePrism.
+- `ServiceExternalIP` allocates a LoadBalancer IP.
+- `BGPPolicy` establishes with FRR from every node and advertises the pod CIDRs
+  and the LoadBalancer IP — verified by reading FRR's table, not Antrea's.
+
+`run-bgp-test.sh` deploys FRR as an ordinary pod standing in for the UDM: same
+ASNs as `unifi/bgp.conf` (64512/64513), dynamic neighbours via a listen range
+rather than named node addresses. Its peer address is read from the running pod
+at apply time rather than written into the manifest.
+
+## Stage 2 — billed, only if stage 1 passes
+
+Hetzner nodes have two interfaces and no shared L2, so this is the only place
+the two remaining requirements can be tested:
+
+```bash
+CNI=antrea ./kubernetes/bootstrap.sh hcloudtest
+```
+
+Uncomment `transportInterface` and `trafficEncapMode: hybrid` in
+`antrea-values.yaml` first, pointing the former at the private-network NIC while
+`site-hcloud.yaml` keeps the kubelet node IP where it is. Then confirm:
+
+- pod east-west leaves on the nominated interface while the node IP does not
+  move — the property Cilium cannot express;
+- `hybrid` encapsulates across a routed network, which is what the Hetzner
+  cluster needs and the homelab does not.
+
+`hcloudtest` bills by the hour on three cx43s and tears down from its EXIT trap.
+If a run is killed before the trap fires, `./kubernetes/bootstrap.sh hcloudclean`
+deletes by label.
+
+## What this harness does not cover
+
+- **Policy migration.** 59 `CiliumNetworkPolicy` objects across 34 files, using
+  `fromEntities`/`toEntities` in 15, typed ICMP in 24, and `toFQDNs` in
+  `trmnl-ns.yaml`. Antrea has equivalents including FQDN egress in OSS, but the
+  port is real work and is not tested here.
+- **Multus.** Required either way — Antrea's native secondary networks are VLAN
+  and SR-IOV only, and all six NADs are macvlan or host-device. Unchanged by a
+  CNI swap.
+- **In-place migration.** There isn't one. Talos documents CNI choice as
+  bootstrap-time only, and a Sidero maintainer's guidance is *"much easier to
+  redeploy the cluster from scratch"*. Adopting Antrea means a cold rebuild.
+
+## Cleanup
+
+```bash
+kubectl delete namespace cni-eval
+kubectl delete bgppolicy cni-eval-peer
+kubectl delete externalippool cni-eval-lb
+```

kubernetes/cni-eval/antrea-values.yaml

@@ -0,0 +1,33 @@
+# Antrea Helm values for the CNI evaluation. Not reconciled by Flux; used only
+# by `CNI=antrea ./bootstrap.sh vmtest` and `hcloudtest`.
+
+# Talos blocks CAP_SYS_MODULE for workloads and builds openvswitch into its
+# kernel, so the modprobe init step is skipped rather than granted.
+agent:
+  dontLoadKernelModules: true
+  installCNI:
+    securityContext:
+      capabilities: []
+
+# talos/patches/cluster.yaml sets cluster.proxy.disabled, so there is no
+# kube-proxy: AntreaProxy has to serve every Service type, and the agent has to
+# reach the API server through KubePrism rather than a Service IP.
+antreaProxy:
+  proxyAll: true
+kubeAPIServerOverride: "localhost:7445"
+
+# BGPPolicy is alpha and off by default. ServiceExternalIP is the LoadBalancer
+# IPAM; on by default since 2.3, named here so the manifest states it.
+featureGates:
+  BGPPolicy: true
+  ServiceExternalIP: true
+
+# Matches --service-cluster-ip-range in talos/patches/cluster.yaml.
+serviceCIDR: 10.96.0.0/12
+serviceCIDRv6: fd10:96::/112
+
+# Stage 2 only (hcloudtest, two NICs). Pins pod east-west to a named interface
+# while the kubelet node IP stays where it is — the property Cilium cannot
+# express. Left unset here because the QEMU provisioner gives one flat NIC.
+# transportInterface: ""
+# trafficEncapMode: hybrid

kubernetes/cni-eval/bgp-test.yaml

@@ -0,0 +1,48 @@
+# What the test asks Antrea to advertise. Applied after frr-peer.yaml.
+# __FRR_PEER_IP__ is substituted by run-bgp-test.sh from the running FRR pod —
+# not written here, because it is a lease, not an identity.
+---
+apiVersion: crd.antrea.io/v1beta1
+kind: ExternalIPPool
+metadata:
+  name: cni-eval-lb
+spec:
+  ipRanges:
+    - start: 10.99.99.10
+      end: 10.99.99.20
+  nodeSelector: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: lb-probe
+  namespace: cni-eval
+  annotations:
+    service.antrea.io/external-ip-pool: cni-eval-lb
+spec:
+  type: LoadBalancer
+  selector:
+    app: frr
+  ports:
+    - port: 179
+      targetPort: 179
+---
+apiVersion: crd.antrea.io/v1alpha1
+kind: BGPPolicy
+metadata:
+  name: cni-eval-peer
+spec:
+  nodeSelector:
+    matchLabels:
+      kubernetes.io/os: linux
+  localASN: 64513
+  listenPort: 179
+  advertisements:
+    pod: {}
+    service:
+      ipTypes:
+        - LoadBalancerIP
+  bgpPeers:
+    - address: __FRR_PEER_IP__
+      asn: 64512
+      port: 179

kubernetes/cni-eval/frr-peer.yaml

@@ -0,0 +1,106 @@
+# FRR standing in for the UDM, so Antrea's BGP can be tested without one.
+# Mirrors unifi/bgp.conf where it matters: same ASNs, dynamic neighbours via a
+# listen range rather than named node addresses.
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cni-eval
+  labels:
+    # Talos enforces baseline, which rejects the NET_ADMIN and NET_RAW zebra
+    # needs. Throwaway cluster only.
+    pod-security.kubernetes.io/enforce: privileged
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: frr-config
+  namespace: cni-eval
+data:
+  daemons: |
+    bgpd=yes
+    zebra=yes
+    vtysh_enable=yes
+    bgpd_options="-A 127.0.0.1"
+    zebra_options="-A 127.0.0.1 -s 90000000"
+  frr.conf: |
+    frr defaults traditional
+    hostname frr-eval
+    log stdout informational
+    !
+    router bgp 64512
+     bgp router-id 172.31.255.1
+     no bgp ebgp-requires-policy
+     bgp listen range 0.0.0.0/0 peer-group K8S
+     bgp listen range ::/0 peer-group K8S
+     neighbor K8S peer-group
+     neighbor K8S remote-as 64513
+     neighbor K8S timers 30 90
+     !
+     address-family ipv4 unicast
+      neighbor K8S activate
+      neighbor K8S soft-reconfiguration inbound
+     exit-address-family
+     !
+     address-family ipv6 unicast
+      neighbor K8S activate
+      neighbor K8S soft-reconfiguration inbound
+     exit-address-family
+    !
+    line vty
+    !
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: frr
+  namespace: cni-eval
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: frr
+  template:
+    metadata:
+      labels:
+        app: frr
+    spec:
+      # /etc/frr must be writable for the daemons and vtysh; a ConfigMap mount
+      # is not. Seeded from the image's own defaults plus the ConfigMap.
+      initContainers:
+        - name: seed-config
+          image: quay.io/frrouting/frr:10.2.1
+          command:
+            - sh
+            - -c
+            - cp -a /etc/frr/. /frr-etc/ && cp /config/frr.conf /config/daemons /frr-etc/
+          volumeMounts:
+            - name: config
+              mountPath: /config
+            - name: etc
+              mountPath: /frr-etc
+      containers:
+        - name: frr
+          image: quay.io/frrouting/frr:10.2.1
+          securityContext:
+            capabilities:
+              add:
+                - NET_ADMIN
+                - NET_BIND_SERVICE
+                - NET_RAW
+          ports:
+            - containerPort: 179
+              name: bgp
+          volumeMounts:
+            - name: etc
+              mountPath: /etc/frr
+            - name: run
+              mountPath: /var/run/frr
+      volumes:
+        - name: config
+          configMap:
+            name: frr-config
+        - name: etc
+          emptyDir: {}
+        - name: run
+          emptyDir: {}

kubernetes/cni-eval/run-bgp-test.sh

@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Verifies Antrea's alpha BGPPolicy against an FRR peer. Answers exactly one
+# question: does the session establish and carry the routes the homelab needs
+# (pod CIDRs and LoadBalancer IPs). Everything it asserts is read back from
+# FRR, not from Antrea's own view of what it advertised.
+#
+#   kubernetes/cni-eval/run-bgp-test.sh
+#
+# Expects a cluster already built with CNI=antrea. Leaves the cni-eval
+# namespace behind for inspection; the cleanup line is printed at the end.
+
+DIR="$(cd "$(dirname "$0")" && pwd)"
+RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'; NC=$'\033[0m'
+fail() { echo "${RED}FAIL${NC} $*" >&2; exit 1; }
+pass() { echo "${GREEN}PASS${NC} $*"; }
+info() { echo "${YELLOW}==>${NC} $*"; }
+
+# vmtest and hcloudtest export their isolated KUBECONFIG only inside their own
+# process. A shell that runs this afterwards gets the default kubeconfig, which
+# is production. Refuse unless pointed at a throwaway cluster explicitly.
+[[ -n "${KUBECONFIG:-}" ]] \
+    || fail "set KUBECONFIG to the throwaway cluster's kubeconfig — the default one is production"
+server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
+case "$server" in
+    *k8s.renner.uno*|*10.2.40.*) fail "KUBECONFIG points at production ($server) — refusing" ;;
+esac
+node_names=$(kubectl get nodes -o name)
+if grep -qx 'node/k8s-[123]' <<<"$node_names"; then
+    fail "this cluster has production node names — refusing"
+fi
+pass "target is a throwaway cluster ($server)"
+
+info "Checking Antrea is the running CNI..."
+kubectl -n kube-system get daemonset antrea-agent >/dev/null \
+    || fail "antrea-agent DaemonSet not found — was the cluster built with CNI=antrea?"
+
+info "Confirming the BGPPolicy feature gate is on..."
+gates=$(kubectl -n kube-system get configmap antrea-config -o jsonpath='{.data.antrea-agent\.conf}')
+grep -q 'BGPPolicy: *true' <<<"$gates" \
+    || fail "BGPPolicy feature gate is not enabled in antrea-config"
+pass "BGPPolicy enabled"
+
+info "Deploying the FRR peer..."
+kubectl apply -f "$DIR/frr-peer.yaml"
+kubectl -n cni-eval rollout status deployment/frr --timeout=180s \
+    || fail "FRR did not become ready"
+
+# Read the address rather than assign one: the peer is a pod, and its address
+# is whatever the CNI under test gave it.
+frr_ip=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].status.podIP}')
+[[ -n "$frr_ip" ]] || fail "FRR pod has no IP"
+pass "FRR peer at $frr_ip"
+
+info "Applying ExternalIPPool, LoadBalancer Service and BGPPolicy..."
+sed "s|__FRR_PEER_IP__|$frr_ip|" "$DIR/bgp-test.yaml" | kubectl apply -f -
+
+info "Waiting for a LoadBalancer IP (ServiceExternalIP)..."
+lb=""
+for _ in $(seq 1 30); do
+    lb=$(kubectl -n cni-eval get svc lb-probe -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+    [[ -n "$lb" ]] && break
+    sleep 5
+done
+[[ -n "$lb" ]] || fail "no LoadBalancer IP assigned after 150s — ServiceExternalIP did not allocate"
+pass "LoadBalancer IP $lb"
+
+frr_pod=$(kubectl -n cni-eval get pod -l app=frr -o jsonpath='{.items[0].metadata.name}')
+
+info "Waiting for BGP sessions to establish..."
+nodes=$(kubectl get nodes -o name | wc -l)
+established=0
+for _ in $(seq 1 36); do
+    # The text summary shows an established peer as a prefix count, not a state
+    # name; the per-neighbour JSON carries bgpState explicitly.
+    neighbors=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp neighbor json' || true)
+    established=$(jq '[.[] | select(type == "object" and .bgpState == "Established")] | length' <<<"$neighbors" || echo 0)
+    established=${established:-0}
+    [[ "$established" -ge "$nodes" ]] && break
+    sleep 5
+done
+kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp summary'
+[[ "$established" -ge "$nodes" ]] \
+    || fail "only $established of $nodes sessions established after 180s"
+pass "$established/$nodes sessions established"
+
+info "Checking the advertised routes as FRR sees them..."
+kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp'
+routes=$(kubectl -n cni-eval exec "$frr_pod" -- vtysh -c 'show ip bgp json')
+
+# Exact prefix keys rather than substring matches.
+advertised() { jq -e --arg p "$1" '.routes | has($p)' <<<"$routes" >/dev/null; }
+
+pod_cidrs=$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}')
+missing=0
+while read -r cidr; do
+    [[ -z "$cidr" ]] && continue
+    if advertised "$cidr"; then
+        pass "pod CIDR $cidr advertised"
+    else
+        echo "${RED}FAIL${NC} pod CIDR $cidr NOT advertised"
+        missing=1
+    fi
+done <<<"$pod_cidrs"
+
+if grep -qF "$lb" <<<"$routes"; then
+    pass "LoadBalancer IP $lb advertised"
+else
+    echo "${RED}FAIL${NC} LoadBalancer IP $lb NOT advertised"
+    missing=1
+fi
+
+[[ "$missing" -eq 0 ]] || fail "one or more expected prefixes were not advertised"
+
+echo
+pass "Antrea BGPPolicy carried pod CIDRs and LoadBalancer IPs to FRR."
+echo "Clean up: kubectl delete namespace cni-eval && kubectl delete bgppolicy cni-eval-peer && kubectl delete externalippool cni-eval-lb"

kubernetes/infrastructure/post-crds/cilium-bgp.yaml

@@ -25,13 +25,13 @@ spec:
       stop: "fd99::ff"
 ---
 # Cilium BGPv2 control-plane config peering each node with the UDM over the
-# management LAN — never over v30iot, which is the untrusted IoT segment.
+# ingress VLAN 47 — never over v30iot, which is the untrusted IoT segment.
 # The source address comes from the kernel routing decision, so no explicit
 # localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
-#   - v4 session: peer 10.2.40.1, family IPv4 unicast only
-#   - v6 session: peer fd40::1, family IPv6 unicast only
+#   - v4 session: peer 10.2.47.1, family IPv4 unicast only
+#   - v6 session: peer fd02:47::1, family IPv6 unicast only
 # This avoids the "v6 NLRIs over v4 transport" path, which requires
 # bilateral extended-nexthop capability negotiation (RFC 5549/8950).
 # Cilium 1.19 BGPv2 advertises the capability only for the IPv4 Unicast
@@ -74,14 +74,14 @@ spec:
       peers:
         - name: udm-v6
           peerASN: 64512
-          # ULA — stable across ISP renumber events. UDM's br40 has
-          # fd40::1/64; nodes get fd40::<eui-64> via SLAAC on eth0.
-          peerAddress: fd40::1
+          # ULA — stable across ISP renumber events. UDM's br47 has
+          # fd02:47::1/64; nodes get fd02:47::<eui-64> via SLAAC on ingress0.
+          peerAddress: fd02:47::1
           peerConfigRef:
             name: bgp-peer-udm-v6
         - name: udm-v4
           peerASN: 64512
-          peerAddress: 10.2.40.1
+          peerAddress: 10.2.47.1
           peerConfigRef:
             name: bgp-peer-udm-v4
 ---

kubernetes/talos/patches/k8s-1.yaml

@@ -17,9 +17,19 @@ machine:
     # pod0 is a closed L2 fabric between the three nodes: statically
     # addressed, no gateway, nothing to learn from an RA.
     net.ipv6.conf.pod0.accept_ra: "0"
-    # ingress0 takes its prefix from the RA but not the default route — that
-    # stays on mgmt0 until the split flips it, and two default routes at the
-    # same metric tie.
+    # ingress0 keeps RAs, at the default accept_ra of 2: an IPv6 default
+    # router is a link-local address learned only from the RA, and with
+    # net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
+    # entirely below 2. That is how mgmt0 has its IPv6 default today.
+    #
+    # The prefix is announced correctly, so SLAAC addresses this link the same
+    # way mgmt0 gets its fd40:: address: derived from the MAC, stable across
+    # reboots, nothing to allocate. Only IPv4 is declared here, and only
+    # because a second DHCP client is what broke the node's DNS record.
+    #
+    # Just the default route is suppressed, because mgmt0 owns it until the
+    # split flips it and two defaults at one metric tie. Dropping this line is
+    # what hands the IPv6 default over.
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # That network also carries the mgmt VLAN's ARP. Answer ARP only on the
     # link that owns the address, or the UDM learns the node IP against
@@ -164,21 +174,27 @@ addresses:
 # Ingress network, VLAN 47. World-to-pod: the BGP session, the next hop for
 # every advertised LoadBalancer VIP, and the masquerade source for pod egress.
 # Publicly routed, which is what separates it from pod0.
+#
+# Statically addressed, not leased. A second DHCP client on the node announced
+# the same hostname as mgmt0's and the UDM registered the node against this
+# segment instead, so mgmt0's `srv.renner.uno` record disappeared and the
+# lease's resolver displaced the management one. The UDM cannot register a
+# network's clients selectively, so the client goes rather than the record.
+# DHCP is off on VLAN 47 for the same reason.
+#
+# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
+# the segment. The default route moves here in the change that flips it.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
 vlanID: 47
 parent: uplink
 up: true
----
-# Metric loses to mgmt0's default 1024 deliberately: this pass only adds the
-# segment. The default route moves here in the change that flips it, which
-# needs no reboot and is provable with `--mode=try`. DHCPv4Config cannot
-# suppress a default route, only deprioritise it.
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: ingress0
-routeMetric: 2048
+addresses:
+  - address: 10.2.47.11/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication and nothing else. The udev rule in the machine

kubernetes/talos/patches/k8s-2.yaml

@@ -12,8 +12,9 @@ machine:
     # cluster IPv6 egress.
     net.ipv6.conf.uplink.disable_ipv6: "1"
     net.ipv6.conf.mgmt.disable_ipv6: "1"
-    # pod0 is statically addressed with no gateway; ingress0 takes its prefix
-    # but not the RA default route — see k8s-1.yaml.
+    # pod0 is a closed segment with no router, so it ignores RAs outright.
+    # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
+    # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
@@ -117,18 +118,19 @@ addresses:
   - address: 10.2.48.12/24
   - address: fd02:48::12/64
 ---
-# Ingress network, VLAN 47. World-to-pod, publicly routed — see k8s-1.yaml.
+# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
+# addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
 vlanID: 47
 parent: uplink
 up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: ingress0
-routeMetric: 2048
+addresses:
+  - address: 10.2.47.12/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.

kubernetes/talos/patches/k8s-3.yaml

@@ -8,8 +8,9 @@ machine:
     # k8s-1.yaml.
     net.ipv6.conf.uplink.disable_ipv6: "1"
     net.ipv6.conf.mgmt.disable_ipv6: "1"
-    # pod0 is statically addressed with no gateway; ingress0 takes its prefix
-    # but not the RA default route — see k8s-1.yaml.
+    # pod0 is a closed segment with no router, so it ignores RAs outright.
+    # ingress0 keeps RAs: it learns its IPv6 router and its address from them,
+    # and only the default route is held back — see k8s-1.yaml.
     net.ipv6.conf.pod0.accept_ra: "0"
     net.ipv6.conf.ingress0.accept_ra_defrtr: "0"
     # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
@@ -104,18 +105,19 @@ addresses:
   - address: 10.2.48.13/24
   - address: fd02:48::13/64
 ---
-# Ingress network, VLAN 47. World-to-pod, publicly routed — see k8s-1.yaml.
+# Ingress network, VLAN 47. World-to-pod, publicly routed, statically
+# addressed — see k8s-1.yaml for why it is not leased.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
 vlanID: 47
 parent: uplink
 up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: ingress0
-routeMetric: 2048
+addresses:
+  - address: 10.2.47.13/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
 # here on purpose; the link is handed whole to the OSD pod (host-device).

unifi/bgp.conf

@@ -28,10 +28,19 @@ router bgp 64512
  ! advertise extended-nexthop for the IPv6 Unicast SAFI when v6 routes
  ! are carried over a v4 transport, so we run a native v6 session that
  ! carries v6 routes with v6 next-hops and skip the capability dance
- ! entirely. Dynamic neighbors — k8s nodes are SLAAC-addressed on mgmt
- ! LAN; the listen ranges cover the SLAAC suffix space.
+ ! entirely. Dynamic neighbors — nodes are SLAAC-addressed, so the listen
+ ! ranges cover the SLAAC suffix space rather than naming each peer.
+ !
+ ! Both segments are listed while the ingress split lands (#576). The
+ ! cluster peers from mgmt (40) today and moves to ingress (47), and a
+ ! listen range only accepts a session, it does not open one — so having
+ ! both costs nothing and means the move never has a window where the UDM
+ ! is not listening where the nodes are calling from. Drop the 40 ranges
+ ! once every session is established on 47.
  bgp listen range 10.2.40.0/24 peer-group K8S
  bgp listen range fd40::/64 peer-group K8S
+ bgp listen range 10.2.47.0/24 peer-group K8S
+ bgp listen range fd02:47::/64 peer-group K8S
  !
  address-family ipv4 unicast
   neighbor K8S activate