fix(bird): originate the node's own pod CIDR instead of learning it

IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.

Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.

That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.

Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -376,3 +376,29 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
           --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
+
+  # bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
+  # artifact like heating-can; same registry so the audit tracks it.
+  - name: build-bird
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: ci/images/bird
+    depends_on: []
+    when:
+      - event: deployment
+      - event: push
+        path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/bird:$TAG \
+          --destination gitea.renner.uno/gitea_admin/bird:latest

ci/images/bird/Containerfile

@@ -0,0 +1,22 @@
+FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
+
+# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
+# step — the image is the binary plus its runtime.
+#
+# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
+# 3.22/community.
+#
+# Config is read from /etc/bird/bird.conf, not the package default
+# /etc/bird.conf, so the mount cannot shadow what the package ships.
+#
+# kubectl renders the init container's per-node config: go-template-file for
+# the templating, a field selector for excluding the node itself, in-cluster
+# config for the auth. From the distro, so there is no version literal to drift
+# and no download to trust. It is only ever used to read core/v1 Nodes.
+RUN apk add --no-cache bird2 kubectl
+
+# Fails the build rather than the first pod if a binary or its path is wrong.
+RUN /usr/sbin/bird --version \
+ && kubectl version --client > /dev/null
+
+ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -1,103 +1,194 @@
 # Proposal — separate the four traffic classes onto their own VLANs
 
-Status: open, issue #576. Two of the four classes are separated. Pod east-west
-cannot leave the node IP under Cilium. Both ways of moving it within Cilium
-(options A and B below) are rejected by the owner, and replacing the CNI
-(option C) is shelved.
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
 
-## Target
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
 
-Set by the owner on 2026-09-08:
+## The four classes
 
-| class | link |
-|---|---|
-| Ceph replication | X710 p1, VLAN 49 |
-| pod-pod | X710 trunk, a VLAN of its own |
-| world-pod | X710 trunk, a VLAN of its own |
-| node management | 1 Gb onboard NIC |
-
-Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
-on the 10G switch.
-
-## Where the classes are
-
-Measured 2026-09-15.
-
-| class | today | state |
+| class | what it is | where it goes today |
 |---|---|---|
-| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
-| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
-| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
-| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
-| management | `mgmt0`, VLAN 40 | not started |
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
 
-`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
-#833). `pod0` holds addresses and carries no traffic.
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
 
-## Why pod-pod is blocked
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
 
-Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
-next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:
+## Principle
 
-- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
-  fed from `newNode.GetNodeIP(false)`.
-- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
-- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
-  `initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
-  Node object.
-- `--direct-routing-device` selects a device for the BPF NodePort path, not a
-  next-hop.
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
 
-Tested on the cluster, none of these moved the next-hop:
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
 
-| attempt | result |
-|---|---|
-| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
-| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
-| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |
+## The finding that shapes everything else
 
-The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
-`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
-Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
-cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
-has no effect on pod-pod routing.
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
 
-## Options for pod-pod
+`installDirectRoute` takes the next-hop as a parameter:
 
-| | option | status |
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
 |---|---|---|
-| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
-| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
-| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
 
-A per-workload Multus attachment detaches a single workload from the node IP,
-the way Ceph replication is detached. It does not cover east-west traffic in
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
 general.
 
-## Levers per management service
+## Recommended mapping
 
-These hold regardless of which option is chosen:
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
 
-| service | can leave the node IP? | lever |
-|---|---|---|
-| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
-| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
-| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
-| kubelet | no | it is the node IP |
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
 
-## Remaining, independent of pod-pod
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
 
-- **pod→world egress**: default route and masquerade source onto `ingress0`,
-  via route metrics, `devices` and `enable-masquerade-to-route-source`. This
-  moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
-- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
-  removed once ingress on VLAN 47 is settled.
+Under B:
 
-## Sources
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
 
-- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
-  `--direct-routing-device`.
-- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
-- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
-- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
-  `CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.

kubernetes/apps/immich/machine-learning.yaml

@@ -32,7 +32,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: immich-machine-learning
-          image: ghcr.io/immich-app/immich-machine-learning:v3.2.1 # {"$imagepolicy": "flux-system:immich-machine-learning"}
+          image: ghcr.io/immich-app/immich-machine-learning:v3.2.2 # {"$imagepolicy": "flux-system:immich-machine-learning"}
           ports:
             - name: http
               containerPort: 3003

kubernetes/apps/immich/server.yaml

@@ -87,7 +87,7 @@ spec:
               mountPath: /ca
       containers:
         - name: immich-server
-          image: ghcr.io/immich-app/immich-server:v3.2.1 # {"$imagepolicy": "flux-system:immich-server"}
+          image: ghcr.io/immich-app/immich-server:v3.2.2 # {"$imagepolicy": "flux-system:immich-server"}
           ports:
             - name: http
               containerPort: 3001

kubernetes/infrastructure/bird/bird.conf

@@ -0,0 +1,67 @@
+# Identical on every node and free of addresses. defines.conf and peers.conf
+# are rendered per node by the init container; the first is included above the
+# templates that reference it, the second below them.
+include "/etc/bird/defines.conf";
+
+router id from "pod0";
+log stderr all;
+
+protocol device { scan time 10; }
+
+# This node's own pod CIDR, originated rather than read back out of the kernel.
+# Reading it back made BIRD's view depend on the shape of Cilium's route, and
+# the IPv4 one is rejected outright: its next hop sits inside the prefix it
+# describes, which KRT refuses with "strange next-hop". The prefix comes from
+# the Node object either way, so there is nothing to learn.
+#
+# blackhole because this route is only ever advertised. Nothing installs it:
+# the only kernel protocol exporting from these tables takes BGP routes.
+protocol static own4 {
+  ipv4 { table master4; };
+  route OWN_POD4 blackhole;
+}
+protocol static own6 {
+  ipv6 { table master6; };
+  route OWN_POD6 blackhole;
+}
+
+# Only BGP-learned routes reach table 48 — not the static above, and not
+# anything else that may end up in the master tables.
+filter to_t48 {
+  if source != RTS_BGP then reject;
+  accept;
+}
+
+protocol kernel k48v4 {
+  kernel table 48;
+  metric 0;
+  ipv4 { table master4; import none; export filter to_t48; };
+}
+protocol kernel k48v6 {
+  kernel table 48;
+  metric 0;
+  ipv6 { table master6; import none; export filter to_t48; };
+}
+
+# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
+# from ingress0 and is unrelated.
+#
+# direct is explicit because iBGP defaults to multihop, which makes the gateway
+# recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
+# keeps the listener off mgmt0 and ingress0.
+template bgp pod4 {
+  local as 64514;
+  source address MY4;
+  direct;
+  strict bind;
+  ipv4 { next hop self; import all; export all; };
+}
+template bgp pod6 {
+  local as 64514;
+  source address MY6;
+  direct;
+  strict bind;
+  ipv6 { next hop self; import all; export all; };
+}
+
+include "/etc/bird/peers.conf";

kubernetes/infrastructure/bird/daemonset.yaml

@@ -0,0 +1,115 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: bird
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: bird
+rules:
+  # The peer list is read from the nodes' pod0 annotations, so no address is
+  # written into this repo twice.
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: bird
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: bird
+subjects:
+  - kind: ServiceAccount
+    name: bird
+    namespace: kube-system
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: bird
+  namespace: kube-system
+  labels:
+    tier: node
+    app: bird
+spec:
+  selector:
+    matchLabels:
+      app: bird
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: bird
+    spec:
+      # Carries pod east-west once the routing rules land. Without it table 48
+      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
+      priorityClassName: system-node-critical
+      # Writes host routes and peers on host addresses.
+      hostNetwork: true
+      serviceAccountName: bird
+      # Every node here is a control plane node, so this is not optional.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      initContainers:
+        - name: render-config
+          # By digest, not by the dated tag: the tag is rebuilt within the day
+          # it names, and the cluster's image path keeps resolving it to the
+          # first build of that day — imagePullPolicy: Always does not help,
+          # because the tag itself resolves stale. Update this when the image
+          # is rebuilt.
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -eu
+              cp /config/bird.conf /etc/bird/bird.conf
+              kubectl get node "$NODE_NAME" \
+                -o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
+              kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
+                -o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
+              # BIRD's own parser is the check: a node whose annotation is
+              # missing renders as <no value> and fails here rather than
+              # crashlooping the daemon or forming a mesh with a hole.
+              bird -p -c /etc/bird/bird.conf
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          volumeMounts:
+            - name: config
+              mountPath: /config
+            - name: etc-bird
+              mountPath: /etc/bird
+      containers:
+        - name: bird
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
+          resources:
+            requests:
+              cpu: "50m"
+              memory: "32Mi"
+            limits:
+              cpu: "200m"
+              memory: "128Mi"
+          securityContext:
+            capabilities:
+              drop: ["ALL"]
+              # NET_ADMIN writes the routes; NET_BIND_SERVICE binds 179.
+              add: ["NET_ADMIN", "NET_BIND_SERVICE"]
+          volumeMounts:
+            - name: etc-bird
+              mountPath: /etc/bird
+      volumes:
+        - name: config
+          configMap:
+            name: bird-config
+        - name: etc-bird
+          emptyDir: {}

kubernetes/infrastructure/bird/defines.tmpl

@@ -0,0 +1,7 @@
+define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
+define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
+{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
+     IPv6 prefix into an ipv4 protocol, which fails the parse in the init
+     container rather than passing silently. */}}
+define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
+define OWN_POD6 = {{ index .spec.podCIDRs 1 }};

kubernetes/infrastructure/bird/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - daemonset.yaml
+configMapGenerator:
+  # Hashed name, so editing any of these rolls the DaemonSet.
+  - name: bird-config
+    namespace: kube-system
+    files:
+      - bird.conf
+      - defines.tmpl
+      - peers.tmpl

kubernetes/infrastructure/bird/peers.tmpl

@@ -0,0 +1,7 @@
+{{- range $i, $n := .items -}}
+{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
+{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
+# {{ $n.metadata.name }}
+protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
+protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
+{{ end -}}

kubernetes/infrastructure/kustomization.yaml

@@ -7,6 +7,7 @@ resources:
   - cluster-settings/
   - cilium/
   - multus/
+  - bird/
   - cert-manager/
   - snapshot-controller/
   - metrics-server/

kubernetes/talos/patches/k8s-1.yaml

@@ -52,6 +52,12 @@ machine:
     # this label instead of listing hostnames, so which boxes carry storage is
     # stated once, here, by the machine that knows.
     storage.renner.uno/ceph-osd: "true"
+  # Publishes this node's pod0 addresses, declared below in its VLANConfig, so
+  # the bird DaemonSet reads its peers from the node list rather than from a
+  # copy. Annotations, not labels: a label value cannot contain a colon.
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::11
+    net.renner.uno/pod0-v4: 10.2.48.11
   udev:
     # Renames every port this node uses to its role at boot, matched by
     # burned-in MAC — kernel names derive from PCI topology and shift when
@@ -182,8 +188,14 @@ addresses:
 # network's clients selectively, so the client goes rather than the record.
 # DHCP is off on VLAN 47 for the same reason.
 #
-# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
-# the segment. The default route moves here in the change that flips it.
+# Both defaults live in table 47, not in main, so only the pod-source rules in
+# site-homelab.yaml reach them. mgmt0 keeps main's default and with it every
+# reply the node sends from its own addresses.
+#
+# The v6 gateway is the UDM's configured ULA on this segment, the same address
+# BGP peers with. The RA-learned link-local would be the alternative and is not
+# a value to write down. accept_ra_defrtr stays 0 for that reason: the RA
+# default would land in main, where it does not belong.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
@@ -193,8 +205,10 @@ up: true
 addresses:
   - address: 10.2.47.11/24
 routes:
+  - gateway: fd02:47::1
+    table: "47"
   - gateway: 10.2.47.1
-    metric: 2048
+    table: "47"
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication and nothing else. The udev rule in the machine

kubernetes/talos/patches/k8s-2.yaml

@@ -33,6 +33,10 @@ machine:
     topology.kubernetes.io/zone: saturn
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
+  # pod0 addresses for the bird peer list — see k8s-1.yaml.
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::12
+    net.renner.uno/pod0-v4: 10.2.48.12
   udev:
     # Renames every port to its role at boot — see k8s-1.yaml.
     rules:
@@ -129,8 +133,10 @@ up: true
 addresses:
   - address: 10.2.47.12/24
 routes:
+  - gateway: fd02:47::1
+    table: "47"
   - gateway: 10.2.47.1
-    metric: 2048
+    table: "47"
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.

kubernetes/talos/patches/k8s-3.yaml

@@ -29,6 +29,10 @@ machine:
     topology.kubernetes.io/zone: neptune
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
+  # pod0 addresses for the bird peer list — see k8s-1.yaml.
+  nodeAnnotations:
+    net.renner.uno/pod0-v6: fd02:48::13
+    net.renner.uno/pod0-v4: 10.2.48.13
   udev:
     # Renames every port to its role at boot — see k8s-1.yaml.
     rules:
@@ -116,8 +120,10 @@ up: true
 addresses:
   - address: 10.2.47.13/24
 routes:
+  - gateway: fd02:47::1
+    table: "47"
   - gateway: 10.2.47.1
-    metric: 2048
+    table: "47"
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
 # here on purpose; the link is handed whole to the OSD pod (host-device).

kubernetes/talos/patches/site-homelab.yaml

@@ -20,3 +20,47 @@ machine:
       validSubnets:
         - 10.2.40.0/24
         - fd40::/64
+---
+# Pod egress leaves over ingress0. Matching on the pod source prefix is what
+# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
+# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
+# source onto VLAN 47.
+#
+# The routing decision runs before Cilium's masquerade hook, so the rule sees
+# the pod address and the masquerade then rewrites it to ingress0's.
+#
+# Table 47 holds nothing but the two default routes, so the destination rules
+# below have to run first: without them a pod's packet to a service or to a pod
+# on another node also matches the source rule, finds only the default, and
+# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
+#
+# They send everything inside the cluster and the local networks back to main.
+# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
+# is not foreign traffic.
+#
+# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
+# The two families are separate rule tables, so the priorities need not match;
+# the document name is the priority and must be unique.
+apiVersion: v1alpha1
+kind: RoutingRuleConfig
+name: "990"
+dst: fd00::/8
+table: main
+---
+apiVersion: v1alpha1
+kind: RoutingRuleConfig
+name: "991"
+dst: 10.0.0.0/8
+table: main
+---
+apiVersion: v1alpha1
+kind: RoutingRuleConfig
+name: "1000"
+src: fd10:244::/48
+table: "47"
+---
+apiVersion: v1alpha1
+kind: RoutingRuleConfig
+name: "1001"
+src: 10.244.0.0/16
+table: "47"