fix(bird): take the pod prefixes out of bird.conf too

The first revision removed the pod0 address list but still carried 10.244.0.0/16 and fd10:244::/48 in the filters — the cluster podSubnets from talos/patches/cluster.yaml, copied a second time. Same defect, different addresses.

The Node objects already carry .spec.podCIDRs, so the render script emits defines.conf with this node's own pod CIDRs alongside its pod0 addresses, and the filters match OWN_POD4/OWN_POD6 exactly instead of an aggregate. That is also stricter: the aggregate would have matched the peers' CIDRs as well, leaving the ifname check as the only discriminator.

The BGP channels drop their prefix filters entirely. The only route this node offers is its own CIDR, already selected by own_pod4/6, and the only peers are the other cluster nodes — so there was nothing left for them to decide. source address moves back into the templates, since defines.conf is included above them.

bird.conf now contains no address at all; grep for one finds nothing.

Verified: shellcheck clean; the podCIDR selectors return fd10:244::/64 and 10.244.0.0/24 against the live k8s-1 object; bird -p parses bird.conf with a generated defines.conf and peers.conf inside the 2.17.2 image, exit 0.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci/images/bird/Containerfile

@@ -9,14 +9,14 @@ FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc
 # Config is read from /etc/bird/bird.conf, not the package default
 # /etc/bird.conf, so the mount cannot shadow what the package ships.
 #
-# kubectl renders the init container's per-node config: go-template-file for
-# the templating, a field selector for excluding the node itself, in-cluster
-# config for the auth. From the distro, so there is no version literal to drift
-# and no download to trust. It is only ever used to read core/v1 Nodes.
-RUN apk add --no-cache bird2 kubectl
+# curl and jq are for the init container, which renders the per-node peer list
+# from the Kubernetes API. busybox wget cannot verify against the API server's
+# CA bundle, and the query must not run unverified.
+RUN apk add --no-cache bird2 curl jq
 
 # Fails the build rather than the first pod if a binary or its path is wrong.
 RUN /usr/sbin/bird --version \
- && kubectl version --client > /dev/null
+ && curl --version > /dev/null \
+ && jq --version > /dev/null
 
 ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]

kubernetes/NETWORK-traffic-classes.md

@@ -1,117 +0,0 @@
-# Traffic classes on the k8s nodes
-
-Issue #576. Three of the four classes are separated; management is not.
-
-## Where each class runs
-
-| class | link | mechanism |
-|---|---|---|
-| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
-| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
-| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
-| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
-| management | `mgmt0`, VLAN 40 | not started |
-
-Hard constraint, set by the owner: inter-node and pod traffic never traverses
-the UDM. Everything above stays on the 10G switch.
-
-## Why policy routing rather than moving the node IP
-
-Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
-and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
-`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
-Both were established by test (#837, #838), not by reading.
-
-So the InternalIP decides where pod traffic goes — unless the lookup happens in
-a different table. The rules do that, and nothing about the node's identity
-moves. The routing decision runs before Cilium's masquerade hook, so a rule
-matching the pod source sees the pod address, and the masquerade then rewrites
-it to the egress link's.
-
-## Rule order
-
-Lower priority runs first.
-
-| prio | match | table |
-|---|---|---|
-| 980 | src pod, dst pod | 48 |
-| 981 | src pod, dst pod (v4) | 48 |
-| 990 | dst `fd00::/8` | main |
-| 991 | dst `10.0.0.0/8` | main |
-| 1000 | src pod | 47 |
-| 1001 | src pod (v4) | 47 |
-
-980/981 match source **and** destination. Source alone would also catch the
-node's own traffic to a pod, which would then leave with a `pod0` source — an
-address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
-`world` rather than `host`.
-
-990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
-cluster-internal traffic out of table 47. Without them a pod's packet to a
-service matches the source rule, finds only a default route, and leaves for the
-UDM — measured, in-cluster DNS stopped resolving.
-
-Traffic to a pod on the same node falls through on its own: table 48 holds only
-the peers' CIDRs, so the lookup misses and 990/991 send it to main.
-
-## Where each table comes from
-
-- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
-  The IPv6 gateway is the UDM's configured address on the segment, not the
-  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
-  default does not land in main.
-- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
-  per peer pod CIDR with the peer's `pod0` address as next hop.
-
-## The bird mesh
-
-`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
-and installs what it learns into table 48. Cilium's routes in main are
-untouched, which is why the two coexist.
-
-No address is written into this repo twice. Each node publishes its `pod0`
-addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
-pod CIDRs come from the Node objects. An init container renders both from the
-API with `kubectl -o go-template-file`, excludes the node itself with a field
-selector, and ends with `bird -p` so BIRD's own parser gates startup.
-
-The node's own CIDR is **originated**, not read back from the kernel. Reading
-it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
-rejected outright: its next hop sits inside the prefix it describes, which KRT
-refuses with `strange next-hop`. The IPv6 route is a device route and passed,
-so the failure was silent on one family only.
-
-## Failure behaviour
-
-Benign at every layer. An empty or absent table 48 makes 980/981 fall through
-to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
-holds if the DaemonSet cannot start, which is the case during a cold rebuild
-until Gitea serves its image.
-
-## Management, the remaining class
-
-Not started. The levers differ per service and none of them is the node IP:
-
-| service | can leave the node IP? | lever |
-|---|---|---|
-| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
-| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
-| Talos API | n/a | `apid` answers on every node address; a firewall question |
-| kubelet | no | it *is* the node IP |
-
-Two approaches were rejected by the owner and are not to be proposed again:
-moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
-cluster VLAN while moving etcd and the API VIP to a new management VLAN.
-Replacing the CNI (#839, Antrea) was shelved.
-
-Note that the reasoning which produced that list also concluded pod↔pod was
-blocked, which policy routing disproved. Treat it as a record of decisions, not
-as an exhaustive analysis.
-
-## Sources
-
-Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
-`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
-`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
-`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
-#855; closed unmerged: #830, #838, #839.

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -0,0 +1,194 @@
+# Proposal — separate the four traffic classes onto their own VLANs
+
+Status: draft. The question this started with is **answered** — from the Cilium
+source and flag reference, not from a test — and the answer changes the shape
+of the proposal rather than confirming it.
+
+Supersedes the framing in issue #576, which described this as "give node
+management its own NIC, and Ceph public its own VLAN". That was two of four
+classes and one of them turned out already solved.
+
+## The four classes
+
+| class | what it is | where it goes today |
+|---|---|---|
+| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
+| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
+| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
+| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+
+Three of four share one interface. That is not a design decision — until the
+bare-metal conversion there was one NIC to work with.
+
+Ceph client I/O living on the pod network is correct rather than a leftover:
+that traffic *is* cluster-internal. Ceph is done.
+
+## Principle
+
+**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
+separate, changeable decision.**
+
+Defining classes by VLAN means the separation survives recabling, port failures
+and capacity changes, and it does not require one port per class. Defining them
+by interface would freeze a capacity decision into the architecture.
+
+## The finding that shapes everything else
+
+The original open question was whether `nodePort.directRoutingDevice` moves the
+next-hop of Cilium's direct routes, or only selects an interface. It is
+answerable from the source, and the answer is: **only the interface.**
+
+`installDirectRoute` takes the next-hop as a parameter:
+
+```go
+func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
+    skipUnreachable bool) (routeSpec *netlink.Route, err error)
+```
+
+and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
+in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
+same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
+
+`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
+an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
+`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
+cluster there is no path by which Cilium's node IP differs from the kubelet's
+`InternalIP`.
+
+`--direct-routing-device` is documented as "Device name used to connect nodes in
+direct routing mode" — a device, not an address.
+
+**So cluster-internal traffic is welded to the node IP, and the node IP is the
+kubelet InternalIP.** Two consequences:
+
+- The class cannot be moved off whatever VLAN the node IP lives on.
+- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
+  from the node IP; native mode uses it as the direct-route next-hop. Native
+  routing buys nothing here, so the disruptive, non-rolling step this proposal
+  previously contained is simply gone.
+
+## Consequence: draw the line the other way
+
+If cluster-internal cannot leave the node IP, then separation has to come from
+moving *management services* off it — and each one has to be checked
+individually, because they do not share a mechanism.
+
+| management service | can it leave the node IP? | lever |
+|---|---|---|
+| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
+| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
+| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
+| kubelet | **no** | it *is* the node IP, by definition |
+
+So four VLANs are reachable, but the boundary does not fall where the class
+names suggest: **kubelet stays with cluster-internal.** That is the honest
+result, and it is worth stating plainly rather than filing kubelet under
+management and hoping.
+
+The remaining classes keep the levers established earlier:
+
+- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
+  For eBGP the session source is the next-hop, so pinning the session pins
+  where return traffic arrives. The session to the UDM has been running for
+  months.
+- **cluster-external egress** — eBPF masquerading follows the output device, so
+  this is a default-route decision plus `enable-masquerade-to-route-source`
+  (today `false`).
+- **Ceph replication** — Multus macvlan on VLAN 49, done.
+
+## The general escape hatch
+
+Ceph is separated because its OSDs got a second interface and stopped using the
+pod network for replication. That is the *only* mechanism that fully detaches a
+workload from the node IP, and it generalises: anything that must not ride the
+node IP gets its own `NetworkAttachmentDefinition`.
+
+It is also expensive per workload — an attachment, an IPAM pool, a pod spec
+change. Worth it for Ceph replication. Not worth it for east-west traffic in
+general.
+
+## Recommended mapping
+
+Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+
+**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
+and Talos API. Cleanest match to the class names, but it changes every node's
+identity, which is the one address the whole cluster is administered through.
+
+**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
+VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
+Identical end state, and no node ever changes its InternalIP.
+
+Under B:
+
+| VLAN | carries |
+|---|---|
+| 49 | Ceph replication |
+| 40 (relabelled *cluster*) | pod east-west, kubelet |
+| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
+| new *external* | LoadBalancer ingress, egress default route |
+
+The property the user asked for — management carries only management traffic —
+holds under B. The label moves; the property is what matters.
+
+## Migration shape
+
+Every step is rolling and reversible. The routing-mode change is gone.
+
+1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
+2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
+   Still unused; this only proves the L2 works.
+3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
+   waiting for a healthy quorum between each.
+4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
+   `k8s.renner.uno` follows.
+5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
+   Reversible by reverting the override; failure is visible immediately in
+   `cilium-dbg shell -- bgp/peers`.
+6. Verify VLAN 40 by what is *absent* from it, not by what still works.
+
+Step 3 is the one that deserves care — etcd is the only step where a mistake
+costs quorum rather than connectivity.
+
+## What this does not buy
+
+Management currently shares a 10 Gb link and is not contended. Nothing observed
+today is slow because of this. The case for the split is that traffic classes
+with different failure modes and security postures should not be
+indistinguishable — not that something is broken.
+
+## Sources and measurements
+
+Source, Cilium v1.20.1 (the running version):
+
+- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
+  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
+  a next-hop that is not directly reachable.
+- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
+  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
+  Kubernetes Node.
+- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
+  to connect nodes in direct routing mode".
+
+Talos:
+
+- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
+  defaults to it. kube-apiserver and Talos API listen addresses are not
+  covered there.
+- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
+  `10.2.40.60` with `link: mgmt0`.
+
+Measured on the cluster:
+
+- `apid` answered on a node address that is not the InternalIP, so the Talos
+  API is not bound to one address.
+- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
+  InternalIP had changed since the last session, which is why nothing here
+  names an address as fixed.
+- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
+  unset and no direct-routing device configured.
+
+Not verified: whether the kube-apiserver's `--advertise-address` follows the
+node IP independently of the VIP. It does not affect the recommendation, since
+clients reach the API through the VIP either way.

kubernetes/bootstrap.sh

@@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
     # Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
     # would not match the amd64 schematic this cluster installs. That leaves the
     # cx*/cpx*2 families in the German locations.
-    # Two segments only, unlike the homelab's split in
-    # NETWORK-traffic-classes.md: here the node address carries management,
+    # Today's traffic-class split, not the target one in
+    # PROPOSAL-traffic-class-separation.md: the node address carries management,
     # cluster-internal and external alike, and Ceph replication has its own.
     #
     # The order is load-bearing: it decides which kernel interface each segment

kubernetes/infrastructure/bird/bird.conf

@@ -1,32 +1,53 @@
-# Identical on every node and free of addresses. defines.conf and peers.conf
-# are rendered per node by the init container; the first is included above the
-# templates that reference it, the second below them.
+# Identical on every node, and free of addresses: everything node-specific is
+# rendered by render-peers.sh from the Kubernetes API.
+#
+# defines.conf first, because the templates below reference it; peers.conf last,
+# because it instantiates them.
 include "/etc/bird/defines.conf";
 
 router id from "pod0";
 log stderr all;
 
+# Mirrors of the kernel main table, filtered to this node's own pod CIDR.
+# master4/master6 are implicit and hold what BGP exchanges.
+ipv4 table kmain4;
+ipv6 table kmain6;
+
 protocol device { scan time 10; }
 
-# This node's own pod CIDR, originated rather than read back out of the kernel.
-# Reading it back made BIRD's view depend on the shape of Cilium's route, and
-# the IPv4 one is rejected outright: its next hop sits inside the prefix it
-# describes, which KRT refuses with "strange next-hop". The prefix comes from
-# the Node object either way, so there is nothing to learn.
-#
-# blackhole because this route is only ever advertised. Nothing installs it:
-# the only kernel protocol exporting from these tables takes BGP routes.
-protocol static own4 {
-  ipv4 { table master4; };
-  route OWN_POD4 blackhole;
+# Two conditions, both needed. The prefix alone would also match Cilium's
+# routes for the OTHER nodes, which are protocol kernel as well; the interface
+# alone would also match cilium_host's own host address. ifname is set for the
+# gatewayless IPv6 device route too, which is why this works in both families.
+filter own_pod4 {
+  if net != OWN_POD4 then reject;
+  if ifname !~ "cilium_host" then reject;
+  accept;
 }
-protocol static own6 {
-  ipv6 { table master6; };
-  route OWN_POD6 blackhole;
+filter own_pod6 {
+  if net != OWN_POD6 then reject;
+  if ifname !~ "cilium_host" then reject;
+  accept;
 }
 
-# Only BGP-learned routes reach table 48 — not the static above, and not
-# anything else that may end up in the master tables.
+# learn all, not learn: plain learn skips routes marked proto kernel, which is
+# what Cilium's are.
+protocol kernel klearn4 {
+  learn all;
+  scan time 10;
+  ipv4 { table kmain4; import filter own_pod4; export none; };
+}
+protocol kernel klearn6 {
+  learn all;
+  scan time 10;
+  ipv6 { table kmain6; import filter own_pod6; export none; };
+}
+
+protocol pipe own4 { table kmain4; peer table master4; export all; import none; }
+protocol pipe own6 { table kmain6; peer table master6; export all; import none; }
+
+# Only BGP-learned routes reach table 48. The local CIDR stays in main, where
+# cilium_host already serves it.
 filter to_t48 {
   if source != RTS_BGP then reject;
   accept;
@@ -46,6 +67,9 @@ protocol kernel k48v6 {
 # iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
 # from ingress0 and is unrelated.
 #
+# No prefix filters: the only route this node offers is its own pod CIDR, which
+# own_pod4/6 already selected, and the only peers are the other cluster nodes.
+#
 # direct is explicit because iBGP defaults to multihop, which makes the gateway
 # recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
 # keeps the listener off mgmt0 and ingress0.

kubernetes/infrastructure/bird/daemonset.yaml

@@ -9,7 +9,7 @@ kind: ClusterRole
 metadata:
   name: bird
 rules:
-  # The peer list is read from the nodes' pod0 annotations, so no address is
+  # The peer list comes from the nodes' pod0 annotations, so no address is
   # written into this repo twice.
   - apiGroups: [""]
     resources: ["nodes"]
@@ -59,26 +59,12 @@ spec:
         - operator: Exists
           effect: NoSchedule
       initContainers:
-        - name: render-config
-          # By digest, not by the dated tag: the tag is rebuilt within the day
-          # it names, and the cluster's image path keeps resolving it to the
-          # first build of that day — imagePullPolicy: Always does not help,
-          # because the tag itself resolves stale. Update this when the image
-          # is rebuilt.
-          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -eu
-              cp /config/bird.conf /etc/bird/bird.conf
-              kubectl get node "$NODE_NAME" \
-                -o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
-              kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
-                -o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
-              # BIRD's own parser is the check: a node whose annotation is
-              # missing renders as <no value> and fails here rather than
-              # crashlooping the daemon or forming a mesh with a hole.
-              bird -p -c /etc/bird/bird.conf
+        - name: render-peers
+          image: gitea.renner.uno/gitea_admin/bird:2026-09-16
+          # The dated tag was rebuilt today to add curl and jq, so a node may
+          # hold the earlier build of the same tag.
+          imagePullPolicy: Always
+          command: ["/bin/sh", "/config/render-peers.sh"]
           env:
             - name: NODE_NAME
               valueFrom:
@@ -91,7 +77,8 @@ spec:
               mountPath: /etc/bird
       containers:
         - name: bird
-          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
+          image: gitea.renner.uno/gitea_admin/bird:2026-09-16
+          imagePullPolicy: Always
           resources:
             requests:
               cpu: "50m"

kubernetes/infrastructure/bird/defines.tmpl

@@ -1,7 +0,0 @@
-define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
-define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
-{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
-     IPv6 prefix into an ipv4 protocol, which fails the parse in the init
-     container rather than passing silently. */}}
-define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
-define OWN_POD6 = {{ index .spec.podCIDRs 1 }};

kubernetes/infrastructure/bird/kustomization.yaml

@@ -3,10 +3,9 @@ kind: Kustomization
 resources:
   - daemonset.yaml
 configMapGenerator:
-  # Hashed name, so editing any of these rolls the DaemonSet.
+  # Hashed name, so editing either file rolls the DaemonSet.
   - name: bird-config
     namespace: kube-system
     files:
       - bird.conf
-      - defines.tmpl
-      - peers.tmpl
+      - render-peers.sh

kubernetes/infrastructure/bird/peers.tmpl

@@ -1,7 +0,0 @@
-{{- range $i, $n := .items -}}
-{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
-{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
-# {{ $n.metadata.name }}
-protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
-protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
-{{ end -}}

kubernetes/infrastructure/bird/render-peers.sh

@@ -0,0 +1,50 @@
+#!/bin/sh
+# Renders everything node-specific from the Kubernetes API, so no address is
+# written into this repo twice. pod0 addresses come from annotations set beside
+# the addresses themselves in kubernetes/talos/patches/k8s-*.yaml; pod CIDRs
+# come from the Node objects, where Kubernetes assigns them.
+set -eu
+
+SA=/var/run/secrets/kubernetes.io/serviceaccount
+A6=net.renner.uno/pod0-v6
+A4=net.renner.uno/pod0-v4
+
+nodes=$(curl -sS --fail \
+  --cacert "$SA/ca.crt" \
+  -H "Authorization: Bearer $(cat "$SA/token")" \
+  https://kubernetes.default.svc/api/v1/nodes)
+
+# -e throughout, so a node missing an annotation or a pod CIDR fails here,
+# loudly, rather than forming a mesh with a hole in it.
+self=$(printf '%s' "$nodes" | jq -er --arg n "$NODE_NAME" \
+  '.items[] | select(.metadata.name == $n)')
+
+my6=$(printf '%s' "$self" | jq -er --arg a "$A6" '.metadata.annotations[$a]')
+my4=$(printf '%s' "$self" | jq -er --arg a "$A4" '.metadata.annotations[$a]')
+
+# podCIDRs is ordered by the cluster's family list, so pick by content rather
+# than by index.
+pod6=$(printf '%s' "$self" | jq -er '.spec.podCIDRs[] | select(contains(":"))')
+pod4=$(printf '%s' "$self" | jq -er '.spec.podCIDRs[] | select(contains(":") | not)')
+
+cp /config/bird.conf /etc/bird/bird.conf
+
+cat > /etc/bird/defines.conf <<EOF
+define MY6 = $my6;
+define MY4 = $my4;
+define OWN_POD6 = $pod6;
+define OWN_POD4 = $pod4;
+EOF
+
+# Protocol names are BIRD symbols, so the node name is sanitised.
+printf '%s' "$nodes" | jq -r \
+  --arg n "$NODE_NAME" --arg a6 "$A6" --arg a4 "$A4" '
+  .items[]
+  | select(.metadata.name != $n)
+  | select(.metadata.annotations[$a6] != null and .metadata.annotations[$a4] != null)
+  | (.metadata.name | gsub("[^a-zA-Z0-9]"; "_")) as $p
+  | "protocol bgp p_\($p)_v6 from pod6 { neighbor \(.metadata.annotations[$a6]) as 64514; }",
+    "protocol bgp p_\($p)_v4 from pod4 { neighbor \(.metadata.annotations[$a4]) as 64514; }"
+' > /etc/bird/peers.conf
+
+echo "$NODE_NAME: $(grep -c . /etc/bird/peers.conf) peer protocols, own pod CIDRs $pod6 $pod4"

kubernetes/talos/patches/k8s-1.yaml

@@ -53,8 +53,8 @@ machine:
     # stated once, here, by the machine that knows.
     storage.renner.uno/ceph-osd: "true"
   # Publishes this node's pod0 addresses, declared below in its VLANConfig, so
-  # the bird DaemonSet reads its peers from the node list rather than from a
-  # copy. Annotations, not labels: a label value cannot contain a colon.
+  # the bird DaemonSet builds its peer list from the node list rather than from
+  # a copy. Annotations, not labels: a label value cannot contain a colon.
   nodeAnnotations:
     net.renner.uno/pod0-v6: fd02:48::11
     net.renner.uno/pod0-v4: 10.2.48.11

kubernetes/talos/patches/site-homelab.yaml

@@ -41,29 +41,6 @@ machine:
 # Priorities are below main (32766) and clear of the rule Cilium installs at 9.
 # The two families are separate rule tables, so the priorities need not match;
 # the document name is the priority and must be unique.
-# Pod to pod between nodes goes over pod0, via table 48, which the bird mesh
-# fills with one route per peer pod CIDR and the peer's pod0 address as the
-# next hop. Ahead of 990/991 so it wins for that traffic, and matching source
-# AND destination so nothing else lands there: the node's own traffic to a pod
-# would otherwise leave with a pod0 source, which CiliumNode does not list, so
-# Cilium would classify it as world rather than host.
-#
-# Traffic to a pod on this node falls through — table 48 holds only the peers'
-# CIDRs, so the lookup misses and 990/991 send it to main and cilium_host.
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
-name: "980"
-src: fd10:244::/48
-dst: fd10:244::/48
-table: "48"
----
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
-name: "981"
-src: 10.244.0.0/16
-dst: 10.244.0.0/16
-table: "48"
----
 apiVersion: v1alpha1
 kind: RoutingRuleConfig
 name: "990"