docs(#576 ): record the rejected options in the traffic-class proposal

The proposal still recommended option B (node IP stays on VLAN 40, etcd and the API VIP move to a new management VLAN), which the owner has rejected repeatedly, and it described the cluster as running in tunnel mode. Rewritten to the measured state of 2026-09-15: world-pod is on VLAN 47 since #835/#836, the cluster runs native routing since #837, pod-pod still follows the kubelet InternalIP on mgmt0. Options A (#838) and B are marked rejected, C (#839, Antrea) shelved. The owner's 2026-09-08 target and the no-UDM constraint are kept as the reference. Refs #576 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 13:45:33 +02:00
15 changed files with 91 additions and 526 deletions
@@ -376,29 +376,3 @@ steps:
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
-
-  # bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
-  # artifact like heating-can; same registry so the audit tracks it.
-  - name: build-bird
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
-    directory: ci/images/bird
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/bird:$TAG \
-          --destination gitea.renner.uno/gitea_admin/bird:latest
@@ -1,22 +0,0 @@
-FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
-
-# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
-# step — the image is the binary plus its runtime.
-#
-# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
-# 3.22/community.
-#
-# Config is read from /etc/bird/bird.conf, not the package default
-# /etc/bird.conf, so the mount cannot shadow what the package ships.
-#
-# kubectl renders the init container's per-node config: go-template-file for
-# the templating, a field selector for excluding the node itself, in-cluster
-# config for the auth. From the distro, so there is no version literal to drift
-# and no download to trust. It is only ever used to read core/v1 Nodes.
-RUN apk add --no-cache bird2 kubectl
-
-# Fails the build rather than the first pod if a binary or its path is wrong.
-RUN /usr/sbin/bird --version \
- && kubectl version --client > /dev/null
-
-ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]
@@ -1,194 +1,103 @@
 # Proposal — separate the four traffic classes onto their own VLANs

-Status: draft. The question this started with is **answered** — from the Cilium
-source and flag reference, not from a test — and the answer changes the shape
-of the proposal rather than confirming it.
+Status: open, issue #576. Two of the four classes are separated. Pod east-west
+cannot leave the node IP under Cilium. Both ways of moving it within Cilium
+(options A and B below) are rejected by the owner, and replacing the CNI
+(option C) is shelved.

-Supersedes the framing in issue #576, which described this as "give node
-management its own NIC, and Ceph public its own VLAN". That was two of four
-classes and one of them turned out already solved.
+## Target

-## The four classes
+Set by the owner on 2026-09-08:

-| class | what it is | where it goes today |
+| class | link |
+|---|---|
+| Ceph replication | X710 p1, VLAN 49 |
+| pod-pod | X710 trunk, a VLAN of its own |
+| world-pod | X710 trunk, a VLAN of its own |
+| node management | 1 Gb onboard NIC |
+
+Hard constraint: inter-node and pod traffic never traverses the UDM. It stays
+on the 10G switch.
+
+## Where the classes are
+
+Measured 2026-09-15.
+
+| class | today | state |
 |---|---|---|
-| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
-| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
-| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
-| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
+| Ceph replication | `ceph-san`, VLAN 49, handed whole to the OSD pod (Multus `host-device`) | done |
+| world→pod | BGP from each node's `ingress0` (VLAN 47) to `fd02:47::1` / `10.2.47.1`; LB VIPs and PodCIDRs are advertised with that next-hop | done (#835, #836) |
+| pod→world | default routes via `mgmt0`: v4 metric 1024 against `ingress0`'s 2048, v6 from SLAAC on `mgmt0`; `devices` unset | open |
+| pod-pod | native routing, `<PodCIDR> via <peer InternalIP> dev mgmt0`, VLAN 40 | blocked, see below |
+| management | `mgmt0`, VLAN 40 | not started |

-Three of four share one interface. That is not a design decision — until the
-bare-metal conversion there was one NIC to work with.
+`ingress0` (VLAN 47) and `pod0` (VLAN 48) exist on all three nodes (#831,
+#833). `pod0` holds addresses and carries no traffic.

-Ceph client I/O living on the pod network is correct rather than a leftover:
-that traffic *is* cluster-internal. Ceph is done.
+## Why pod-pod is blocked

-## Principle
+Cilium's route to a peer's pods uses the peer's Kubernetes InternalIP as the
+next-hop, in native mode, and as the VXLAN endpoint in tunnel mode:

-**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
-separate, changeable decision.**
+- `pkg/datapath/linux/node.go`: `installDirectRoute(…, nodeIP net.IP, …)` is
+  fed from `newNode.GetNodeIP(false)`.
+- `pkg/node/types/node.go`: `GetNodeIP` prefers `NodeInternalIP`.
+- `pkg/node/sync/local_node_sync.go`: `initFromK8s()` runs after
+  `initFromConfig()` and overwrites `--ipv4-node` / `--ipv6-node` from the
+  Node object.
+- `--direct-routing-device` selects a device for the BPF NodePort path, not a
+  next-hop.

-Defining classes by VLAN means the separation survives recabling, port failures
-and capacity changes, and it does not require one port per class. Defining them
-by interface would freeze a capacity decision into the architecture.
+Tested on the cluster, none of these moved the next-hop:

-## The finding that shapes everything else
+| attempt | result |
+|---|---|
+| `ipv4-node` / `ipv6-node` via `CiliumNodeConfig` | agent logged both flags, still published its `mgmt0` address |
+| `nodePort.directRoutingDevice: pod0` (#837) | agent logged `direct-routing-device=pod0`, still installed routes via `mgmt0` |
+| tunnel and native mode, v4 and v6 | endpoints and routes keyed to the peer's InternalIP |

-The original open question was whether `nodePort.directRoutingDevice` moves the
-next-hop of Cilium's direct routes, or only selects an interface. It is
-answerable from the source, and the answer is: **only the interface.**
+The InternalIP comes from `machine.kubelet.nodeIP.validSubnets` in
+`talos/patches/site-homelab.yaml`, currently `10.2.40.0/24` and `fd40::/64`.
+Pod-pod traffic therefore sits on whichever VLAN holds the InternalIP. The
+cluster has been in native routing mode since #837. `directRoutingDevice: pod0`
+has no effect on pod-pod routing.

-`installDirectRoute` takes the next-hop as a parameter:
+## Options for pod-pod

-```go
-func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
-    skipUnreachable bool) (routeSpec *netlink.Route, err error)
-```
-
-and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
-in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
-same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
-
-`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
-an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
-`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
-cluster there is no path by which Cilium's node IP differs from the kubelet's
-`InternalIP`.
-
-`--direct-routing-device` is documented as "Device name used to connect nodes in
-direct routing mode" — a device, not an address.
-
-**So cluster-internal traffic is welded to the node IP, and the node IP is the
-kubelet InternalIP.** Two consequences:
-
- The class cannot be moved off whatever VLAN the node IP lives on.
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
-  from the node IP; native mode uses it as the direct-route next-hop. Native
-  routing buys nothing here, so the disruptive, non-rolling step this proposal
-  previously contained is simply gone.
-
-## Consequence: draw the line the other way
-
-If cluster-internal cannot leave the node IP, then separation has to come from
-moving *management services* off it — and each one has to be checked
-individually, because they do not share a mechanism.
-
-| management service | can it leave the node IP? | lever |
+| | option | status |
 |---|---|---|
-| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
-| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
-| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
-| kubelet | **no** | it *is* the node IP, by definition |
+| A | move the node IP and etcd to `pod0` | **rejected** by the owner; #838 closed 2026-09-08 |
+| B | keep the node IP on VLAN 40, relabel VLAN 40 as the cluster VLAN, move etcd and the API VIP to a new management VLAN | **rejected** by the owner, repeatedly. Not to be proposed again |
+| C | replace Cilium with a CNI that picks the transport interface independently of the node IP (Antrea `transportInterface`) | **shelved** 2026-09-10; #839 closed. Needs a cold rebuild, rewriting the CiliumNetworkPolicy objects, and Antrea's BGP is alpha |

-So four VLANs are reachable, but the boundary does not fall where the class
-names suggest: **kubelet stays with cluster-internal.** That is the honest
-result, and it is worth stating plainly rather than filing kubelet under
-management and hoping.
-
-The remaining classes keep the levers established earlier:
-
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
-  For eBGP the session source is the next-hop, so pinning the session pins
-  where return traffic arrives. The session to the UDM has been running for
-  months.
- **cluster-external egress** — eBPF masquerading follows the output device, so
-  this is a default-route decision plus `enable-masquerade-to-route-source`
-  (today `false`).
- **Ceph replication** — Multus macvlan on VLAN 49, done.
-
-## The general escape hatch
-
-Ceph is separated because its OSDs got a second interface and stopped using the
-pod network for replication. That is the *only* mechanism that fully detaches a
-workload from the node IP, and it generalises: anything that must not ride the
-node IP gets its own `NetworkAttachmentDefinition`.
-
-It is also expensive per workload — an attachment, an IPAM pool, a pod spec
-change. Worth it for Ceph replication. Not worth it for east-west traffic in
+A per-workload Multus attachment detaches a single workload from the node IP,
+the way Ceph replication is detached. It does not cover east-west traffic in
 general.

-## Recommended mapping
+## Levers per management service

-Two ways to arrange this, and they differ only in which VLAN gets relabelled:
+These hold regardless of which option is chosen:

-**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
-and Talos API. Cleanest match to the class names, but it changes every node's
-identity, which is the one address the whole cluster is administered through.
+| service | can leave the node IP? | lever |
+|---|---|---|
+| etcd | yes | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it); needs a reboot per node |
+| kube-apiserver endpoint | yes | VIP, `Layer2VIPConfig` with a `link:` field |
+| Talos API | n/a | `apid` answers on every node address; a client and firewall choice |
+| kubelet | no | it is the node IP |

-**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
-VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
-Identical end state, and no node ever changes its InternalIP.
+## Remaining, independent of pod-pod

-Under B:
+- **pod→world egress**: default route and masquerade source onto `ingress0`,
+  via route metrics, `devices` and `enable-masquerade-to-route-source`. This
+  moves the nodes' own egress (image pulls, NTP) to VLAN 47 too.
+- **BGP rollback path**: the UDM still accepts sessions from VLAN 40; to be
+  removed once ingress on VLAN 47 is settled.

-| VLAN | carries |
-|---|---|
-| 49 | Ceph replication |
-| 40 (relabelled *cluster*) | pod east-west, kubelet |
-| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
-| new *external* | LoadBalancer ingress, egress default route |
+## Sources

-The property the user asked for — management carries only management traffic —
-holds under B. The label moves; the property is what matters.
-
-## Migration shape
-
-Every step is rolling and reversible. The routing-mode change is gone.
-
-1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
-2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
-   Still unused; this only proves the L2 works.
-3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
-   waiting for a healthy quorum between each.
-4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
-   `k8s.renner.uno` follows.
-5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
-   Reversible by reverting the override; failure is visible immediately in
-   `cilium-dbg shell -- bgp/peers`.
-6. Verify VLAN 40 by what is *absent* from it, not by what still works.
-
-Step 3 is the one that deserves care — etcd is the only step where a mistake
-costs quorum rather than connectivity.
-
-## What this does not buy
-
-Management currently shares a 10 Gb link and is not contended. Nothing observed
-today is slow because of this. The case for the split is that traffic classes
-with different failure modes and security postures should not be
-indistinguishable — not that something is broken.
-
-## Sources and measurements
-
-Source, Cilium v1.20.1 (the running version):
-
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
-  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
-  a next-hop that is not directly reachable.
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
-  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
-  Kubernetes Node.
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
-  to connect nodes in direct routing mode".
-
-Talos:
-
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
-  defaults to it. kube-apiserver and Talos API listen addresses are not
-  covered there.
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
-  `10.2.40.60` with `link: mgmt0`.
-
-Measured on the cluster:
-
- `apid` answered on a node address that is not the InternalIP, so the Talos
-  API is not bound to one address.
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
-  InternalIP had changed since the last session, which is why nothing here
-  names an address as fixed.
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
-  unset and no direct-routing device configured.
-
-Not verified: whether the kube-apiserver's `--advertise-address` follows the
-node IP independently of the VIP. It does not affect the recommendation, since
-clients reach the API through the VIP either way.
+- Cilium v1.20.1 source files listed above; `cilium-agent` flag reference for
+  `--direct-routing-device`.
+- Talos multihoming guide: `cluster.etcd.advertisedSubnets`, `listenSubnets`.
+- PRs #831, #833, #835, #836, #837, #838, #839 and the comments on #576.
+- Live state 2026-09-15: `kubectl get nodes -o wide`, `cilium-config`,
+  `CiliumBGPNodeConfig` status, node routing tables on k8s-1 and k8s-2.
@@ -32,7 +32,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: immich-machine-learning
-          image: ghcr.io/immich-app/immich-machine-learning:v3.2.2 # {"$imagepolicy": "flux-system:immich-machine-learning"}
+          image: ghcr.io/immich-app/immich-machine-learning:v3.2.1 # {"$imagepolicy": "flux-system:immich-machine-learning"}
          ports:
            - name: http
              containerPort: 3003
@@ -87,7 +87,7 @@ spec:
              mountPath: /ca
      containers:
        - name: immich-server
-          image: ghcr.io/immich-app/immich-server:v3.2.2 # {"$imagepolicy": "flux-system:immich-server"}
+          image: ghcr.io/immich-app/immich-server:v3.2.1 # {"$imagepolicy": "flux-system:immich-server"}
          ports:
            - name: http
              containerPort: 3001
@@ -1,107 +0,0 @@
-# Identical on every node and free of addresses. defines.conf and peers.conf
-# are rendered per node by the init container; the first is included above the
-# templates that reference it, the second below them.
-include "/etc/bird/defines.conf";
-
-router id from "pod0";
-log stderr all;
-
-protocol device { scan time 10; }
-
-# Everything a pod may need to reach, in one table, resolved by longest prefix:
-# the node's attached networks, the peers' pod CIDRs, and a default for the
-# rest. No rule has to carve out exceptions, because the specific routes win on
-# their own.
-#
-# The table is BIRD's alone. If this daemon stops, it empties, the rule finds
-# nothing and falls through to main — pod traffic returns to VLAN 40,
-# unseparated rather than misrouted. That is why the default route lives here
-# and not in the Talos config: a default that outlived the specific routes
-# would send pod traffic to the UDM.
-
-# The node's attached networks, taken from the interfaces rather than written
-# down. The usual advice is not to export these to the kernel because it
-# maintains them itself — true of main, not of a table the kernel never fills.
-protocol direct attached {
-  interface "mgmt0", "ingress0", "pod0";
-  ipv4 { table master4; };
-  ipv6 { table master6; };
-}
-
-# Pod egress. GW4/GW6 are the UDM on the ingress segment, published as node
-# annotations beside the address that defines them.
-protocol static egress4 {
-  ipv4 { table master4; };
-  route 0.0.0.0/0 via GW4;
-}
-protocol static egress6 {
-  ipv6 { table master6; };
-  route ::/0 via GW6;
-}
-
-# This node's own pod CIDR, originated rather than read back out of the kernel.
-# Reading it back made BIRD's view depend on the shape of Cilium's route, and
-# the IPv4 one is rejected outright: its next hop sits inside the prefix it
-# describes, which KRT refuses with "strange next-hop".
-#
-# blackhole because it is only ever advertised — it is filtered out below.
-protocol static own4 {
-  ipv4 { table master4; };
-  route OWN_POD4 blackhole;
-}
-protocol static own6 {
-  ipv6 { table master6; };
-  route OWN_POD6 blackhole;
-}
-
-# Everything except this node's own CIDR, which cilium_host already serves in
-# main and which is a blackhole here.
-filter to_pod_table {
-  if source = RTS_STATIC && dest = RTD_BLACKHOLE then reject;
-  accept;
-}
-
-protocol kernel k4 {
-  kernel table 47;
-  metric 0;
-  ipv4 { table master4; import none; export filter to_pod_table; };
-}
-protocol kernel k6 {
-  kernel table 47;
-  metric 0;
-  ipv6 { table master6; import none; export filter to_pod_table; };
-}
-
-# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
-# from ingress0 and is unrelated.
-#
-# direct is explicit because iBGP defaults to multihop, which makes the gateway
-# recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
-# keeps the listener off mgmt0 and ingress0.
-#
-# Only the node's own CIDR is offered: the export filter drops everything the
-# static protocols and the direct protocol produce.
-template bgp pod4 {
-  local as 64514;
-  source address MY4;
-  direct;
-  strict bind;
-  ipv4 {
-    next hop self;
-    import all;
-    export filter { if net = OWN_POD4 then accept; reject; };
-  };
-}
-template bgp pod6 {
-  local as 64514;
-  source address MY6;
-  direct;
-  strict bind;
-  ipv6 {
-    next hop self;
-    import all;
-    export filter { if net = OWN_POD6 then accept; reject; };
-  };
-}
-
-include "/etc/bird/peers.conf";
@@ -1,115 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: bird
-  namespace: kube-system
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: bird
-rules:
-  # The peer list is read from the nodes' pod0 annotations, so no address is
-  # written into this repo twice.
-  - apiGroups: [""]
-    resources: ["nodes"]
-    verbs: ["get", "list"]
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: bird
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: bird
-subjects:
-  - kind: ServiceAccount
-    name: bird
-    namespace: kube-system
---
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: bird
-  namespace: kube-system
-  labels:
-    tier: node
-    app: bird
-spec:
-  selector:
-    matchLabels:
-      app: bird
-  updateStrategy:
-    type: RollingUpdate
-  template:
-    metadata:
-      labels:
-        tier: node
-        app: bird
-    spec:
-      # Carries pod east-west once the routing rules land. Without it table 48
-      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
-      priorityClassName: system-node-critical
-      # Writes host routes and peers on host addresses.
-      hostNetwork: true
-      serviceAccountName: bird
-      # Every node here is a control plane node, so this is not optional.
-      tolerations:
-        - operator: Exists
-          effect: NoSchedule
-      initContainers:
-        - name: render-config
-          # By digest, not by the dated tag: the tag is rebuilt within the day
-          # it names, and the cluster's image path keeps resolving it to the
-          # first build of that day — imagePullPolicy: Always does not help,
-          # because the tag itself resolves stale. Update this when the image
-          # is rebuilt.
-          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -eu
-              cp /config/bird.conf /etc/bird/bird.conf
-              kubectl get node "$NODE_NAME" \
-                -o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
-              kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
-                -o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
-              # BIRD's own parser is the check: a node whose annotation is
-              # missing renders as <no value> and fails here rather than
-              # crashlooping the daemon or forming a mesh with a hole.
-              bird -p -c /etc/bird/bird.conf
-          env:
-            - name: NODE_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: spec.nodeName
-          volumeMounts:
-            - name: config
-              mountPath: /config
-            - name: etc-bird
-              mountPath: /etc/bird
-      containers:
-        - name: bird
-          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
-          resources:
-            requests:
-              cpu: "50m"
-              memory: "32Mi"
-            limits:
-              cpu: "200m"
-              memory: "128Mi"
-          securityContext:
-            capabilities:
-              drop: ["ALL"]
-              # NET_ADMIN writes the routes; NET_BIND_SERVICE binds 179.
-              add: ["NET_ADMIN", "NET_BIND_SERVICE"]
-          volumeMounts:
-            - name: etc-bird
-              mountPath: /etc/bird
-      volumes:
-        - name: config
-          configMap:
-            name: bird-config
-        - name: etc-bird
-          emptyDir: {}
@@ -1,9 +0,0 @@
-define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
-define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
-define GW6 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v6" }};
-define GW4 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v4" }};
-{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
-     IPv6 prefix into an ipv4 protocol, which fails the parse in the init
-     container rather than passing silently. */}}
-define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
-define OWN_POD6 = {{ index .spec.podCIDRs 1 }};
@@ -1,12 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - daemonset.yaml
-configMapGenerator:
-  # Hashed name, so editing any of these rolls the DaemonSet.
-  - name: bird-config
-    namespace: kube-system
-    files:
-      - bird.conf
-      - defines.tmpl
-      - peers.tmpl
@@ -1,7 +0,0 @@
-{{- range $i, $n := .items -}}
-{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
-{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
-# {{ $n.metadata.name }}
-protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
-protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
-{{ end -}}
@@ -7,7 +7,6 @@ resources:
  - cluster-settings/
  - cilium/
  - multus/
-  - bird/
  - cert-manager/
  - snapshot-controller/
  - metrics-server/
@@ -52,12 +52,6 @@ machine:
    # this label instead of listing hostnames, so which boxes carry storage is
    # stated once, here, by the machine that knows.
    storage.renner.uno/ceph-osd: "true"
-  # Publishes this node's pod0 addresses, declared below in its VLANConfig, so
-  # the bird DaemonSet reads its peers from the node list rather than from a
-  # copy. Annotations, not labels: a label value cannot contain a colon.
-  nodeAnnotations:
-    net.renner.uno/pod0-v6: fd02:48::11
-    net.renner.uno/pod0-v4: 10.2.48.11
  udev:
    # Renames every port this node uses to its role at boot, matched by
    # burned-in MAC — kernel names derive from PCI topology and shift when
@@ -188,13 +182,8 @@ addresses:
 # network's clients selectively, so the client goes rather than the record.
 # DHCP is off on VLAN 47 for the same reason.
 #
-# No route here. Table 47 belongs to bird, which fills it with this node's
-# attached networks, the peers' pod CIDRs and the egress default, and empties it
-# if it stops — leaving the rule to fall through to main. A default declared
-# here would outlive the specific routes and send pod traffic to the UDM.
-#
-# accept_ra_defrtr stays 0 on this link so the RA default does not land in main,
-# where mgmt0 owns it.
+# The metric loses to mgmt0's default 1024 deliberately: this pass only adds
+# the segment. The default route moves here in the change that flips it.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
@@ -203,6 +192,9 @@ parent: uplink
 up: true
 addresses:
  - address: 10.2.47.11/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
@@ -33,10 +33,6 @@ machine:
    topology.kubernetes.io/zone: saturn
    # Holds a Ceph OSD drive — see k8s-1.yaml.
    storage.renner.uno/ceph-osd: "true"
-  # pod0 addresses for the bird peer list — see k8s-1.yaml.
-  nodeAnnotations:
-    net.renner.uno/pod0-v6: fd02:48::12
-    net.renner.uno/pod0-v4: 10.2.48.12
  udev:
    # Renames every port to its role at boot — see k8s-1.yaml.
    rules:
@@ -132,6 +128,9 @@ parent: uplink
 up: true
 addresses:
  - address: 10.2.47.12/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.
@@ -29,10 +29,6 @@ machine:
    topology.kubernetes.io/zone: neptune
    # Holds a Ceph OSD drive — see k8s-1.yaml.
    storage.renner.uno/ceph-osd: "true"
-  # pod0 addresses for the bird peer list — see k8s-1.yaml.
-  nodeAnnotations:
-    net.renner.uno/pod0-v6: fd02:48::13
-    net.renner.uno/pod0-v4: 10.2.48.13
  udev:
    # Renames every port to its role at boot — see k8s-1.yaml.
    rules:
@@ -119,6 +115,9 @@ parent: uplink
 up: true
 addresses:
  - address: 10.2.47.13/24
+routes:
+  - gateway: 10.2.47.1
+    metric: 2048
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
 # here on purpose; the link is handed whole to the OSD pod (host-device).
@@ -20,38 +20,3 @@ machine:
      validSubnets:
        - 10.2.40.0/24
        - fd40::/64
-  # The UDM on the ingress segment. Published here rather than as a route
-  # because bird owns table 47 and originates the default from it; this is the
-  # one place the address is stated.
-  nodeAnnotations:
-    net.renner.uno/egress-gw-v6: fd02:47::1
-    net.renner.uno/egress-gw-v4: 10.2.47.1
---
-# All pod traffic goes to table 47, and the table decides — longest prefix
-# match, no exceptions to carve out. bird fills it with the node's attached
-# networks, the peers' pod CIDRs and the egress default, so a packet to a node
-# on the mgmt segment takes the connected route, a packet to a pod on another
-# node takes pod0, and anything else takes the default over ingress0.
-#
-# Matching the source is what keeps the node's own traffic out: it answers on
-# mgmt0, and a reply leaving over ingress0 would carry an mgmt0 source onto
-# VLAN 47. The routing decision runs before Cilium's masquerade hook, so the
-# rule sees the pod address and the masquerade then rewrites it.
-#
-# If bird stops, table 47 empties, the lookup misses and the rule falls through
-# to main — pod traffic returns to VLAN 40, unseparated rather than misrouted.
-#
-# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
-# The two families are separate rule tables, so the priorities need not match;
-# the document name is the priority and must be unique.
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
-name: "1000"
-src: fd10:244::/48
-table: "47"
---
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
-name: "1001"
-src: 10.244.0.0/16
-table: "47"