feat(ceph): serve reads from the local OSD (#576)

Measuring VLAN 40 for the traffic-class split turned up a share of the
management segment that is not management: the node's Ceph client
talking to OSDs on the other two nodes, sourced from the node address
on mgmt0.

Ceph sends every client read to the acting primary of the PG, which
with three OSDs is remote two times out of three. Read affinity has the
client read from the replica in its own CRUSH location instead.

The preconditions hold here. replicapool and cephfs/data0 are size 3
over failureDomain host across three nodes, so every object has a copy
on every node. The OSDs are created under zone buckets
(root=default host=k8s-N zone=<name>) and the nodes carry
topology.kubernetes.io/zone, which is in Rook's default
crushLocationLabels. Ceph is v20.2.4 and the kernel 6.18, against a
documented minimum of 5.8.

Writes are unaffected: they always go to the primary, and the
replication behind them already rides the VLAN 49 cluster network.

cephfs-ci is size 2, so a local replica is not guaranteed there and the
client falls back to the primary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/NETWORK-traffic-classes.md

@@ -0,0 +1,117 @@
+# Traffic classes on the k8s nodes
+
+Issue #576. Three of the four classes are separated; management is not.
+
+## Where each class runs
+
+| class | link | mechanism |
+|---|---|---|
+| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
+| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
+| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
+| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
+| management | `mgmt0`, VLAN 40 | not started |
+
+Hard constraint, set by the owner: inter-node and pod traffic never traverses
+the UDM. Everything above stays on the 10G switch.
+
+## Why policy routing rather than moving the node IP
+
+Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
+and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
+`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
+Both were established by test (#837, #838), not by reading.
+
+So the InternalIP decides where pod traffic goes — unless the lookup happens in
+a different table. The rules do that, and nothing about the node's identity
+moves. The routing decision runs before Cilium's masquerade hook, so a rule
+matching the pod source sees the pod address, and the masquerade then rewrites
+it to the egress link's.
+
+## Rule order
+
+Lower priority runs first.
+
+| prio | match | table |
+|---|---|---|
+| 980 | src pod, dst pod | 48 |
+| 981 | src pod, dst pod (v4) | 48 |
+| 990 | dst `fd00::/8` | main |
+| 991 | dst `10.0.0.0/8` | main |
+| 1000 | src pod | 47 |
+| 1001 | src pod (v4) | 47 |
+
+980/981 match source **and** destination. Source alone would also catch the
+node's own traffic to a pod, which would then leave with a `pod0` source — an
+address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
+`world` rather than `host`.
+
+990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
+cluster-internal traffic out of table 47. Without them a pod's packet to a
+service matches the source rule, finds only a default route, and leaves for the
+UDM — measured, in-cluster DNS stopped resolving.
+
+Traffic to a pod on the same node falls through on its own: table 48 holds only
+the peers' CIDRs, so the lookup misses and 990/991 send it to main.
+
+## Where each table comes from
+
+- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
+  The IPv6 gateway is the UDM's configured address on the segment, not the
+  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
+  default does not land in main.
+- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
+  per peer pod CIDR with the peer's `pod0` address as next hop.
+
+## The bird mesh
+
+`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
+and installs what it learns into table 48. Cilium's routes in main are
+untouched, which is why the two coexist.
+
+No address is written into this repo twice. Each node publishes its `pod0`
+addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
+pod CIDRs come from the Node objects. An init container renders both from the
+API with `kubectl -o go-template-file`, excludes the node itself with a field
+selector, and ends with `bird -p` so BIRD's own parser gates startup.
+
+The node's own CIDR is **originated**, not read back from the kernel. Reading
+it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
+rejected outright: its next hop sits inside the prefix it describes, which KRT
+refuses with `strange next-hop`. The IPv6 route is a device route and passed,
+so the failure was silent on one family only.
+
+## Failure behaviour
+
+Benign at every layer. An empty or absent table 48 makes 980/981 fall through
+to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
+holds if the DaemonSet cannot start, which is the case during a cold rebuild
+until Gitea serves its image.
+
+## Management, the remaining class
+
+Not started. The levers differ per service and none of them is the node IP:
+
+| service | can leave the node IP? | lever |
+|---|---|---|
+| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
+| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
+| Talos API | n/a | `apid` answers on every node address; a firewall question |
+| kubelet | no | it *is* the node IP |
+
+Two approaches were rejected by the owner and are not to be proposed again:
+moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
+cluster VLAN while moving etcd and the API VIP to a new management VLAN.
+Replacing the CNI (#839, Antrea) was shelved.
+
+Note that the reasoning which produced that list also concluded pod↔pod was
+blocked, which policy routing disproved. Treat it as a record of decisions, not
+as an exhaustive analysis.
+
+## Sources
+
+Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
+`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
+`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
+`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
+#855; closed unmerged: #830, #838, #839.

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -1,194 +0,0 @@
-# Proposal — separate the four traffic classes onto their own VLANs
-
-Status: draft. The question this started with is **answered** — from the Cilium
-source and flag reference, not from a test — and the answer changes the shape
-of the proposal rather than confirming it.
-
-Supersedes the framing in issue #576, which described this as "give node
-management its own NIC, and Ceph public its own VLAN". That was two of four
-classes and one of them turned out already solved.
-
-## The four classes
-
-| class | what it is | where it goes today |
-|---|---|---|
-| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
-| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
-| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
-| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
-
-Three of four share one interface. That is not a design decision — until the
-bare-metal conversion there was one NIC to work with.
-
-Ceph client I/O living on the pod network is correct rather than a leftover:
-that traffic *is* cluster-internal. Ceph is done.
-
-## Principle
-
-**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
-separate, changeable decision.**
-
-Defining classes by VLAN means the separation survives recabling, port failures
-and capacity changes, and it does not require one port per class. Defining them
-by interface would freeze a capacity decision into the architecture.
-
-## The finding that shapes everything else
-
-The original open question was whether `nodePort.directRoutingDevice` moves the
-next-hop of Cilium's direct routes, or only selects an interface. It is
-answerable from the source, and the answer is: **only the interface.**
-
-`installDirectRoute` takes the next-hop as a parameter:
-
-```go
-func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
-    skipUnreachable bool) (routeSpec *netlink.Route, err error)
-```
-
-and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
-in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
-same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
-
-`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
-an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
-`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
-cluster there is no path by which Cilium's node IP differs from the kubelet's
-`InternalIP`.
-
-`--direct-routing-device` is documented as "Device name used to connect nodes in
-direct routing mode" — a device, not an address.
-
-**So cluster-internal traffic is welded to the node IP, and the node IP is the
-kubelet InternalIP.** Two consequences:
-
-- The class cannot be moved off whatever VLAN the node IP lives on.
-- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
-  from the node IP; native mode uses it as the direct-route next-hop. Native
-  routing buys nothing here, so the disruptive, non-rolling step this proposal
-  previously contained is simply gone.
-
-## Consequence: draw the line the other way
-
-If cluster-internal cannot leave the node IP, then separation has to come from
-moving *management services* off it — and each one has to be checked
-individually, because they do not share a mechanism.
-
-| management service | can it leave the node IP? | lever |
-|---|---|---|
-| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
-| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
-| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
-| kubelet | **no** | it *is* the node IP, by definition |
-
-So four VLANs are reachable, but the boundary does not fall where the class
-names suggest: **kubelet stays with cluster-internal.** That is the honest
-result, and it is worth stating plainly rather than filing kubelet under
-management and hoping.
-
-The remaining classes keep the levers established earlier:
-
-- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
-  For eBGP the session source is the next-hop, so pinning the session pins
-  where return traffic arrives. The session to the UDM has been running for
-  months.
-- **cluster-external egress** — eBPF masquerading follows the output device, so
-  this is a default-route decision plus `enable-masquerade-to-route-source`
-  (today `false`).
-- **Ceph replication** — Multus macvlan on VLAN 49, done.
-
-## The general escape hatch
-
-Ceph is separated because its OSDs got a second interface and stopped using the
-pod network for replication. That is the *only* mechanism that fully detaches a
-workload from the node IP, and it generalises: anything that must not ride the
-node IP gets its own `NetworkAttachmentDefinition`.
-
-It is also expensive per workload — an attachment, an IPAM pool, a pod spec
-change. Worth it for Ceph replication. Not worth it for east-west traffic in
-general.
-
-## Recommended mapping
-
-Two ways to arrange this, and they differ only in which VLAN gets relabelled:
-
-**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
-and Talos API. Cleanest match to the class names, but it changes every node's
-identity, which is the one address the whole cluster is administered through.
-
-**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
-VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
-Identical end state, and no node ever changes its InternalIP.
-
-Under B:
-
-| VLAN | carries |
-|---|---|
-| 49 | Ceph replication |
-| 40 (relabelled *cluster*) | pod east-west, kubelet |
-| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
-| new *external* | LoadBalancer ingress, egress default route |
-
-The property the user asked for — management carries only management traffic —
-holds under B. The label moves; the property is what matters.
-
-## Migration shape
-
-Every step is rolling and reversible. The routing-mode change is gone.
-
-1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
-2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
-   Still unused; this only proves the L2 works.
-3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
-   waiting for a healthy quorum between each.
-4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
-   `k8s.renner.uno` follows.
-5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
-   Reversible by reverting the override; failure is visible immediately in
-   `cilium-dbg shell -- bgp/peers`.
-6. Verify VLAN 40 by what is *absent* from it, not by what still works.
-
-Step 3 is the one that deserves care — etcd is the only step where a mistake
-costs quorum rather than connectivity.
-
-## What this does not buy
-
-Management currently shares a 10 Gb link and is not contended. Nothing observed
-today is slow because of this. The case for the split is that traffic classes
-with different failure modes and security postures should not be
-indistinguishable — not that something is broken.
-
-## Sources and measurements
-
-Source, Cilium v1.20.1 (the running version):
-
-- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
-  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
-  a next-hop that is not directly reachable.
-- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
-- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
-  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
-  Kubernetes Node.
-- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
-  to connect nodes in direct routing mode".
-
-Talos:
-
-- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
-  defaults to it. kube-apiserver and Talos API listen addresses are not
-  covered there.
-- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
-  `10.2.40.60` with `link: mgmt0`.
-
-Measured on the cluster:
-
-- `apid` answered on a node address that is not the InternalIP, so the Talos
-  API is not bound to one address.
-- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
-  InternalIP had changed since the last session, which is why nothing here
-  names an address as fixed.
-- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
-  unset and no direct-routing device configured.
-
-Not verified: whether the kube-apiserver's `--advertise-address` follows the
-node IP independently of the VIP. It does not affect the recommendation, since
-clients reach the API through the VIP either way.

kubernetes/apps/zone-configurator/deployment.yaml

@@ -21,7 +21,7 @@ spec:
       # Service; nothing on this path needs the host's network namespace.
       containers:
         - name: zone-configurator
-          image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.4 # {"$imagepolicy": "flux-system:mmwave-configurator"}
+          image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.5 # {"$imagepolicy": "flux-system:mmwave-configurator"}
           env:
             - name: HA_BASE_URL
               value: "http://home-assistant:80"

kubernetes/bootstrap.sh

@@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
     # Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
     # would not match the amd64 schematic this cluster installs. That leaves the
     # cx*/cpx*2 families in the German locations.
-    # Today's traffic-class split, not the target one in
-    # PROPOSAL-traffic-class-separation.md: the node address carries management,
+    # Two segments only, unlike the homelab's split in
+    # NETWORK-traffic-classes.md: here the node address carries management,
     # cluster-internal and external alike, and Ceph replication has its own.
     #
     # The order is load-bearing: it decides which kernel interface each segment

kubernetes/infrastructure/bird/bird.conf

@@ -6,84 +6,98 @@ include "/etc/bird/defines.conf";
 router id from "pod0";
 log stderr all;
 
-# Mirrors of the kernel main table, filtered to this node's own pod CIDR.
-# master4/master6 are implicit and hold what BGP exchanges.
-ipv4 table kmain4;
-ipv6 table kmain6;
-
 protocol device { scan time 10; }
 
-# Two conditions. cilium_host also carries its own host address, which the
-# length check drops; and Cilium's routes for the OTHER nodes are protocol
-# kernel too, which the interface check drops. ifname is set for the
-# gatewayless IPv6 device route as well, so this works in both families.
-filter own_pod4 {
-  if ifname !~ "cilium_host" then reject;
-  if net.len = 32 then reject;
-  accept;
-}
-filter own_pod6 {
-  if ifname !~ "cilium_host" then reject;
-  if net.len = 128 then reject;
-  accept;
+# Everything a pod may need to reach, in one table, resolved by longest prefix:
+# the node's attached networks, the peers' pod CIDRs, and a default for the
+# rest. No rule has to carve out exceptions, because the specific routes win on
+# their own.
+#
+# The table is BIRD's alone. If this daemon stops, it empties, the rule finds
+# nothing and falls through to main — pod traffic returns to VLAN 40,
+# unseparated rather than misrouted. That is why the default route lives here
+# and not in the Talos config: a default that outlived the specific routes
+# would send pod traffic to the UDM.
+
+# The node's attached networks, taken from the interfaces rather than written
+# down. The usual advice is not to export these to the kernel because it
+# maintains them itself — true of main, not of a table the kernel never fills.
+protocol direct attached {
+  interface "mgmt0", "ingress0", "pod0";
+  ipv4 { table master4; };
+  ipv6 { table master6; };
 }
 
-# learn all, not learn: plain learn skips routes marked proto kernel, which is
-# what Cilium's are.
-protocol kernel klearn4 {
-  learn all;
-  scan time 10;
-  ipv4 { table kmain4; import filter own_pod4; export none; };
+# Pod egress. GW4/GW6 are the UDM on the ingress segment, published as node
+# annotations beside the address that defines them.
+protocol static egress4 {
+  ipv4 { table master4; };
+  route 0.0.0.0/0 via GW4;
 }
-protocol kernel klearn6 {
-  learn all;
-  scan time 10;
-  ipv6 { table kmain6; import filter own_pod6; export none; };
+protocol static egress6 {
+  ipv6 { table master6; };
+  route ::/0 via GW6;
 }
 
-protocol pipe own4 { table kmain4; peer table master4; export all; import none; }
-protocol pipe own6 { table kmain6; peer table master6; export all; import none; }
-
-# Only BGP-learned routes reach table 48. The local CIDR stays in main, where
-# cilium_host already serves it.
-filter to_t48 {
-  if source != RTS_BGP then reject;
-  accept;
+# This node's own pod CIDR, via the interface Cilium serves it on. Originated
+# rather than read back out of the kernel: reading it back made BIRD's view
+# depend on the shape of Cilium's route, and the IPv4 one is rejected outright,
+# its next hop sitting inside the prefix it describes.
+#
+# It belongs in the table, not just in the BGP announcement. A reply from a pod
+# on another node is forwarded traffic with a pod source, so the rule matches it
+# too; without this route it would find only the default and leave for the UDM.
+# Measured — cross-node pod traffic, and with it cluster DNS, stopped.
+protocol static own4 {
+  ipv4 { table master4; };
+  route OWN_POD4 via "cilium_host";
+}
+protocol static own6 {
+  ipv6 { table master6; };
+  route OWN_POD6 via "cilium_host";
 }
 
-protocol kernel k48v4 {
-  kernel table 48;
+protocol kernel k4 {
+  kernel table 47;
   metric 0;
-  ipv4 { table master4; import none; export filter to_t48; };
+  ipv4 { table master4; import none; export all; };
 }
-protocol kernel k48v6 {
-  kernel table 48;
+protocol kernel k6 {
+  kernel table 47;
   metric 0;
-  ipv6 { table master6; import none; export filter to_t48; };
+  ipv6 { table master6; import none; export all; };
 }
 
 # iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
 # from ingress0 and is unrelated.
 #
-# No prefix filters: the only route this node offers is its own pod CIDR, which
-# own_pod4/6 already selected, and the only peers are the other cluster nodes.
-#
 # direct is explicit because iBGP defaults to multihop, which makes the gateway
 # recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
 # keeps the listener off mgmt0 and ingress0.
+#
+# Only the node's own CIDR is offered: the export filter drops everything the
+# static protocols and the direct protocol produce.
 template bgp pod4 {
   local as 64514;
   source address MY4;
   direct;
   strict bind;
-  ipv4 { next hop self; import all; export all; };
+  ipv4 {
+    next hop self;
+    import all;
+    export filter { if net = OWN_POD4 then accept; reject; };
+  };
 }
 template bgp pod6 {
   local as 64514;
   source address MY6;
   direct;
   strict bind;
-  ipv6 { next hop self; import all; export all; };
+  ipv6 {
+    next hop self;
+    import all;
+    export filter { if net = OWN_POD6 then accept; reject; };
+  };
 }
 
 include "/etc/bird/peers.conf";

kubernetes/infrastructure/bird/daemonset.yaml

@@ -60,9 +60,12 @@ spec:
           effect: NoSchedule
       initContainers:
         - name: render-config
-          image: gitea.renner.uno/gitea_admin/bird:2026-09-16
-          # The dated tag is rebuilt within the day it names.
-          imagePullPolicy: Always
+          # By digest, not by the dated tag: the tag is rebuilt within the day
+          # it names, and the cluster's image path keeps resolving it to the
+          # first build of that day — imagePullPolicy: Always does not help,
+          # because the tag itself resolves stale. Update this when the image
+          # is rebuilt.
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
           command: ["/bin/sh", "-c"]
           args:
             - |
@@ -88,8 +91,7 @@ spec:
               mountPath: /etc/bird
       containers:
         - name: bird
-          image: gitea.renner.uno/gitea_admin/bird:2026-09-16
-          imagePullPolicy: Always
+          image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
           resources:
             requests:
               cpu: "50m"

kubernetes/infrastructure/bird/defines.tmpl

@@ -1,2 +1,9 @@
 define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
 define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
+define GW6 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v6" }};
+define GW4 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v4" }};
+{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
+     IPv6 prefix into an ipv4 protocol, which fails the parse in the init
+     container rather than passing silently. */}}
+define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
+define OWN_POD6 = {{ index .spec.podCIDRs 1 }};

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -126,6 +126,16 @@ spec:
     addressRanges:
       cluster:
         - 10.99.0.0/29
+  # Reads go to the OSD in the client's own CRUSH location. crushLocationLabels
+  # is unset, so Rook's default list applies; of those the nodes carry
+  # topology.kubernetes.io/zone, one zone per node, matching the zone buckets
+  # the OSDs are created under.
+  #
+  # Passed as a krbd/mount option at map time, so a volume keeps its current
+  # behaviour until it is remounted.
+  csi:
+    readAffinity:
+      enabled: true
   # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
   # re-provisions an OSD on any node whose device still carries a BlueStore
   # signature, so purging an OSD before pulling its drive is impossible — it

kubernetes/talos/patches/k8s-1.yaml

@@ -188,14 +188,13 @@ addresses:
 # network's clients selectively, so the client goes rather than the record.
 # DHCP is off on VLAN 47 for the same reason.
 #
-# Both defaults live in table 47, not in main, so only the pod-source rules in
-# site-homelab.yaml reach them. mgmt0 keeps main's default and with it every
-# reply the node sends from its own addresses.
+# No route here. Table 47 belongs to bird, which fills it with this node's
+# attached networks, the peers' pod CIDRs and the egress default, and empties it
+# if it stops — leaving the rule to fall through to main. A default declared
+# here would outlive the specific routes and send pod traffic to the UDM.
 #
-# The v6 gateway is the UDM's configured ULA on this segment, the same address
-# BGP peers with. The RA-learned link-local would be the alternative and is not
-# a value to write down. accept_ra_defrtr stays 0 for that reason: the RA
-# default would land in main, where it does not belong.
+# accept_ra_defrtr stays 0 on this link so the RA default does not land in main,
+# where mgmt0 owns it.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ingress0
@@ -204,11 +203,6 @@ parent: uplink
 up: true
 addresses:
   - address: 10.2.47.11/24
-routes:
-  - gateway: fd02:47::1
-    table: "47"
-  - gateway: 10.2.47.1
-    table: "47"
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication and nothing else. The udev rule in the machine

kubernetes/talos/patches/k8s-2.yaml

@@ -132,11 +132,6 @@ parent: uplink
 up: true
 addresses:
   - address: 10.2.47.12/24
-routes:
-  - gateway: fd02:47::1
-    table: "47"
-  - gateway: 10.2.47.1
-    table: "47"
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.

kubernetes/talos/patches/k8s-3.yaml

@@ -119,11 +119,6 @@ parent: uplink
 up: true
 addresses:
   - address: 10.2.47.13/24
-routes:
-  - gateway: fd02:47::1
-    table: "47"
-  - gateway: 10.2.47.1
-    table: "47"
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
 # here on purpose; the link is handed whole to the OSD pod (host-device).

kubernetes/talos/patches/site-homelab.yaml

@@ -20,41 +20,32 @@ machine:
       validSubnets:
         - 10.2.40.0/24
         - fd40::/64
+  # The UDM on the ingress segment. Published here rather than as a route
+  # because bird owns table 47 and originates the default from it; this is the
+  # one place the address is stated.
+  nodeAnnotations:
+    net.renner.uno/egress-gw-v6: fd02:47::1
+    net.renner.uno/egress-gw-v4: 10.2.47.1
 ---
-# Pod egress leaves over ingress0. Matching on the pod source prefix is what
-# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
-# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
-# source onto VLAN 47.
+# All pod traffic goes to table 47, and the table decides — longest prefix
+# match, no exceptions to carve out. bird fills it with the node's attached
+# networks, the peers' pod CIDRs and the egress default, so a packet to a node
+# on the mgmt segment takes the connected route, a packet to a pod on another
+# node takes pod0, and anything else takes the default over ingress0.
 #
-# The routing decision runs before Cilium's masquerade hook, so the rule sees
-# the pod address and the masquerade then rewrites it to ingress0's.
+# Matching the source is what keeps the node's own traffic out: it answers on
+# mgmt0, and a reply leaving over ingress0 would carry an mgmt0 source onto
+# VLAN 47. The routing decision runs before Cilium's masquerade hook, so the
+# rule sees the pod address and the masquerade then rewrites it.
 #
-# Table 47 holds nothing but the two default routes, so the destination rules
-# below have to run first: without them a pod's packet to a service or to a pod
-# on another node also matches the source rule, finds only the default, and
-# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
-#
-# They send everything inside the cluster and the local networks back to main.
-# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
-# is not foreign traffic.
+# If bird stops, table 47 empties, the lookup misses and the rule falls through
+# to main — pod traffic returns to VLAN 40, unseparated rather than misrouted.
 #
 # Priorities are below main (32766) and clear of the rule Cilium installs at 9.
 # The two families are separate rule tables, so the priorities need not match;
 # the document name is the priority and must be unique.
 apiVersion: v1alpha1
 kind: RoutingRuleConfig
-name: "990"
-dst: fd00::/8
-table: main
----
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
-name: "991"
-dst: 10.0.0.0/8
-table: main
----
-apiVersion: v1alpha1
-kind: RoutingRuleConfig
 name: "1000"
 src: fd10:244::/48
 table: "47"