docs(proposal): public edge measured from the proxies — cooldown is the lever

Watchdog journals on proxy-1/proxy-2 for the morning roll, plus a
controlled experiment (three routing-peer kills probed from proxy-2 at
2 s; the LAN cannot probe this path, the UDM hairpin-DNATs the proxy
addresses to the VIP):

- a pinned-peer loss wedges the whole proxy process; the flood watchdog
  restarts it within 2 s, the probe sees one failed round (~3 s)
- inside the flood watchdog 600 s cooldown the restart is suppressed and
  the URL watchdog needs three one-minute rounds: 3.5 min of 000 on every
  public domain, measured; the morning k8s-2 roll hit the same suppression
- subnet-target services: HTTP mode + IP target -> Host header is the IP,
  Traefik 404; HTTP mode + domain target -> SERVFAIL, no mesh DNS on the
  proxy; TLS passthrough (mode tls, listen_port 443) + subnet target works
  end to end and coexists on 443 -- but dies with the wedge as long as any
  domain resource exists

Stage 5 rewritten: 5a cooldown 600->60 s (ships first), 5b identity
persistence as a hypothesis to measure, 5c all-passthrough as the
structural exit with its VIP reconciler cost. Side findings: proxy clocks
3 min slow, ACME on the v6-only member ~20 min late, no proxy syslog in
Loki.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kubernetes/PROPOSAL-node-roll-downtime.md

@@ -0,0 +1,304 @@
+# Proposal — a node roll without the ingress hiccup
+
+Status: draft, grounded in one measured roll. On 2026-09-07 all three MS-01
+were drained, patched and rebooted back to back (06:52–07:08) for the SAN
+rollout. Everything below is read off that window: Traefik access logs and
+Cilium BGP logs in Loki, the drain logs, and the live cluster afterwards.
+
+The short version: there are two ingresses and they failed differently.
+**The LAN ingress (Traefik behind the BGP VIP) did not hiccup** — it
+answered every request; the 5xx it returned were for backends mid-move.
+**The public edge (proxies → NetBird mesh → Traefik) went dark for 40–80 s
+on two of the three rolls, and would have for 3½ minutes on the third had
+its routes not recovered by luck** — a known client bug the repo already
+documents and works around with a watchdog, whose 600 s restart cooldown is
+what turns a 3-second blip into minutes when rolls follow each other. So
+this is two projects: make evictions cheap, and make a peer loss on the
+mesh cost seconds, deterministically.
+
+## What was measured
+
+Traefik, 06:48–07:12, 24 minutes, 3 391 requests, of which 5xx:
+
+| router | status | count |
+|---|---|---|
+| gitea | 503 | 56 |
+| home-assistant | 503 | 36 |
+| shelly-manager | 502 | 25 |
+| immich | 503 | 19 |
+| openbao | 503 | 13 |
+| grafana | 503 | 10 |
+| music-assistant | 503 | 9 |
+| crowdsec-lapi | 503 | 7 |
+| wordpress | 503 | 6 |
+| woodpecker | 503 | 4 |
+
+The quiet 24 minutes before the roll had zero 5xx. Every entry above is a
+single-replica workload (or its single database) that was evicted and had to
+come up on another node — pod reschedule, RBD volume detach and re-attach,
+process start. That is 30–90 s of 503 per app per drain.
+
+Two things made it worse than one move per app:
+
+- **Double moves.** The drain of k8s-1 pushed pods onto k8s-2, which was
+  drained next: `ha-pg-1`, `ak-pg-1`, `im-pg-1`, `pprl-pg-2`, `tr-pg-1`,
+  `wp-pg-1`, `influxdb`, `protonmail-bridge`, `prometheus`, `wordpress-mariadb`
+  each appear in two drain logs. Half the downtime was paid twice.
+- **`--disable-eviction`.** Ten PodDisruptionBudgets allow zero disruptions
+  (every CNPG primary, gitea, influxdb, syncthing, protonmail-bridge,
+  woodpecker-server). A plain `kubectl drain` hangs on them, so the roll used
+  `--disable-eviction`, which deletes pods outright. For the two-instance CNPG
+  clusters (authentik, immich, paperless) that skipped the switchover CNPG
+  would have done: the operator's documented behaviour is "if a node is to be
+  drained and contains a cluster's primary instance, a switchover happens
+  ahead of the drain" — but only when the drain goes through the eviction API.
+  We turned a 2-second switchover into a 30–60 s primary restart.
+
+Beside the HTTP path, one reboot side effect took a dependency down for
+half an hour: openbao-0 came back **sealed**, and because the StatefulSet is
+`OrderedReady`, openbao-1/2 were never recreated after their eviction until
+openbao-0 was unsealed by hand. ESO could not refresh a secret for 35 minutes.
+
+### The public edge
+
+Public traffic reaches Traefik with client address `127.0.0.1` (the rootless
+netbird-ingress peer forwards from inside its pod through socket-LB), LAN
+traffic with `10.2.x`. That splits the access log by path. At that hour the
+only public traffic was the proxies' own watchdog — two hosts probed once a
+minute from each of two proxies, a flat 4 requests/min — which makes it a
+clean synthetic probe of the whole external chain:
+
+| minute (UTC) | LAN req | public req | what was happening |
+|---|---|---|---|
+| 04:55 | 128 | 4 | k8s-1 rebooting (04:55:10) |
+| 04:56 | 109 | 2 | k8s-1 back, its peer re-enrolled 04:56:24 |
+| 04:57 | 127 | **0** | — |
+| 04:58 | 126 | 4 | proxies restarted by their watchdog |
+| 05:00–05:03 | 186–233 | 4 | k8s-2 roll: **no dip** |
+| 05:06–05:07 | 97–135 | 2 | k8s-3 roll: half |
+
+The external chain was down for roughly two minutes at the k8s-1 roll, one
+at k8s-3, and not at all at k8s-2. The mechanism is in
+`ansible/roles/netbird-proxy-watchdog/defaults/main.yml`: the proxy's
+embedded NetBird client pins each domain route to one routing-peer
+*identity*; the netbird-ingress DaemonSet enrols with a setup key, so every
+pod start is a **new identity** (the k8s-1 pod registered fresh at
+04:56:24); and upstream's dns-interceptor never rebinds a route whose
+owning peer is gone ("already routed by peer … HA routing disabled"). The
+proxy then serves 502 for every domain until its container restarts —
+which the watchdog does after three failed one-minute probes, or at once
+when the flood watchdog sees the wedge in the proxy log. Which roll hurts
+depends on which of the three peers the proxies happened to pin. k8s-2's
+peer was pinned by neither.
+
+Note what did *not* help: the DaemonSet's preStop (`netbird down`) took the
+peer out of the mesh cleanly at 04:55:11, and the peer was back 73 s later.
+Neither matters to a client that will not rebind and will not recognise
+the returning pod as the same peer.
+
+## What was not the cause
+
+**BGP.** Per node the UDM sessions were down for 47–51 s (Peer Down at the
+Cilium shutdown, Peer Up ~50 s later). That gap did not carry traffic: every
+LoadBalancer here is `externalTrafficPolicy: Local`, so a VIP is only
+advertised from nodes with a local endpoint, and the drain had already
+evicted those endpoints — Cilium withdrew the routes while the session was
+still healthy (the "soft reset out" bursts during each drain). The 90 s hold
+time and 120 s graceful-restart window never came into play. Leave them.
+
+**Traefik.** Two replicas on k8s-1/k8s-2, one evicted per roll, the other
+carried the VIP alone. No PDB, no spread constraint, no preStop — which held
+by luck (the two pods happened to sit on different nodes), and is fixed below
+because luck is not a design.
+
+## Target state, in order of measured impact
+
+### 1. Drain in a way that lets operators do their job (procedure only)
+
+- **Cordon the next node before draining the current one.** Evicted pods
+  then land on the node that stays up, and nothing moves twice. Order:
+  cordon B → drain A → reboot A → uncordon A → cordon C → drain B → …
+- **Never `--disable-eviction`.** It exists in the roll only because ten
+  PDBs say "never". Stage 2 makes those PDBs honest, after which a plain
+  `kubectl drain --ignore-daemonsets --delete-emptydir-data` completes and
+  CNPG switches primaries over before its replicas are evicted.
+- **After every reboot, three gates before uncordon:** `00-multus.conf` on
+  the host and Multus + whereabouts Running (k8s-1's OSD came up
+  Cilium-only today because it beat the Multus DaemonSet by 30 s); OpenBao
+  unsealed (`bao status` on every pod that exists, then the ones the
+  StatefulSet creates after it); Ceph back at baseline.
+- This is `rollout-node.sh` from today's window, minus `--disable-eviction`,
+  plus the OpenBao gate, checked into `scripts/`. A reboot runbook that lives
+  in a scratchpad is a runbook nobody else can run.
+
+### 2. PodDisruptionBudgets that mean what they say
+
+A `minAvailable: 1` PDB on a single-replica workload does not protect
+anything; it only forces the operator to bypass all PDBs. Two changes:
+
+- **CNPG single-instance clusters** (grafana-db, home-assistant, trmnl,
+  woodpecker): `spec.enablePDB: false`. CNPG documents exactly this — "for
+  single-instance clusters the operator will prevent draining the node" and
+  recommends moving to direct control of disruption budgets. The instance
+  restarts on drain (it does today too, just through the back door), and the
+  multi-instance clusters keep their switchover-before-eviction behaviour.
+- **gitea, influxdb, syncthing, protonmail-bridge, woodpecker-server**: drop
+  the `minAvailable: 1` PDBs. They are single replicas with RWO volumes; the
+  PDB cannot buy availability, only stalls.
+
+### 3. The ingress itself: three replicas, one per node, drained gracefully
+
+Traefik goes to `replicas: 3` with a required `podAntiAffinity` on
+hostname, `PodDisruptionBudget maxUnavailable: 1`, and a preStop that lets
+in-flight requests finish before the endpoint is withdrawn:
+
+```yaml
+# chart keys as in traefik/values.yaml
+deployment:
+  replicas: 3
+  lifecycle:
+    preStop:
+      sleep:
+        seconds: 5
+affinity:
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      - topologyKey: kubernetes.io/hostname
+        labelSelector:
+          matchLabels:
+            app.kubernetes.io/name: traefik
+podDisruptionBudget:
+  enabled: true
+  maxUnavailable: 1
+ports:
+  websecure:
+    transport:
+      lifeCycle:
+        graceTimeOut: 15s
+```
+
+With `Local` traffic policy every node then advertises the VIP; a roll takes
+one of three ECMP paths away instead of one of two, and the UDM sees the
+withdrawal before the pod stops accepting. Also gives the UDM a route from
+k8s-3, which today advertises nothing for the ingress.
+
+### 4. Databases the apps actually wait on
+
+The 503 table is a priority list. Each of these is a single Postgres today:
+
+| app | 5xx | cluster | today |
+|---|---|---|---|
+| home-assistant | 36 | ha-pg | `instances: 1` |
+| grafana | 10 | gf-pg | `instances: 1` |
+| woodpecker | 4 | wp-pg | `instances: 1` |
+| trmnl | — | tr-pg | `instances: 1` |
+
+`instances: 2` for ha-pg and gf-pg turns a drain into a CNPG switchover
+(seconds). The cost is one more Postgres pod and a second RBD volume each.
+trmnl and woodpecker can stay at one: they were barely visible.
+
+gitea (56) is the largest number and has no cheap answer: one StatefulSet,
+one RWO volume, an application that does not run two instances against the
+same data directory without shared storage. It moves once per roll (with
+stage 1) and that is its floor unless its repositories move to CephFS.
+
+### 5. The public edge: what the proxies' own journals say
+
+This is the only *complete* outage in the window, so despite the numbering
+it ships right after stage 1. It was measured twice: from the proxies'
+watchdog journals for the morning roll, and in a controlled afternoon
+experiment (three routing-peer kills, probed from proxy-2 itself at 2 s —
+the LAN cannot probe this path, the UDM hairpin-DNATs the proxy addresses
+straight to the VIP). Proxy clocks run ~3 min behind the cluster.
+
+**What a peer loss does.** When the routing peer the proxy has pinned
+disappears, the proxy's route manager wedges (`couldn't fetch peer state`
+burst) and the *whole* process stops answering — every domain, and in the
+experiment also services that never used a domain route. The flood
+watchdog restarts the container within 2 s of the burst; the probe saw
+one failed round (≈3 s) at the first two kills. The third kill fell inside
+the flood watchdog's **600 s restart cooldown**: restart suppressed, wedge
+left standing, and the URL watchdog needed its three one-minute rounds —
+**3½ minutes of 000 on every public domain**. The morning roll had the
+same shape: k8s-1 (both proxies restarted, 40–80 s), k8s-2 (wedge at
+06:58, restart suppressed on both proxies, 278 s after the previous one —
+the routes happened to recover on their own), k8s-3 (restart, ~10 s).
+Three rolls in sixteen minutes guarantee the suppressed case.
+
+- **5a. Cooldown 600 s → 60 s** in
+  `ansible/roles/netbird-proxy-watchdog/defaults/main.yml`. The cooldown
+  exists so a genuinely broken backend cannot restart-loop the proxy; a
+  minute still does that (the URL watchdog needs three minutes to fire
+  anyway). With it, every peer loss — rolls back to back included — is a
+  ~3 s blip instead of a lottery between 3 s and 3½ min. Cheapest change
+  in this document, ships first.
+- **5b. Persist the peer identity.** Replace the DaemonSet with a
+  StatefulSet of three, required anti-affinity on hostname,
+  `PodDisruptionBudget maxUnavailable: 1`, and a small RBD volume per
+  replica holding NetBird's state directory. A drain then evicts the pod
+  through the PDB (the DaemonSet pod is not evicted at all today, it dies
+  with the reboot) and it returns within ~30 s **as the same peer**. Whether
+  the route manager then un-wedges without a restart is the hypothesis to
+  measure; the setup-key constraint (ephemeral peers reaped after >10 min
+  offline, NeedsLogin liveness restart) holds unchanged.
+- **5c. The structural exit: no domain routes at all.** The subnet-target
+  experiment settled what works: an HTTP-mode service with an IP target
+  forwards `Host: <ip>` and Traefik answers 404 (NetBird rejects a Host
+  override); an HTTP-mode service with a domain target needs mesh DNS on
+  the proxy, which answers SERVFAIL there; a **TLS-passthrough service
+  (`mode: tls`, `listen_port: 443`) with a subnet target works** — SNI and
+  Host reach Traefik untouched, it coexists with the HTTP listener on 443,
+  no `skip_tls_verify`. If every public service became such a passthrough
+  over a subnet resource covering the Cilium LB pool (`10.2.99.0/24`,
+  `fd99::/64`), there would be no domain resource left to wedge on, and
+  peer loss would be NetBird's documented HA failover (seconds). Costs: the
+  target's `host` is the VIP, so a reconciler (the external-dns-unifi
+  pattern, `PUT /api/reverse-proxies/services/{id}` from the Traefik
+  Service's `status.loadBalancer.ingress`) has to own it; the proxy's L7
+  features (auth, path rules) are lost — none are in use; and the pure
+  failover time is unmeasured, because today every kill wedged the domain
+  routes first. Decide after 5a and 5b have been measured on a real roll.
+- **Track the upstream bug.** The repo's note dates the dns-interceptor
+  behaviour to 2026-07; a search of NetBird's releases and issues turned up
+  no fix for the "already routed by peer / HA routing disabled" path.
+  Re-check on each NetBird bump.
+- **Side findings, each its own small change:** both proxy clocks are
+  ~3 min slow (the proxy logs SCT timestamps "in the future"; TLS
+  validity windows care) — NTP on proxy-1/proxy-2; a new service's
+  certificate reaches the IPv6-only member only after ~20 min of ACME
+  retries (Let's Encrypt validates over IPv4 and lands on the other member,
+  which has no token for it); proxy-1 and proxy-2 do not ship syslog to
+  Loki (edge-1 does), so every public-edge timeline is reconstructed over
+  SSH — point them at the alloy-syslog VIP.
+
+## Expected result
+
+| stage | removes |
+|---|---|
+| 1 | every double move (half of today's 5xx), the primary restarts, the 35-minute OpenBao/ESO outage, the Multus race |
+| 2 | the need for `--disable-eviction` at all |
+| 3 | the single-node ingress exposure; ECMP loses 1/3 instead of 1/2 |
+| 4 | home-assistant and grafana 503s entirely |
+| 5a | the 3½-minute public-edge blackout of a second peer loss within 10 min; every peer loss becomes a ~3 s restart |
+| 5b/5c | the restart itself, if the returning identity (5b) or the absence of domain routes (5c) proves to un-wedge — to be measured |
+
+What remains after all of it: one move per single-replica app per roll
+(gitea, immich, shelly-manager, music-assistant, wordpress), 30–60 s each,
+once, and a few seconds of public edge per roll. That is the honest floor
+for single-replica services on RWO storage and a mesh client that pins.
+
+## Transition
+
+1. Stage 5a is one Ansible default and a proxy deploy — ships first.
+2. Stage 1 is a script and a runbook change — no cluster change.
+3. Stage 5b's StatefulSet changes how the public edge enrols, so it gets
+   its own roll to measure, probed from a proxy.
+4. Stage 2 is one field on four CNPG clusters and five deleted PDBs; each
+   Kustomization applies without disruption.
+5. Stage 3 is a Traefik HelmRelease change; the rollout itself is a rolling
+   update under the new PDB.
+6. Stage 4 is `instances: 2` on two clusters; CNPG adds the replica online.
+7. Stage 5c only if 5a+5b leave a measurable public gap.
+
+Measure the next roll the same way: 5xx by router, the proxies' watchdog
+journals (or their syslog in Loki), BGP Down→Up gaps, pods per drain log.

kubernetes/bootstrap.sh

@@ -1379,8 +1379,8 @@ EOF
             vm_addr="${NODE_ADDRS[$idx]}"
             vols=$(talosctl -n "$vm_addr" -e "$vm_addr" get discoveredvolumes 2>&1) \
                 || fail "could not read volumes from $vm_node ($vm_addr): ${vols//$'\n'/ | }"
-            awk '$NF == "r-ceph-osd" {found=1} END {exit !found}' <<<"$vols" \
-                || fail "$vm_node has no r-ceph-osd partition — RawVolumeConfig did not take"
+            awk '$NF == "r-osd-data" {found=1} END {exit !found}' <<<"$vols" \
+                || fail "$vm_node has no r-osd-data partition — RawVolumeConfig did not take"
             kubectl get node "$vm_node" \
                 -o jsonpath='{.metadata.labels.storage\.renner\.uno/ceph-osd}' \
                 | grep -q true \

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -95,12 +95,18 @@ spec:
   # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
   # Multus. The attachment is san-network-attachment.yaml.
   #
-  # Only `cluster` is selected. Do not add `public`, and do not reach for
-  # `addressRanges` — that is honoured only under provider: host, which puts
-  # every daemon including the mons on host networking. A mon's address then
-  # becomes the node's DHCP lease, and when a lease moves the mon crash-loops
-  # on "Cannot assign requested address" with no node free to fail over to,
-  # because all three already hold the mon host ports.
+  # Only `cluster` is selected. Do not add `public`: under provider: host
+  # that puts every daemon including the mons on host networking. A mon's
+  # address then becomes the node's DHCP lease, and when a lease moves the mon
+  # crash-loops on "Cannot assign requested address" with no node free to
+  # fail over to, because all three already hold the mon host ports.
+  #
+  # addressRanges.cluster is stated, not discovered. Without it Rook runs a
+  # canary pod on an OSD node to read the CIDR off the attachment on EVERY
+  # reconcile; the attachment is host-device, so the OSD on that node already
+  # holds the NIC and the canary can never attach — the reconcile blocks for
+  # 15 minutes, fails, and repeats. The value must equal the whereabouts range
+  # in san-network-attachment.yaml.
   #
   # With `public` unset, client traffic stays on the pod network and VLAN 49
   # carries Ceph-internal traffic only — which is also why Rook's Multus
@@ -117,6 +123,9 @@ spec:
     provider: multus
     selectors:
       cluster: rook-ceph/ceph-san
+    addressRanges:
+      cluster:
+        - 10.99.0.0/29
   # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
   # re-provisions an OSD on any node whose device still carries a BlueStore
   # signature, so purging an OSD before pulling its drive is impossible — it
@@ -144,10 +153,12 @@ spec:
     useAllDevices: false
     # Two forms while the production drives are migrated:
     #
-    #   by-partlabel/r-ceph-osd  the target. Talos names the drive in the
-    #                            machine config (RawVolumeConfig ceph-osd), so
+    #   by-partlabel/r-osd-data  the target. Talos names the drive in the
+    #                            machine config (RawVolumeConfig osd-data), so
     #                            this file needs to know nothing about models,
-    #                            serials or hostnames.
+    #                            serials or hostnames. The label must not
+    #                            contain "ceph": ceph-volume rejects any such
+    #                            partition as a legacy ceph-disk member.
     #   by-id/nvme-SAMSUNG…      the three MS-01 drives as they are today:
     #                            bluestore written straight onto the raw disk,
     #                            with no partition table and therefore no
@@ -159,7 +170,7 @@ spec:
     #
     # The second half goes when those three are repartitioned, one node at a
     # time. Until then it is what keeps the running OSDs claimed.
-    devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
+    devicePathFilter: "^/dev/disk/by-(partlabel/r-osd-data|id/nvme-SAMSUNG_MZQLB960HAJR)"
     config:
       osdsPerDevice: "1"
   # Highest built-in class: kubelet evicts these last and the kernel gives

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -3,33 +3,26 @@
 # without every Ceph daemon landing on host networking to get there. The
 # alternative and why it was rejected is in cephcluster.yaml.
 #
+# host-device: the SAN NIC is dedicated to Ceph and has exactly one consumer
+# per node — the OSD pod — so it is handed to that pod whole, not virtualised
+# under it. The device keeps its identity (MAC, MTU) and moves back to the
+# host when the pod goes away.
+#
+# `device` names the role, not hardware: every machine renames its own SAN
+# interface to ceph-san at boot (udev rule in its Talos patch). No kernel
+# interface name appears in this file.
+#
 # ISOLATION
 #
-# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
-# are deliberate:
+# VLAN 49 carries Ceph-internal traffic and nothing else. No gateway and no
+# routes: there is no router on this segment and there must not be one.
+# whereabouts adds neither unless asked, so nothing here can leak onto or off
+# the SAN.
 #
-#   No gateway and no routes. There is no router on this segment and there
-#   must not be one. whereabouts adds neither unless asked, so nothing here
-#   can leak onto or off the SAN. This also means no `sbr` chaining is
-#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
-#   route to confine, because there is no DHCP and no default route.
-#
-#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
-#   migration ran, because the nodes still held .1-.3 for as long as Ceph
-#   needed host networking to reach this link — three addresses for three
-#   OSDs, no spare. The nodes have since given those up (the link is only a
-#   macvlan parent now and needs no address of its own), so the reservation
-#   has nothing left to avoid.
-#
-#   Keeping the pool inside the /29 rather than moving it elsewhere on the
-#   segment is not cosmetic. There is no router here, so an address outside
-#   the /29 would be unreachable from anything still using one inside it —
-#   which is exactly what would break a future rolling change, where migrated
-#   and un-migrated OSDs have to keep talking to each other.
-#
-#   Six addresses for three OSDs is the headroom the migration could not
-#   afford: room for a leaked allocation to sit until the reconciler collects
-#   it, and room to add a second OSD per node without touching this file.
+# The pool is the whole usable /29, .1 through .6 — headroom for a leaked
+# allocation to sit until the reconciler collects it. One address per node is
+# in use; a second OSD on a node would need a second SAN interface, which is a
+# hardware change, not a config one.
 #
 # IPAM
 #
@@ -37,29 +30,18 @@
 # a server that must not exist here, and host-local keeps its state per node
 # with no cluster-wide coordination, so two nodes would hand out the same
 # address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
-#
-# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
-# it is stated so that a change to the parent does not silently halve
-# replication throughput without anything in the tree recording the
-# expectation.
 apiVersion: k8s.cni.cncf.io/v1
 kind: NetworkAttachmentDefinition
 metadata:
   name: ceph-san
   namespace: rook-ceph
 spec:
-  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
-  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
-  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
-  # reboots.
   config: |
     {
       "cniVersion": "0.3.1",
       "name": "ceph-san",
-      "type": "macvlan",
-      "master": "enp2s0f1np1",
-      "mode": "bridge",
-      "mtu": 9000,
+      "type": "host-device",
+      "device": "ceph-san",
       "ipam": {
         "type": "whereabouts",
         "range": "10.99.0.0/29",

kubernetes/talos/patches/hcloud-1.yaml

@@ -57,7 +57,7 @@ up: true
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     # Two disks: the system disk and the attached volume. Matching on transport

kubernetes/talos/patches/hcloud-2.yaml

@@ -42,7 +42,7 @@ up: true
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     match: '!system_disk'

kubernetes/talos/patches/hcloud-3.yaml

@@ -42,7 +42,7 @@ up: true
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     match: '!system_disk'

kubernetes/talos/patches/k8s-1.yaml

@@ -14,6 +14,13 @@ machine:
     # Disabling v6 on a VLAN parent does not affect the VLAN child.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    # That network also carries the mgmt VLAN's ARP. Answer ARP only on the
+    # link that owns the address, or the UDM learns the node IP against
+    # enp87s0's MAC and delivers IPv4 to a NIC Cilium is not attached to (#793).
+    net.ipv4.conf.all.arp_ignore: "1"
+    net.ipv4.conf.default.arp_ignore: "1"
+    net.ipv4.conf.all.arp_announce: "2"
+    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select disks by serial only. A broad selector such as
     # `disk.transport == 'nvme'` matches every drive in the machine, and the
@@ -28,6 +35,13 @@ machine:
     # this label instead of listing hostnames, so which boxes carry storage is
     # stated once, here, by the machine that knows.
     storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
+    # role name at boot, before the network stack opens links — a rule applied
+    # to a RUNNING node cannot rename (Resource busy) and takes effect at the
+    # next reboot.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -124,14 +138,15 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
-# wire. OSD-to-OSD replication and nothing else.
+# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
+# section above renames the port to ceph-san at boot, so shared manifests name
+# the role and only this file knows the hardware.
 #
-# The node holds NO address here on purpose. Rook attaches the link to the OSD
-# pods through a Multus macvlan, and a macvlan parent only needs to be up.
-# Giving the node an address would put it on a segment reserved for
-# Ceph-internal traffic.
+# The node holds NO address here on purpose: the link is handed whole to the
+# OSD pod (host-device attachment). Giving the node an address would put it on
+# a segment reserved for Ceph-internal traffic.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f1np1
+name: ceph-san
 up: true
 mtu: 9000

kubernetes/talos/patches/k8s-2.yaml

@@ -12,6 +12,11 @@ machine:
     # cluster IPv6 egress.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
+    net.ipv4.conf.all.arp_ignore: "1"
+    net.ipv4.conf.default.arp_ignore: "1"
+    net.ipv4.conf.all.arp_announce: "2"
+    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select the installer target by serial so it cannot land on the OSD drive
     # or on EPHEMERAL, whatever order the kernel enumerates them in.
@@ -23,6 +28,10 @@ machine:
     topology.kubernetes.io/zone: saturn
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -95,11 +104,12 @@ up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.
+# Renamed to the role name by the udev rule above — see k8s-1.yaml.
 #
-# The node holds NO address here on purpose. Rook attaches the link to the OSD
-# pods through a Multus macvlan, and a macvlan parent only needs to be up.
+# The node holds NO address here on purpose: the link is handed whole to the
+# OSD pod (host-device attachment).
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f1np1
+name: ceph-san
 up: true
 mtu: 9000

kubernetes/talos/patches/k8s-3.yaml

@@ -8,6 +8,11 @@ machine:
     # k8s-1.yaml.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
+    # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
+    net.ipv4.conf.all.arp_ignore: "1"
+    net.ipv4.conf.default.arp_ignore: "1"
+    net.ipv4.conf.all.arp_announce: "2"
+    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select the installer target by serial so it cannot land on the OSD drive
     # or on EPHEMERAL, whatever order the kernel enumerates them in.
@@ -19,6 +24,10 @@ machine:
     topology.kubernetes.io/zone: neptune
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
+  udev:
+    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
+    rules:
+      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -78,9 +87,9 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
-# here on purpose; the link is only a macvlan parent for the OSD pods.
+# here on purpose; the link is handed whole to the OSD pod (host-device).
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f1np1
+name: ceph-san
 up: true
 mtu: 9000

kubernetes/talos/patches/vm-1.yaml

@@ -43,3 +43,15 @@ apiVersion: v1alpha1
 kind: HostnameConfig
 auto: off
 hostname: vm-1
+---
+# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
+# DHCPv4Config below is load-bearing.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ceph-san
+parent: enp0s2
+vlanID: 30
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp0s2

kubernetes/talos/patches/vm-2.yaml

@@ -21,7 +21,7 @@ auto: off
 hostname: vm-2
 ---
 # The OSD device, named by the machine that has it. Ceph then consumes
-# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
+# /dev/disk/by-partlabel/r-osd-data and needs to know nothing about models,
 # serials or hostnames — the same CephCluster works here and on the MS-01s.
 #
 # The selector is a CEL expression, so each machine says whatever identifies its
@@ -29,9 +29,33 @@ hostname: vm-2
 # real nodes it will name the Samsung.
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
+---
+# The Ceph cluster network link. Production gives it a second physical port;
+# this machine has one NIC, so the segment is a VLAN on it.
+#
+# A dummy link would be simpler and worthless — no traffic between nodes, so
+# Ceph would replicate over an interface going nowhere and the test would pass
+# having proven nothing. A VLAN rides the provisioner's bridge that every VM is
+# on, so the SAN carries real frames on its own tag.
+#
+# No MTU, unlike production's 9000: the qemu NIC is created with host_mtu=1500.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ceph-san
+parent: enp0s2
+vlanID: 30
+---
+# Load-bearing. Declaring any network document makes Talos configure this link
+# explicitly, and the implicit DHCP that gives the node its address stops. A
+# first attempt with only the VLANConfig took vm-4 off the network — NotReady,
+# unreachable over the Talos API. Verified on a second throwaway node that this
+# document keeps the parent addressed.
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp0s2

kubernetes/talos/patches/vm-3.yaml

@@ -15,9 +15,21 @@ hostname: vm-3
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
+---
+# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
+# DHCPv4Config below is load-bearing.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ceph-san
+parent: enp0s2
+vlanID: 30
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp0s2

kubernetes/talos/patches/vm-4.yaml

@@ -19,9 +19,21 @@ hostname: vm-4
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: ceph-osd
+name: osd-data
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
+---
+# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
+# DHCPv4Config below is load-bearing.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ceph-san
+parent: enp0s2
+vlanID: 30
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: enp0s2