chore(rook-ceph): return mon_data_avail_warn to the Ceph default

The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).

Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.

Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.

mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.

kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.

Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/MIGRATION-sops-openbao.md

@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```
 
 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).

kubernetes/RUNBOOK-cold-rebuild.md

@@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).
 
 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods
 
 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
 
-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```
 
 Verify the store recovers before moving on:
@@ -170,7 +174,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```
 
 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.
 
 ## Step 5 — apps: let the rest reconcile
 

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -12,6 +12,7 @@ resources:
   - prometheusrule-netbird-availability.yaml
   - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
   # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -275,8 +275,11 @@ spec:
           - alertmanager-smtp
       # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
       # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
       # monitoring ns is not blocked by any CNP today.
       config:
         global:

kubernetes/apps/music-assistant/deployment.yaml

@@ -55,7 +55,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -65,7 +65,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/infrastructure/openbao/helmrelease.yaml

@@ -42,12 +42,28 @@ spec:
       # the agent-injector mutating webhook is unused surface.
       enabled: false
     server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
       ha:
         enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
         raft:
           enabled: true
           setNodeId: true
@@ -65,8 +81,32 @@ spec:
               cluster_address = "[::]:8201"
             }
 
+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
             storage "raft" {
               path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
             }
 
             service_registration "kubernetes" {}

kubernetes/infrastructure/openbao/snapshot-cronjob.yaml

@@ -53,7 +53,13 @@ spec:
               command: ["/bin/sh", "-ec"]
               args:
                 - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                   BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                     role=openbao-snapshot \
                     jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -8,6 +8,17 @@ spec:
     image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
   mon:
     count: 3
     allowMultiplePerNode: false
@@ -36,26 +47,27 @@ spec:
   # (paperless + audiobookshelf + immich + ... all clone at once).
   cephConfig:
     mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
       #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
       # Stop compacting the whole RocksDB prefix on every paxos trim.
       #
       # Upstream default is true — "Compact a certain prefix (including paxos)
@@ -147,19 +159,65 @@ spec:
     ssl: false
   monitoring:
     enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
   network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
   # Nodes are listed explicitly rather than useAllNodes, so that a node can be
   # taken out of the storage fleet by removing it here. Under useAllNodes the
   # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +251,15 @@ spec:
   #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
   #   S437NF0M501176    PM983     — this OSD
   storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
     useAllNodes: false
     useAllDevices: false
     nodes:

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
   - cephcluster.yaml
   - toolbox.yaml
   - storageclass.yaml

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -0,0 +1,84 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }

kubernetes/talos/patches/k8s-1.yaml

@@ -20,6 +20,29 @@ machine:
     # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
     # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are ordinary block devices to Talos here: the Kingston
     # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -117,13 +164,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29

kubernetes/talos/patches/k8s-2.yaml

@@ -9,6 +9,15 @@ machine:
     # macvlan parent for the v30iot pod attachments; host addresses on that L2
     # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -89,10 +98,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29

kubernetes/talos/patches/k8s-3.yaml

@@ -8,6 +8,15 @@ machine:
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29