Merge pull request 'feat(rook-ceph): attach the cluster network via Multus (phase B of two)' (#642) from ceph-provider-phase-b into master

Reviewed-on: k.renner/homelab#642

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -158,37 +158,51 @@ spec:
     ssl: false
   monitoring:
     enabled: true
-  # PHASE A of moving the cluster network onto Multus. This is an intermediate
-  # state and is not where this should be left — san-network-attachment.yaml
-  # exists and is unused until phase B sets `provider: multus`.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
   #
-  # The detour is forced by Rook's own CRD validation:
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
   #
-  #   rule:    self == '' || oldSelf == '' || self == oldSelf
-  #   message: network provider must be disabled (reverted to empty string)
-  #            before a new provider is enabled
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
   #
-  # So host -> multus is rejected outright and has to go through empty. An
-  # empty provider means the default CNI, and `addressRanges` only takes
-  # effect under `provider: host`, so it goes with it.
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
   #
-  # WHAT THIS COSTS WHILE IT IS IN PLACE
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
   #
-  # There is no cluster network at all. OSD heartbeat, replication and
-  # recovery move onto the pod network and share it with everything else,
-  # instead of the dedicated X710 SFP+ link on VLAN 49. Slower, and each OSD
-  # restart triggers backfill that pays that price — but not a correctness or
-  # safety problem, and the OSDs remain able to reach one another throughout,
-  # which is what matters.
+  # ISOLATION IS PRESERVED
   #
-  # The mons do not move here. Rook leaves existing mons on the network they
-  # were created with; they only change on failover, which is a separate
-  # deliberate step.
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
   #
   # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
   # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
   network:
-    provider: ""
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
   # Nodes are listed explicitly rather than useAllNodes, so that a node can be
   # taken out of the storage fleet by removing it here. Under useAllNodes the
   # operator re-provisions an OSD on any node whose device still carries a