From a775b675d406350669c435d5689a5f2650895f8c Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 26 Aug 2026 21:52:38 +0200 Subject: [PATCH] feat(rook-ceph): attach the cluster network via Multus (phase B of two) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting every Ceph daemon on host networking to get there. The cluster network is an OSD concern. Per the Ceph documentation it carries OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on the public network". Reaching it through addressRanges required provider: host, which is not per-daemon — the mons went onto host networking as a side effect, making a mon's address the node's DHCP lease. That failure is documented rather than hypothetical. When a lease moved, the mon could not bind ("Cannot assign requested address") and crash-looped. Rook detected it and began a failover, but with three mons on three nodes each holding the mon host ports the replacement could not be scheduled anywhere — the canary sat Pending on "didn't have free ports". Two of three nodes hit it during the Talos v1.12.11 upgrade; both needed hand intervention. Selecting only `cluster` leaves `public` on the pod network, so mons, MDS, MGR and the CSI pods keep Rook-managed identities and hold no host ports. That removes the failure mode rather than working around it. Isolation is not weakened. Rook's Multus prerequisites about hosts routing to the Multus network apply to a Multus *public* network; selecting only `cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic and nothing else, with no router and no other participants. MUST NOT be applied before phase A. Verified rather than assumed — kubectl diff against the current cluster refuses it: spec.network.provider: Invalid value: "multus": network provider must be disabled (reverted to empty string) before a new provider is enabled Second of the two rolling OSD restarts, bounded by the same two settings: one OSD at a time, only once PGs are clean. Follow-up, not in this commit: once this settles, the node addresses on VLAN 49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos configs, after which the attachment's pool can take the whole /29 instead of the three addresses it reserves today. Co-Authored-By: Claude Opus 5 (1M context) --- .../rook-ceph-cluster/cephcluster.yaml | 56 ++++++++++++------- 1 file changed, 35 insertions(+), 21 deletions(-) diff --git a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml index d90c8a0d..d2507e8c 100644 --- a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml +++ b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml @@ -158,37 +158,51 @@ spec: ssl: false monitoring: enabled: true - # PHASE A of moving the cluster network onto Multus. This is an intermediate - # state and is not where this should be left — san-network-attachment.yaml - # exists and is unused until phase B sets `provider: multus`. + # Dedicated Ceph cluster network on VLAN 49, carried by the second X710 + # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus. # - # The detour is forced by Rook's own CRD validation: + # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK # - # rule: self == '' || oldSelf == '' || self == oldSelf - # message: network provider must be disabled (reverted to empty string) - # before a new provider is enabled + # Because it reaches it for every daemon, not just the ones that need it. + # The cluster network is an OSD concern: per the Ceph documentation it + # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always + # operate on the public network". `addressRanges` is only honoured under + # provider: host, so using it put the mons on host networking as a side + # effect — making a mon's address the node's DHCP lease. # - # So host -> multus is rejected outright and has to go through empty. An - # empty provider means the default CNI, and `addressRanges` only takes - # effect under `provider: host`, so it goes with it. + # That is not theoretical. When a lease moved, the mon could no longer bind + # ("Cannot assign requested address") and crash-looped. Rook detected it and + # started a failover, but with three mons on three nodes each holding the + # mon host ports, the replacement could not be scheduled anywhere — the + # canary sat Pending on "didn't have free ports". Two of three nodes hit + # this during the Talos v1.12.11 upgrade and both needed hand intervention. # - # WHAT THIS COSTS WHILE IT IS IN PLACE + # Only `cluster` is selected here. With `public` left unset Rook puts client + # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep + # Rook-managed identities and hold no host ports — which is what removes + # that failure mode rather than working around it. # - # There is no cluster network at all. OSD heartbeat, replication and - # recovery move onto the pod network and share it with everything else, - # instead of the dedicated X710 SFP+ link on VLAN 49. Slower, and each OSD - # restart triggers backfill that pays that price — but not a correctness or - # safety problem, and the OSDs remain able to reach one another throughout, - # which is what matters. + # ISOLATION IS PRESERVED # - # The mons do not move here. Rook leaves existing mons on the network they - # were created with; they only change on failover, which is a separate - # deliberate step. + # Rook's Multus prerequisites about hosts routing to the Multus network + # apply to a Multus *public* network. Selecting only `cluster` means they do + # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing + # else, no router, no other participants. After this settles, the node + # addresses on that segment (10.99.0.1-.3) have no remaining purpose and + # should come out of the Talos configs — at which point the attachment's + # pool can take the whole /29. + # + # Reached from provider: "" rather than directly from host, which Rook's CRD + # rejects. See the phase A commit. # # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel # above: one OSD restarts at a time and only once PGs are clean again. + # + # The attachment itself is san-network-attachment.yaml. network: - provider: "" + provider: multus + selectors: + cluster: rook-ceph/ceph-san # Nodes are listed explicitly rather than useAllNodes, so that a node can be # taken out of the storage fleet by removing it here. Under useAllNodes the # operator re-provisions an OSD on any node whose device still carries a