diff --git a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml index d90c8a0d..d2507e8c 100644 --- a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml +++ b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml @@ -158,37 +158,51 @@ spec: ssl: false monitoring: enabled: true - # PHASE A of moving the cluster network onto Multus. This is an intermediate - # state and is not where this should be left — san-network-attachment.yaml - # exists and is unused until phase B sets `provider: multus`. + # Dedicated Ceph cluster network on VLAN 49, carried by the second X710 + # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus. # - # The detour is forced by Rook's own CRD validation: + # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK # - # rule: self == '' || oldSelf == '' || self == oldSelf - # message: network provider must be disabled (reverted to empty string) - # before a new provider is enabled + # Because it reaches it for every daemon, not just the ones that need it. + # The cluster network is an OSD concern: per the Ceph documentation it + # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always + # operate on the public network". `addressRanges` is only honoured under + # provider: host, so using it put the mons on host networking as a side + # effect — making a mon's address the node's DHCP lease. # - # So host -> multus is rejected outright and has to go through empty. An - # empty provider means the default CNI, and `addressRanges` only takes - # effect under `provider: host`, so it goes with it. + # That is not theoretical. When a lease moved, the mon could no longer bind + # ("Cannot assign requested address") and crash-looped. Rook detected it and + # started a failover, but with three mons on three nodes each holding the + # mon host ports, the replacement could not be scheduled anywhere — the + # canary sat Pending on "didn't have free ports". Two of three nodes hit + # this during the Talos v1.12.11 upgrade and both needed hand intervention. # - # WHAT THIS COSTS WHILE IT IS IN PLACE + # Only `cluster` is selected here. With `public` left unset Rook puts client + # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep + # Rook-managed identities and hold no host ports — which is what removes + # that failure mode rather than working around it. # - # There is no cluster network at all. OSD heartbeat, replication and - # recovery move onto the pod network and share it with everything else, - # instead of the dedicated X710 SFP+ link on VLAN 49. Slower, and each OSD - # restart triggers backfill that pays that price — but not a correctness or - # safety problem, and the OSDs remain able to reach one another throughout, - # which is what matters. + # ISOLATION IS PRESERVED # - # The mons do not move here. Rook leaves existing mons on the network they - # were created with; they only change on failover, which is a separate - # deliberate step. + # Rook's Multus prerequisites about hosts routing to the Multus network + # apply to a Multus *public* network. Selecting only `cluster` means they do + # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing + # else, no router, no other participants. After this settles, the node + # addresses on that segment (10.99.0.1-.3) have no remaining purpose and + # should come out of the Talos configs — at which point the attachment's + # pool can take the whole /29. + # + # Reached from provider: "" rather than directly from host, which Rook's CRD + # rejects. See the phase A commit. # # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel # above: one OSD restarts at a time and only once PGs are clean again. + # + # The attachment itself is san-network-attachment.yaml. network: - provider: "" + provider: multus + selectors: + cluster: rook-ceph/ceph-san # Nodes are listed explicitly rather than useAllNodes, so that a node can be # taken out of the storage fleet by removing it here. Under useAllNodes the # operator re-provisions an OSD on any node whose device still carries a