feat(rook-ceph): attach the cluster network via Multus (phase B of two) #642
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#642
Reference in New Issue
Block a user
Re-targeted at master. PR 641 carried the same commit but was opened against the ceph-provider-phase-a branch so its diff would show only the phase B change. That branch had already merged into master and was stale, so merging 641 landed the change on a dead-end branch and it never reached the cluster. My mistake in choosing the base, not a problem with the change. Content is unchanged.
Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting every Ceph daemon on host networking to get there. The cluster network is an OSD concern — per Ceph documentation it carries OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on the public network". Reaching it through addressRanges required provider host, which is not per-daemon: the mons went onto host networking as a side effect, making a mon address the node DHCP lease. Two of three nodes hit that during the Talos v1.12.11 upgrade, and Rook could not place a replacement mon because all three nodes already held the mon host ports.
Selecting only cluster leaves public on the pod network, so mons, MDS, MGR and the CSI pods keep Rook-managed identities and hold no host ports.
Precondition, learned the hard way in phase A. addressRanges was removed from the CR, but public_network and cluster_network persisted in Ceph config database — removing them from the file does not clear them there. The first OSD crash-looped on "unable to find any IPv4 address in networks 10.2.40.0/24". Both were cleared by hand, and ceph config dump currently shows no network settings at all. Phase B will have Rook write cluster_network afresh from the attachment.
What to watch during the rollout, given that experience:
Current state: 217 PGs active+clean, 3 OSDs up, 3 mons in quorum, all OSDs on the pod network.
Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting every Ceph daemon on host networking to get there. The cluster network is an OSD concern. Per the Ceph documentation it carries OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on the public network". Reaching it through addressRanges required provider: host, which is not per-daemon — the mons went onto host networking as a side effect, making a mon's address the node's DHCP lease. That failure is documented rather than hypothetical. When a lease moved, the mon could not bind ("Cannot assign requested address") and crash-looped. Rook detected it and began a failover, but with three mons on three nodes each holding the mon host ports the replacement could not be scheduled anywhere — the canary sat Pending on "didn't have free ports". Two of three nodes hit it during the Talos v1.12.11 upgrade; both needed hand intervention. Selecting only `cluster` leaves `public` on the pod network, so mons, MDS, MGR and the CSI pods keep Rook-managed identities and hold no host ports. That removes the failure mode rather than working around it. Isolation is not weakened. Rook's Multus prerequisites about hosts routing to the Multus network apply to a Multus *public* network; selecting only `cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic and nothing else, with no router and no other participants. MUST NOT be applied before phase A. Verified rather than assumed — kubectl diff against the current cluster refuses it: spec.network.provider: Invalid value: "multus": network provider must be disabled (reverted to empty string) before a new provider is enabled Second of the two rolling OSD restarts, bounded by the same two settings: one OSD at a time, only once PGs are clean. Follow-up, not in this commit: once this settles, the node addresses on VLAN 49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos configs, after which the attachment's pool can take the whole /29 instead of the three addresses it reserves today. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>