feat(rook-ceph): attach the cluster network via Multus (phase B of two) #641
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#641
Reference in New Issue
Block a user
Based on #640, not on master. Must not be applied before phase A has landed and the cluster is back to 217 PGs
active+clean.Puts the OSDs back on the VLAN 49 link that phase A gives up — without putting every Ceph daemon on host networking to get there.
Why not
provider: host, which also reaches that link. Because it reaches it for every daemon, not just the ones that need it. The cluster network is an OSD concern: per Ceph's documentation it carries OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on the public network".addressRangesis only honoured underprovider: host, so using it put the mons on host networking as a side effect — making a mon's address the node's DHCP lease.Documented, not hypothetical: when a lease moved the mon could not bind (
Cannot assign requested address) and crash-looped. Rook began a failover, but with three mons on three nodes each holding the mon host ports the replacement could not be scheduled anywhere — the canary satPendingon "didn't have free ports". Two of three nodes hit this during the Talos v1.12.11 upgrade; both needed hand intervention.Selecting only
clusterleavespublicon the pod network, so mons, MDS, MGR and the CSI pods keep Rook-managed identities and hold no host ports. That removes the failure mode rather than working around it.Isolation is not weakened. Rook's Multus prerequisites about hosts routing to the Multus network apply to a Multus public network; selecting only
clustermeans they do not apply. VLAN 49 stays Ceph-internal traffic and nothing else — no router, no other participants.Ordering is enforced by Rook, and I verified it rather than assuming.
kubectl diffagainst the current cluster refuses this outright:Second of the two rolling OSD restarts, bounded by the same settings from #639: one OSD at a time, only once PGs are clean.
Follow-up, deliberately not in this PR: once this settles, the node addresses on VLAN 49 (
10.99.0.1-.3) have no remaining purpose and should come out of the Talos configs — after which the attachment's pool can take the whole /29 instead of the three addresses it reserves today. The mons also still need failing over one at a time to leave host networking.Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting every Ceph daemon on host networking to get there. The cluster network is an OSD concern. Per the Ceph documentation it carries OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on the public network". Reaching it through addressRanges required provider: host, which is not per-daemon — the mons went onto host networking as a side effect, making a mon's address the node's DHCP lease. That failure is documented rather than hypothetical. When a lease moved, the mon could not bind ("Cannot assign requested address") and crash-looped. Rook detected it and began a failover, but with three mons on three nodes each holding the mon host ports the replacement could not be scheduled anywhere — the canary sat Pending on "didn't have free ports". Two of three nodes hit it during the Talos v1.12.11 upgrade; both needed hand intervention. Selecting only `cluster` leaves `public` on the pod network, so mons, MDS, MGR and the CSI pods keep Rook-managed identities and hold no host ports. That removes the failure mode rather than working around it. Isolation is not weakened. Rook's Multus prerequisites about hosts routing to the Multus network apply to a Multus *public* network; selecting only `cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic and nothing else, with no router and no other participants. MUST NOT be applied before phase A. Verified rather than assumed — kubectl diff against the current cluster refuses it: spec.network.provider: Invalid value: "multus": network provider must be disabled (reverted to empty string) before a new provider is enabled Second of the two rolling OSD restarts, bounded by the same two settings: one OSD at a time, only once PGs are clean. Follow-up, not in this commit: once this settles, the node addresses on VLAN 49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos configs, after which the attachment's pool can take the whole /29 instead of the three addresses it reserves today. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>