claude-botandClaude Opus 5
296f79469e
Step 2 of moving Ceph's cluster network off host networking. Inert on its own: a NetworkAttachmentDefinition does nothing until a pod references it, and nothing will until cephcluster.yaml names it under network.selectors.cluster. Why this exists at all: `addressRanges` only takes effect under `provider: host`, so carrying replication over the dedicated 10G link meant setting a provider that is not per-daemon. It puts mons on host networking too, which makes a mon's address the node's DHCP lease — and when that lease moves the mon cannot bind. Three mons on three nodes then each hold the mon host ports, so Rook cannot schedule the replacement it just decided to create and the failover stalls. Both happened during the v1.12.11 upgrade, on two of three nodes. None of that serves the goal. Per the Ceph documentation the cluster network carries OSD heartbeat, replication and recovery only — "Ceph Monitors always operate on the public network". Attaching it here gives the OSDs the link and leaves every other daemon on the pod network. Its own subnet rather than the existing /29. The nodes hold 10.99.0.1-.3 for as long as Ceph still uses host networking, which leaves exactly three addresses for three OSDs — no headroom during the one phase where things actually move, and a leaked allocation from a hard node loss would mean an OSD without an address. OSDs never need to reach the node addresses, since the cluster network is OSD-to-OSD, so the pool sits in its own range on the same segment instead. A /24 for three addresses costs nothing: nothing else can ever live on this segment, and it takes address count off the table when OSDs are added. After the provider switch is confirmed, the node addresses on VLAN 49 have no remaining purpose and can come out of the Talos configs. No gateway and no routes, because there is no router on this segment and there must not be one. That also means no `sbr` chaining, unlike the v30iot attachments — there is no DHCP-supplied default route to confine here. whereabouts for IPAM: dhcp needs a server that must not exist on the SAN, and host-local keeps state per node with no cluster-wide coordination, so two nodes would hand out the same address on shared L2. MTU 9000 matches the link. macvlan inherits it, but stating it means a change to the parent cannot silently halve replication throughput with nothing in the tree recording the expectation. Verified: whereabouts from step 1 is running on all three nodes with its binary installed next to macvlan; the NAD passes a server-side dry run; the embedded CNI JSON parses and carries the intended master, mtu, ipam type and range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>