The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link
required CephCluster.spec.network.provider: host, which is not per-daemon and
put every Ceph daemon on host networking. Rook now attaches the link to the
OSD pods through a Multus macvlan, and a macvlan parent only needs to be up —
it does not need an address of its own for its children to work.
By the rule this segment is built on — Ceph-internal traffic and nothing else,
no router, no other participants — the node has no business holding an address
there either. This is stricter than before, not looser.
The link keeps up: true and mtu: 9000. Both still matter: the parent must be
up for the macvlan children to exist, and the MTU is what the children
inherit.
Not in this commit, deliberately: widening the attachment's whereabouts pool
from .4-.6 to the full /29. That must not land before these addresses are
actually gone from the running nodes, or whereabouts could hand .1 to an OSD
while a node still holds it. Follow-up once these configs are applied and
verified.
Also records the one-time step that the manifests cannot express. Removing
addressRanges did not clear public_network and cluster_network from Ceph's
config database — the same trap the bluestore comment in the same file
describes. The first OSD of the phase A rollout came up on the pod network,
still had public_network 10.2.40.0/24 in the config database, found no
interface in it and crash-looped on "Failed to pick public address". Both keys
were removed by hand. A rebuild from scratch will not hit this because the
config database starts empty; another provider change on a live cluster will.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 5