chore(talos): drop the node addresses on the Ceph SAN #643

2026-08-26T20:17:39Z

claude-bot commented

2026-08-26 20:17:39 +00:00

The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link required provider: host, which is not per-daemon and put every Ceph daemon on host networking. Rook now attaches the link to the OSD pods through a Multus macvlan, and a macvlan parent only needs to be up — it does not need an address of its own for its children to work.

By the rule this segment is built on — Ceph-internal traffic and nothing else, no router, no other participants — the node has no business holding an address there either. Stricter than before, not looser.

The link keeps up: true and mtu: 9000. Both still matter: the parent must be up for the macvlan children to exist, and the MTU is what the children inherit.

Deliberately NOT in this PR: widening the attachment pool from .4-.6 to the full /29. That must not land before these addresses are gone from the running nodes, or whereabouts could hand .1 to an OSD while a node still holds it. Follow-up once these configs are applied and verified.

Also records a one-time step the manifests cannot express. Removing addressRanges did not clear public_network and cluster_network from Ceph own config database — the same trap the bluestore comment in the same file describes. The first OSD of the phase A rollout came up on the pod network, still had public_network 10.2.40.0/24 in the config database, found no interface in it and crash-looped on "Failed to pick public address". Both keys were removed by hand; Rook later wrote cluster_network afresh from the attachment. A rebuild from scratch will not hit this — the config database starts empty — but another provider change on a live cluster will.

Current state after phase B: all three OSDs on back_addr 10.99.0.4/.5/.6 with front_addr on the pod network, mons on pod addresses, 217 PGs active+clean.

Applying the Talos side is a talosctl patch machineconfig per node, not a Flux action — these patches are not reconciled automatically.

The nodes held `10.99.0.1-.3/29` on `enp2s0f1np1` only because reaching that link required `provider: host`, which is not per-daemon and put every Ceph daemon on host networking. Rook now attaches the link to the OSD pods through a Multus macvlan, and a macvlan parent only needs to be **up** — it does not need an address of its own for its children to work. By the rule this segment is built on — Ceph-internal traffic and nothing else, no router, no other participants — the node has no business holding an address there either. Stricter than before, not looser. The link keeps `up: true` and `mtu: 9000`. Both still matter: the parent must be up for the macvlan children to exist, and the MTU is what the children inherit. **Deliberately NOT in this PR:** widening the attachment pool from `.4-.6` to the full /29. That must not land before these addresses are gone from the running nodes, or whereabouts could hand `.1` to an OSD while a node still holds it. Follow-up once these configs are applied and verified. **Also records a one-time step the manifests cannot express.** Removing `addressRanges` did not clear `public_network` and `cluster_network` from Ceph own config database — the same trap the bluestore comment in the same file describes. The first OSD of the phase A rollout came up on the pod network, still had `public_network 10.2.40.0/24` in the config database, found no interface in it and crash-looped on *"Failed to pick public address"*. Both keys were removed by hand; Rook later wrote `cluster_network` afresh from the attachment. A rebuild from scratch will not hit this — the config database starts empty — but another provider change on a live cluster will. Current state after phase B: all three OSDs on `back_addr` `10.99.0.4/.5/.6` with `front_addr` on the pod network, mons on pod addresses, 217 PGs `active+clean`. Applying the Talos side is a `talosctl patch machineconfig` per node, not a Flux action — these patches are not reconciled automatically.

claude-bot added 1 commit 2026-08-26 20:17:39 +00:00

chore(talos): drop the node addresses on the Ceph SAN

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

b5a1818896

The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link
required CephCluster.spec.network.provider: host, which is not per-daemon and
put every Ceph daemon on host networking. Rook now attaches the link to the
OSD pods through a Multus macvlan, and a macvlan parent only needs to be up —
it does not need an address of its own for its children to work.

By the rule this segment is built on — Ceph-internal traffic and nothing else,
no router, no other participants — the node has no business holding an address
there either. This is stricter than before, not looser.

The link keeps up: true and mtu: 9000. Both still matter: the parent must be
up for the macvlan children to exist, and the MTU is what the children
inherit.

Not in this commit, deliberately: widening the attachment's whereabouts pool
from .4-.6 to the full /29. That must not land before these addresses are
actually gone from the running nodes, or whereabouts could hand .1 to an OSD
while a node still holds it. Follow-up once these configs are applied and
verified.

Also records the one-time step that the manifests cannot express. Removing
addressRanges did not clear public_network and cluster_network from Ceph's
config database — the same trap the bluestore comment in the same file
describes. The first OSD of the phase A rollout came up on the pod network,
still had public_network 10.2.40.0/24 in the config database, found no
interface in it and crash-looped on "Failed to pick public address". Both keys
were removed by hand. A rebuild from scratch will not hit this because the
config database starts empty; another provider change on a live cluster will.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 96acff3456 into master

2026-08-26 20:18:18 +00:00

k.renner referenced this issue from a commit

2026-08-26 20:18:19 +00:00

Merge pull request 'chore(talos): drop the node addresses on the Ceph SAN' (#643) from ceph-san-cleanup into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#643