fix(rook-ceph): raise cephfs-ci to size 2 — size 1 blocks all OSD maintenance #547

2026-08-21T20:21:42Z

claude-bot commented

2026-08-21 20:21:42 +00:00

The unreplicated CI pools introduced in #542 make every OSD on the cluster permanently un-restartable. Unblocks #545.

The failure

ceph osd ok-to-stop refuses while stopping an OSD would take PGs offline. A size-1 pool puts every PG on exactly one OSD, so the check fails for all three:

Error EBUSY: unsafe to stop osd(s) at this time (20 PGs are or would become offline)
bad_become_inactive: 13.6 13.9 13.d 13.f 14.0 ... 14.1c 14.1d

Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2 pools are not implicated — they run min_size 1 and appear under ok_become_degraded, which is permitted:

pool  1 'replicapool'         size 2  min_size 1
pool  6 'cephfs-data0'        size 2  min_size 1
pool 13 'cephfs-ci-metadata'  size 1  min_size 1   <- blocker
pool 14 'cephfs-ci-data0'     size 1  min_size 1   <- blocker

Rook gates OSD restarts on that check, so this blocks CephX key rotation, Ceph upgrades, node drains and reboots, and drive replacement.

How it presented

It deadlocked #545. Rook retried several times a second and could never succeed:

op-osd: OSD 0 is not ok-to-stop. will try updating it again later
op-osd: OSD 1 is not ok-to-stop. will try updating it again later
op-osd: OSD 2 is not ok-to-stop. will try updating it again later
op-osd: waiting... 3 of 3 OSD prepare jobs have finished and 0 of 3 OSDs have been updated

Throughout, ceph -s reported 217 pgs active+clean. Nothing is unhealthy — the restarts are simply refused. No health check covers this, which is why it was invisible until something needed to restart an OSD.

What #542 got wrong

The reasoning there was that unreplicated CI data only risks in-flight pipelines, since every byte is a fresh git clone and the worst case is a re-run. That part holds. What it missed is that the cost is not scoped to the pool — one unreplicated pool freezes maintenance for the entire cluster.

Most of #542's benefit came from keeping CI churn off cephfs, whose metadataPool is size 3. That survives this change. Only the second-order gain from dropping the replica is given up, and the backfill is trivial: cephfs-ci-data0 stores 0 B, cephfs-ci-metadata 182 MiB.

size 1 becomes viable again once these pools sit on OSDs of their own, where the permanently un-stoppable OSD holds nothing but throwaway data — planned with the retired QLC drives.

Alerting follows

POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was excluded because cephfs-ci was expected to keep it permanently active. No pool is unreplicated now, so the exclusion would only hide a real one losing its replicas.
CephPoolLostRedundancy drops its cephfs-ci exemption for the same reason, and its description now states the maintenance consequence rather than only data loss.
The AUTH_INSECURE_* exclusion is reworded. Calling it "known-accepted" reads as a decision that the risk is fine; it is a kernel-blocked remediation (aes256k over the CSI mount path needs kernel 7.0+, nodes run 6.18.18-talos).

After merge

Rook should stop retrying and roll the three OSDs one at a time. Verify:

kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}'
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s

status.cephx.osd should reach keyGeneration: 2 with keyType: aes256k, completing #545. Watch that PGs return to active+clean between OSDs — size 2 means a restarting OSD leaves its PGs on a single copy.

Refs #542, #545, #527

🤖 Generated with Claude Code

The unreplicated CI pools introduced in #542 make **every OSD on the cluster permanently un-restartable**. Unblocks #545. ## The failure `ceph osd ok-to-stop` refuses while stopping an OSD would take PGs offline. A size-1 pool puts every PG on exactly one OSD, so the check fails for all three: ``` Error EBUSY: unsafe to stop osd(s) at this time (20 PGs are or would become offline) bad_become_inactive: 13.6 13.9 13.d 13.f 14.0 ... 14.1c 14.1d ``` Pools 13 and 14 are `cephfs-ci-metadata` and `cephfs-ci-data0`. The size-2 pools are not implicated — they run `min_size 1` and appear under `ok_become_degraded`, which is permitted: ``` pool 1 'replicapool' size 2 min_size 1 pool 6 'cephfs-data0' size 2 min_size 1 pool 13 'cephfs-ci-metadata' size 1 min_size 1 <- blocker pool 14 'cephfs-ci-data0' size 1 min_size 1 <- blocker ``` Rook gates OSD restarts on that check, so this blocks CephX key rotation, Ceph upgrades, node drains and reboots, and drive replacement. ## How it presented It deadlocked #545. Rook retried several times a second and could never succeed: ``` op-osd: OSD 0 is not ok-to-stop. will try updating it again later op-osd: OSD 1 is not ok-to-stop. will try updating it again later op-osd: OSD 2 is not ok-to-stop. will try updating it again later op-osd: waiting... 3 of 3 OSD prepare jobs have finished and 0 of 3 OSDs have been updated ``` Throughout, `ceph -s` reported `217 pgs active+clean`. **Nothing is unhealthy — the restarts are simply refused.** No health check covers this, which is why it was invisible until something needed to restart an OSD. ## What #542 got wrong The reasoning there was that unreplicated CI data only risks in-flight pipelines, since every byte is a fresh git clone and the worst case is a re-run. That part holds. What it missed is that **the cost is not scoped to the pool** — one unreplicated pool freezes maintenance for the entire cluster. Most of #542's benefit came from keeping CI churn off `cephfs`, whose metadataPool is size 3. That survives this change. Only the second-order gain from dropping the replica is given up, and the backfill is trivial: `cephfs-ci-data0` stores 0 B, `cephfs-ci-metadata` 182 MiB. size 1 becomes viable again once these pools sit on OSDs of their own, where the permanently un-stoppable OSD holds nothing but throwaway data — planned with the retired QLC drives. ## Alerting follows - **`POOL_NO_REDUNDANCY` is no longer excluded from `CephHealthWarn`.** It was excluded because cephfs-ci was expected to keep it permanently active. No pool is unreplicated now, so the exclusion would only hide a real one losing its replicas. - **`CephPoolLostRedundancy` drops its cephfs-ci exemption** for the same reason, and its description now states the maintenance consequence rather than only data loss. - **The `AUTH_INSECURE_*` exclusion is reworded.** Calling it "known-accepted" reads as a decision that the risk is fine; it is a kernel-blocked remediation (aes256k over the CSI mount path needs kernel 7.0+, nodes run 6.18.18-talos). ## After merge Rook should stop retrying and roll the three OSDs one at a time. Verify: ``` kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}' kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s ``` `status.cephx.osd` should reach `keyGeneration: 2` with `keyType: aes256k`, completing #545. Watch that PGs return to `active+clean` between OSDs — size 2 means a restarting OSD leaves its PGs on a single copy. Refs #542, #545, #527 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-21 20:21:43 +00:00

fix(rook-ceph): raise cephfs-ci to size 2 — size 1 blocks all OSD maintenance

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

85996991b6

The unreplicated CI pools introduced in #542 make every OSD on the
cluster permanently un-restartable.

`ceph osd ok-to-stop` refuses while stopping an OSD would take PGs
offline. A size-1 pool puts every PG on exactly one OSD, so the check
fails for all three:

  Error EBUSY: unsafe to stop osd(s) at this time
               (20 PGs are or would become offline)
  bad_become_inactive: 13.6 13.9 ... 14.1c 14.1d

Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2
pools are not implicated — they run min_size 1 and appear under
ok_become_degraded, which is allowed.

Rook gates OSD restarts on that check, so this blocks CephX key
rotation, Ceph upgrades, node drains and reboots, and drive
replacement. It surfaced when #545 deadlocked: Rook retried
ok-to-stop against all three OSDs several times a second and could
never succeed, while `ceph -s` reported 217 pgs active+clean
throughout. Nothing is unhealthy; the restarts are simply refused.

The reasoning in #542 was that unreplicated CI data only risks
in-flight pipelines, since every byte is a fresh git clone. That part
holds. What it missed is that the cost is not scoped to the pool: one
unreplicated pool freezes maintenance for the entire cluster.

Most of #542's saving came from keeping CI churn off `cephfs`, whose
metadataPool is size 3 — that survives this change. Only the second-
order gain from dropping the replica is given up, and cephfs-ci-data0
stores 0 B with cephfs-ci-metadata at 182 MiB, so the backfill is
trivial.

size 1 becomes viable again once these pools sit on OSDs of their own,
where the permanently un-stoppable OSD holds nothing but throwaway
data. That is planned with the retired QLC drives.

Alerting follows the same change:

- POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was
  excluded because cephfs-ci was expected to keep it permanently
  active; no pool is unreplicated now, so the exclusion would only
  hide a real one losing its replicas.
- CephPoolLostRedundancy drops its cephfs-ci exemption for the same
  reason, and its description now states the maintenance consequence,
  not just the data-loss one.
- The AUTH_INSECURE_* exclusion is reworded. Calling it
  "known-accepted" reads as a decision that the risk is fine; it is a
  kernel-blocked remediation (aes256k over the CSI mount path needs
  kernel 7.0+, nodes run 6.18).

Refs #542, #545, #527