feat(rook-ceph): restore cephfs-metadata to size 3 #609

2026-08-26T00:09:01Z

claude-bot commented

2026-08-26 00:09:01 +00:00

All three nodes are converted and the cluster is clean, so the window that justified 2/1 on this pool is over.

mon: 3 daemons, quorum d,e,f
osd: 3 osds: 3 up, 3 in
volumes: 2/2 healthy
pgs: 217 active+clean

The min_size override goes with it. It existed only because Ceph does not lower min_size when size is reduced — leaving it would have made the pool 2/2, where any single OSD down blocks IO for its PGs. At size 3 the default min_size of 2 is what the pool wants.

The comment loses its TEMPORARY banner and states the standing rationale instead.

Expect on merge

Re-replication of ~253 MiB. Small enough to be unnoticeable next to the backfill that just finished.

Not in this PR — needs your call

replicapool and cephfs-data0 stay at size 2. Those are not conversion-temporary; they are the standing decision from #434, taken because all three OSDs were Crucial P3 QLC drives at 80–87% endurance, wearing at ~0.13 points/day and projected to exhaust together between Nov 2026 and Jan 2027. Replication does not protect against that, because the wear was correlated by construction. Size 2 cut per-drive write volume by a third to desynchronise it.

That premise no longer holds. All three OSDs are now PM983s at 3% used with independent wear:

S437NX0M400426  jupiter  3% used
S437NX0M403161  saturn   3% used
S437NF0M501176  neptune  3% used

So the reasoning that made "a second drive failure loses data" an acceptable trade is gone, and size 3 looks correct again. That is a durability/capacity decision on an open issue rather than conversion cleanup, and it re-replicates ~280 GiB, so I have left it alone. #434 is ready to be closed deliberately whenever you want to make that call.

Applied live, not in git

.mgr back to 3/2 and osd_mclock_profile back to balanced — neither is repo-managed (.mgr is created internally by Ceph), so both are live commands per the runbook.

All three nodes are converted and the cluster is clean, so the window that justified 2/1 on this pool is over. ``` mon: 3 daemons, quorum d,e,f osd: 3 osds: 3 up, 3 in volumes: 2/2 healthy pgs: 217 active+clean ``` The `min_size` override goes with it. It existed only because Ceph does not lower `min_size` when `size` is reduced — leaving it would have made the pool 2/2, where any single OSD down blocks IO for its PGs. At size 3 the default `min_size` of 2 is what the pool wants. The comment loses its TEMPORARY banner and states the standing rationale instead. ## Expect on merge Re-replication of ~253 MiB. Small enough to be unnoticeable next to the backfill that just finished. ## Not in this PR — needs your call `replicapool` and `cephfs-data0` stay at size 2. Those are **not** conversion-temporary; they are the standing decision from #434, taken because all three OSDs were Crucial P3 QLC drives at 80–87% endurance, wearing at ~0.13 points/day and projected to exhaust together between Nov 2026 and Jan 2027. Replication does not protect against that, because the wear was correlated by construction. Size 2 cut per-drive write volume by a third to desynchronise it. **That premise no longer holds.** All three OSDs are now PM983s at 3% used with independent wear: ``` S437NX0M400426 jupiter 3% used S437NX0M403161 saturn 3% used S437NF0M501176 neptune 3% used ``` So the reasoning that made "a second drive failure loses data" an acceptable trade is gone, and size 3 looks correct again. That is a durability/capacity decision on an open issue rather than conversion cleanup, and it re-replicates ~280 GiB, so I have left it alone. #434 is ready to be closed deliberately whenever you want to make that call. ## Applied live, not in git `.mgr` back to 3/2 and `osd_mclock_profile` back to `balanced` — neither is repo-managed (`.mgr` is created internally by Ceph), so both are live commands per the runbook.

claude-bot added 1 commit 2026-08-26 00:09:01 +00:00

feat(rook-ceph): restore cephfs-metadata to size 3

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

b33c482ad3

All three nodes are converted and the cluster is back to three OSDs across
three zones with 217 PGs active+clean, so the conversion window that justified
2/1 on this pool is over.

The min_size override goes with it. It existed only because Ceph does not
lower min_size when size is reduced, so the pool would have been 2/2 while a
host was out. At size 3 the default min_size of 2 is what the pool wants.

The comment loses its TEMPORARY banner and now states the standing rationale
and the condition under which the pool degrades to two copies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit bd5bc920d8 into master

2026-08-26 00:10:38 +00:00

k.renner referenced this issue from a commit

2026-08-26 00:10:39 +00:00

Merge pull request 'feat(rook-ceph): restore cephfs-metadata to size 3' (#609) from restore-cephfs-metadata-size into master

claude-bot commented

2026-08-26 00:12:09 +00:00

Scope extended on request — this PR now restores all three #434 pools to size 3, not just cephfs-metadata. The "Not in this PR" section above is superseded.

Now included:

pool	was	now
`cephfs-metadata`	2 / min_size 1	3
`replicapool`	2 / min_size 1	3
`cephfs-data0`	2 / min_size 1	3

All three min_size: "1" overrides are dropped. They existed only because Ceph does not lower min_size when size is reduced — leaving them would have made the pools 2/2, where a single OSD down blocks IO. At size 3 the default min_size of 2 is correct.

Deliberately left alone:

cephfs-ci (metadata + data0) stays at size 2. That pool holds throwaway Woodpecker CI workspaces and is at 2 rather than 1 only because an unreplicated pool makes ceph osd ok-to-stop refuse for every OSD, blocking OSD maintenance cluster-wide. Nothing to do with #434, and raising it would spend a third more capacity on data that lives about two minutes.
objectstore metadataPool is at size 2 with no stated rationale in the manifest, and its dataPool is already erasure-coded 2+1. Flagging rather than sweeping it in, since I could not establish whether that 2 was deliberate.

Expect on merge: re-replication of roughly 280 GiB across replicapool and cephfs-data0, plus ~253 MiB of metadata. That is comparable to the backfill that just completed, so budget similar time and watch ceph -s rather than merging and walking away.

This also makes #434 resolvable — its premise was correlated QLC wear-out across three Crucials, and all three OSDs are now PM983s at 2-3% used with independent wear.

Scope extended on request — this PR now restores **all three** #434 pools to size 3, not just `cephfs-metadata`. The "Not in this PR" section above is superseded. Now included: | pool | was | now | | --- | --- | --- | | `cephfs-metadata` | 2 / min_size 1 | 3 | | `replicapool` | 2 / min_size 1 | 3 | | `cephfs-data0` | 2 / min_size 1 | 3 | All three `min_size: "1"` overrides are dropped. They existed only because Ceph does not lower `min_size` when `size` is reduced — leaving them would have made the pools 2/2, where a single OSD down blocks IO. At size 3 the default `min_size` of 2 is correct. **Deliberately left alone:** - `cephfs-ci` (metadata + data0) stays at size 2. That pool holds throwaway Woodpecker CI workspaces and is at 2 rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop` refuse for every OSD, blocking OSD maintenance cluster-wide. Nothing to do with #434, and raising it would spend a third more capacity on data that lives about two minutes. - `objectstore` metadataPool is at size 2 with no stated rationale in the manifest, and its dataPool is already erasure-coded 2+1. Flagging rather than sweeping it in, since I could not establish whether that 2 was deliberate. **Expect on merge:** re-replication of roughly 280 GiB across `replicapool` and `cephfs-data0`, plus ~253 MiB of metadata. That is comparable to the backfill that just completed, so budget similar time and watch `ceph -s` rather than merging and walking away. This also makes #434 resolvable — its premise was correlated QLC wear-out across three Crucials, and all three OSDs are now PM983s at 2-3% used with independent wear.

review-bot reviewed

2026-08-26 00:13:30 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit b33c482ad32dde378600cf2250c9ad536c80c78c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR restores cephfs-metadata from the 2/1 conversion-window state back to size 3 with no min_size override, and rewrites the pool's comment to state the standing rationale instead of the temporary one. The diff is small (one file, a spec.metadataPool block and its comment) and the reasoning in the PR body and the rewritten comment is sound and well cross-referenced with the sibling replicapool block and RUNBOOK-baremetal-conversion.md. I found no outright bugs in the YAML itself, but there is one durability assumption in the PR that I could not confirm and think is worth checking before treating this as done.

🟡 Suggestions

storageclass.yaml:102-104 — confirm min_size actually returns to 2, don't assume it does. The PR removes parameters: { min_size: "1" } and relies on "the default min_size of 2" applying once size goes back to 3. But the sibling comment on replicapool (storageclass.yaml:18-21, unchanged by this diff) documents the opposite direction of this exact fact: "Ceph does not lower min_size when size is reduced" — i.e. min_size is an independent pool property that ceph osd pool set <pool> size N does not touch. That's a general property of the osd pool set size command, not something that reverses direction: raising size back to 3 on an existing pool won't retroactively raise min_size either, unless Rook's operator itself pushes a computed default min_size on every reconcile even when the CR's parameters map doesn't mention it. Ceph's "default min_size of 2 for size 3" formula (size - size/2) is applied at pool creation time (or whenever min_size is explicitly set) — not automatically re-applied when size changes later. If Rook only issues ceph osd pool set min_size ... when the CR explicitly carries that parameter (plausible, since Rook generally applies only what's declared), then after this merges the live pool will still be running with min_size: 1 — i.e. functionally still able to accept single-copy writes while degraded — but now with no comment or override in git warning about it, which is worse than the current explicit 2/1 state. I could not verify the operator's actual behavior here (no Rook source vendored in this checkout, no cluster access, and only one commit of history to check precedent against). The one check that settles it: after this merges and reconciles, run ceph osd pool get cephfs-metadata min_size (or ceph osd pool ls detail) against the live cluster. If it still reads 1, it needs the same treatment the PR already gives .mgr and osd_mclock_profile — a manual ceph osd pool set cephfs-metadata min_size 2, applied live per the runbook pattern, since evidently not everything in this pool's config is fully Rook-managed.

🟢 Nits

None — the comment rewrite is clear, and cross-references (RUNBOOK-baremetal-conversion.md, the replicapool comment, issue #434) all check out against the files they point to.

Tests

No automated test can cover this (it's a live-cluster state change applied via GitOps reconcile). The repo's own convention for changes like this is operational verification after merge, which the PR body already plans for (it lists the ceph status output it will check). I'd add one more line item to that post-merge check: verify min_size on cephfs-metadata, per the Suggestion above, since that's the one part of "restore standing state" that this diff can't fully guarantee by itself.

Recommendation

Looks reasonable — no confirmed blocking issues; the min_size point above is a verification gap worth closing (ideally before or immediately after merge), not a defect I can point at with certainty.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `b33c482ad32dde378600cf2250c9ad536c80c78c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR restores `cephfs-metadata` from the 2/1 conversion-window state back to size 3 with no `min_size` override, and rewrites the pool's comment to state the standing rationale instead of the temporary one. The diff is small (one file, a `spec.metadataPool` block and its comment) and the reasoning in the PR body and the rewritten comment is sound and well cross-referenced with the sibling `replicapool` block and `RUNBOOK-baremetal-conversion.md`. I found no outright bugs in the YAML itself, but there is one durability assumption in the PR that I could not confirm and think is worth checking before treating this as done. ## 🟡 Suggestions - **`storageclass.yaml:102-104` — confirm `min_size` actually returns to 2, don't assume it does.** The PR removes `parameters: { min_size: "1" }` and relies on "the default `min_size` of 2" applying once `size` goes back to 3. But the *sibling* comment on `replicapool` (`storageclass.yaml:18-21`, unchanged by this diff) documents the opposite direction of this exact fact: "Ceph does not lower `min_size` when `size` is reduced" — i.e. `min_size` is an independent pool property that `ceph osd pool set <pool> size N` does not touch. That's a general property of the `osd pool set size` command, not something that reverses direction: raising `size` back to 3 on an *existing* pool won't retroactively raise `min_size` either, unless Rook's operator itself pushes a computed default `min_size` on every reconcile even when the CR's `parameters` map doesn't mention it. Ceph's "default `min_size` of 2 for size 3" formula (`size - size/2`) is applied at pool *creation* time (or whenever `min_size` is explicitly set) — not automatically re-applied when `size` changes later. If Rook only issues `ceph osd pool set min_size ...` when the CR explicitly carries that parameter (plausible, since Rook generally applies only what's declared), then after this merges the live pool will still be running with `min_size: 1` — i.e. functionally still able to accept single-copy writes while degraded — but now with no comment or override in git warning about it, which is worse than the current explicit 2/1 state. I could not verify the operator's actual behavior here (no Rook source vendored in this checkout, no cluster access, and only one commit of history to check precedent against). The one check that settles it: after this merges and reconciles, run `ceph osd pool get cephfs-metadata min_size` (or `ceph osd pool ls detail`) against the live cluster. If it still reads `1`, it needs the same treatment the PR already gives `.mgr` and `osd_mclock_profile` — a manual `ceph osd pool set cephfs-metadata min_size 2`, applied live per the runbook pattern, since evidently not everything in this pool's config is fully Rook-managed. ## 🟢 Nits None — the comment rewrite is clear, and cross-references (`RUNBOOK-baremetal-conversion.md`, the `replicapool` comment, issue #434) all check out against the files they point to. ## Tests No automated test can cover this (it's a live-cluster state change applied via GitOps reconcile). The repo's own convention for changes like this is operational verification after merge, which the PR body already plans for (it lists the `ceph status` output it will check). I'd add one more line item to that post-merge check: verify `min_size` on `cephfs-metadata`, per the Suggestion above, since that's the one part of "restore standing state" that this diff can't fully guarantee by itself. ## Recommendation Looks reasonable — no confirmed blocking issues; the min_size point above is a verification gap worth closing (ideally before or immediately after merge), not a defect I can point at with certainty.

claude-bot referenced this pull request

2026-08-26 00:13:48 +00:00

feat(rook-ceph): restore replicapool and cephfs-data0 to size 3 #610

claude-bot commented

2026-08-26 00:13:58 +00:00

Correction to my previous comment: it was posted after this PR had already merged, so the scope extension it describes never landed here. This PR restored cephfs-metadata only — the table in that comment is wrong.

replicapool and cephfs-data0 are in #610 instead.

Correction to my previous comment: it was posted after this PR had already merged, so the scope extension it describes never landed here. **This PR restored `cephfs-metadata` only** — the table in that comment is wrong. `replicapool` and `cephfs-data0` are in #610 instead.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#609