chore(rook-ceph): return mon_data_avail_warn to the Ceph default #655

2026-08-27T06:38:08Z

claude-bot commented

2026-08-27 06:38:08 +00:00

The override existed because the mons share the node /var with containerd, and on the VMs that was ~60 GB — 30% free measured image and snapshot churn rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the image cache filled and GC could not reclaim it (#434).

Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so 30% free is a real reserve again and the ~106 MB mon database is irrelevant against it. Reaching 70% would be a genuine signal about the node rather than noise about the image cache. The comment still described the 60 GB VM /var and ended with "BARE-METAL MIGRATION: revisit this" — now done.

Set to 30 rather than deleted, deliberately. ceph config keeps a value that only disappears from this file:

mon   advanced   mon_data_avail_warn   20

Removing the line would leave that in the mon config database indefinitely. It is the trap the bluestore settings below already document — and the one that crash-looped an OSD during the network provider change, when public_network survived being deleted here and the OSD came up unable to pick an address.

mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in monitoring-rules still watches /var directly for the real risk, which is the node filling rather than the mon growing.

Diff against the live cluster is the value and the generation bump, nothing else. After it applies, ceph config get mon mon_data_avail_warn returning 30 is the proof the database entry was overwritten rather than merely edited in git.

Supersedes the remaining half of #503, closed earlier as overtaken by the bare-metal migration.

The override existed because the mons share the node `/var` with containerd, and on the VMs that was ~60 GB — 30% free measured image and snapshot churn rather than anything about Ceph, so k8s-2 tripped `MON_DISK_LOW` whenever the image cache filled and GC could not reclaim it (#434). **Bare metal removed the premise.** `/var` is now ~931 GB per node at 7-8% used, so 30% free is a real reserve again and the ~106 MB mon database is irrelevant against it. Reaching 70% would be a genuine signal about the node rather than noise about the image cache. The comment still described the 60 GB VM `/var` and ended with *"BARE-METAL MIGRATION: revisit this"* — now done. **Set to 30 rather than deleted, deliberately.** `ceph config` keeps a value that only disappears from this file: ``` mon advanced mon_data_avail_warn 20 ``` Removing the line would leave that in the mon config database indefinitely. It is the trap the bluestore settings below already document — and the one that crash-looped an OSD during the network provider change, when `public_network` survived being deleted here and the OSD came up unable to pick an address. `mon_data_avail_crit` stays at the 5% default, and `CephMonDiskLow` in monitoring-rules still watches `/var` directly for the real risk, which is the node filling rather than the mon growing. **Diff against the live cluster** is the value and the generation bump, nothing else. After it applies, `ceph config get mon mon_data_avail_warn` returning 30 is the proof the database entry was overwritten rather than merely edited in git. Supersedes the remaining half of #503, closed earlier as overtaken by the bare-metal migration.

claude-bot added 1 commit 2026-08-27 06:38:08 +00:00

chore(rook-ceph): return mon_data_avail_warn to the Ceph default

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

dd311f4b01

The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).

Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.

Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.

mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.

kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.

Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-27 06:40:01 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit dd311f4b013c282f5a7caf7a38875257b6583441.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR updates the Ceph monitor disk-warning threshold from 20% to 30% (Ceph's default) to reflect the bare-metal migration, where /var is now ~931 GB per node rather than the previous 60 GB VMs. The change is well-reasoned, properly documented, and follows the project's established pattern for updating Ceph config values.

🟢 Nits

The PR body references issue #503 as "supersedes the remaining half of #503" — consider linking it as closes #503 if that issue can be fully closed by this change, to maintain issue tracking.

Tests

No tests are needed for this configuration change. The configuration will be applied to the cluster and can be verified at runtime with ceph config get mon mon_data_avail_warn, which should return 30 after applying.

Recommendation

Looks reasonable — no blocking issues; the change is correct and well-documented.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `dd311f4b013c282f5a7caf7a38875257b6583441`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR updates the Ceph monitor disk-warning threshold from 20% to 30% (Ceph's default) to reflect the bare-metal migration, where `/var` is now ~931 GB per node rather than the previous 60 GB VMs. The change is well-reasoned, properly documented, and follows the project's established pattern for updating Ceph config values. ## 🟢 Nits - The PR body references issue #503 as "supersedes the remaining half of #503" — consider linking it as `closes #503` if that issue can be fully closed by this change, to maintain issue tracking. ## Tests No tests are needed for this configuration change. The configuration will be applied to the cluster and can be verified at runtime with `ceph config get mon mon_data_avail_warn`, which should return `30` after applying. ## Recommendation Looks reasonable — no blocking issues; the change is correct and well-documented.

k.renner merged commit cdebaba2d3 into master

2026-08-27 06:40:39 +00:00

k.renner referenced this issue from a commit

2026-08-27 06:40:39 +00:00

Merge pull request 'chore(rook-ceph): return mon_data_avail_warn to the Ceph default' (#655) from ceph-mon-disk-warn-default into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#655