fix(ceph): set mon_data_avail_warn to 20%, matching the shared /var #502

2026-08-18T19:41:28Z

claude-bot commented

2026-08-18 19:41:28 +00:00

Addresses #434. Stops MON_DISK_LOW firing for something that is not a Ceph condition.

Why 30% is the wrong number here

Ceph's default assumes the mon owns its filesystem, where 30% free is a meaningful reserve. In this cluster:


mon storage	`dataDirHostPath: /var/lib/rook` — the node's `/var`
shares that volume with	~29 GB of containerd image + snapshot storage
mon DB actual size	~106 MB

So a 106 MB database is judged against a 60 GB filesystem it merely shares, and k8s-2 trips MON_DISK_LOW whenever its image cache fills. The check is measuring container churn, not Ceph.

Tuning image GC was tried and does not work

Measured on k8s-2 (see #501): dropping imageGCHighThresholdPercent to 60 deleted 28 images and freed ~1.3 GB of blobs, moving free space 31.49% → 31.47%. /var/lib/containerd is 20.5 GB of overlayfs snapshots against 8.6 GB of content, and those snapshots are unpacked layers held by running containers — image GC cannot reclaim them. That change has been reverted.

What this does

mon_data_avail_warn: "20" via cephConfig, so it reconciles from git rather than living as a drifting ceph config set.

20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database — ample. What is deliberately not touched:

mon_data_avail_crit stays at 5%, so a genuinely full volume still raises HEALTH_ERR.
CephMonDiskLow (added in #500) keeps watching /var free space directly, which is where the real risk lives — the node filling, not the mon growing. Lowering the Ceph threshold does not blind that.

Noted for the bare-metal migration

The manifest comment carries this so it is not lost:

BARE-METAL MIGRATION: revisit this. Give the mons their own volume and the default 30% becomes meaningful again, at which point this override should be dropped rather than carried over.

That cannot be done on the current cluster, for two independent reasons established in #501:

Every StorageClass here is Ceph-backed (rook-ceph.*.csi.ceph.com), so a mon PVC would be circular — the mons are the quorum authority, and on a cold start there would be no Ceph to serve the volume they need to boot.
Rook only applies mon.volumeClaimTemplate to newly created mons — it never migrates existing ones.

Bare metal is the natural point to give mons a dedicated volume, which is why the reminder is pinned to the setting it would replace.

Verification

kubectl kustomize builds; rendered spec.cephConfig contains the new mon section alongside the existing mgr and osd ones.
kubectl apply --dry-run=server accepts the CephCluster.
Live baseline confirmed via the toolbox from #500: ceph config get mon mon_data_avail_warn → 30, mon_data_avail_crit → 5, /var/lib/rook → 106 MB.

Addresses #434. Stops MON_DISK_LOW firing for something that is not a Ceph condition. ## Why 30% is the wrong number here Ceph's default assumes the mon owns its filesystem, where 30% free is a meaningful reserve. In this cluster: | | | |---|---| | mon storage | `dataDirHostPath: /var/lib/rook` — the node's `/var` | | shares that volume with | ~29 GB of containerd image + snapshot storage | | **mon DB actual size** | **~106 MB** | So a 106 MB database is judged against a 60 GB filesystem it merely shares, and k8s-2 trips MON_DISK_LOW whenever its image cache fills. The check is measuring container churn, not Ceph. ## Tuning image GC was tried and does not work Measured on k8s-2 (see #501): dropping `imageGCHighThresholdPercent` to 60 deleted 28 images and freed ~1.3 GB of blobs, moving free space **31.49% → 31.47%**. `/var/lib/containerd` is 20.5 GB of overlayfs snapshots against 8.6 GB of content, and those snapshots are unpacked layers held by running containers — image GC cannot reclaim them. That change has been reverted. ## What this does `mon_data_avail_warn: "20"` via `cephConfig`, so it reconciles from git rather than living as a drifting `ceph config set`. 20% of a 60 GB `/var` still leaves ~12 GB behind a 106 MB database — ample. What is deliberately *not* touched: - **`mon_data_avail_crit` stays at 5%**, so a genuinely full volume still raises HEALTH_ERR. - **`CephMonDiskLow`** (added in #500) keeps watching `/var` free space directly, which is where the real risk lives — the node filling, not the mon growing. Lowering the Ceph threshold does not blind that. ## Noted for the bare-metal migration The manifest comment carries this so it is not lost: > **BARE-METAL MIGRATION: revisit this.** Give the mons their own volume and the default 30% becomes meaningful again, at which point this override should be dropped rather than carried over. That cannot be done on the current cluster, for two independent reasons established in #501: 1. **Every StorageClass here is Ceph-backed** (`rook-ceph.*.csi.ceph.com`), so a mon PVC would be circular — the mons are the quorum authority, and on a cold start there would be no Ceph to serve the volume they need to boot. 2. **Rook only applies `mon.volumeClaimTemplate` to newly created mons** — it never migrates existing ones. Bare metal is the natural point to give mons a dedicated volume, which is why the reminder is pinned to the setting it would replace. ## Verification - `kubectl kustomize` builds; rendered `spec.cephConfig` contains the new `mon` section alongside the existing `mgr` and `osd` ones. - `kubectl apply --dry-run=server` accepts the CephCluster. - Live baseline confirmed via the toolbox from #500: `ceph config get mon mon_data_avail_warn` → `30`, `mon_data_avail_crit` → `5`, `/var/lib/rook` → 106 MB.

claude-bot added 1 commit 2026-08-18 19:41:28 +00:00

fix(ceph): set mon_data_avail_warn to 20%, matching the shared /var

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

57acf5d8f2

Addresses #434.

Ceph's 30% default assumes the mon owns its filesystem, where 30% free is
a real reserve. Our mons sit on dataDirHostPath /var/lib/rook — the node's
/var, shared with ~29 GB of containerd storage — and the mon DB is ~106 MB.
The default therefore measures container churn rather than Ceph state, and
k8s-2 trips MON_DISK_LOW whenever its image cache fills.

Tuning image GC does not help: measured on k8s-2, dropping the GC high
threshold to 60 deleted 28 images and moved free space by 0.02 percentage
points, because most of /var/lib/containerd is overlayfs snapshots held by
running containers, which image GC cannot reclaim.

20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database.
mon_data_avail_crit is untouched at 5%, and CephMonDiskLow watches /var
directly for the real risk — the node filling, not the mon growing.

Set through cephConfig so it is reconciled from git rather than applied as
a live `ceph config set`.

The comment records that this override should be revisited, and dropped,
when mons get their own volume at bare-metal migration. That is not
possible on the current cluster: every StorageClass is Ceph-backed, so a
mon PVC would be circular, and Rook applies mon.volumeClaimTemplate only
to newly created mons.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

claude-bot referenced this pull request

2026-08-18 19:42:10 +00:00

Bare-metal migration: give Ceph mons their own volume and drop the mon_data_avail_warn override #503

review-bot reviewed

2026-08-18 19:43:18 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 57acf5d8f21ccc1c7f42ad7e178069cfee30c2bb.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR lowers the Ceph mon_data_avail_warn threshold from 30% to 20% to address false MON_DISK_LOW alerts caused by the architectural constraint that mons share /var with ~29 GB of containerd storage. The fix is well-reasoned, includes defense-in-depth safeguards, and has been verified in the live cluster.

🔴 Blocking

None.

🟡 Suggestions

Value type consistency — mon_data_avail_warn: "20" is specified as a string. While this works (verified in the live cluster), Ceph configuration parameters are typically integers. Consider whether mon_data_avail_warn: 20 (unquoted) would be more idiomatic. This is a minor style point if the current form is working as verified; confirm with your Ceph/Rook team's convention if there's any doubt.

🟢 Nits

The comment is excellent and explains the architectural constraints, the trade-offs, and the migration path clearly. This level of documentation is exactly what infrastructure-as-code needs for maintainability.

Tests

This is a Kubernetes manifest change, not application code, so unit tests don't apply. The PR includes appropriate verification:

kubectl kustomize builds without errors (cephcluster.yaml:27-47)
kubectl apply --dry-run=server accepts the CephCluster
Live verification confirmed the setting is applied correctly via ceph config get mon mon_data_avail_warn

The change properly preserves defense-in-depth: mon_data_avail_crit stays at 5% (the hard limit), and the PR references CephMonDiskLow from #500 that watches /var directly for actual disk pressure.

Recommendation

Looks reasonable — no blocking issues; the suggestion above is optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `57acf5d8f21ccc1c7f42ad7e178069cfee30c2bb`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR lowers the Ceph `mon_data_avail_warn` threshold from 30% to 20% to address false MON_DISK_LOW alerts caused by the architectural constraint that mons share `/var` with ~29 GB of containerd storage. The fix is well-reasoned, includes defense-in-depth safeguards, and has been verified in the live cluster. ## 🔴 Blocking None. ## 🟡 Suggestions 1. **Value type consistency** — `mon_data_avail_warn: "20"` is specified as a string. While this works (verified in the live cluster), Ceph configuration parameters are typically integers. Consider whether `mon_data_avail_warn: 20` (unquoted) would be more idiomatic. This is a minor style point if the current form is working as verified; confirm with your Ceph/Rook team's convention if there's any doubt. ## 🟢 Nits The comment is excellent and explains the architectural constraints, the trade-offs, and the migration path clearly. This level of documentation is exactly what infrastructure-as-code needs for maintainability. ## Tests This is a Kubernetes manifest change, not application code, so unit tests don't apply. The PR includes appropriate verification: - `kubectl kustomize` builds without errors (cephcluster.yaml:27-47) - `kubectl apply --dry-run=server` accepts the CephCluster - Live verification confirmed the setting is applied correctly via `ceph config get mon mon_data_avail_warn` The change properly preserves defense-in-depth: `mon_data_avail_crit` stays at 5% (the hard limit), and the PR references `CephMonDiskLow` from #500 that watches `/var` directly for actual disk pressure. ## Recommendation Looks reasonable — no blocking issues; the suggestion above is optional.

k.renner merged commit 283f7b1fee into master

2026-08-18 19:45:26 +00:00

k.renner referenced this issue from a commit

2026-08-18 19:45:28 +00:00

Merge pull request 'fix(ceph): set mon_data_avail_warn to 20%, matching the shared /var' (#502) from ceph-mon-data-avail-warn into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#502