fix: [alert] CephMonDiskLow (monitoring) (#569) #572

2026-08-25T16:44:21Z

issue-fix-bot commented

2026-08-25 16:44:21 +00:00

What this PR does

CephMonDiskLow fired for k8s-3 (10.2.40.168, /var, /dev/vda5) because free space fell to 19.76% during a wave of pod rescheduling. Verifying live this run, the acute condition has already cleared on its own: k8s-3 is back to 24.78% free and the trend is positive, and MON_DISK_LOW is no longer an active Ceph health warning. The alert is still firing only because it sits at a 30% early-warning margin. That leaves one real, fixable defect — and it is a documentation defect the issue correctly identified: the alert's description asserts that 30% "is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW", which is stale. cephcluster.yaml:58 overrides mon_data_avail_warn to 20. The wording turns an early warning into a false statement of fact about Ceph state, and it demonstrably misled triage on this very alert. This PR rewords the annotation to say what the 30% margin actually means and how to check whether Ceph is genuinely warning.

Changes

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml — reworded the CephMonDiskLow summary and description. The description now states that 30% is an early-warning margin, that cephcluster.yaml overrides mon_data_avail_warn to 20 so MON_DISK_LOW only fires below 20% free, and points the responder at ceph_health_detail{name="MON_DISK_LOW"} == 1 to check whether Ceph is actually warning before treating the alert as a Ceph problem. The existing talosctl usage guidance is kept verbatim. No threshold, expr, for, or severity was changed — the alert still fires at exactly the same time.

Verification

Verified live this run (all via ./.ci-diag/kubectl):

ALERTS{alertname="CephMonDiskLow"} → alertstate="firing" for instance="10.2.40.168:9100". The alert is genuinely still firing.
100 * node_filesystem_avail_bytes / node_filesystem_size_bytes{mountpoint="/var"} → k8s-3 24.78%, k8s-2 37.53%. k8s-3 has recovered from the 19.76% in the issue and is back above Ceph's real 20% threshold.
ceph_health_detail == 1 → active warnings are MON_DOWN, PG_DEGRADED, TOO_FEW_OSDS, AUTH_INSECURE_*. MON_DISK_LOW is absent — it has cleared since the issue was filed. This is the key correction: the issue's central claim that "MON_DISK_LOW is already an active HEALTH_WARN" is no longer true.
deriv(node_filesystem_avail_bytes{...}[30m]) * 3600 → +63 MiB/h. Free space is recovering, not draining. The issue's "−271 MiB/h → ~33 hours of headroom" no longer holds.
kubectl get nodes → only k8s-2 and k8s-3 exist, both uncordoned and untainted. k8s-1 is no longer in the cluster at all.
kubectl get pods -n rook-ceph -o wide → rook-ceph-mon-a on k8s-3 and mon-b on k8s-2 are 1/1 Running; rook-ceph-mon-c is Pending (63m) and rook-ceph-mon-d-canary is Pending — unschedulable because there is no third node.

Reviewer confirmation: this is an annotation-only change, so nothing about firing behaviour changes. After Flux reconciles, kubectl -n monitoring get prometheusrule → the CephMonDiskLow description should no longer mention 30% as mon_data_avail_warn. The alert itself will stop firing once k8s-3 /var passes 30% free for 30m; it is at 24.78% and rising.

Notes

What the issue got wrong / what has changed since it was filed:

The primary recommended action is not actionable. The issue says "k8s-1 is already back … rebalance the pods that landed on k8s-3 back onto k8s-1". k8s-1 is not in the cluster — kubectl get nodes returns only k8s-2 and k8s-3. Deleting k8s-3-resident pods now would have nowhere to reschedule them except k8s-3 itself. Do not follow that step.
MON_DISK_LOW has cleared and the disk trend has reversed to positive, so the urgency framing (33h to mon_data_avail_crit) is stale. Part of the recovery is plausibly the merged immich server/machine-learning scaledown (#570) freeing snapshots on k8s-3, though I did not attribute the recovery to a specific workload.
The issue's documentation-defect finding was correct, and it is the only part I acted on.

Now the more serious condition, and it is not repo-fixable: Ceph is MON_DOWN with mon-c and a mon-d-canary both Pending, running on two of three mons — quorum is intact but has no redundancy left. Losing k8s-2 or k8s-3 loses quorum and the storage cluster. This is the expected in-flight state of the bare-metal conversion documented at cephcluster.yaml:52-57, not something a manifest change should paper over, so I made no edits for it. A human should either return a third node to the cluster to restore mon quorum redundancy, or consciously accept the two-mon window for its duration.

Deliberately not done: no change to imageGCHighThresholdPercent (the file already records that tuning it moved free space by 0.02 points), no change to mon_data_avail_warn, and no change to the 30% threshold or for: duration — that would be silencing the alert rather than fixing anything. The durable capacity fix (larger /var on the opentofu/ VM definitions, or a dedicated mon volume) is a human capacity decision that cephcluster.yaml already flags as blocked on the bare-metal migration.

Closes #569

## What this PR does `CephMonDiskLow` fired for k8s-3 (`10.2.40.168`, `/var`, `/dev/vda5`) because free space fell to 19.76% during a wave of pod rescheduling. Verifying live this run, the acute condition has **already cleared on its own**: k8s-3 is back to **24.78% free** and the trend is **positive**, and `MON_DISK_LOW` is **no longer an active Ceph health warning**. The alert is still firing only because it sits at a 30% early-warning margin. That leaves one real, fixable defect — and it is a documentation defect the issue correctly identified: the alert's `description` asserts that 30% "is Ceph's `mon_data_avail_warn` threshold, so the mon on this node **will** report MON_DISK_LOW", which is stale. `cephcluster.yaml:58` overrides `mon_data_avail_warn` to `20`. The wording turns an early warning into a false statement of fact about Ceph state, and it demonstrably misled triage on this very alert. This PR rewords the annotation to say what the 30% margin actually means and how to check whether Ceph is genuinely warning. ## Changes - `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` — reworded the `CephMonDiskLow` `summary` and `description`. The description now states that 30% is an early-warning margin, that `cephcluster.yaml` overrides `mon_data_avail_warn` to 20 so `MON_DISK_LOW` only fires below 20% free, and points the responder at `ceph_health_detail{name="MON_DISK_LOW"} == 1` to check whether Ceph is actually warning before treating the alert as a Ceph problem. The existing `talosctl usage` guidance is kept verbatim. No threshold, `expr`, `for`, or severity was changed — the alert still fires at exactly the same time. ## Verification Verified live this run (all via `./.ci-diag/kubectl`): - **`ALERTS{alertname="CephMonDiskLow"}`** → `alertstate="firing"` for `instance="10.2.40.168:9100"`. The alert is genuinely still firing. - **`100 * node_filesystem_avail_bytes / node_filesystem_size_bytes{mountpoint="/var"}`** → k8s-3 **24.78%**, k8s-2 **37.53%**. k8s-3 has recovered from the 19.76% in the issue and is **back above Ceph's real 20% threshold**. - **`ceph_health_detail == 1`** → active warnings are `MON_DOWN`, `PG_DEGRADED`, `TOO_FEW_OSDS`, `AUTH_INSECURE_*`. **`MON_DISK_LOW` is absent** — it has cleared since the issue was filed. This is the key correction: the issue's central claim that "MON_DISK_LOW is already an active HEALTH_WARN" is no longer true. - **`deriv(node_filesystem_avail_bytes{...}[30m]) * 3600`** → **+63 MiB/h**. Free space is *recovering*, not draining. The issue's "−271 MiB/h → ~33 hours of headroom" no longer holds. - **`kubectl get nodes`** → only **`k8s-2` and `k8s-3`** exist, both uncordoned and untainted. **k8s-1 is no longer in the cluster at all.** - **`kubectl get pods -n rook-ceph -o wide`** → `rook-ceph-mon-a` on k8s-3 and `mon-b` on k8s-2 are `1/1 Running`; `rook-ceph-mon-c` is **`Pending` (63m)** and `rook-ceph-mon-d-canary` is `Pending` — unschedulable because there is no third node. Reviewer confirmation: this is an annotation-only change, so nothing about firing behaviour changes. After Flux reconciles, `kubectl -n monitoring get prometheusrule` → the `CephMonDiskLow` description should no longer mention 30% as `mon_data_avail_warn`. The alert itself will stop firing once k8s-3 `/var` passes 30% free for 30m; it is at 24.78% and rising. ## Notes **What the issue got wrong / what has changed since it was filed:** 1. **The primary recommended action is not actionable.** The issue says "k8s-1 is already back … rebalance the pods that landed on k8s-3 back onto k8s-1". k8s-1 is **not** in the cluster — `kubectl get nodes` returns only k8s-2 and k8s-3. Deleting k8s-3-resident pods now would have nowhere to reschedule them except k8s-3 itself. **Do not follow that step.** 2. **`MON_DISK_LOW` has cleared** and the disk trend has reversed to positive, so the urgency framing (33h to `mon_data_avail_crit`) is stale. Part of the recovery is plausibly the merged immich `server`/`machine-learning` scaledown (#570) freeing snapshots on k8s-3, though I did not attribute the recovery to a specific workload. 3. **The issue's documentation-defect finding was correct**, and it is the only part I acted on. **Now the more serious condition, and it is not repo-fixable:** Ceph is `MON_DOWN` with `mon-c` and a `mon-d-canary` both Pending, running on **two of three mons** — quorum is intact but has **no redundancy left**. Losing k8s-2 or k8s-3 loses quorum and the storage cluster. This is the expected in-flight state of the bare-metal conversion documented at `cephcluster.yaml:52-57`, not something a manifest change should paper over, so I made no edits for it. A human should either return a third node to the cluster to restore mon quorum redundancy, or consciously accept the two-mon window for its duration. **Deliberately not done:** no change to `imageGCHighThresholdPercent` (the file already records that tuning it moved free space by 0.02 points), no change to `mon_data_avail_warn`, and no change to the 30% threshold or `for:` duration — that would be silencing the alert rather than fixing anything. The durable capacity fix (larger `/var` on the `opentofu/` VM definitions, or a dedicated mon volume) is a human capacity decision that `cephcluster.yaml` already flags as blocked on the bare-metal migration. Closes #569

issue-fix-bot added 1 commit 2026-08-25 16:44:21 +00:00

fix: [alert] CephMonDiskLow (monitoring) (#569 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

f6dba0eefc

issue-fix-bot referenced this pull request

2026-08-25 16:44:22 +00:00

[alert] CephMonDiskLow (monitoring) #569

review-bot reviewed

2026-08-25 16:46:20 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit f6dba0eefce6c75f676e95d577d82f01aa25b1ac.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes a documentation defect in the CephMonDiskLow alert annotation. The old description incorrectly claimed that 30% free is Ceph's mon_data_avail_warn threshold, but the actual configuration in cephcluster.yaml:58 sets it to 20%. The change correctly documents this distinction and adds actionable guidance for verifying whether Ceph is actually warning. No alert behavior, thresholds, or severity are changed.

🔴 Blocking

None — no correctness issues found.

🟡 Suggestions

None — the change is clear, accurate, and well-documented.

🟢 Nits

None.

Tests

This is an annotation-only change to a PrometheusRule; no tests are needed. The alert's expression, for duration, severity, and labels remain unchanged. The PR includes excellent verification confirming that the alert still fires as expected and that cephcluster.yaml:58 does indeed override mon_data_avail_warn to 20 (as claimed in the description).

Recommendation

Looks reasonable — no blocking issues; this is a straightforward documentation fix that improves the clarity and accuracy of the alert annotation.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `f6dba0eefce6c75f676e95d577d82f01aa25b1ac`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes a documentation defect in the `CephMonDiskLow` alert annotation. The old description incorrectly claimed that 30% free is Ceph's `mon_data_avail_warn` threshold, but the actual configuration in `cephcluster.yaml:58` sets it to 20%. The change correctly documents this distinction and adds actionable guidance for verifying whether Ceph is actually warning. No alert behavior, thresholds, or severity are changed. ## 🔴 Blocking None — no correctness issues found. ## 🟡 Suggestions None — the change is clear, accurate, and well-documented. ## 🟢 Nits None. ## Tests This is an annotation-only change to a PrometheusRule; no tests are needed. The alert's expression, `for` duration, severity, and labels remain unchanged. The PR includes excellent verification confirming that the alert still fires as expected and that `cephcluster.yaml:58` does indeed override `mon_data_avail_warn` to 20 (as claimed in the description). ## Recommendation Looks reasonable — no blocking issues; this is a straightforward documentation fix that improves the clarity and accuracy of the alert annotation.

k.renner closed this pull request

2026-08-26 00:11:14 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#572