fix: [alert] CephHealthWarn (rook-ceph) (#562) #563

2026-08-24T12:47:13Z

issue-fix-bot commented

2026-08-24 12:47:13 +00:00

What this PR does

CephHealthWarn fired for BLUESTORE_SLOW_OP_ALERT and self-cleared ~24h
later, which is the sticky-check behaviour the rule's own comment already
documents. The underlying stall is not repo-fixable (QLC NVMe outrunning
their SLC cache under the weekly Velero restore load), so nothing about the
alert's expr or for changes. What was fixable is the triage gap the
issue hit: the rule tells the operator that "a repeat on the same OSD means
check that drive", but its runbook pointed only at
kubectl exec ... ceph health detail, which needs cluster-admin exec and
returns nothing once the check clears — so the one fact the rule asks for was
the one fact nobody could get. This run established the OSD is recoverable
from Loki, so the change records the per-OSD attribution for all three
episodes and puts the Loki query in the alert description.

Changes

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml — comment: third
episode (Sun Aug 23) added, and per-OSD attribution recorded for all three
(Jul 26 + Aug 2 hit osd.0/1/2 together = load-driven; Aug 23 hit osd.1
alone), plus why no metric can supply the OSD id. Description: added the
Loki query that names the OSD and works after the check clears, kept the
exec command for while-still-active detail, and corrected the restore-test
window from 08:00-13:00Z to 08:00-14:00Z. expr and for untouched.

Verification

Live, this run:

kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath=... → HEALTH_WARN.
Prometheus ceph_health_detail==1 → only AUTH_INSECURE_CLIENT_KEY_TYPE,
AUTH_INSECURE_KEYS_ALLOWED, AUTH_INSECURE_KEYS_CREATABLE (all excluded by
design). BLUESTORE_SLOW_OP_ALERT is gone — the blip did resolve.
Prometheus ALERTS{alertname="CephHealthWarn",name="BLUESTORE_SLOW_OP_ALERT"}
→ pending from 2026-08-23T12:00Z, firing 18:00Z → last sample 2026-08-24T11:30Z,
absent at 12:42Z.
OSD identified: Loki
{namespace="rook-ceph",pod=~"rook-ceph-osd-.*"} |= "slow operation observed"
over Aug 23 08:00-13:00Z returned lines from rook-ceph-osd-1 only, e.g.
2026-08-23T11:32:07.603+0000 ... bluestore(/var/lib/ceph/osd/ceph-1) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.616281986s
— i.e. osd.1, 92 lines in a single burst.
Same query aggregated sum by (pod) (count_over_time(...[1d])) over 30d → exactly
three episode days: Jul 26 (osd.0 985 / osd.1 2957 / osd.2 913), Aug 2 (793 / 321 /
721), Aug 23 (osd.1 92 only).
max_over_time(ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon=~"osd.*"}[30d])
→ 0 for all three OSDs, confirming that metric cannot supply the OSD id.
kubernetes/infrastructure/velero/restore-test-cronjob.yaml:32,41 →
schedule: "0 8 * * 0" + activeDeadlineSeconds: 21600, i.e. Sun 08:00-14:00Z.
The 11:32Z stall is inside it.
kubectl create --dry-run=client -f <file> parses; description renders with
{{ $labels.name }} intact and the escaped quotes resolved.

Reviewer check after Flux reconciles: kubectl -n monitoring get prometheusrule ceph
picks up the new annotation; the alert's firing behaviour is unchanged by design, so
nothing should start or stop firing as a result of this PR.

Notes

Things the issue got wrong, corrected here:

It concluded the OSD "can't be identified" without pods/exec on
rook-ceph-tools. It can — the OSD daemon logs it, and Loki has 90d
retention. This is what let me name osd.1.
It called this "the same recurring pattern" as the prior two episodes. The
signature is actually different: Jul 26 and Aug 2 stalled all three OSDs
with thousands of log lines over several hours, whereas Aug 23 was osd.1
alone, 92 lines in one ~1-minute burst. Both still fall inside the restore
window, so this does not yet meet the rule's "check that drive" bar — but
it is the first single-OSD episode, which is why the comment now flags a
repeat of osd.1-alone specifically.
It was right that ceph_health_detail carries no OSD label, but missed that
ceph_daemon_health_metrics{type="SLOW_OPS"} exists with a ceph_daemon
label. I checked it: it reads 0 at every scrape over 30d, because a stall is
shorter than the scrape interval. So the conclusion held, for a different
reason — now recorded so nobody re-investigates that metric.

Caveats:

The ~26h durations for the Jul 26 and Aug 2 episodes are carried over from
the pre-existing comment; I did not re-verify them. I verified the Aug 23
episode's ~24h window (mon "Health check failed" at 11:32:14Z Aug 23, and the
alert series ending 11:30Z Aug 24 with the check now absent).
Drive wear (80-87% endurance) is quoted from the existing comment /
storageclass.yaml; the diagnostics SA cannot read SMART data, so that is
not re-verified. If a human wants the endurance figures refreshed before
acting on osd.1, that still needs ceph device health-metrics via exec.
Deliberately did not touch expr, for, or thresholds — the alert is
correct as written and the issue does not establish otherwise.

Closes #562

## What this PR does `CephHealthWarn` fired for `BLUESTORE_SLOW_OP_ALERT` and self-cleared ~24h later, which is the sticky-check behaviour the rule's own comment already documents. The underlying stall is not repo-fixable (QLC NVMe outrunning their SLC cache under the weekly Velero restore load), so nothing about the alert's `expr` or `for` changes. What *was* fixable is the triage gap the issue hit: the rule tells the operator that "a repeat on the same OSD means check that drive", but its runbook pointed only at `kubectl exec ... ceph health detail`, which needs cluster-admin exec and returns nothing once the check clears — so the one fact the rule asks for was the one fact nobody could get. This run established the OSD *is* recoverable from Loki, so the change records the per-OSD attribution for all three episodes and puts the Loki query in the alert description. ## Changes - `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` — comment: third episode (Sun Aug 23) added, and per-OSD attribution recorded for all three (Jul 26 + Aug 2 hit osd.0/1/2 together = load-driven; Aug 23 hit osd.1 alone), plus why no metric can supply the OSD id. Description: added the Loki query that names the OSD and works after the check clears, kept the exec command for while-still-active detail, and corrected the restore-test window from `08:00-13:00Z` to `08:00-14:00Z`. `expr` and `for` untouched. ## Verification Live, this run: - `kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath=...` → `HEALTH_WARN`. - Prometheus `ceph_health_detail==1` → only `AUTH_INSECURE_CLIENT_KEY_TYPE`, `AUTH_INSECURE_KEYS_ALLOWED`, `AUTH_INSECURE_KEYS_CREATABLE` (all excluded by design). `BLUESTORE_SLOW_OP_ALERT` is gone — the blip did resolve. - Prometheus `ALERTS{alertname="CephHealthWarn",name="BLUESTORE_SLOW_OP_ALERT"}` → pending from 2026-08-23T12:00Z, firing 18:00Z → last sample 2026-08-24T11:30Z, absent at 12:42Z. - **OSD identified**: Loki `{namespace="rook-ceph",pod=~"rook-ceph-osd-.*"} |= "slow operation observed"` over Aug 23 08:00-13:00Z returned lines from `rook-ceph-osd-1` only, e.g. `2026-08-23T11:32:07.603+0000 ... bluestore(/var/lib/ceph/osd/ceph-1) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.616281986s` — i.e. **osd.1**, 92 lines in a single burst. - Same query aggregated `sum by (pod) (count_over_time(...[1d]))` over 30d → exactly three episode days: Jul 26 (osd.0 985 / osd.1 2957 / osd.2 913), Aug 2 (793 / 321 / 721), Aug 23 (osd.1 92 only). - `max_over_time(ceph_daemon_health_metrics{type="SLOW_OPS",ceph_daemon=~"osd.*"}[30d])` → `0` for all three OSDs, confirming that metric cannot supply the OSD id. - `kubernetes/infrastructure/velero/restore-test-cronjob.yaml:32,41` → `schedule: "0 8 * * 0"` + `activeDeadlineSeconds: 21600`, i.e. Sun 08:00-14:00Z. The 11:32Z stall is inside it. - `kubectl create --dry-run=client -f <file>` parses; description renders with `{{ $labels.name }}` intact and the escaped quotes resolved. Reviewer check after Flux reconciles: `kubectl -n monitoring get prometheusrule ceph` picks up the new annotation; the alert's firing behaviour is unchanged by design, so nothing should start or stop firing as a result of this PR. ## Notes Things the issue got wrong, corrected here: - It concluded the OSD "can't be identified" without `pods/exec` on `rook-ceph-tools`. It can — the OSD daemon logs it, and Loki has 90d retention. This is what let me name **osd.1**. - It called this "the same recurring pattern" as the prior two episodes. The signature is actually different: Jul 26 and Aug 2 stalled all three OSDs with thousands of log lines over several hours, whereas Aug 23 was osd.1 alone, 92 lines in one ~1-minute burst. Both still fall inside the restore window, so this does not yet meet the rule's "check that drive" bar — but it is the first single-OSD episode, which is why the comment now flags a repeat of osd.1-alone specifically. - It was right that `ceph_health_detail` carries no OSD label, but missed that `ceph_daemon_health_metrics{type="SLOW_OPS"}` exists with a `ceph_daemon` label. I checked it: it reads 0 at every scrape over 30d, because a stall is shorter than the scrape interval. So the conclusion held, for a different reason — now recorded so nobody re-investigates that metric. Caveats: - The `~26h` durations for the Jul 26 and Aug 2 episodes are carried over from the pre-existing comment; I did not re-verify them. I verified the Aug 23 episode's ~24h window (mon "Health check failed" at 11:32:14Z Aug 23, and the alert series ending 11:30Z Aug 24 with the check now absent). - Drive wear (80-87% endurance) is quoted from the existing comment / `storageclass.yaml`; the diagnostics SA cannot read SMART data, so that is not re-verified. If a human wants the endurance figures refreshed before acting on osd.1, that still needs `ceph device health-metrics` via exec. - Deliberately did not touch `expr`, `for`, or thresholds — the alert is correct as written and the issue does not establish otherwise. Closes #562

issue-fix-bot added 1 commit 2026-08-24 12:47:14 +00:00

fix: [alert] CephHealthWarn (rook-ceph) (#562 )

ci/woodpecker/pr/ci-manifests-test Pipeline is pending

Details

ci/woodpecker/pr/ci-scripts-test Pipeline is pending

Details

ci/woodpecker/pr/pr-review Pipeline is pending

Details

b1901047a0

issue-fix-bot referenced this pull request

2026-08-24 12:47:14 +00:00

[alert] CephHealthWarn (rook-ceph) #562

k.renner closed this pull request

2026-08-25 05:16:02 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#563