fix: [alert] CephHealthWarn (rook-ceph) (#557) #558

Merged
k.renner merged 1 commits from alert-fix/issue-557 into master 2026-08-23 20:59:30 +00:00
Member

What this PR does

CephHealthWarn fired for BLUESTORE_SLOW_OP_ALERT on osd.1. The issue read this
as a one-off, unexplained transient ("possible transient disk/controller hiccup",
trigger unconfirmed) and concluded nothing was fixable. The transient part is right;
the "one-off, unexplained" part is not. Over the 30d to 2026-08-23 there were
two episodes, not one, and both began inside the weekly Velero restore-test
window (Sundays 08:00–13:00Z)
— a 5-hour sustained restore that outruns the SLC
cache on these QLC NVMe, whose own post-cache baseline is already documented in
cephcluster.yaml at 0.89 s average write latency. Each episode then stayed WARN
for ~26 h and cleared with no intervention, because Ceph holds this check for
bluestore_slow_ops_warn_lifetime after the last slow op — far past the rule's
for: 6h, so it pages every time while live latency is already normal.

There is no config change that stops a QLC drive from stalling under a 5-hour
restore, and excluding the check would be silencing the earliest warning of a
genuinely failing drive on hardware at 80–87 % endurance used. So this PR changes
no firing behaviour at all — expr and for are untouched. It fixes the
alert's description, which asserted the one thing that is false here ("has been
active for six hours, so it is not a self-clearing blip … so this is a new
condition") and sent triage hunting a live latency problem that had ended five
hours earlier, and records the measured mechanism next to the rule.

Changes

  • kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml — on CephHealthWarn:
    dropped the "not a self-clearing blip / new condition" claim from the description,
    which is untrue for BLUESTORE_SLOW_OP_ALERT, and replaced it with the actual
    triage question (was the stall inside the Sunday restore-test window? then it is
    expected and self-clearing; outside it, or repeating on the same OSD, go check
    that drive). Added a comment block recording the 24 h stickiness, the measured
    episode durations, the Velero correlation, and — explicitly — why the check is
    not excluded despite reliably outliving the 6h for.

Verification

All timestamps UTC. Live this run:

  • kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.ceph}' →
    "health":"HEALTH_WARN", BLUESTORE_SLOW_OP_ALERT: "1 OSD(s) experiencing slow operations in BlueStore", lastChecked: 2026-08-23T18:41:26Z. Still active now —
    the issue's core claim holds.
  • osd.1 log (logs rook-ceph-osd-1-845d6ff994-cdpjl --since=8h) → 92 lines
    matching slow operation observed, matching the issue's count exactly. The
    11:32:09–11:32:10Z clustering of those lines is the issue's reading, which I did
    not re-derive; the count corroborates it.
  • Prometheus query_range on max(ceph_health_detail{name="BLUESTORE_SLOW_OP_ALERT"}),
    30 d at 6 h step, then re-run at 600 s step around each transition — exactly two
    episodes
    :
    • #1 started between 10:03Z and 10:13Z on 2026-08-02, cleared between 12:27Z
      and 12:43Z on 2026-08-03 → ~26.5 h, self-cleared, ~23.5 h after the last
      plausible stall. This is the load-bearing measurement: it is what makes the
      condition demonstrably self-clearing rather than persistent.
    • #2 started 2026-08-23 ~11:37Z, still active.
  • Velero correlation, both episodes:
    • kubectl -n velero get jobs → restore-test-29791200 ran
      2026-08-23T08:00:00Z → 13:06:39Z. The 11:32Z stall is inside it.
    • Job objects for 2026-08-02 are long gone, so via Prometheus instead:
      max(kube_job_status_active{job_name=~"restore-test.*"}) → active
      2026-08-02 08:30Z → 13:00Z. Episode #1's ~10:08Z start is inside it too.
    • restore-test schedule is 0 8 * * 0; 2026-08-02 and 2026-08-23 were both
      Sundays.

For the reviewer: this PR cannot make the alert stop or start firing — diff
expr/for to confirm. BLUESTORE_SLOW_OP_ALERT should clear on its own by
roughly 2026-08-24 13:00Z (~24 h after today's restore window closed), matching
episode #1's behaviour; if it is still set well past that, the drive theory below
gets much stronger. After Flux reconciles, the next CephHealthWarn email carries
the corrected description.

Notes

  • What the issue got wrong. It called this a single burst with an unconfirmed
    trigger and checked only k8s-3 node events for corroboration (correctly finding
    none — the load is cluster-wide restore traffic, not a node fault). It never
    looked for prior occurrences, so it missed the 2026-08-02 episode and therefore
    the weekly pattern. Its recommendation to close as "a one-off transient blip" if
    it self-clears would have been wrong twice over: it will self-clear, and it is
    not a one-off.
  • One detail not verified live. bluestore_slow_ops_warn_lifetime = 24h is the
    Ceph Squid default (cluster runs 19.2.6); I could not read it back, since this
    agent has no pods/exec in rook-ceph and no Secrets access. The measured ~26.5 h
    episode ending ~24 h after the restore window is a good fit, but confirm with
    kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config get osd bluestore_slow_ops_warn_lifetime
    before trusting the "24h" figure in the comment.
  • Deliberately not done, per "never silence the alert instead of fixing the
    cause": no exclusion of BLUESTORE_SLOW_OP_ALERT from expr, no lengthened for
    (anything >26 h would suppress it permanently), no inhibition. The check is
    noisy-but-correct, and on drives with roughly four months of rated life left it is
    worth keeping loud.
  • Recommended human follow-up (not repo work): smartctl on k8s-3's nvme0n1,
    and talosctl -n k8s-3 dmesg around 2026-08-23T11:32Z, to separate "QLC under
    sustained load" from early drive failure. The decisive evidence is cheaper,
    though: if a future stall lands outside a Sunday 08:00–13:00Z restore window, or
    the same OSD trips repeatedly while the other two never do, treat it as the drive.
    So far only osd.1 has ever tripped this, which is the one fact that keeps the
    hardware theory alive — worth watching, not yet worth acting on.
  • Separate latent issue, not touched here. ceph_health_detail carries a pod
    label for the reporting mgr, so a mgr failover starts a new series and restarts
    the for: 6h timer. It only ever delays firing, and mgr restarts are rare, but a
    flapping mgr would suppress CephHealthWarn entirely. Out of scope for this issue.

Closes #557

## What this PR does `CephHealthWarn` fired for `BLUESTORE_SLOW_OP_ALERT` on osd.1. The issue read this as a one-off, unexplained transient ("possible transient disk/controller hiccup", trigger unconfirmed) and concluded nothing was fixable. The transient part is right; the "one-off, unexplained" part is not. Over the 30d to 2026-08-23 there were **two** episodes, not one, and **both began inside the weekly Velero `restore-test` window (Sundays 08:00–13:00Z)** — a 5-hour sustained restore that outruns the SLC cache on these QLC NVMe, whose own post-cache baseline is already documented in `cephcluster.yaml` at 0.89 s average write latency. Each episode then stayed WARN for ~26 h and cleared with no intervention, because Ceph holds this check for `bluestore_slow_ops_warn_lifetime` after the *last* slow op — far past the rule's `for: 6h`, so it pages every time while live latency is already normal. There is no config change that stops a QLC drive from stalling under a 5-hour restore, and excluding the check would be silencing the earliest warning of a genuinely failing drive on hardware at 80–87 % endurance used. So this PR changes **no firing behaviour at all** — `expr` and `for` are untouched. It fixes the alert's *description*, which asserted the one thing that is false here ("has been active for six hours, so it is not a self-clearing blip … so this is a new condition") and sent triage hunting a live latency problem that had ended five hours earlier, and records the measured mechanism next to the rule. ## Changes - `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` — on `CephHealthWarn`: dropped the "not a self-clearing blip / new condition" claim from the description, which is untrue for `BLUESTORE_SLOW_OP_ALERT`, and replaced it with the actual triage question (was the stall inside the Sunday restore-test window? then it is expected and self-clearing; outside it, or repeating on the same OSD, go check that drive). Added a comment block recording the 24 h stickiness, the measured episode durations, the Velero correlation, and — explicitly — why the check is *not* excluded despite reliably outliving the 6h `for`. ## Verification All timestamps UTC. Live this run: - `kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.ceph}'` → `"health":"HEALTH_WARN"`, `BLUESTORE_SLOW_OP_ALERT: "1 OSD(s) experiencing slow operations in BlueStore"`, `lastChecked: 2026-08-23T18:41:26Z`. Still active now — the issue's core claim holds. - osd.1 log (`logs rook-ceph-osd-1-845d6ff994-cdpjl --since=8h`) → **92** lines matching `slow operation observed`, matching the issue's count exactly. The 11:32:09–11:32:10Z clustering of those lines is the issue's reading, which I did not re-derive; the count corroborates it. - Prometheus `query_range` on `max(ceph_health_detail{name="BLUESTORE_SLOW_OP_ALERT"})`, 30 d at 6 h step, then re-run at 600 s step around each transition — **exactly two episodes**: - **#1** started between 10:03Z and 10:13Z on 2026-08-02, cleared between 12:27Z and 12:43Z on 2026-08-03 → **~26.5 h**, self-cleared, ~23.5 h after the last plausible stall. This is the load-bearing measurement: it is what makes the condition demonstrably self-clearing rather than persistent. - **#2** started 2026-08-23 ~11:37Z, still active. - Velero correlation, both episodes: - `kubectl -n velero get jobs` → `restore-test-29791200` ran **2026-08-23T08:00:00Z → 13:06:39Z**. The 11:32Z stall is inside it. - Job objects for 2026-08-02 are long gone, so via Prometheus instead: `max(kube_job_status_active{job_name=~"restore-test.*"})` → active **2026-08-02 08:30Z → 13:00Z**. Episode #1's ~10:08Z start is inside it too. - `restore-test` schedule is `0 8 * * 0`; 2026-08-02 and 2026-08-23 were both Sundays. For the reviewer: this PR cannot make the alert stop or start firing — diff `expr`/`for` to confirm. `BLUESTORE_SLOW_OP_ALERT` should clear on its own by roughly **2026-08-24 13:00Z** (~24 h after today's restore window closed), matching episode #1's behaviour; if it is still set well past that, the drive theory below gets much stronger. After Flux reconciles, the next `CephHealthWarn` email carries the corrected description. ## Notes - **What the issue got wrong.** It called this a single burst with an unconfirmed trigger and checked only k8s-3 node events for corroboration (correctly finding none — the load is cluster-wide restore traffic, not a node fault). It never looked for prior occurrences, so it missed the 2026-08-02 episode and therefore the weekly pattern. Its recommendation to close as "a one-off transient blip" if it self-clears would have been wrong twice over: it *will* self-clear, and it is *not* a one-off. - **One detail not verified live.** `bluestore_slow_ops_warn_lifetime = 24h` is the Ceph Squid default (cluster runs 19.2.6); I could not read it back, since this agent has no `pods/exec` in `rook-ceph` and no Secrets access. The measured ~26.5 h episode ending ~24 h after the restore window is a good fit, but confirm with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config get osd bluestore_slow_ops_warn_lifetime` before trusting the "24h" figure in the comment. - **Deliberately not done**, per "never silence the alert instead of fixing the cause": no exclusion of `BLUESTORE_SLOW_OP_ALERT` from `expr`, no lengthened `for` (anything >26 h would suppress it permanently), no inhibition. The check is noisy-but-correct, and on drives with roughly four months of rated life left it is worth keeping loud. - **Recommended human follow-up** (not repo work): `smartctl` on k8s-3's `nvme0n1`, and `talosctl -n k8s-3 dmesg` around 2026-08-23T11:32Z, to separate "QLC under sustained load" from early drive failure. The decisive evidence is cheaper, though: if a future stall lands *outside* a Sunday 08:00–13:00Z restore window, or the same OSD trips repeatedly while the other two never do, treat it as the drive. So far only osd.1 has ever tripped this, which is the one fact that keeps the hardware theory alive — worth watching, not yet worth acting on. - **Separate latent issue, not touched here.** `ceph_health_detail` carries a `pod` label for the reporting mgr, so a mgr failover starts a new series and restarts the `for: 6h` timer. It only ever delays firing, and mgr restarts are rare, but a flapping mgr would suppress `CephHealthWarn` entirely. Out of scope for this issue. Closes #557
issue-fix-bot added 1 commit 2026-08-23 18:46:51 +00:00
fix: [alert] CephHealthWarn (rook-ceph) (#557)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
8386f755d1
review-bot left a comment
Member

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 8386f755d1db93dd7440d9588b9b011543b1341d.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.


Summary

This PR updates the Prometheus alert description for CephHealthWarn to correctly characterize the behavior of BLUESTORE_SLOW_OP_ALERT, which the issue identified as a transient condition that is sticky for ~24h but self-clearing. The change is documentation-only — no alert logic or expressions are modified. The new description removes false claims, adds evidence-based triage guidance, and clarifies why the alert isn't excluded despite being self-clearing.

🟢 Observations

Correctness of the update: The old description claimed "BLUESTORE_SLOW_OP_ALERT is not a self-clearing blip ... this is a new condition," which contradicts the issue's measured data (two episodes, both ~26.5h, both self-cleared untouched). The new description correctly states it "is sticky for 24h after the last stall and clears itself" — a fact-based reframing backed by Prometheus metrics and Ceph logs in the PR body.

Triage guidance: The replacement description provides actionable next steps: check whether the stall fell inside the Sunday 08:00–13:00Z Velero restore-test window (expected behavior) or outside it (drive investigation needed). This is materially more helpful for oncall than the old framing.

Scope adherence: The PR correctly limits changes to annotations (comments + description). The expr and for fields remain unchanged, so alert firing behavior is unaffected. Comments explicitly document why BLUESTORE_SLOW_OP_ALERT is not excluded despite its self-clearing nature (early signal of drive failure; drives at 80–87% endurance).

Verification: The PR body provides detailed verification: Prometheus query_range data showing exactly two episodes (2026-08-02 and 2026-08-23), pod logs matching the count, and a Velero job timeline showing restore-test ran during both stalls. The claim about bluestore_slow_ops_warn_lifetime = 24h (Ceph Squid default) is noted as unverified live but plausible given the measured ~26.5h duration and ~24h offset.

Minor potential drift: The description now hardcodes a time window ("Sunday 08:00–13:00Z") and endurance state ("80–87%"). These are correct now but could drift if Velero schedules change or hardware ages. The description does provide a way to verify it ("check whether the stall fell inside..."), so the guidance remains actionable even if the specifics shift.

YAML syntax: Indentation and quoting are correct; the file remains valid Kubernetes/Prometheus YAML.

Recommendation

Looks reasonable — no blocking issues. The change is a necessary correction to alert documentation, not a workaround or band-aid. Comments above are optional but generally positive observations about verification quality.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `8386f755d1db93dd7440d9588b9b011543b1341d`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR updates the Prometheus alert description for `CephHealthWarn` to correctly characterize the behavior of `BLUESTORE_SLOW_OP_ALERT`, which the issue identified as a transient condition that is sticky for ~24h but self-clearing. The change is documentation-only — no alert logic or expressions are modified. The new description removes false claims, adds evidence-based triage guidance, and clarifies why the alert isn't excluded despite being self-clearing. ## 🟢 Observations **Correctness of the update**: The old description claimed "`BLUESTORE_SLOW_OP_ALERT` is not a self-clearing blip ... this is a new condition," which contradicts the issue's measured data (two episodes, both ~26.5h, both self-cleared untouched). The new description correctly states it "is sticky for 24h after the last stall and clears itself" — a fact-based reframing backed by Prometheus metrics and Ceph logs in the PR body. **Triage guidance**: The replacement description provides actionable next steps: check whether the stall fell inside the Sunday 08:00–13:00Z Velero restore-test window (expected behavior) or outside it (drive investigation needed). This is materially more helpful for oncall than the old framing. **Scope adherence**: The PR correctly limits changes to annotations (comments + description). The `expr` and `for` fields remain unchanged, so alert firing behavior is unaffected. Comments explicitly document why `BLUESTORE_SLOW_OP_ALERT` is not excluded despite its self-clearing nature (early signal of drive failure; drives at 80–87% endurance). **Verification**: The PR body provides detailed verification: Prometheus `query_range` data showing exactly two episodes (2026-08-02 and 2026-08-23), pod logs matching the count, and a Velero job timeline showing restore-test ran during both stalls. The claim about `bluestore_slow_ops_warn_lifetime = 24h` (Ceph Squid default) is noted as unverified live but plausible given the measured ~26.5h duration and ~24h offset. **Minor potential drift**: The description now hardcodes a time window ("Sunday 08:00–13:00Z") and endurance state ("80–87%"). These are correct now but could drift if Velero schedules change or hardware ages. The description does provide a way to verify it ("check whether the stall fell inside..."), so the guidance remains actionable even if the specifics shift. **YAML syntax**: Indentation and quoting are correct; the file remains valid Kubernetes/Prometheus YAML. ## Recommendation Looks reasonable — no blocking issues. The change is a necessary correction to alert documentation, not a workaround or band-aid. Comments above are optional but generally positive observations about verification quality. <!-- pr-review:sha=8386f755d1db93dd7440d9588b9b011543b1341d -->
k.renner merged commit 3f8e9fe333 into master 2026-08-23 20:59:30 +00:00
Sign in to join this conversation.