konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

[alert] CephHealthWarn (rook-ceph) #562

New Issue
Closed
opened 2026-08-24 11:50:15 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-24 11:50:15 +00:00
Member
Copy Link
Copy Source

What fired: CephHealthWarn (severity warning, namespace rook-ceph), reported as a blip — active within the last ~35 min and already resolved by the time this ran (fingerprint 249e4dc23d4b4981).

Evidence

  • Live CephCluster status (kubectl -n rook-ceph get cephcluster rook-ceph) is currently HEALTH_WARN, but the only active ceph_health_detail checks right now are AUTH_INSECURE_CLIENT_KEY_TYPE / AUTH_INSECURE_KEYS_ALLOWED / AUTH_INSECURE_KEYS_CREATABLE (all excluded from this rule by design, see below) — every other check, including BLUESTORE_SLOW_OP_ALERT, reads 0.
  • ALERTS{alertname="CephHealthWarn"} (Prometheus query_range) shows the firing series was name="BLUESTORE_SLOW_OP_ALERT", continuously firing from before 2026-08-24T02:53Z through 2026-08-24T11:28Z, then gone — confirming the blip classification (it did resolve).
  • Loki, rook-ceph-mon-a cluster log:
    • 2026-08-23T11:32:14Z [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)
    • 2026-08-24T11:32:11Z [INF] Health check cleared: BLUESTORE_SLOW_OP_ALERT (was: 1 OSD(s) experiencing slow operations in BlueStore)
    • That's a ~24h00m active window, matching Ceph's bluestore_slow_ops_warn_lifetime (24h) sticky-clear behavior.

Root cause

Verified mechanism: this is the same recurring pattern already documented in kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml (lines 26-41) — BLUESTORE_SLOW_OP_ALERT is deliberately not excluded from CephHealthWarn (unlike AUTH_INSECURE_*) because it self-clears ~24-26h after the last stall regardless of whether the drive is actually failing. The stall here started 2026-08-23T11:32Z, which falls inside the documented Sunday 08:00-13:00Z Velero restore-test window — consistent with the two prior episodes the rule's comment already measured (both ran ~26h, both began in that window), and expected on these QLC NVMe OSDs, which are at 80-87% rated endurance already (kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:11).

What is NOT confirmed: which of the three OSDs (rook-ceph-osd-0/1/2) actually logged the slow op. The mon cluster-log lines above don't include an OSD id, the ceph_healthcheck_slow_ops Prometheus metric carries no per-OSD label, and the CI diagnostics ServiceAccount cannot exec into rook-ceph-tools (pods/exec is forbidden) to run ceph health detail, so I could not check per-OSD ceph_health_detail metadata or SMART/wear stats. This matters because the rule's own author kept this check un-excluded specifically so a human checks: "a repeat on the same OSD... means check that drive (#557)" — with all three OSDs already past 80% endurance, that distinction (expected wear-driven blip vs. an early failing-drive signal) can't be made from what's available here.

Where to fix / what to do

Not a repo config change — this needs a human with cluster-admin exec access to:

  1. Run kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail (or check historical Alertmanager notifications) to identify which osd.N stalled this time.
  2. Compare against the OSD(s) implicated in the two prior episodes referenced in kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:33-35 — if it's the same OSD again, treat it as the early-failure signal the comment warns about and start replacement planning per kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:24; if it's a different OSD (or varies), this is just the expected weekly restore-load pattern and can be closed with a note.
  3. No PrometheusRule or manifest change is needed unless this OSD-identification step turns out to be wanted going forward, in which case kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml's CephHealthWarn annotation could be extended to surface the OSD id (if ceph_health_detail's label set can be made to carry one — it currently doesn't).

Filed automatically by the alert-monitor pipeline (2026-08-24). The issue-fix pipeline may pick this up and open a PR.

**What fired**: `CephHealthWarn` (severity `warning`, namespace `rook-ceph`), reported as a blip — active within the last ~35 min and already resolved by the time this ran (fingerprint `249e4dc23d4b4981`). **Evidence** - Live `CephCluster` status (`kubectl -n rook-ceph get cephcluster rook-ceph`) is currently `HEALTH_WARN`, but the only active `ceph_health_detail` checks right now are `AUTH_INSECURE_CLIENT_KEY_TYPE` / `AUTH_INSECURE_KEYS_ALLOWED` / `AUTH_INSECURE_KEYS_CREATABLE` (all excluded from this rule by design, see below) — every other check, including `BLUESTORE_SLOW_OP_ALERT`, reads `0`. - `ALERTS{alertname="CephHealthWarn"}` (Prometheus `query_range`) shows the firing series was `name="BLUESTORE_SLOW_OP_ALERT"`, continuously firing from before 2026-08-24T02:53Z through 2026-08-24T11:28Z, then gone — confirming the blip classification (it did resolve). - Loki, `rook-ceph-mon-a` cluster log: - `2026-08-23T11:32:14Z [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)` - `2026-08-24T11:32:11Z [INF] Health check cleared: BLUESTORE_SLOW_OP_ALERT (was: 1 OSD(s) experiencing slow operations in BlueStore)` - That's a ~24h00m active window, matching Ceph's `bluestore_slow_ops_warn_lifetime` (24h) sticky-clear behavior. **Root cause** Verified mechanism: this is the same recurring pattern already documented in `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` (lines 26-41) — `BLUESTORE_SLOW_OP_ALERT` is deliberately *not* excluded from `CephHealthWarn` (unlike `AUTH_INSECURE_*`) because it self-clears ~24-26h after the last stall regardless of whether the drive is actually failing. The stall here started 2026-08-23T11:32Z, which falls inside the documented Sunday 08:00-13:00Z Velero restore-test window — consistent with the two prior episodes the rule's comment already measured (both ran ~26h, both began in that window), and expected on these QLC NVMe OSDs, which are at 80-87% rated endurance already (`kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:11`). **What is NOT confirmed**: which of the three OSDs (`rook-ceph-osd-0/1/2`) actually logged the slow op. The mon cluster-log lines above don't include an OSD id, the `ceph_healthcheck_slow_ops` Prometheus metric carries no per-OSD label, and the CI diagnostics ServiceAccount cannot `exec` into `rook-ceph-tools` (`pods/exec` is forbidden) to run `ceph health detail`, so I could not check per-OSD `ceph_health_detail` metadata or SMART/wear stats. This matters because the rule's own author kept this check un-excluded specifically so a human checks: "a repeat on the same OSD... means check that drive (#557)" — with all three OSDs already past 80% endurance, that distinction (expected wear-driven blip vs. an early failing-drive signal) can't be made from what's available here. **Where to fix / what to do** Not a repo config change — this needs a human with cluster-admin exec access to: 1. Run `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` (or check historical Alertmanager notifications) to identify which `osd.N` stalled this time. 2. Compare against the OSD(s) implicated in the two prior episodes referenced in `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:33-35` — if it's the same OSD again, treat it as the early-failure signal the comment warns about and start replacement planning per `kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:24`; if it's a different OSD (or varies), this is just the expected weekly restore-load pattern and can be closed with a note. 3. No PrometheusRule or manifest change is needed unless this OSD-identification step turns out to be wanted going forward, in which case `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml`'s `CephHealthWarn` annotation could be extended to surface the OSD id (if `ceph_health_detail`'s label set can be made to carry one — it currently doesn't). --- <!-- alert-monitor:fingerprint=249e4dc23d4b4981 --> _Filed automatically by the alert-monitor pipeline (2026-08-24). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-24 12:47:14 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#563 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/563 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#562