claude-botandClaude Opus 5.5
221b350ea2
- AUTH_INSECURE_* is no longer excluded. #527 is done: every CephX key is aes256k since 2026-09-23 (#931, #934, #935) and all six AUTH_INSECURE_* checks report 0, so a new one is a real regression. - The description now only says what to do: BLUESTORE_SLOW_OP_ALERT outlives the stall by 24h; find the OSD in `ceph health detail` and check that drive's SMART data. No history in the annotation. - Comment: drop the AUTH_INSECURE paragraph and the restore-test note; the "80-87% endurance used" claim was wrong for the OSDs (nvme2 at 2-3% per smartctl_device_percentage_used, the 83-91% drives are the nvme1 EPHEMERAL disks), so it goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
306 lines
20 KiB
YAML
306 lines
20 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: ceph
|
|
namespace: monitoring
|
|
labels:
|
|
# kube-prometheus-stack's Prometheus resource selects on this label;
|
|
# without it the rule is created but never loaded into Prometheus.
|
|
release: kube-prometheus-stack
|
|
spec:
|
|
groups:
|
|
- name: ceph.rules
|
|
rules:
|
|
- alert: CephHealthWarn
|
|
# Per-check rather than ceph_health_status, so one permanently-active
|
|
# check cannot mask every other Ceph warning.
|
|
#
|
|
# POOL_NO_REDUNDANCY is in scope. An unreplicated pool makes
|
|
# `ceph osd ok-to-stop` refuse for every OSD, so it reports blocked
|
|
# maintenance and not just data risk — do not exclude it for a pool
|
|
# that is unreplicated deliberately.
|
|
#
|
|
# 6h: Ceph warns for self-clearing conditions. Only a warning that
|
|
# does not clear is worth paging on.
|
|
#
|
|
# BLUESTORE_SLOW_OP_ALERT breaks that premise and is still NOT
|
|
# excluded (#557). Ceph holds the check for
|
|
# bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
|
|
# one multi-second stall outlives the 6h `for` and always pages,
|
|
# with latency back to normal long before anyone looks. Measured:
|
|
# both episodes in the 30d to 2026-08-23 ran ~26h and cleared
|
|
# untouched, the first ~24h after the last stall.
|
|
#
|
|
# Not excludable: it is also the earliest signal of a failing drive.
|
|
expr: ceph_health_detail == 1
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Ceph health check {{ $labels.name }} active for 6h"
|
|
description: "Ceph health check {{ $labels.name }} has been active for six hours. BLUESTORE_SLOW_OP_ALERT stays for 24h after the last stall even when latency is back to normal; find the OSD in the detail output and check that drive's SMART data (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
|
|
|
|
- alert: CephPoolLostRedundancy
|
|
# Names the pool, which POOL_NO_REDUNDANCY does not.
|
|
#
|
|
# There is no pool-size metric, but for a REPLICATED pool
|
|
# bytes_used/stored IS the replication factor. The >1e6 floor drops
|
|
# pools too small to measure meaningfully, empty ones included.
|
|
#
|
|
# ERASURE-CODED POOLS ARE EXCLUDED, AND MUST STAY EXCLUDED (REN-88).
|
|
#
|
|
# That ratio is not a replication factor for an EC pool, it is the
|
|
# coding overhead: k+m shards over k data shards. For the two ec:2+1
|
|
# pools here that is 3/2 = exactly 1.5 — sitting precisely on this
|
|
# rule's own threshold, on the wrong side of it half the time.
|
|
# Measured on `cephfs-bulk` (pool 15): used 480042086400 / stored
|
|
# 320028049408 = 1.5000000384, a margin of 12 KiB in 480 GB, so which
|
|
# way the comparison lands is decided by the last bits of a float
|
|
# division. It paged critical for 199 minutes on 2026-09-25 (12:37 to
|
|
# 15:56 UTC, two emails) with the cluster HEALTH_OK, zero degraded
|
|
# PGs and every OSD up, then cleared on its own. 50 of the 210
|
|
# samples in the preceding 30h were below 1.5; only that run held the
|
|
# 30m `for`. `migration.rgw.buckets.data` (pool 12) is the same ec:2+1
|
|
# shape and was one drift away from doing the same thing.
|
|
#
|
|
# The premise fails for EC too, not just the arithmetic: the old
|
|
# description claimed a single OSD loss destroys data, but 2+1 carries
|
|
# a parity shard and survives one OSD loss, same as replica-2.
|
|
#
|
|
# Do NOT "fix" this by nudging the threshold to 1.4 or 1.49. That
|
|
# keeps a knife-edge, just a quieter one, and it also narrows the real
|
|
# replica-2 -> replica-1 signal this rule exists to catch (1.0).
|
|
#
|
|
# EC pools keep full redundancy coverage without this rule: a real
|
|
# loss raises POOL_NO_REDUNDANCY, which is a live series in
|
|
# `ceph_health_detail` and is deliberately in scope for
|
|
# CephHealthWarn above. That check reads Ceph's own verdict instead of
|
|
# inferring one from a byte ratio, which is the right instrument here
|
|
# — an EC pool that loses a shard does not change its stored/used
|
|
# ratio at all, because Ceph does not delete the surviving shards.
|
|
# So there is no gap to reopen, and no second EC-specific threshold
|
|
# worth inventing.
|
|
#
|
|
# `on(pool_id)` is deliberately the whole matcher. ceph_pool_metadata
|
|
# carries labels ceph_pool_bytes_used does not (name, type,
|
|
# description, application, compression_mode), so default all-label
|
|
# matching would never join. Adding `cluster` would be defensive
|
|
# against a second Ceph cluster that does not exist, at the cost of
|
|
# the rule silently matching nothing if that label is ever dropped —
|
|
# an alert that fails open invisibly is the worse trade.
|
|
expr: |
|
|
ceph_pool_bytes_used / ceph_pool_stored < 1.5
|
|
and on(pool_id) ceph_pool_metadata{type="replicated"} == 1
|
|
and ceph_pool_stored > 1e6
|
|
for: 30m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Ceph replicated pool {{ $labels.pool_id }} is running unreplicated"
|
|
description: "Replicated pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning it is down to a single copy and one OSD loss destroys data. No replicated pool is meant to be unreplicated — it also blocks OSD maintenance for as long as it lasts, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Erasure-coded pools are out of scope for this rule and are covered by POOL_NO_REDUNDANCY via CephHealthWarn (REN-88). Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
|
|
|
|
- alert: CephPoolLostRedundancyBlind
|
|
# Canary for the rule directly above. Keep the two adjacent.
|
|
#
|
|
# CephPoolLostRedundancy is `critical` and it fails CLOSED and
|
|
# SILENTLY. Its expression joins three metrics on one label and
|
|
# filters on one specific label VALUE, and if any of those stops
|
|
# matching the expression returns nothing, which is indistinguishable
|
|
# from "every pool is fine". Health stays ok, no email is sent, and
|
|
# nothing anywhere reports that a critical storage alert has gone
|
|
# dark. Measured at REN-88 review time: `type` carried exactly two
|
|
# values, `replicated` and `erasure`, across 15 series — so the
|
|
# dependency is real, and a Rook or Ceph upgrade that renames the
|
|
# label, renames the value, or drops it in a relabel rule is enough.
|
|
#
|
|
# This expression is CephPoolLostRedundancy's with both thresholds
|
|
# removed, wrapped in absent(): "can that rule produce any series at
|
|
# all". It goes to 1 if either metric disappears, if `pool_id` is
|
|
# dropped from either side of the join, if the `type` label is
|
|
# renamed, or if `replicated` stops being its value.
|
|
#
|
|
# The one deliberate difference is that `ceph_pool_stored > 1e6` is
|
|
# NOT mirrored here. That floor is about data volume, not structure —
|
|
# replicated pools that are all legitimately tiny is a quiet state,
|
|
# not a broken dependency, and copying it in would make this fire for
|
|
# a benign reason.
|
|
#
|
|
# ONE CANARY PER FRAGILE JOIN, NOT ONE PER RULE GROUP. Judged on
|
|
# drift: a group-level canary has to enumerate every metric the group
|
|
# reads, and adding a rule to the group does not update it. Six months
|
|
# on it silently under-covers while LOOKING comprehensive, which is the
|
|
# same failure shape as a discovery step that passes having checked
|
|
# nothing. This one's scope is exactly one rule sitting six lines
|
|
# above it, so "is it still true" is answerable by reading the two
|
|
# blocks together. It can still drift — someone edits the expr above
|
|
# and not this — but that drift shows up in the same diff hunk, and
|
|
# when it does bite it FIRES rather than going quiet. For a guard,
|
|
# loud and wrong beats silent and wrong.
|
|
#
|
|
# 1h, not the 30m above. The exporter moved through 17 distinct mgr
|
|
# instances in 30d, and every one of those failovers makes all ceph_*
|
|
# metrics absent for as long as it takes the new mgr to be scraped;
|
|
# anything under ~15m would page on routine failover, and noise is
|
|
# what REN-88 was about. A long `for` costs nothing in detection value
|
|
# because the failure this catches is a schema change — permanent, not
|
|
# transient — and a blindness that self-heals inside an hour is not
|
|
# worth an email. Transient exporter loss is already covered by the
|
|
# chart's own TargetDown.
|
|
#
|
|
# `warning`, not `critical`: an alert stopped being able to fire.
|
|
# Storage is not broken.
|
|
expr: |
|
|
absent(
|
|
ceph_pool_bytes_used / ceph_pool_stored
|
|
and on(pool_id) ceph_pool_metadata{type="replicated"} == 1
|
|
)
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CephPoolLostRedundancy cannot fire — its metrics no longer join"
|
|
description: "The CephPoolLostRedundancy alert has been unable to produce any series for an hour, so a `critical` storage alert is silently dark. It joins ceph_pool_bytes_used / ceph_pool_stored against ceph_pool_metadata{type=\"replicated\"} on `pool_id`; one of those metrics, the `pool_id` label, the `type` label, or the `replicated` value has stopped matching — most likely a Rook or Ceph upgrade, or a relabel rule. Check what the exporter emits now with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail` and by querying `ceph_pool_metadata` directly, then fix the join in kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml — do not delete this canary (REN-93)."
|
|
|
|
- alert: CephHealthError
|
|
# HEALTH_ERR means data availability or redundancy is at risk.
|
|
expr: ceph_health_status == 2
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Ceph is HEALTH_ERR"
|
|
description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."
|
|
|
|
- alert: CephMonDiskLow
|
|
# Named explicitly rather than left to CephHealthWarn, because the
|
|
# mons sit on the node's /var alongside container storage: this
|
|
# fires for a full node, not for Ceph growth. The mon's own data is
|
|
# ~106 MB, so 30% of a 60 GB shared filesystem says nothing about
|
|
# Ceph.
|
|
# Image GC does not clear this — measured on k8s-2, the reclaimable
|
|
# content store is a small fraction of /var/lib/containerd next to
|
|
# ~20 GB of overlayfs snapshots held by running containers. Look at
|
|
# what actually grew before reaching for a threshold.
|
|
expr: |
|
|
100 * node_filesystem_avail_bytes{mountpoint="/var"}
|
|
/ node_filesystem_size_bytes{mountpoint="/var"} < 30
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
|
|
description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."
|
|
|
|
- alert: CephOSDNearfullApproaching
|
|
# Warns BEFORE Ceph's own nearfull check, which is the incident and
|
|
# not a warning. mon_osd_nearfull_ratio is 0.85 and is not
|
|
# overridden here.
|
|
#
|
|
# There is no rebalance to hide behind: failureDomain is host with
|
|
# size 3 on exactly three OSDs, so every OSD holds a copy of every
|
|
# replicated object and all three cross 0.85 within seconds of each
|
|
# other. Measured 2026-09-26: 81.86 / 81.87 / 81.83%.
|
|
#
|
|
# 0.75 is 0.85 minus the largest bulk-tier burst on record. Measured
|
|
# at 30s resolution over the 14d to 2026-09-26: the fullest OSD went
|
|
# 71.63% -> 81.87% between 05:18 and 05:43Z (+10.24 pp, ~275 GiB raw)
|
|
# and reclaimed within the hour. At a SUSTAINED 75%, one more burst of
|
|
# that size reaches nearfull, which is the point at which the level
|
|
# itself is the finding.
|
|
#
|
|
# `for: 30m` is the shortest wait that outlives a self-reclaiming
|
|
# burst. That same burst held >= 75% for 19.0 min, so 30m clears it by
|
|
# 11 min; a permanent step holds it indefinitely. Do NOT shorten this
|
|
# to 15m — the 09-26 burst then pages, with the cluster reclaiming on
|
|
# its own while someone reads the mail.
|
|
#
|
|
# Nor should it be lengthened to 1h. The fastest observed SUSTAINED
|
|
# climb was +11.44 pp in one hour (2026-09-25 08:02-09:02Z, a real
|
|
# ingest that never reclaimed). From 75% that reaches 0.85 in ~53 min,
|
|
# so a 1h pending window would fire after nearfull, not before it.
|
|
# 30m fires at ~80.7% in that worst case — about 22 min of lead — and
|
|
# at 2.24 pp/h (the worst 6h-sustained rate) it fires at ~76.1%, about
|
|
# 4h of lead. The 30m/1h choice is the whole trade: a burst reclaims
|
|
# in ~20 min and a step does not, so ~30m is the earliest honest
|
|
# verdict. REN-162 is measuring the burst envelope; if a burst is ever
|
|
# seen holding >= 75% for more than ~25 min, this `for` moves out, not
|
|
# the threshold.
|
|
#
|
|
# Replay over the 14d to 2026-09-26 08:00Z: the worst 30m-sustained
|
|
# floor on any OSD was 71.632% (09-26 04:31Z), so this rule fires ZERO
|
|
# times on the observed normal shape, including the 09-26 excursion to
|
|
# 81.87% and the 09-25 step from 59% to 71%. Margin 3.37 pp.
|
|
#
|
|
# `max by (ceph_daemon)` is not cosmetic. rook-ceph-exporter pods are
|
|
# replaced often — 32 distinct series for three OSDs in 14 days — and
|
|
# the raw expression carries `pod` and `instance`, so every restart
|
|
# would change the alert identity and reset a pending window.
|
|
expr: |
|
|
max by (ceph_daemon) (
|
|
ceph_osd_stat_bytes_used / ceph_osd_stat_bytes
|
|
) >= 0.75
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Ceph {{ $labels.ceph_daemon }} sustained above 75% — one bulk burst from nearfull"
|
|
description: "{{ $labels.ceph_daemon }} has been at or above 75% utilisation for 30 minutes ({{ $value | humanizePercentage }} now). That is 0.85 nearfull minus the largest bulk-tier burst measured here (+10.24 pp in 25 minutes), so a single repeat of that burst trips mon_osd_nearfull_ratio — and with failureDomain host, size 3 and three OSDs, it trips on all three at once with no rebalance available. This is sustained, not a spike: a burst clears in about 20 minutes and never holds this for 30. Find what grew with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df` and `ceph osd df tree`; the bulk tier (cephfs-bulk, pool 15) is the usual mover."
|
|
|
|
- alert: CephReplicapoolHeadroomLow
|
|
# The `replicapool` counterpart to the OSD rule above, in the units a
|
|
# reader can act on: GiB that still fit.
|
|
#
|
|
# These two are NOT independent signals and must not be read as
|
|
# corroboration. `ceph_pool_max_avail` for a size-3 pool on three
|
|
# equal OSDs is exactly (full_ratio - fullest_OSD_utilisation) x
|
|
# osd_size; verified against live data, the identity holds at 0.95 to
|
|
# five decimals. So this rule is the OSD series rescaled — a second
|
|
# TIER, deliberately set to trip after the OSD rule, not a second
|
|
# opinion. Ceph's own nearfull line of 0.85 sits at 89.4 GiB here.
|
|
#
|
|
# 160 GiB is what must still fit, measured:
|
|
# 89 GiB max_avail at the 0.85 nearfull line, i.e. the part of
|
|
# max_avail that is not ours to spend
|
|
# 38 GiB ha-pg's unconsumed allocation after the approved 20Gi ->
|
|
# 32Gi expansion: 2 instances x (31.33 GiB provisioned -
|
|
# 12.4 GiB used) as of 2026-09-26
|
|
# 35 GiB replicapool churn allowance — the worst observed single
|
|
# day (+14.3 GiB, 09-23) plus eleven days at the 14d mean
|
|
# of 1.9 GiB/day
|
|
# = 162 GiB, rounded DOWN to 160 so the rounding costs quiet, not
|
|
# headroom.
|
|
#
|
|
# Selecting on the pool NAME through ceph_pool_metadata, not on a
|
|
# hardcoded pool_id: ids are assignable and replicapool is only id 1
|
|
# by history. `on(pool_id)` is the whole matcher for the same reason
|
|
# given on CephPoolLostRedundancy above — ceph_pool_metadata carries
|
|
# labels ceph_pool_max_avail does not, so default matching never
|
|
# joins.
|
|
#
|
|
# `max by (pool_id)` twice over: it collapses the churning mgr `pod`
|
|
# and `instance` labels (8 series in 14 days), and during an mgr
|
|
# failover it takes the live value rather than a stale low one.
|
|
#
|
|
# `for: 30m` also covers a real exporter artefact: ceph_pool_max_avail
|
|
# reads exactly 0 for a single 30s sample after an mgr restart, twice
|
|
# in the 14d window (e.g. 09-23 12:27Z), on every mgr series at once.
|
|
# A `for` shorter than a few minutes would page on that alone. Note
|
|
# the glitch is one-directional, so it cannot cancel a genuine breach
|
|
# — 0 is still below the floor.
|
|
#
|
|
# Replay over the 14d to 2026-09-26 08:00Z: the lowest 30m-sustained
|
|
# ceiling was 208.97 GiB (09-26 04:31Z), so ZERO fires on the observed
|
|
# normal shape. The 09-26 burst took max_avail to 117.4 GiB for about
|
|
# 10 minutes and did not come close to holding it.
|
|
expr: |
|
|
max by (pool_id) (
|
|
ceph_pool_max_avail
|
|
and on(pool_id) ceph_pool_metadata{name="replicapool"}
|
|
) < 160 * 1024^3
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "replicapool usable headroom below 160 GiB"
|
|
description: "replicapool has had under 160 GiB usable for 30 minutes ({{ $value | humanize1024 }}B now). 160 GiB is the 89 GiB that max_avail still holds above Ceph's 0.85 nearfull line, plus the 38 GiB ha-pg has not yet consumed of its 32Gi expansion, plus 35 GiB of observed replicapool churn — below this, what is already committed no longer fits before nearfull. This is the same series as CephOSDNearfullApproaching rescaled (max_avail = (0.95 - fullest OSD) x OSD size), so it is a second tier and not independent confirmation. Check `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df`; growth here is usually the bulk tier on the shared OSDs, not replicapool itself."
|