Compare commits

...
Author SHA1 Message Date
issue-fix-bot 4c3300488a fix: [alert] CephPoolLostRedundancy (rook-ceph) (#968)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
2026-09-25 17:44:07 +00:00
@@ -52,19 +52,23 @@ spec:
- alert: CephPoolLostRedundancy
# Names the pool, which POOL_NO_REDUNDANCY does not.
#
# There is no pool-size metric, but bytes_used/stored IS the
# replication factor. The >1e6 floor drops pools too small to measure
# meaningfully, empty ones included.
# There is no pool-size metric, but bytes_used/stored is the
# replication factor for replicated pools and (k+m)/k for EC pools.
# The EC 2+1 pools (cephfs-bulk, rgw buckets.data) sit at exactly
# 1.5 with ~1e-7 accounting drift either side, so the cut-off is
# midway between them and an unreplicated pool (1.0) (#968).
# The >1e6 floor drops pools too small to measure meaningfully,
# empty ones included.
#
# No pool is exempt.
expr: |
ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
ceph_pool_bytes_used / ceph_pool_stored < 1.25 and ceph_pool_stored > 1e6
for: 30m
labels:
severity: critical
annotations:
summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
description: "Pool {{ $labels.pool_id }} has a raw/stored ratio below 1.25 (replicated size 1), meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
- alert: CephHealthError
# HEALTH_ERR means data availability or redundancy is at risk.