fix(monitoring): exclude EC pools from CephPoolLostRedundancy (REN-88)

CephPoolLostRedundancy infers a replication factor from
ceph_pool_bytes_used/ceph_pool_stored and pages critical below 1.5. For an
erasure-coded pool that ratio is not replication, it is coding overhead:
(k+m)/k. Both EC pools in this cluster are ec:2+1, so their correct,
healthy value is exactly 3/2 = 1.5 — the rule's own threshold.

cephfs-bulk (pool 15) reads 480042086400/320028049408 = 1.5000000384. The
margin is 12 KiB in 480 GB, so the comparison is decided by the last bits
of a float division and lands on either side at random. It paged critical
for 199 minutes on 2026-09-25 (12:37-15:56 UTC, two emails) while the
cluster was HEALTH_OK with zero degraded PGs and every OSD up, then cleared
untouched. migration.rgw.buckets.data (pool 12) is the same ec:2+1 shape.

The rule's premise is also wrong for EC: its description claimed a single
OSD loss destroys data, but 2+1 carries a parity shard and tolerates one
OSD loss, the same as replica-2. Summary and description now say
"replicated" and drop that claim.

Gated on the exporter's own `type` label rather than moving the threshold.
Nudging 1.5 to 1.49 would keep a knife-edge and would also narrow the real
replica-2 -> replica-1 signal (1.0) this rule exists to catch.

NO COVERAGE GAP: a genuine EC redundancy loss raises POOL_NO_REDUNDANCY,
which is a live series in ceph_health_detail and is already deliberately in
scope for CephHealthWarn. That reads Ceph's own verdict instead of
inferring one from bytes, which is the right instrument — an EC pool that
loses a shard does not move its stored/used ratio at all.

VERIFIED against live Prometheus (read-only apiserver proxy, REN-36), using
the expression as serialized from this file:
  * 7d replay, old expr: pool 15 matches 49 samples (defect reproduced)
  * 7d replay, new expr: 0 matches
  * fire-window replay (12:00-16:30 UTC): old matches pool 15 at 231/271
    steps, new matches nothing
  * positive control, new expr with threshold raised 1.5->3.5: selects
    replicated pools 1,3,5,6,7,13,14 and excludes both EC pools (12,15) —
    the filter removes exactly the EC pools and nothing else
  * evaluated cleanly across 17 distinct mgr instances over 30d, so the
    on(pool_id) join does not break on mgr failover

BLAST RADIUS: none to workloads. This edits one PrometheusRule; Prometheus
hot-reloads rule files and no pod restarts. Worst case is alerting scope on
one rule. Nothing writes to the cluster, no Ceph object is touched.

REVERT: revert this commit. One file, no state left behind.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -52,19 +52,61 @@ spec:
         - alert: CephPoolLostRedundancy
           # Names the pool, which POOL_NO_REDUNDANCY does not.
           #
-          # There is no pool-size metric, but bytes_used/stored IS the
-          # replication factor. The >1e6 floor drops pools too small to measure
-          # meaningfully, empty ones included.
+          # There is no pool-size metric, but for a REPLICATED pool
+          # bytes_used/stored IS the replication factor. The >1e6 floor drops
+          # pools too small to measure meaningfully, empty ones included.
           #
-          # No pool is exempt.
+          # ERASURE-CODED POOLS ARE EXCLUDED, AND MUST STAY EXCLUDED (REN-88).
+          #
+          # That ratio is not a replication factor for an EC pool, it is the
+          # coding overhead: k+m shards over k data shards. For the two ec:2+1
+          # pools here that is 3/2 = exactly 1.5 — sitting precisely on this
+          # rule's own threshold, on the wrong side of it half the time.
+          # Measured on `cephfs-bulk` (pool 15): used 480042086400 / stored
+          # 320028049408 = 1.5000000384, a margin of 12 KiB in 480 GB, so which
+          # way the comparison lands is decided by the last bits of a float
+          # division. It paged critical for 199 minutes on 2026-09-25 (12:37 to
+          # 15:56 UTC, two emails) with the cluster HEALTH_OK, zero degraded
+          # PGs and every OSD up, then cleared on its own. 50 of the 210
+          # samples in the preceding 30h were below 1.5; only that run held the
+          # 30m `for`. `migration.rgw.buckets.data` (pool 12) is the same ec:2+1
+          # shape and was one drift away from doing the same thing.
+          #
+          # The premise fails for EC too, not just the arithmetic: the old
+          # description claimed a single OSD loss destroys data, but 2+1 carries
+          # a parity shard and survives one OSD loss, same as replica-2.
+          #
+          # Do NOT "fix" this by nudging the threshold to 1.4 or 1.49. That
+          # keeps a knife-edge, just a quieter one, and it also narrows the real
+          # replica-2 -> replica-1 signal this rule exists to catch (1.0).
+          #
+          # EC pools keep full redundancy coverage without this rule: a real
+          # loss raises POOL_NO_REDUNDANCY, which is a live series in
+          # `ceph_health_detail` and is deliberately in scope for
+          # CephHealthWarn above. That check reads Ceph's own verdict instead of
+          # inferring one from a byte ratio, which is the right instrument here
+          # — an EC pool that loses a shard does not change its stored/used
+          # ratio at all, because Ceph does not delete the surviving shards.
+          # So there is no gap to reopen, and no second EC-specific threshold
+          # worth inventing.
+          #
+          # `on(pool_id)` is deliberately the whole matcher. ceph_pool_metadata
+          # carries labels ceph_pool_bytes_used does not (name, type,
+          # description, application, compression_mode), so default all-label
+          # matching would never join. Adding `cluster` would be defensive
+          # against a second Ceph cluster that does not exist, at the cost of
+          # the rule silently matching nothing if that label is ever dropped —
+          # an alert that fails open invisibly is the worse trade.
           expr: |
-            ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
+            ceph_pool_bytes_used / ceph_pool_stored < 1.5
+              and on(pool_id) ceph_pool_metadata{type="replicated"} == 1
+              and ceph_pool_stored > 1e6
           for: 30m
           labels:
             severity: critical
           annotations:
-            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
-            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
+            summary: "Ceph replicated pool {{ $labels.pool_id }} is running unreplicated"
+            description: "Replicated pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning it is down to a single copy and one OSD loss destroys data. No replicated pool is meant to be unreplicated — it also blocks OSD maintenance for as long as it lasts, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Erasure-coded pools are out of scope for this rule and are covered by POOL_NO_REDUNDANCY via CephHealthWarn (REN-88). Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
 
         - alert: CephHealthError
           # HEALTH_ERR means data availability or redundancy is at risk.