feat(monitoring): make a stalled WAL archiver reportable (REN-129)

ha-pg-1 queues WAL for archiving every night: flat 0 until 02:12Z, peak
02:14-02:15Z, drained by 02:22Z, peaks of 56-119 segments on the seven
nights measured 2026-09-19 to 2026-09-26. Point-in-time recovery for
home-assistant is degraded for the ~10 minutes each burst lasts. Nothing
fires, because CNPGWalArchiveBacklog's `for` was 30m.

The cause is a write burst, not IO contention. Drain was a flat 13.0-14.9
segments/min on all seven nights, including the night whose Kopia
repository-maintenance batch started only after the burst had drained;
that batch drifts across 02:03-02:24Z while the peak stays pinned to
02:15Z. Fill is what varies, 18.7-39.7 segments/min. 02:12Z is 04:12
local, when Home Assistant's recorder purge runs. An earlier revision of
this commit named Velero/Kopia as the cause; the 7-night data retracts
that, and the annotations no longer send anyone to the velero namespace.

Three changes, all alert-rule text. Nothing restarts, no storage and no
traffic path is touched.

- CNPGWalArchiveBacklog `for`: 30m -> 15m. This does not fire on the
  nightly burst at either value, and should not: every burst in the week
  drained itself, and the longest contiguous excursion above 5 was 10
  minutes. 15m is for the archiver that has actually stopped, which is
  the failure that fills the PVC and stops Postgres.

- New CNPGWalArchiveBacklogSurge, >64 segments (1 GiB) for 8m, warning.
  Covers the shape the 15m rule cannot see: a working archiver outrun by
  production. 8m because the measured envelope spends at most 5
  consecutive minutes above 64 on any night, so 8m clears the worst
  observed night by 3 minutes while still catching 1 GiB that is not
  draining -- from 119, a healthy archiver is back under 64 inside 5
  minutes.

- Both exprs wrapped in `max by (namespace, pod)`. `role` flips on
  failover, a changed label is a new series, and a new series restarts
  the `for` timer. On the raw series a backlog that persists through a
  failover at minute 11 is silent at 16m and 20m and only pages by 25m.

Verified with promtool 3.6.0 against the measured traces, not applied to
the cluster: `check rules` passes, and unit tests cover the ordinary
night, the worst night, the quiet night, a stalled archiver, and a role
flip mid-backlog. Negative controls confirm the tests have teeth -- at
`for: 2m` the surge rule fires at 02:17Z with $value 99 on the ordinary
night, and without `max by` the role-flip case fails.

Revert: `git revert <this commit>` restores both rules exactly. No state
outside Prometheus's rule evaluation is affected either way.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -13,7 +13,7 @@ spec:
       rules:
         # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
         # probes that S3 endpoint, which proves the bucket is reachable and
-        # nothing more — these two rules check that something is actually being
+        # nothing more — these rules check that something is actually being
         # written to it.
 
         - alert: CNPGWalArchiveBacklog
@@ -23,22 +23,106 @@ spec:
           # that moment, and pg_wal keeps growing until it fills the PVC and
           # takes Postgres down.
           #
-          # This is the load-bearing alert of the two. It reads the
+          # This is the load-bearing alert of the group. It reads the
           # instance-level collector directly, so unlike the backup-age metric
           # discussed below it does not depend on Cluster status being
           # populated.
           #
-          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # Threshold: >5 segments sustained for 15m. Brief non-zero values are
           # normal under write bursts — a segment is 16 MiB, so 5 is roughly
           # 80 MiB of un-archived WAL, well below anything that threatens the
           # volume but far above idle noise.
-          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
-          for: 30m
+          #
+          # `for` was 30m and is now 15m (REN-129). To be clear about what that
+          # does and does not buy: this rule does NOT fire on the nightly write
+          # burst documented under the Surge rule below, at 30m or at 15m, and
+          # that is correct — across seven consecutive measured nights the
+          # longest contiguous excursion above 5 was 10 minutes and every one
+          # of them drained itself. What 15m is for is the case that actually
+          # costs something: an archiver that has stopped. Outside those
+          # nightly windows the metric is a hard 0 on all 12 CNPG instances in
+          # this cluster, so there was never a noise floor that needed 30m to
+          # wait out — the extra 15m only delayed that page.
+          #
+          # `max by (namespace, pod)` is not cosmetic. The raw series carries
+          # `role`, which flips on failover; a changed label is a new series,
+          # and a new series restarts the `for` timer. Aggregating `role` away
+          # means a backlog that persists across a failover still pages 15m
+          # after the backlog started rather than 15m after the failover.
+          expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 5
+          for: 15m
           labels:
             severity: warning
           annotations:
             summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+            description: "{{ $value }} WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGWalArchiveBacklogSurge
+          # Companion to CNPGWalArchiveBacklog above, for the opposite failure
+          # shape: not a stuck archiver, but a working archiver that cannot
+          # drain as fast as Postgres is producing.
+          #
+          # CNPGWalArchiveBacklog cannot see that shape at any `for` value that
+          # is also safe against a flapping archiver, because the event is over
+          # before the window closes.
+          #
+          # What is normal here, measured on ha-pg-1 at 1-minute steps over the
+          # seven nights 2026-09-19 to 2026-09-26 (REN-129). Every night is the
+          # same burst, and nothing in that week exceeds 5 at any other hour:
+          #
+          #   flat 0 until 02:12Z; fill from 02:13Z; peak 02:14-02:15Z;
+          #   drained by 02:22Z. Nightly peaks 56, 91, 100, 101, 106, 113, 119.
+          #
+          # This is a write burst, not IO contention. Drain was 13.0-14.9
+          # segments/min on all seven nights — including the night whose Kopia
+          # repository-maintenance batch did not start until after the burst had
+          # already drained, and the night when two such jobs were saturating
+          # the RBD devices. That batch drifts across 02:03-02:24Z while the
+          # peak stays pinned to 02:15Z, and a drifting cause cannot produce an
+          # effect fixed to the clock. What varies is the fill (18.7-39.7
+          # segments/min), so the height of the backlog is set by how much WAL
+          # is produced against an archiver with a flat ~14 segments/min
+          # ceiling. 02:12Z is 04:12 local, which is when Home Assistant's
+          # recorder purge runs.
+          #
+          # So the nightly burst is a benign, self-clearing fact of this
+          # cluster and this rule has to stay silent through it. Threshold:
+          # >64 segments (1 GiB of un-archived WAL) for 8m.
+          #
+          #   - 64, not something nearer the >5 above, because this rule is
+          #     deliberately not a faster copy of that one. 1 GiB is where the
+          #     backlog becomes a material fraction of a CNPG PVC's free space
+          #     (ha-pg's volume is 20Gi, holding steady at 6.4-7.7 GiB free),
+          #     and a rule that trips on every write spike is how the
+          #     load-bearing rule above gets ignored.
+          #   - 8m because the measured envelope spends at most 5 consecutive
+          #     minutes above 64 on any of the seven nights, so 8m clears the
+          #     worst observed night by 3 minutes. An earlier revision of this
+          #     rule used 2m; that fires at ~02:16Z on 6 of the 7 nights, which
+          #     is two owner emails a night at 04:16 and 04:22 local for a
+          #     condition that repairs itself.
+          #
+          # What 8m still catches: 1 GiB queued that is not moving. From 119,
+          # the highest peak in the week, a healthy archiver is back under 64
+          # inside 5 minutes. Sitting above it for 8 means the archiver has
+          # lost most of its throughput, and that is worth a 04:00 look.
+          #
+          # When it does fire, the fix is drain rate or headroom, never a
+          # higher threshold here. ObjectStore `wal.maxParallel`
+          # (kubernetes/apps/home-assistant/ha-pg-objectstore.yaml) is unset,
+          # which the barman-cloud plugin reads as one WAL segment at a time;
+          # that is where the ~14 segments/min ceiling comes from. Headroom is
+          # `spec.storage.size` in ha-pg-cluster.yaml. Both are REN-135.
+          #
+          # `max by (namespace, pod)` for the same reason as the rule above:
+          # `role` flips on failover and would restart the `for` timer.
+          expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 64
+          for: 8m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving cannot keep up on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129)."
 
         - alert: CNPGBackupFailed
           # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0