diff --git a/kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml b/kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml index 6400a9ac..d5386595 100644 --- a/kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml +++ b/kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml @@ -13,7 +13,7 @@ spec: rules: # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown # probes that S3 endpoint, which proves the bucket is reachable and - # nothing more — these two rules check that something is actually being + # nothing more — these rules check that something is actually being # written to it. - alert: CNPGWalArchiveBacklog @@ -23,22 +23,106 @@ spec: # that moment, and pg_wal keeps growing until it fills the PVC and # takes Postgres down. # - # This is the load-bearing alert of the two. It reads the + # This is the load-bearing alert of the group. It reads the # instance-level collector directly, so unlike the backup-age metric # discussed below it does not depend on Cluster status being # populated. # - # Threshold: >5 segments sustained for 30m. Brief non-zero values are + # Threshold: >5 segments sustained for 15m. Brief non-zero values are # normal under write bursts — a segment is 16 MiB, so 5 is roughly # 80 MiB of un-archived WAL, well below anything that threatens the # volume but far above idle noise. - expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5 - for: 30m + # + # `for` was 30m and is now 15m (REN-129). To be clear about what that + # does and does not buy: this rule does NOT fire on the nightly write + # burst documented under the Surge rule below, at 30m or at 15m, and + # that is correct — across seven consecutive measured nights the + # longest contiguous excursion above 5 was 10 minutes and every one + # of them drained itself. What 15m is for is the case that actually + # costs something: an archiver that has stopped. Outside those + # nightly windows the metric is a hard 0 on all 12 CNPG instances in + # this cluster, so there was never a noise floor that needed 30m to + # wait out — the extra 15m only delayed that page. + # + # `max by (namespace, pod)` is not cosmetic. The raw series carries + # `role`, which flips on failover; a changed label is a new series, + # and a new series restarts the `for` timer. Aggregating `role` away + # means a backlog that persists across a failover still pages 15m + # after the backlog started rather than 15m after the failover. + expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 5 + for: 15m labels: severity: warning annotations: summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}" - description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`." + description: "{{ $value }} WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`." + + - alert: CNPGWalArchiveBacklogSurge + # Companion to CNPGWalArchiveBacklog above, for the opposite failure + # shape: not a stuck archiver, but a working archiver that cannot + # drain as fast as Postgres is producing. + # + # CNPGWalArchiveBacklog cannot see that shape at any `for` value that + # is also safe against a flapping archiver, because the event is over + # before the window closes. + # + # What is normal here, measured on ha-pg-1 at 1-minute steps over the + # seven nights 2026-09-19 to 2026-09-26 (REN-129). Every night is the + # same burst, and nothing in that week exceeds 5 at any other hour: + # + # flat 0 until 02:12Z; fill from 02:13Z; peak 02:14-02:15Z; + # drained by 02:22Z. Nightly peaks 56, 91, 100, 101, 106, 113, 119. + # + # This is a write burst, not IO contention. Drain was 13.0-14.9 + # segments/min on all seven nights — including the night whose Kopia + # repository-maintenance batch did not start until after the burst had + # already drained, and the night when two such jobs were saturating + # the RBD devices. That batch drifts across 02:03-02:24Z while the + # peak stays pinned to 02:15Z, and a drifting cause cannot produce an + # effect fixed to the clock. What varies is the fill (18.7-39.7 + # segments/min), so the height of the backlog is set by how much WAL + # is produced against an archiver with a flat ~14 segments/min + # ceiling. 02:12Z is 04:12 local, which is when Home Assistant's + # recorder purge runs. + # + # So the nightly burst is a benign, self-clearing fact of this + # cluster and this rule has to stay silent through it. Threshold: + # >64 segments (1 GiB of un-archived WAL) for 8m. + # + # - 64, not something nearer the >5 above, because this rule is + # deliberately not a faster copy of that one. 1 GiB is where the + # backlog becomes a material fraction of a CNPG PVC's free space + # (ha-pg's volume is 20Gi, holding steady at 6.4-7.7 GiB free), + # and a rule that trips on every write spike is how the + # load-bearing rule above gets ignored. + # - 8m because the measured envelope spends at most 5 consecutive + # minutes above 64 on any of the seven nights, so 8m clears the + # worst observed night by 3 minutes. An earlier revision of this + # rule used 2m; that fires at ~02:16Z on 6 of the 7 nights, which + # is two owner emails a night at 04:16 and 04:22 local for a + # condition that repairs itself. + # + # What 8m still catches: 1 GiB queued that is not moving. From 119, + # the highest peak in the week, a healthy archiver is back under 64 + # inside 5 minutes. Sitting above it for 8 means the archiver has + # lost most of its throughput, and that is worth a 04:00 look. + # + # When it does fire, the fix is drain rate or headroom, never a + # higher threshold here. ObjectStore `wal.maxParallel` + # (kubernetes/apps/home-assistant/ha-pg-objectstore.yaml) is unset, + # which the barman-cloud plugin reads as one WAL segment at a time; + # that is where the ~14 segments/min ceiling comes from. Headroom is + # `spec.storage.size` in ha-pg-cluster.yaml. Both are REN-135. + # + # `max by (namespace, pod)` for the same reason as the rule above: + # `role` flips on failover and would restart the `for` timer. + expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 64 + for: 8m + labels: + severity: warning + annotations: + summary: "CNPG WAL archiving cannot keep up on {{ $labels.namespace }}/{{ $labels.pod }}" + description: "{{ $value }} WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129)." - alert: CNPGBackupFailed # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0