feat(monitoring): alert on CNPG WAL archiving and backup failure #523

2026-08-19T13:43:26Z

claude-bot commented

2026-08-19 13:43:26 +00:00

Refs #522. Found while verifying ha-pg phase 1 (#521).

The gap

Seven Postgres clusters back up to Hetzner and nothing alerts if that stops.

The only related rule is HetznerBackupEndpointDown, which probes the S3 endpoint — it proves the bucket is reachable, not that anything was written to it.

This is the same failure shape as the WordPress backup that failed nightly for three days against a bucket declared in tofu but never applied: the status surface looked fine because nothing checked the thing that mattered.

What this adds

CNPGWalArchiveBacklog — the load-bearing one.

cnpg_collector_pg_wal_archive_status{value="ready"} counts WAL segments waiting in pg_wal/archive_status. Steady state is 0 on every cluster (verified). A sustained backlog means the archiver is failing, so:

point-in-time recovery is silently broken from that moment, and
pg_wal grows until it fills the PVC and takes Postgres down.

Threshold >5 sustained for 30m — a segment is 16 MiB, so ~80 MiB of un-archived WAL: well below anything threatening the volume, far above idle noise. It reads only the instance-level collector, so it works despite #522.

CNPGBackupFailed — base backup failure within 24h, guarded on timestamp > 0 so the epoch-zero default is not read as "failed in 1970".

Deliberately NOT added: the backup-age alert

The rule you would most want here is (time() - cnpg_collector_last_available_backup_timestamp) > 48h. It cannot be used today.

That metric reads exactly 0 on every instance in the cluster — verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg. CNPG has no record of any successful backup anywhere, because the Backup CRs it derives that from are deleted within a minute of completing (#522).

Adding it now would fire permanently for all seven clusters and be silenced within a day — worse than not having it. It belongs in the fix for #522.

The backups themselves DO work. Verified independently by listing the bucket with barman-cloud-backup-list and restoring a WAL segment back out of S3. The gap is in the record, not the data.

Validation

Both expressions were checked against live Prometheus before commit, rather than written from metric names:

selector matches 8 series      (0 would mean a DEAD rule)
CNPGWalArchiveBacklog firing:  0   (correct — archiving healthy)
CNPGBackupFailed firing:       0   (correct — only recorded failure is ~91d old)

The label is value, not status — confirmed against the live series rather than assumed. That check exists because all four rules in prometheusrule-woodpecker-rotation.yaml were dead on arrival for exactly this class of mistake.

🤖 Generated with Claude Code

Refs #522. Found while verifying ha-pg phase 1 (#521). ## The gap **Seven Postgres clusters back up to Hetzner and nothing alerts if that stops.** The only related rule is `HetznerBackupEndpointDown`, which probes the S3 endpoint — it proves the bucket is *reachable*, not that anything was *written* to it. This is the same failure shape as the WordPress backup that failed nightly for three days against a bucket declared in tofu but never applied: the status surface looked fine because nothing checked the thing that mattered. ## What this adds **`CNPGWalArchiveBacklog`** — the load-bearing one. `cnpg_collector_pg_wal_archive_status{value="ready"}` counts WAL segments waiting in `pg_wal/archive_status`. Steady state is **0 on every cluster** (verified). A sustained backlog means the archiver is failing, so: - point-in-time recovery is **silently broken** from that moment, and - `pg_wal` grows until it fills the PVC and takes Postgres down. Threshold `>5` sustained for 30m — a segment is 16 MiB, so ~80 MiB of un-archived WAL: well below anything threatening the volume, far above idle noise. It reads only the instance-level collector, so **it works despite #522**. **`CNPGBackupFailed`** — base backup failure within 24h, guarded on `timestamp > 0` so the epoch-zero default is not read as "failed in 1970". ## Deliberately NOT added: the backup-age alert The rule you would most want here is `(time() - cnpg_collector_last_available_backup_timestamp) > 48h`. It **cannot be used today**. That metric reads **exactly 0 on every instance in the cluster** — verified across `gf-pg`, `ak-pg-1/2`, `pprl-pg-2/3`, `im-pg`, `wp-pg`, `ha-pg`. CNPG has no record of any successful backup anywhere, because the Backup CRs it derives that from are deleted within a minute of completing (#522). Adding it now would fire permanently for all seven clusters and be silenced within a day — worse than not having it. It belongs in the fix for #522. **The backups themselves DO work.** Verified independently by listing the bucket with `barman-cloud-backup-list` and restoring a WAL segment back out of S3. The gap is in the *record*, not the data. ## Validation Both expressions were checked against live Prometheus **before commit**, rather than written from metric names: ``` selector matches 8 series (0 would mean a DEAD rule) CNPGWalArchiveBacklog firing: 0 (correct — archiving healthy) CNPGBackupFailed firing: 0 (correct — only recorded failure is ~91d old) ``` The label is `value`, not `status` — confirmed against the live series rather than assumed. That check exists because all four rules in `prometheusrule-woodpecker-rotation.yaml` were dead on arrival for exactly this class of mistake. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-19 13:43:26 +00:00

feat(monitoring): alert on CNPG WAL archiving and backup failure

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

1260165af1

Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.

CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.

CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".

DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.

The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.

Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.

Refs #522

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-19 13:47:07 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 1260165af1b08ce482b24d814968f82e6eb0aba2.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a new PrometheusRule (prometheusrule-cnpg-backup.yaml) with two alerts — CNPGWalArchiveBacklog and CNPGBackupFailed — wired into kubernetes/apps/monitoring-rules/kustomization.yaml, plus a well-reasoned comment block explaining why the more obvious backup-age alert is deliberately omitted (blocked on #522). The rule structure, labeling (release: kube-prometheus-stack), and PromQL both look correct and match the conventions of sibling files (prometheusrule-hetzner-backup.yaml, prometheusrule-woodpecker-rotation.yaml). No blocking issues found.

🟡 Suggestions

prometheusrule-cnpg-backup.yaml:59 — the CNPGBackupFailed description tells the responder to "see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md". I read that file in full and grepped the repo for barman-cloud-backup-list / barman: the string only appears in this new alert file itself. The doc has a "Confirm the backup actually landed in the bucket rather than trusting the CR status" line (line 65-67) but no barman-cloud-backup-list command or recipe. Whoever gets paged by this alert and follows the pointer will find nothing actionable there — worth either adding the actual command to that doc (or a new one) or pointing at wherever it really lives.
prometheusrule-cnpg-backup.yaml:23-44 vs prometheusrule-hetzner-backup.yaml:13-31 — CNPGWalArchiveBacklog's own comment says it's "the load-bearing" alert here and describes the same failure mode HetznerBackupEndpointDown warns about at severity: critical (WAL queuing until it fills the PVC and takes Postgres down), yet this rule is severity: warning. Not a functional problem today — helmrelease-prometheus.yaml's Alertmanager route has no severity-based routing, everything lands on the same email receiver (kubernetes/apps/monitoring/helmrelease-prometheus.yaml:349-384) — but it's an inconsistency in how urgency is communicated, and would start to matter the moment severity-based routing is added.

🟢 Nits

kustomization.yaml:16-19 — the prometheusrule-*.yaml entries are alphabetical (ceph, cilium, hetzner-backup, loki, netbird-*, velero, woodpecker-rotation); prometheusrule-cnpg-backup.yaml is appended after woodpecker-rotation instead of between cilium and hetzner-backup where alphabetical order would put it. Cosmetic only — kustomize doesn't care about resource order.

Tests

No unit-test coverage exists for Prometheus rules in this repo (no promtool test rules, and no sibling prometheusrule-*.yaml has one either), so this isn't a gap specific to this PR. CI's kustomize-build step (.woodpecker/ci-manifests-test.yml) will catch YAML/kustomize wiring errors but can't validate PromQL semantics or that the referenced metric/label names (cnpg_collector_pg_wal_archive_status{value="ready"}, cnpg_collector_last_failed_backup_timestamp) actually exist with those names on the live collector. The PR body states both expressions were checked against live Prometheus before commit (8 matching series, 0 currently firing) — plausible and internally consistent (7 clusters, 2 of which run 2 instances = 8 series), but I have no way to independently verify metric/label names from the static checkout.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `1260165af1b08ce482b24d814968f82e6eb0aba2`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a new `PrometheusRule` (`prometheusrule-cnpg-backup.yaml`) with two alerts — `CNPGWalArchiveBacklog` and `CNPGBackupFailed` — wired into `kubernetes/apps/monitoring-rules/kustomization.yaml`, plus a well-reasoned comment block explaining why the more obvious backup-age alert is deliberately omitted (blocked on #522). The rule structure, labeling (`release: kube-prometheus-stack`), and PromQL both look correct and match the conventions of sibling files (`prometheusrule-hetzner-backup.yaml`, `prometheusrule-woodpecker-rotation.yaml`). No blocking issues found. ## 🟡 Suggestions - `prometheusrule-cnpg-backup.yaml:59` — the `CNPGBackupFailed` description tells the responder to "see the barman-cloud-backup-list recipe in `kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md`". I read that file in full and grepped the repo for `barman-cloud-backup-list` / `barman`: the string only appears in this new alert file itself. The doc has a "Confirm the backup actually landed in the bucket rather than trusting the CR status" line (line 65-67) but no `barman-cloud-backup-list` command or recipe. Whoever gets paged by this alert and follows the pointer will find nothing actionable there — worth either adding the actual command to that doc (or a new one) or pointing at wherever it really lives. - `prometheusrule-cnpg-backup.yaml:23-44` vs `prometheusrule-hetzner-backup.yaml:13-31` — `CNPGWalArchiveBacklog`'s own comment says it's "the load-bearing" alert here and describes the same failure mode `HetznerBackupEndpointDown` warns about at `severity: critical` (WAL queuing until it fills the PVC and takes Postgres down), yet this rule is `severity: warning`. Not a functional problem today — `helmrelease-prometheus.yaml`'s Alertmanager `route` has no severity-based routing, everything lands on the same `email` receiver (kubernetes/apps/monitoring/helmrelease-prometheus.yaml:349-384) — but it's an inconsistency in how urgency is communicated, and would start to matter the moment severity-based routing is added. ## 🟢 Nits - `kustomization.yaml:16-19` — the `prometheusrule-*.yaml` entries are alphabetical (`ceph`, `cilium`, `hetzner-backup`, `loki`, `netbird-*`, `velero`, `woodpecker-rotation`); `prometheusrule-cnpg-backup.yaml` is appended after `woodpecker-rotation` instead of between `cilium` and `hetzner-backup` where alphabetical order would put it. Cosmetic only — kustomize doesn't care about resource order. ## Tests No unit-test coverage exists for Prometheus rules in this repo (no `promtool test rules`, and no sibling `prometheusrule-*.yaml` has one either), so this isn't a gap specific to this PR. CI's `kustomize-build` step (`.woodpecker/ci-manifests-test.yml`) will catch YAML/kustomize wiring errors but can't validate PromQL semantics or that the referenced metric/label names (`cnpg_collector_pg_wal_archive_status{value="ready"}`, `cnpg_collector_last_failed_backup_timestamp`) actually exist with those names on the live collector. The PR body states both expressions were checked against live Prometheus before commit (8 matching series, 0 currently firing) — plausible and internally consistent (7 clusters, 2 of which run 2 instances = 8 series), but I have no way to independently verify metric/label names from the static checkout. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit a2239515e9 into master

2026-08-20 04:39:21 +00:00

k.renner referenced this issue from a commit

2026-08-20 04:39:23 +00:00

Merge pull request 'feat(monitoring): alert on CNPG WAL archiving and backup failure' (#523) from fix-cnpg-backup-alerting into master

claude-bot referenced this pull request

2026-08-20 06:09:33 +00:00

CNPG Backup CRs disappear cluster-wide — backups verifiable only by reaching into S3 #522

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#523