konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

CNPG Backup CRs disappear cluster-wide — backups verifiable only by reaching into S3 #522

New Issue
Closed
opened 2026-08-19 13:40:37 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-08-19 13:40:37 +00:00
Member
Copy Link
Copy Source

Found while verifying ha-pg phase 1 (#521). The backup DATA is fine; the Kubernetes-side record is not.

Observation

There are zero CNPG Backup objects anywhere in the cluster:

$ kubectl get backup -A
wordpress   wordpress-mariadb-backup   2d17h

That single survivor belongs to a different operator (mariadb-operator) and is declared in git. None of the six CNPG clusters — ak-pg, im-pg, pprl-pg, gf-pg, wp-pg, tr-pg, ha-pg — has a single Backup CR, despite all having ScheduledBackup resources with immediate: true.

Observed directly on ha-pg, which was created fresh at 13:25 today:

13:26:15  Backup ha-pg-daily-20260819132545 created
13:26:28  "Backup completed"        (cluster log + k8s event)
13:27:0x  kubectl get backup -> No resources found

Under a minute from completion to gone.

The backups themselves ARE working

Verified against the bucket rather than trusting CR status:

$ barman-cloud-backup-list --endpoint-url https://fsn1.your-objectstorage.com \
    s3://renner-uno-k8s-cnpg/ha-pg ha-pg
Backup ID         End Time              Begin Wal                  Name
20260819T132615   2026-08-19 13:26:22   000000010000000000000003   backup-20260819132615

WAL archiving also confirmed live and readable: ContinuousArchiving=True, and barman-cloud-wal-restore fetched segment 000000010000000000000003 back out of S3 with rc=0.

So this is an observability/verifiability defect, not (currently) a data-loss one.

Why it matters

  • kubectl get backup can never confirm a backup ran. The only verification path is exec-ing a pod with S3 credentials and running barman-cloud-backup-list — which nobody does routinely.
  • Any alerting built on Backup CR phase is silently blind. Worth checking whether anything in monitoring-rules assumes those objects exist.
  • CNPG surfaces backup state partly through these objects, so the operator’s own view of backup history may be degraded.

This is the same failure shape as the WordPress backup that failed nightly for three days against a bucket declared in tofu but never applied: the status surface looked fine because nothing was actually checking the thing that mattered.

Mechanism: NOT yet established

Filing the observation deliberately without a cause, because the obvious hypothesis is disproven:

  • Flux pruning them — ruled out. Flux only prunes resources in its own inventory. The CNPG Backups are created by the operator and were never applied by Flux, so they are not in it. Consistent with the evidence: kustomize-controller logs show no prune of these objects, and the one Backup that does survive is the wordpress one, which appears in Flux apply output as Backup/wordpress/wordpress-mariadb-backup: unchanged because it is declared in git.
  • force=false on the Kustomizations, so it is not delete/recreate on apply.

Candidates not yet tested:

  • backupOwnerReference: self on the ScheduledBackups — if the owner is ever replaced, owned Backups cascade-delete. Worth testing whether the operator replaces rather than updates the ScheduledBackup.
  • Some retention/GC in the CNPG operator or the barman-cloud plugin that reaps Backup CRs after upload.
  • Something else entirely.

Cheap decisive test: create a Backup CR by hand (not via ScheduledBackup, so it carries no Flux labels and has a different owner) and see whether it also disappears. That separates "anything that creates a Backup" from "specifically ScheduledBackup-created ones".

Suggested outcome

Either make the Backup CRs persist so kubectl get backup is meaningful, or — if their deletion turns out to be intended operator behaviour — add an explicit backup-freshness alert that queries something durable, so backup health is monitored rather than assumed.

🤖 Generated with Claude Code

Found while verifying ha-pg phase 1 (#521). The backup DATA is fine; the Kubernetes-side record is not. ## Observation There are **zero CNPG `Backup` objects anywhere in the cluster**: ``` $ kubectl get backup -A wordpress wordpress-mariadb-backup 2d17h ``` That single survivor belongs to a different operator (mariadb-operator) and **is declared in git**. None of the six CNPG clusters — `ak-pg`, `im-pg`, `pprl-pg`, `gf-pg`, `wp-pg`, `tr-pg`, `ha-pg` — has a single Backup CR, despite all having `ScheduledBackup` resources with `immediate: true`. Observed directly on ha-pg, which was created fresh at 13:25 today: ``` 13:26:15 Backup ha-pg-daily-20260819132545 created 13:26:28 "Backup completed" (cluster log + k8s event) 13:27:0x kubectl get backup -> No resources found ``` Under a minute from completion to gone. ## The backups themselves ARE working Verified against the bucket rather than trusting CR status: ``` $ barman-cloud-backup-list --endpoint-url https://fsn1.your-objectstorage.com \ s3://renner-uno-k8s-cnpg/ha-pg ha-pg Backup ID End Time Begin Wal Name 20260819T132615 2026-08-19 13:26:22 000000010000000000000003 backup-20260819132615 ``` WAL archiving also confirmed live and **readable**: `ContinuousArchiving=True`, and `barman-cloud-wal-restore` fetched segment `000000010000000000000003` back out of S3 with rc=0. So this is an observability/verifiability defect, not (currently) a data-loss one. ## Why it matters - `kubectl get backup` can never confirm a backup ran. The only verification path is exec-ing a pod with S3 credentials and running `barman-cloud-backup-list` — which nobody does routinely. - Any alerting built on Backup CR phase is **silently blind**. Worth checking whether anything in `monitoring-rules` assumes those objects exist. - CNPG surfaces backup state partly through these objects, so the operator’s own view of backup history may be degraded. This is the same failure shape as the WordPress backup that failed nightly for three days against a bucket declared in tofu but never applied: the status surface looked fine because nothing was actually checking the thing that mattered. ## Mechanism: NOT yet established Filing the observation deliberately without a cause, because the obvious hypothesis is disproven: - **Flux pruning them — ruled out.** Flux only prunes resources in its own inventory. The CNPG Backups are created by the operator and were never applied by Flux, so they are not in it. Consistent with the evidence: `kustomize-controller` logs show no prune of these objects, and the one Backup that *does* survive is the wordpress one, which appears in Flux apply output as `Backup/wordpress/wordpress-mariadb-backup: unchanged` because it is declared in git. - `force=false` on the Kustomizations, so it is not delete/recreate on apply. Candidates not yet tested: - `backupOwnerReference: self` on the ScheduledBackups — if the owner is ever replaced, owned Backups cascade-delete. Worth testing whether the operator replaces rather than updates the ScheduledBackup. - Some retention/GC in the CNPG operator or the barman-cloud plugin that reaps Backup CRs after upload. - Something else entirely. **Cheap decisive test:** create a Backup CR by hand (not via ScheduledBackup, so it carries no Flux labels and has a different owner) and see whether it also disappears. That separates "anything that creates a Backup" from "specifically ScheduledBackup-created ones". ## Suggested outcome Either make the Backup CRs persist so `kubectl get backup` is meaningful, or — if their deletion turns out to be intended operator behaviour — add an explicit backup-freshness alert that queries something durable, so backup health is monitored rather than assumed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-20 06:09:33 +00:00
Author
Member
Copy Link
Copy Source

RETRACTING the premise of this issue — it was my query that was wrong, not the cluster.

The Backup CRs were never missing

There are 113 of them:

$ kubectl get backups.postgresql.cnpg.io -A --no-headers | wc -l
113

authentik   ak-pg-daily-20260820023000   3h   ak-pg   plugin  completed
immich      im-pg-daily-20260820030000   3h   im-pg   plugin  completed
trmnl       tr-pg-daily-20260820030000   3h   tr-pg   plugin  completed
...

Newest are ~3h old and completed. The only failures in the whole set are three from 2026-05-20, 92 days ago.

Why I got it wrong

kubectl get backup is ambiguous — three CRDs claim that short name:

backups.k8s.mariadb.com        created 2026-08-16   <- four days ago
backups.postgresql.cnpg.io
backups.velero.io

kubectl resolved it to backups.k8s.mariadb.com, so every "no CNPG backups anywhere" observation in this issue was actually listing mariadb Backups. The single "surviving" Backup I cited (wordpress-mariadb-backup) is a mariadb object — the only one of that type, which is exactly why it looked like the lone survivor.

The ambiguity is recent: the mariadb CRD was installed 2026-08-16, so kubectl get backup would have behaved differently before that.

The test that exposed it was creating a Backup by hand — the delete-confirmation came back as backups.k8s.mariadb.com "manual-probe-522" not found while the object had been created successfully as a CNPG Backup. It completed in 75s and persisted.

So backups are verifiable from kubectl after all. Use the fully-qualified name.

What IS real, and is a different defect

cnpg_collector_last_available_backup_timestamp genuinely reads 0 on every instance — but not because Backup CRs vanish. CNPG never populates the field it derives from:

authentik/ak-pg       lastSuccessfulBackup=""
home-assistant/ha-pg  lastSuccessfulBackup=""
immich/im-pg          lastSuccessfulBackup=""
monitoring/gf-pg      lastSuccessfulBackup=""
paperless/pprl-pg     lastSuccessfulBackup=""
trmnl/tr-pg           lastSuccessfulBackup=""
woodpecker/wp-pg      lastSuccessfulBackup=""

while lastFailedBackup IS populated (with the stale 2026-05-20 value). So the failure path writes cluster status and the success path does not. That looks specific to the barman-cloud plugin rather than the deprecated in-tree barman support.

Consequence: backup-AGE alerting via that metric is impossible, which is why #523 deliberately omitted it. That omission was right; the reason I gave for it was wrong.

Correction to #523

Its comment says the metric is 0 "because the Backup CRs it derives that from are deleted within a minute of completing". That is false. The metric is 0 because Cluster.status.lastSuccessfulBackup is never written. The alerts themselves are unaffected — CNPGWalArchiveBacklog reads the instance collector, and CNPGBackupFailed keys on the failed-timestamp metric, which does work.

Reframing this issue

Renaming in spirit: not "Backup CRs disappear" but "CNPG barman-cloud plugin does not populate lastSuccessfulBackup, so backup-age alerting is impossible".

Worth doing as a follow-up, since backup freshness is genuinely unmonitored:

  • expose CNPG Backup CRs to kube-state-metrics via a CustomResourceStateMetrics config, giving a real backup-age series to alert on; or
  • check whether a newer barman-cloud plugin release populates the field, and upgrade.

Neither is urgent — backups are running and verifiable — but "no alert on backup freshness" is a real gap and should not be closed on the false premise I opened it with.

🤖 Generated with Claude Code

RETRACTING the premise of this issue — it was my query that was wrong, not the cluster. ## The Backup CRs were never missing There are **113** of them: ``` $ kubectl get backups.postgresql.cnpg.io -A --no-headers | wc -l 113 authentik ak-pg-daily-20260820023000 3h ak-pg plugin completed immich im-pg-daily-20260820030000 3h im-pg plugin completed trmnl tr-pg-daily-20260820030000 3h tr-pg plugin completed ... ``` Newest are ~3h old and `completed`. The only failures in the whole set are three from **2026-05-20, 92 days ago**. ## Why I got it wrong `kubectl get backup` is **ambiguous** — three CRDs claim that short name: ``` backups.k8s.mariadb.com created 2026-08-16 <- four days ago backups.postgresql.cnpg.io backups.velero.io ``` kubectl resolved it to **`backups.k8s.mariadb.com`**, so every "no CNPG backups anywhere" observation in this issue was actually listing mariadb Backups. The single "surviving" Backup I cited (`wordpress-mariadb-backup`) is a mariadb object — the only one of that type, which is exactly why it looked like the lone survivor. The ambiguity is recent: the mariadb CRD was installed 2026-08-16, so `kubectl get backup` would have behaved differently before that. The test that exposed it was creating a Backup by hand — the delete-confirmation came back as `backups.k8s.mariadb.com "manual-probe-522" not found` while the object had been created successfully as a CNPG Backup. It completed in 75s and persisted. **So backups are verifiable from kubectl after all.** Use the fully-qualified name. ## What IS real, and is a different defect `cnpg_collector_last_available_backup_timestamp` genuinely reads **0** on every instance — but not because Backup CRs vanish. CNPG never populates the field it derives from: ``` authentik/ak-pg lastSuccessfulBackup="" home-assistant/ha-pg lastSuccessfulBackup="" immich/im-pg lastSuccessfulBackup="" monitoring/gf-pg lastSuccessfulBackup="" paperless/pprl-pg lastSuccessfulBackup="" trmnl/tr-pg lastSuccessfulBackup="" woodpecker/wp-pg lastSuccessfulBackup="" ``` while `lastFailedBackup` IS populated (with the stale 2026-05-20 value). So the failure path writes cluster status and the success path does not. That looks specific to the **barman-cloud plugin** rather than the deprecated in-tree barman support. Consequence: backup-AGE alerting via that metric is impossible, which is why #523 deliberately omitted it. That omission was right; the reason I gave for it was wrong. ## Correction to #523 Its comment says the metric is 0 "because the Backup CRs it derives that from are deleted within a minute of completing". That is false. The metric is 0 because `Cluster.status.lastSuccessfulBackup` is never written. The alerts themselves are unaffected — `CNPGWalArchiveBacklog` reads the instance collector, and `CNPGBackupFailed` keys on the failed-timestamp metric, which does work. ## Reframing this issue Renaming in spirit: not "Backup CRs disappear" but **"CNPG barman-cloud plugin does not populate lastSuccessfulBackup, so backup-age alerting is impossible"**. Worth doing as a follow-up, since backup freshness is genuinely unmonitored: - expose CNPG Backup CRs to kube-state-metrics via a CustomResourceStateMetrics config, giving a real backup-age series to alert on; or - check whether a newer barman-cloud plugin release populates the field, and upgrade. Neither is urgent — backups are running and verifiable — but "no alert on backup freshness" is a real gap and should not be closed on the false premise I opened it with. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-20 06:15:00 +00:00
Author
Member
Copy Link
Copy Source

Cause identified — it is a plugin feature gap, not skew and not missing CRs.

Upstream barman-cloud plugin v0.6.0 (Aug 2025) added a "Last failed backup status field and metric". The plugin implements the failed status field and never implemented the successful one. That matches the measurement exactly:

lastFailedBackup      = 2026-05-20T02:40:24Z   <- populated
lastSuccessfulBackup  = ""                     <- empty on all seven clusters
firstRecoverabilityPoint = ""

So cnpg_collector_last_available_backup_timestamp reads 0 because the field it derives from is never written by this plugin. Upgrading will not fix it — I checked the v0.13.0 and v0.14.0 release notes and neither adds successful-backup status reporting.

That closes out the "why". The remaining gap is unchanged and real: there is no backup-freshness alert, and the metric that would provide one is permanently 0 with this plugin.

Options, for whenever this is worth doing:

  • Expose the Backup CRs to kube-state-metrics via CustomResourceStateMetrics, deriving age from .metadata.creationTimestamp filtered on .status.phase == completed. That works regardless of what the plugin reports, and the CRs are plentiful (113 of them).
  • Or alert on absence: no Backup CR newer than 48h for a given cluster.

Both are a genuine improvement over today, where backup freshness is checked only by a human running kubectl get backups.postgresql.cnpg.io.

Separately filed #535 for the sidecar/controller version skew found during this investigation (controller v0.14.0, sidecar v0.12.0). Explicitly NOT the cause of this issue, but worth its own fix.

🤖 Generated with Claude Code

Cause identified — it is a plugin feature gap, not skew and not missing CRs. Upstream barman-cloud plugin **v0.6.0 (Aug 2025)** added a *"Last failed backup status field and metric"*. The plugin implements the **failed** status field and never implemented the successful one. That matches the measurement exactly: ``` lastFailedBackup = 2026-05-20T02:40:24Z <- populated lastSuccessfulBackup = "" <- empty on all seven clusters firstRecoverabilityPoint = "" ``` So `cnpg_collector_last_available_backup_timestamp` reads 0 because the field it derives from is never written by this plugin. **Upgrading will not fix it** — I checked the v0.13.0 and v0.14.0 release notes and neither adds successful-backup status reporting. That closes out the "why". The remaining gap is unchanged and real: **there is no backup-freshness alert**, and the metric that would provide one is permanently 0 with this plugin. Options, for whenever this is worth doing: - Expose the Backup CRs to kube-state-metrics via `CustomResourceStateMetrics`, deriving age from `.metadata.creationTimestamp` filtered on `.status.phase == completed`. That works regardless of what the plugin reports, and the CRs are plentiful (113 of them). - Or alert on absence: no Backup CR newer than 48h for a given cluster. Both are a genuine improvement over today, where backup freshness is checked only by a human running `kubectl get backups.postgresql.cnpg.io`. Separately filed #535 for the sidecar/controller version skew found during this investigation (controller v0.14.0, sidecar v0.12.0). Explicitly NOT the cause of this issue, but worth its own fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#522