konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests Packages Projects Releases 1 Wiki Activity

[alert] VeleroBackupStale (velero) #780

New Issue
Closed
opened 2026-09-03 20:05:13 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-09-03 20:05:13 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: VeleroBackupStale, severity critical
  • Namespace/workload: velero / velero-7f4f56fb6c-gmzzw
  • Label: schedule="netbird-backup-daily"
  • Since: 2026-09-03T19:56:27Z

This is not a backup failure. The netbird-backup-daily schedule was deliberately decommissioned. The alert is firing on a leftover metric series, and it will not clear on its own.

Evidence

1. The Schedule object no longer exists. ./.ci-diag/kubectl get schedules.velero.io -n velero returns exactly the 11 schedules defined in kubernetes/infrastructure/post-crds/velero-schedules.yaml (apps-daily … wordpress-daily). netbird-backup-daily is in neither the cluster nor the repo — a repo-wide grep for netbird-backup matches only ansible/roles/netbird-server/tasks/main.yml.

2. Flux pruned it, at a known time. Loki, {namespace="flux-system"} |= "Schedule/velero/netbird-backup-daily" |= "deleted":

2026-09-01T14:58:10.563Z  garbage collection completed: Schedule/velero/netbird-backup-daily deleted
  Kustomization: flux-system/infrastructure-crds

./.ci-diag/kubectl get kustomization -n flux-system infrastructure-crds confirms path=./kubernetes/infrastructure/post-crds, prune=true, reconciled at master@sha1:a667920. So the schedule was removed from git and Flux garbage-collected it as designed.

3. The removal was intentional. ansible/roles/netbird-server/tasks/main.yml:80-95 decommissions the whole NetBird backup pipeline in the same change — it removes the /usr/local/bin/netbird-backup-dump script and revokes the cluster's SSH key on the VPS, with the stated rationale that the management store's backup "put a root key on this host that the cluster held" and that the NetBird server's state "is its own business to protect rather than the cluster's."

4. No backup ever failed. All 28 surviving backups for this schedule are Completed except one PartiallyFailed on 2026-08-26, which recovered the next day. The last run, netbird-backup-daily-20260901074531, finished Completed at 2026-09-01T07:46:08Z with no errors or warnings. There is no failure streak — the schedule simply stopped existing 7 hours later. The backup targeted includedNamespaces: [netbird-backup], and that namespace no longer exists either.

5. The metric is still being exported. Prometheus velero_backup_last_successful_timestamp{schedule!=""} returns 12 series: the 11 live schedules (all with today's timestamps) plus netbird-backup-daily frozen at 1788248768 = 2026-09-01T07:46:08Z — now ~60h old, which is what crossed the rule's threshold.

Root cause

kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml:61-64 matches on the metric alone:

time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
and velero_backup_last_successful_timestamp{schedule!=""} > 0

There is no join against whether the Schedule still exists. Velero holds velero_backup_last_successful_timestamp as an in-process gauge, so deleting a Schedule does not retract the series — it just stops being updated, ages past 60h, and fires critical. Any future schedule removal reproduces this exactly.

The alert will therefore not self-resolve. The velero pod started 2026-08-27T07:10:22Z with restarts=0 (kubectl get pod -n velero velero-7f4f56fb6c-gmzzw -o jsonpath='{.status.startTime}'), so the stale gauge has been held continuously; count_over_time(...[7d]) returns 20159 samples, i.e. an unbroken series.

Hypothesis (not verified — verifying it requires a restart, which is a mutation): Velero only initialises schedule metrics for Schedules that exist at server start, so restarting the velero deployment would drop the orphan series and clear the alert. Confirm after any restart with:
./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=velero_backup_last_successful_timestamp%7Bschedule%3D%22netbird-backup-daily%22%7D"

Where to fix it

Immediate (not repo-fixable): clear the orphan series by restarting the velero deployment (kubectl -n velero rollout restart deploy/velero), or silence VeleroBackupStale{schedule="netbird-backup-daily"} in Alertmanager. Left alone this pages the email receiver indefinitely.

Durable (repo): kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml — make VeleroBackupStale (and, for the same reason, the two per-event rules above it) ignore schedules that no longer exist. Options, most robust first:

  1. Expose Velero Schedule existence as a metric via a kube-state-metrics CustomResourceState config, then gate the rule with unless on(schedule) absent(...). Note the repo has no CustomResourceState configuration today, and Prometheus currently carries no metric describing Velero schedules — this is new wiring under kubernetes/apps/monitoring/.
  2. Add a max_over_time/staleness bound so a gauge that has not moved in N days stops alerting rather than escalating.
  3. Cheapest but brittle: exclude the decommissioned name with schedule!="netbird-backup-daily", which fixes this instance but not the next decommission.

No action needed on the backup itself. The NetBird management store is by design no longer backed up by the cluster (ansible/roles/netbird-server/tasks/main.yml:80-95). Worth recording as an accepted RPO reduction: NetBird server state now has no cluster-side backup. The 28 existing netbird-backup-daily-* Backup CRs age out on their own — the newest expires 2026-10-01T07:45:31Z (ttl: 720h).

Filed automatically by the alert-monitor pipeline (2026-09-03). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `VeleroBackupStale`, severity **critical** - **Namespace/workload:** `velero` / `velero-7f4f56fb6c-gmzzw` - **Label:** `schedule="netbird-backup-daily"` - **Since:** 2026-09-03T19:56:27Z **This is not a backup failure.** The `netbird-backup-daily` schedule was deliberately decommissioned. The alert is firing on a leftover metric series, and it will not clear on its own. ## Evidence **1. The Schedule object no longer exists.** `./.ci-diag/kubectl get schedules.velero.io -n velero` returns exactly the 11 schedules defined in `kubernetes/infrastructure/post-crds/velero-schedules.yaml` (`apps-daily` … `wordpress-daily`). `netbird-backup-daily` is in neither the cluster nor the repo — a repo-wide grep for `netbird-backup` matches only `ansible/roles/netbird-server/tasks/main.yml`. **2. Flux pruned it, at a known time.** Loki, `{namespace="flux-system"} |= "Schedule/velero/netbird-backup-daily" |= "deleted"`: ``` 2026-09-01T14:58:10.563Z garbage collection completed: Schedule/velero/netbird-backup-daily deleted Kustomization: flux-system/infrastructure-crds ``` `./.ci-diag/kubectl get kustomization -n flux-system infrastructure-crds` confirms `path=./kubernetes/infrastructure/post-crds`, `prune=true`, reconciled at `master@sha1:a667920`. So the schedule was removed from git and Flux garbage-collected it as designed. **3. The removal was intentional.** `ansible/roles/netbird-server/tasks/main.yml:80-95` decommissions the whole NetBird backup pipeline in the same change — it removes the `/usr/local/bin/netbird-backup-dump` script and revokes the cluster's SSH key on the VPS, with the stated rationale that the management store's backup "put a root key on this host that the cluster held" and that the NetBird server's state "is its own business to protect rather than the cluster's." **4. No backup ever failed.** All 28 surviving backups for this schedule are `Completed` except one `PartiallyFailed` on 2026-08-26, which recovered the next day. The last run, `netbird-backup-daily-20260901074531`, finished `Completed` at 2026-09-01T07:46:08Z with no errors or warnings. There is no failure streak — the schedule simply stopped existing 7 hours later. The backup targeted `includedNamespaces: [netbird-backup]`, and that namespace no longer exists either. **5. The metric is still being exported.** Prometheus `velero_backup_last_successful_timestamp{schedule!=""}` returns 12 series: the 11 live schedules (all with today's timestamps) plus `netbird-backup-daily` frozen at `1788248768` = 2026-09-01T07:46:08Z — now ~60h old, which is what crossed the rule's threshold. ## Root cause `kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml:61-64` matches on the metric alone: ```promql time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600 and velero_backup_last_successful_timestamp{schedule!=""} > 0 ``` There is no join against whether the Schedule still exists. Velero holds `velero_backup_last_successful_timestamp` as an in-process gauge, so deleting a Schedule does not retract the series — it just stops being updated, ages past 60h, and fires critical. **Any** future schedule removal reproduces this exactly. The alert will therefore not self-resolve. The velero pod started 2026-08-27T07:10:22Z with `restarts=0` (`kubectl get pod -n velero velero-7f4f56fb6c-gmzzw -o jsonpath='{.status.startTime}'`), so the stale gauge has been held continuously; `count_over_time(...[7d])` returns 20159 samples, i.e. an unbroken series. **Hypothesis (not verified — verifying it requires a restart, which is a mutation):** Velero only initialises schedule metrics for Schedules that exist at server start, so restarting the velero deployment would drop the orphan series and clear the alert. Confirm after any restart with: `./.ci-diag/kubectl get --raw "/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=velero_backup_last_successful_timestamp%7Bschedule%3D%22netbird-backup-daily%22%7D"` ## Where to fix it **Immediate (not repo-fixable):** clear the orphan series by restarting the velero deployment (`kubectl -n velero rollout restart deploy/velero`), or silence `VeleroBackupStale{schedule="netbird-backup-daily"}` in Alertmanager. Left alone this pages the `email` receiver indefinitely. **Durable (repo):** `kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml` — make `VeleroBackupStale` (and, for the same reason, the two per-event rules above it) ignore schedules that no longer exist. Options, most robust first: 1. Expose Velero `Schedule` existence as a metric via a kube-state-metrics `CustomResourceState` config, then gate the rule with `unless on(schedule) absent(...)`. Note the repo has no `CustomResourceState` configuration today, and Prometheus currently carries no metric describing Velero schedules — this is new wiring under `kubernetes/apps/monitoring/`. 2. Add a `max_over_time`/staleness bound so a gauge that has not moved in N days stops alerting rather than escalating. 3. Cheapest but brittle: exclude the decommissioned name with `schedule!="netbird-backup-daily"`, which fixes this instance but not the next decommission. **No action needed on the backup itself.** The NetBird management store is by design no longer backed up by the cluster (`ansible/roles/netbird-server/tasks/main.yml:80-95`). Worth recording as an accepted RPO reduction: NetBird server state now has no cluster-side backup. The 28 existing `netbird-backup-daily-*` Backup CRs age out on their own — the newest expires 2026-10-01T07:45:31Z (`ttl: 720h`). --- <!-- alert-monitor:fingerprint=6d393ab4b5dc3883 --> _Filed automatically by the alert-monitor pipeline (2026-09-03). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-09-03 20:47:38 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#781 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/781 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#780