VeleroBackupStale{schedule="netbird-backup-daily"} has been paging critical since 2026-09-03T19:56Z for a schedule that was deliberately decommissioned and garbage-collected by Flux on 2026-09-01T14:58Z. The cause is not a backup failure: velero_backup_last_successful_timestamp is an in-process gauge that Velero does not retract when a Schedule is deleted, so the orphaned series simply stops advancing, ages past the rule's 60h threshold, and fires forever. The rule matched on that gauge alone, with nothing asserting the schedule still exists — so any future schedule removal reproduces this identically.
The fix gates VeleroBackupStale on velero_backup_last_status, a per-schedule series that Velero does retract on Schedule deletion. It acts as a liveness set for "this schedule still exists", so orphaned schedules drop out of the alert while every real staleness condition still fires.
Changes
kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml — VeleroBackupStale: wrap the existing threshold in parentheses and add and on(schedule) velero_backup_last_status. Threshold, for:, severity, labels and annotations are unchanged; and returns the left-hand labels, so {{ $labels.schedule }} still renders. The rest of the change is a comment recording the failure mode, in the style of the existing 60h rationale block.
Verification
Verified live this run (all read-only, via ./.ci-diag/kubectl):
The alert is real and is a single orphan.query=ALERTS{alertname="VeleroBackupStale"} → exactly one firing series, schedule="netbird-backup-daily", severity="critical".
The schedule is gone; the gauge is not.get schedules.velero.io -n velero → 11 schedules (apps-daily … wordpress-daily), no netbird-backup-daily. velero_backup_last_successful_timestamp{schedule!=""} → 12 series: those 11 plus netbird-backup-daily frozen at 1788248768 (2026-09-01T07:46:08Z). get ns netbird-backup → NotFound.
velero_backup_last_status tracks schedule existence.count(velero_backup_last_status) by (schedule) → exactly the 11 live schedules, no netbird-backup-daily. A range query pinned its last sample to t=1788274660 (2026-09-01T14:57:40Z) — the scrape immediately before the Flux GC deletion at 14:58:10Z. The retraction is tied to the deletion, not to a scrape gap.
The gate creates no false negatives across a Velero restart.count(velero_backup_last_status) by (pod) across the 2026-08-27 rollout shows three pods (…-c9n2g → …-qqxkb → …-gmzzw), each exporting all 13 series from its first scrape. Velero repopulates the metric at startup from the existing Schedules; it does not wait for the next backup run.
The new expression behaves as intended. The exact multi-line expr as committed, sent to Prometheus /api/v1/query, parses (status: success) and returns [] — the alert stops firing. With only the threshold lowered from 60 * 3600 to 0 so every schedule qualifies, the same joined expression returns 11 series, i.e. all live schedules survive the join and only the orphan is dropped. A sampled result retains schedule="paperless-daily" and the full label set.
After Flux reconciles kubernetes/apps/monitoring-rules, VeleroBackupStale{schedule="netbird-backup-daily"} should resolve within one evaluation cycle (no for: wait applies to resolution). Confirm with the ALERTS{alertname="VeleroBackupStale"} query above → empty. Nothing else should change; the other 11 schedules were all well inside the threshold (most recent backups today).
Notes
What the issue got wrong. It proposed also changing the two per-event rules above (VeleroBackupValidationFailure, VeleroBackupFailed) "for the same reason". They do not need it and were left alone: both use increase(<counter>[1h]) > 0, and a frozen orphan counter produces an increase of 0, so it cannot fire spuriously. The issue also listed only three options and ranked a kube-state-metrics CustomResourceState join first, stating Prometheus "carries no metric describing Velero schedules". That is not quite true — velero_backup_last_status is retracted on Schedule deletion and serves the same purpose, which is why no new wiring, RBAC or chart values were needed here.
The restart is no longer necessary. The issue's unverified hypothesis — that restarting Velero would clear the orphan — is consistent with what I observed (each new pod initialises its registry from the live Schedule list), so kubectl -n velero rollout restart deploy/velero would also clear it, and is safe right now (all Backup CRs are in terminal phases; schedules fire 03:15–08:45 UTC). But it is a one-off manual action that fixes only this instance, and this PR removes the need for it: the orphan velero_backup_last_successful_timestamp series may persist until Velero's next restart and the rule will correctly ignore it either way.
Not silencing. The gate suppresses only schedules that no longer exist. A schedule that exists and is failing keeps its velero_backup_last_status series (written as 0 on failed runs, 1 on successful ones), and a schedule that exists but has stopped firing keeps its last value — both still alert. VeleroBackupFailed also still fires on the first failure independently.
Accepted RPO reduction, unchanged by this PR. The NetBird management store now has no cluster-side backup, by design (ansible/roles/netbird-server/tasks/main.yml:80-95 removes the dump script and revokes the cluster's root SSH key on the VPS). Worth recording deliberately somewhere durable. The 28 surviving netbird-backup-daily-* Backup CRs are all Completed and age out on their own; the newest expires 2026-10-01T07:45:31Z. The netbird-backup-default-kopia BackupRepository is still Ready and runs hourly kopia maintenance jobs — expected while those backups are restorable, and it should be left alone until they expire.
Out of scope, ignored as instructed: nothing in the issue asked for changes to CI config, credentials or anything outside the edit allowlist.
## What this PR does
`VeleroBackupStale{schedule="netbird-backup-daily"}` has been paging critical since 2026-09-03T19:56Z for a schedule that was deliberately decommissioned and garbage-collected by Flux on 2026-09-01T14:58Z. The cause is not a backup failure: `velero_backup_last_successful_timestamp` is an in-process gauge that Velero does **not** retract when a `Schedule` is deleted, so the orphaned series simply stops advancing, ages past the rule's 60h threshold, and fires forever. The rule matched on that gauge alone, with nothing asserting the schedule still exists — so **any** future schedule removal reproduces this identically.
The fix gates `VeleroBackupStale` on `velero_backup_last_status`, a per-schedule series that Velero *does* retract on Schedule deletion. It acts as a liveness set for "this schedule still exists", so orphaned schedules drop out of the alert while every real staleness condition still fires.
## Changes
- `kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml` — `VeleroBackupStale`: wrap the existing threshold in parentheses and add `and on(schedule) velero_backup_last_status`. Threshold, `for:`, severity, labels and annotations are unchanged; `and` returns the left-hand labels, so `{{ $labels.schedule }}` still renders. The rest of the change is a comment recording the failure mode, in the style of the existing 60h rationale block.
## Verification
Verified live this run (all read-only, via `./.ci-diag/kubectl`):
1. **The alert is real and is a single orphan.** `query=ALERTS{alertname="VeleroBackupStale"}` → exactly one firing series, `schedule="netbird-backup-daily"`, `severity="critical"`.
2. **The schedule is gone; the gauge is not.** `get schedules.velero.io -n velero` → 11 schedules (`apps-daily` … `wordpress-daily`), no `netbird-backup-daily`. `velero_backup_last_successful_timestamp{schedule!=""}` → **12** series: those 11 plus `netbird-backup-daily` frozen at `1788248768` (2026-09-01T07:46:08Z). `get ns netbird-backup` → `NotFound`.
3. **`velero_backup_last_status` tracks schedule existence.** `count(velero_backup_last_status) by (schedule)` → exactly the 11 live schedules, no `netbird-backup-daily`. A range query pinned its last sample to `t=1788274660` (2026-09-01T14:57:40Z) — the scrape immediately before the Flux GC deletion at 14:58:10Z. The retraction is tied to the deletion, not to a scrape gap.
4. **The gate creates no false negatives across a Velero restart.** `count(velero_backup_last_status) by (pod)` across the 2026-08-27 rollout shows three pods (`…-c9n2g` → `…-qqxkb` → `…-gmzzw`), each exporting all 13 series from its *first* scrape. Velero repopulates the metric at startup from the existing Schedules; it does not wait for the next backup run.
5. **The new expression behaves as intended.** The exact multi-line expr as committed, sent to Prometheus `/api/v1/query`, parses (`status: success`) and returns `[]` — the alert stops firing. With only the threshold lowered from `60 * 3600` to `0` so every schedule qualifies, the same joined expression returns **11** series, i.e. all live schedules survive the join and only the orphan is dropped. A sampled result retains `schedule="paperless-daily"` and the full label set.
After Flux reconciles `kubernetes/apps/monitoring-rules`, `VeleroBackupStale{schedule="netbird-backup-daily"}` should resolve within one evaluation cycle (no `for:` wait applies to resolution). Confirm with the `ALERTS{alertname="VeleroBackupStale"}` query above → empty. Nothing else should change; the other 11 schedules were all well inside the threshold (most recent backups today).
## Notes
- **What the issue got wrong.** It proposed also changing the two per-event rules above (`VeleroBackupValidationFailure`, `VeleroBackupFailed`) "for the same reason". They do not need it and were left alone: both use `increase(<counter>[1h]) > 0`, and a frozen orphan counter produces an increase of 0, so it cannot fire spuriously. The issue also listed only three options and ranked a kube-state-metrics `CustomResourceState` join first, stating Prometheus "carries no metric describing Velero schedules". That is not quite true — `velero_backup_last_status` is retracted on Schedule deletion and serves the same purpose, which is why no new wiring, RBAC or chart values were needed here.
- **The restart is no longer necessary.** The issue's unverified hypothesis — that restarting Velero would clear the orphan — is consistent with what I observed (each new pod initialises its registry from the live Schedule list), so `kubectl -n velero rollout restart deploy/velero` would also clear it, and is safe right now (all Backup CRs are in terminal phases; schedules fire 03:15–08:45 UTC). But it is a one-off manual action that fixes only this instance, and this PR removes the need for it: the orphan `velero_backup_last_successful_timestamp` series may persist until Velero's next restart and the rule will correctly ignore it either way.
- **Not silencing.** The gate suppresses only schedules that no longer exist. A schedule that exists and is failing keeps its `velero_backup_last_status` series (written as `0` on failed runs, `1` on successful ones), and a schedule that exists but has stopped firing keeps its last value — both still alert. `VeleroBackupFailed` also still fires on the first failure independently.
- **Accepted RPO reduction, unchanged by this PR.** The NetBird management store now has no cluster-side backup, by design (`ansible/roles/netbird-server/tasks/main.yml:80-95` removes the dump script and revokes the cluster's root SSH key on the VPS). Worth recording deliberately somewhere durable. The 28 surviving `netbird-backup-daily-*` Backup CRs are all `Completed` and age out on their own; the newest expires 2026-10-01T07:45:31Z. The `netbird-backup-default-kopia` BackupRepository is still `Ready` and runs hourly kopia maintenance jobs — expected while those backups are restorable, and it should be left alone until they expire.
- **Out of scope, ignored as instructed:** nothing in the issue asked for changes to CI config, credentials or anything outside the edit allowlist.
Closes #780
🤖Automated review — model claude-sonnet-5 (complexity: normal), commit 325baf7047039b9d45b89517a2432b89c27c31a4.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
Single-rule change to VeleroBackupStale in kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml: it AND-joins the existing staleness condition (on velero_backup_last_successful_timestamp) with velero_backup_last_status matched on(schedule), so schedules that have been deleted (and whose timestamp gauge Velero never retracts) drop out of the alert instead of paging forever. The change is small, well-scoped, and the PromQL is correct. No blocking issues.
🟡 Suggestions
prometheusrule-velero.yaml:83-84 — velero_backup_last_status is joined unfiltered, while every other selector in this file (including the LHS of this same expression) uses {schedule!=""}. It's harmless today: and is a set filter, and the LHS already excludes schedule="", so a schedule="" sample on the RHS (if one ever exists, e.g. for ad-hoc non-scheduled backups) can't match anything. Still, adding {schedule!=""} here would make the two sides symmetric and self-documenting without changing behavior.
No promtool/CI validation of the rule's PromQL exists in this repo (confirmed — no *promtool* files, no rule_files/promtool test references anywhere), so a typo in an expr: block would only surface once Prometheus tries to load it. That's a pre-existing repo-wide gap, not something this PR introduces, and the PR body documents thorough manual verification (live query dry-runs against Prometheus) in lieu of it, consistent with how prometheusrule-cnpg-backup.yaml and prometheusrule-hetzner-backup.yaml reason about their own expressions in comments.
Tests
No automated test exists or is expected for PrometheusRule expressions in this repo. The PR body documents manual verification against the live Prometheus instance (confirms the orphan series, confirms velero_backup_last_status retracts on Schedule deletion and repopulates on Velero restart without a gap, and dry-runs both the real expression and a threshold-lowered variant to show all 11 live schedules survive the join while the orphan is dropped). That's adequate given the repo's existing conventions for this kind of change.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `325baf7047039b9d45b89517a2432b89c27c31a4`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
Single-rule change to `VeleroBackupStale` in `kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml`: it AND-joins the existing staleness condition (on `velero_backup_last_successful_timestamp`) with `velero_backup_last_status` matched `on(schedule)`, so schedules that have been deleted (and whose timestamp gauge Velero never retracts) drop out of the alert instead of paging forever. The change is small, well-scoped, and the PromQL is correct. No blocking issues.
## 🟡 Suggestions
- `prometheusrule-velero.yaml:83-84` — `velero_backup_last_status` is joined unfiltered, while every other selector in this file (including the LHS of this same expression) uses `{schedule!=""}`. It's harmless today: `and` is a set filter, and the LHS already excludes `schedule=""`, so a `schedule=""` sample on the RHS (if one ever exists, e.g. for ad-hoc non-scheduled backups) can't match anything. Still, adding `{schedule!=""}` here would make the two sides symmetric and self-documenting without changing behavior.
- No `promtool`/CI validation of the rule's PromQL exists in this repo (confirmed — no `*promtool*` files, no `rule_files`/`promtool test` references anywhere), so a typo in an `expr:` block would only surface once Prometheus tries to load it. That's a pre-existing repo-wide gap, not something this PR introduces, and the PR body documents thorough manual verification (live query dry-runs against Prometheus) in lieu of it, consistent with how `prometheusrule-cnpg-backup.yaml` and `prometheusrule-hetzner-backup.yaml` reason about their own expressions in comments.
## Tests
No automated test exists or is expected for `PrometheusRule` expressions in this repo. The PR body documents manual verification against the live Prometheus instance (confirms the orphan series, confirms `velero_backup_last_status` retracts on Schedule deletion and repopulates on Velero restart without a gap, and dry-runs both the real expression and a threshold-lowered variant to show all 11 live schedules survive the join while the orphan is dropped). That's adequate given the repo's existing conventions for this kind of change.
## Recommendation
Looks reasonable — no blocking issues; comments above are optional.
<!-- pr-review:sha=325baf7047039b9d45b89517a2432b89c27c31a4 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
VeleroBackupStale{schedule="netbird-backup-daily"}has been paging critical since 2026-09-03T19:56Z for a schedule that was deliberately decommissioned and garbage-collected by Flux on 2026-09-01T14:58Z. The cause is not a backup failure:velero_backup_last_successful_timestampis an in-process gauge that Velero does not retract when aScheduleis deleted, so the orphaned series simply stops advancing, ages past the rule's 60h threshold, and fires forever. The rule matched on that gauge alone, with nothing asserting the schedule still exists — so any future schedule removal reproduces this identically.The fix gates
VeleroBackupStaleonvelero_backup_last_status, a per-schedule series that Velero does retract on Schedule deletion. It acts as a liveness set for "this schedule still exists", so orphaned schedules drop out of the alert while every real staleness condition still fires.Changes
kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml—VeleroBackupStale: wrap the existing threshold in parentheses and addand on(schedule) velero_backup_last_status. Threshold,for:, severity, labels and annotations are unchanged;andreturns the left-hand labels, so{{ $labels.schedule }}still renders. The rest of the change is a comment recording the failure mode, in the style of the existing 60h rationale block.Verification
Verified live this run (all read-only, via
./.ci-diag/kubectl):query=ALERTS{alertname="VeleroBackupStale"}→ exactly one firing series,schedule="netbird-backup-daily",severity="critical".get schedules.velero.io -n velero→ 11 schedules (apps-daily…wordpress-daily), nonetbird-backup-daily.velero_backup_last_successful_timestamp{schedule!=""}→ 12 series: those 11 plusnetbird-backup-dailyfrozen at1788248768(2026-09-01T07:46:08Z).get ns netbird-backup→NotFound.velero_backup_last_statustracks schedule existence.count(velero_backup_last_status) by (schedule)→ exactly the 11 live schedules, nonetbird-backup-daily. A range query pinned its last sample tot=1788274660(2026-09-01T14:57:40Z) — the scrape immediately before the Flux GC deletion at 14:58:10Z. The retraction is tied to the deletion, not to a scrape gap.count(velero_backup_last_status) by (pod)across the 2026-08-27 rollout shows three pods (…-c9n2g→…-qqxkb→…-gmzzw), each exporting all 13 series from its first scrape. Velero repopulates the metric at startup from the existing Schedules; it does not wait for the next backup run./api/v1/query, parses (status: success) and returns[]— the alert stops firing. With only the threshold lowered from60 * 3600to0so every schedule qualifies, the same joined expression returns 11 series, i.e. all live schedules survive the join and only the orphan is dropped. A sampled result retainsschedule="paperless-daily"and the full label set.After Flux reconciles
kubernetes/apps/monitoring-rules,VeleroBackupStale{schedule="netbird-backup-daily"}should resolve within one evaluation cycle (nofor:wait applies to resolution). Confirm with theALERTS{alertname="VeleroBackupStale"}query above → empty. Nothing else should change; the other 11 schedules were all well inside the threshold (most recent backups today).Notes
VeleroBackupValidationFailure,VeleroBackupFailed) "for the same reason". They do not need it and were left alone: both useincrease(<counter>[1h]) > 0, and a frozen orphan counter produces an increase of 0, so it cannot fire spuriously. The issue also listed only three options and ranked a kube-state-metricsCustomResourceStatejoin first, stating Prometheus "carries no metric describing Velero schedules". That is not quite true —velero_backup_last_statusis retracted on Schedule deletion and serves the same purpose, which is why no new wiring, RBAC or chart values were needed here.kubectl -n velero rollout restart deploy/velerowould also clear it, and is safe right now (all Backup CRs are in terminal phases; schedules fire 03:15–08:45 UTC). But it is a one-off manual action that fixes only this instance, and this PR removes the need for it: the orphanvelero_backup_last_successful_timestampseries may persist until Velero's next restart and the rule will correctly ignore it either way.velero_backup_last_statusseries (written as0on failed runs,1on successful ones), and a schedule that exists but has stopped firing keeps its last value — both still alert.VeleroBackupFailedalso still fires on the first failure independently.ansible/roles/netbird-server/tasks/main.yml:80-95removes the dump script and revokes the cluster's root SSH key on the VPS). Worth recording deliberately somewhere durable. The 28 survivingnetbird-backup-daily-*Backup CRs are allCompletedand age out on their own; the newest expires 2026-10-01T07:45:31Z. Thenetbird-backup-default-kopiaBackupRepository is stillReadyand runs hourly kopia maintenance jobs — expected while those backups are restorable, and it should be left alone until they expire.Closes #780
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit325baf7047039b9d45b89517a2432b89c27c31a4.Summary
Single-rule change to
VeleroBackupStaleinkubernetes/apps/monitoring-rules/prometheusrule-velero.yaml: it AND-joins the existing staleness condition (onvelero_backup_last_successful_timestamp) withvelero_backup_last_statusmatchedon(schedule), so schedules that have been deleted (and whose timestamp gauge Velero never retracts) drop out of the alert instead of paging forever. The change is small, well-scoped, and the PromQL is correct. No blocking issues.🟡 Suggestions
prometheusrule-velero.yaml:83-84—velero_backup_last_statusis joined unfiltered, while every other selector in this file (including the LHS of this same expression) uses{schedule!=""}. It's harmless today:andis a set filter, and the LHS already excludesschedule="", so aschedule=""sample on the RHS (if one ever exists, e.g. for ad-hoc non-scheduled backups) can't match anything. Still, adding{schedule!=""}here would make the two sides symmetric and self-documenting without changing behavior.promtool/CI validation of the rule's PromQL exists in this repo (confirmed — no*promtool*files, norule_files/promtool testreferences anywhere), so a typo in anexpr:block would only surface once Prometheus tries to load it. That's a pre-existing repo-wide gap, not something this PR introduces, and the PR body documents thorough manual verification (live query dry-runs against Prometheus) in lieu of it, consistent with howprometheusrule-cnpg-backup.yamlandprometheusrule-hetzner-backup.yamlreason about their own expressions in comments.Tests
No automated test exists or is expected for
PrometheusRuleexpressions in this repo. The PR body documents manual verification against the live Prometheus instance (confirms the orphan series, confirmsvelero_backup_last_statusretracts on Schedule deletion and repopulates on Velero restart without a gap, and dry-runs both the real expression and a threshold-lowered variant to show all 11 live schedules survive the join while the orphan is dropped). That's adequate given the repo's existing conventions for this kind of change.Recommendation
Looks reasonable — no blocking issues; comments above are optional.