fix(monitoring): raise VeleroBackupStale from 30h to 60h #658
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#658
Reference in New Issue
Block a user
At 30h a single missed cycle pages critical — and a single missed cycle is the normal cost of touching the nodes. Velero moves data through pods, so draining a node kills whatever upload is in flight and the schedule is lost for that day; it is not retried. Backup windows run 03:15–08:45 UTC, so any rolling node work overlapping them produces a guaranteed critical alert per affected schedule.
That is noise about planned work, not signal about backups. It fired for eleven schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailedfires on the first failure regardless, so the first cycle stays visible — only the escalation to critical waits for a second cycle to confirm it.Calibration against the current state:
The new threshold leaves those roughly twelve hours — exactly until today's runs. If those fail too, the alert fires. That is the intended behaviour, not a gap.
Verified: passes a server-side dry run; the expression returns nothing at 60h; and the same query at a lower bound returns the four affected schedules with their ages, so the selector is known to match rather than merely silent.
Context on why this came up: over 30 days each schedule has failed exactly once (home-assistant twice), all on the bare-metal upgrade day and today. The failures are node-disruption artefacts —
dial tcp 10.96.0.1:443: no route to hostduring control-plane reboots on the first day, killed DataUploads on a drained node today.🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commite2f3d67f79200d04d97de3bb92f4b2f67e4ea031.Summary
This PR raises the
VeleroBackupStalealert threshold from 30 hours to 60 hours in the Prometheus monitoring rules. The change is well-reasoned: at 30h a single missed backup triggers a critical alert, but single missed cycles are normal during node maintenance (draining nodes kills in-flight Velero uploads). By moving to 60h, the alert now requires evidence of two consecutive failed backup cycles, which cannot be explained by a single maintenance window. The PR includes thorough documentation of the rationale and verification via dry-run testing.🟢 Nits
Line 62: The arithmetic
60 * 3600is correct (216,000 seconds = 60 hours), though using a named constant likeONE_HOUR_SECONDSin a Prometheus template could improve clarity for future maintainers. That said, this is minor and consistent with how the backup window is expressed in comments (03:15–08:45 UTC as concrete times rather than offsets).Tests
The PR body documents verification: server-side dry run passed, the expression returns nothing at 60h (no false positives among current backups), and queries at lower bounds confirm the selector matches the four affected schedules with their current ages. This is solid testing for a monitoring rule change. No additional tests needed — Prometheus rules are validated by the cluster's Prometheus instance on deployment.
Recommendation
Looks reasonable — no blocking issues; comment above is optional.