monitoring: retire Garage scrape, add Hetzner backup-endpoint alert #26

2026-06-25T14:09:55Z

k.renner commented

2026-06-25 14:09:55 +00:00

Finish line: free nas-1 from cluster-backup monitoring

With all backups now on Hetzner (paperless #23/#24, authentik/immich/velero #25, all validated), nas-1's Garage is no longer a cluster-backup dependency. This removes the monitoring tied to it and adds visibility for the new offsite target.

Removed

Prometheus garage scrape job (s3.nas-1.renner.uno:39033) + garage-metrics-token secret (orphaned).
prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) — would fire every night once nas-1 powers off.

Added

https://fsn1.your-objectstorage.com to the blackbox-http probe (bare endpoint returns 200).
HetznerBackupEndpointDown alert — probe_success == 0 for 10m, critical. Directly detects the WAN→Hetzner outage that both CNPG WAL archiving and Velero depend on.

Deliberately not added

A metric-based CNPG "WAL archiving behind" alert: CNPG pod metrics aren't scraped (enablePodMonitor=false, metric absent), so such an alert would be dead. Enabling CNPG PodMonitors is a worthwhile follow-up. Velero outages are already covered by the existing VeleroBackupValidationFailure / VeleroBackupStale rules.

After this merges, nas-1 has zero cluster-backup dependencies and is clear for the nightly power-off (DSM side: set the power schedule + move the PBS self-backup before shutdown).

🤖 Generated with Claude Code

## Finish line: free nas-1 from cluster-backup monitoring With all backups now on Hetzner (paperless #23/#24, authentik/immich/velero #25, all validated), nas-1's Garage is no longer a cluster-backup dependency. This removes the monitoring tied to it and adds visibility for the new offsite target. ### Removed - Prometheus `garage` scrape job (`s3.nas-1.renner.uno:39033`) + `garage-metrics-token` secret (orphaned). - `prometheusrule-garage.yaml` (`GarageDown` / `GarageScrapeFailing`) — would fire every night once nas-1 powers off. ### Added - `https://fsn1.your-objectstorage.com` to the blackbox-http probe (bare endpoint returns 200). - **`HetznerBackupEndpointDown`** alert — `probe_success == 0 for 10m`, critical. Directly detects the WAN→Hetzner outage that both CNPG WAL archiving and Velero depend on. ### Deliberately not added - A metric-based CNPG "WAL archiving behind" alert: CNPG pod metrics **aren't scraped** (`enablePodMonitor=false`, metric absent), so such an alert would be dead. Enabling CNPG PodMonitors is a worthwhile **follow-up**. Velero outages are already covered by the existing `VeleroBackupValidationFailure` / `VeleroBackupStale` rules. After this merges, nas-1 has **zero cluster-backup dependencies** and is clear for the nightly power-off (DSM side: set the power schedule + move the PBS self-backup before shutdown). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-25 14:09:56 +00:00

monitoring: retire Garage scrape, add Hetzner backup-endpoint alert fe23f18c86

Finish line of the backup migration — nas-1's Garage is no longer a
cluster-backup dependency, so:
- drop the prometheus `garage` scrape job (s3.nas-1.renner.uno:39033) +
  the garage-metrics-token secret (now orphaned)
- delete prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) —
  it would fire every night once nas-1 powers off

Replace it with outage visibility for the new offsite target:
- add https://fsn1.your-objectstorage.com to the blackbox-http probe
  (bare endpoint returns 200)
- new HetznerBackupEndpointDown alert on probe_success == 0 for 10m

Velero outages are already covered by VeleroBackupValidationFailure /
VeleroBackupStale. A granular CNPG "WAL archiving behind" alert needs
CNPG PodMonitors (enablePodMonitor=false today, metrics not scraped) —
left as a follow-up rather than shipping an alert on an absent metric.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 2da8aa0a0d into master

2026-06-25 14:10:39 +00:00

k.renner referenced this issue from a commit

2026-06-25 14:10:40 +00:00

Merge pull request 'monitoring: retire Garage scrape, add Hetzner backup-endpoint alert' (#26) from finish-line-drop-garage into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#26