monitoring: stop wear alerts on EPHEMERAL NVMe drives (REN-419) #1088

Merged
Ghost merged 2 commits from atlas/ren-419-ephemeral-wear-alert into master 2026-10-01 09:19:19 +00:00
2 Commits
Author SHA1 Message Date
atlas-botandClaude Opus 5.5 42e67219e0 monitoring: feed disk-wear case A at 5m so the 6h for completes (REN-438)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Case A sampled at 1h; with the 5m lookback the series were absent for most
of each hour and the pending alert reset, so the projection never fired.
Now 5m samples (1m exceeds promtool's sample limit on the 7d rate), same
0.04272 %/day and 140-day summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:07:58 +00:00
atlas-botandClaude Opus 5.5 18629fd496 monitoring: stop wear alerts on EPHEMERAL NVMe drives (REN-419)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline failed
ci/woodpecker/pr/pr-review Pipeline was successful
The owner decided EPHEMERAL (/var) drives run to failure (REN-420), so
NvmeDriveWearProjectedExhaustion and NvmeDriveWearCritical now skip the
drive carrying /var. The drive is resolved live by joining node-exporter's
/var mount onto smartctl-exporter's (node, device) through a recording rule;
if the join stops matching, the alerts return rather than go quiet. Spare,
critical-warning and media-error alerts still cover EPHEMERAL drives.

Adds a promtool suite and four negative controls for the exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 05:40:32 +00:00