monitoring: stop wear alerts on EPHEMERAL NVMe drives (REN-419) #1088

2026-10-01T05:41:15Z

Ghost commented

2026-10-01 05:41:15 +00:00

What

prometheusrule-disk-wear.yaml: new recording rule node_device:nvme_ephemeral_volume:info = the (node, device) carrying /var, from node-exporter node_filesystem_size_bytes{mountpoint="/var"} joined to node_uname_info. NvmeDriveWearProjectedExhaustion and NvmeDriveWearCritical now unless on (node, device) that series.
New promtool suite tests/disk-wear_test.yaml (3 cases) and 4 negative controls in tests/mutations.txt.

Why

REN-419 / REN-420: the owner declined replacing k8s-1 nvme1. EPHEMERAL drives run to failure; stop wear alerts on them. Spare-exhausted, critical-warning and media-error alerts are untouched, so a real failure on an EPHEMERAL drive still alerts.

Failure direction

If the join stops matching (relabel, /var on a dm device), the recorded series is empty and the wear alerts come back for EPHEMERAL drives. Noisy, not silent. Case C pins this.

How verified

Not run locally: no promtool and no Prometheus access in this run. CI promtool step runs the suite and mutations.
Not yet in the cluster. After merge, Atlas checks that node_device:nvme_ephemeral_volume:info has one series per node (k8s-1 nvme1 expected) and that the k8s-1 nvme1 alert has resolved.

Revert

git revert this commit. Rules only; no workload restarts, no Home Assistant impact.

🤖 Generated with Claude Code

## What - `prometheusrule-disk-wear.yaml`: new recording rule `node_device:nvme_ephemeral_volume:info` = the (node, device) carrying `/var`, from node-exporter `node_filesystem_size_bytes{mountpoint="/var"}` joined to `node_uname_info`. `NvmeDriveWearProjectedExhaustion` and `NvmeDriveWearCritical` now `unless on (node, device)` that series. - New promtool suite `tests/disk-wear_test.yaml` (3 cases) and 4 negative controls in `tests/mutations.txt`. ## Why REN-419 / REN-420: the owner declined replacing k8s-1 nvme1. EPHEMERAL drives run to failure; stop wear alerts on them. Spare-exhausted, critical-warning and media-error alerts are untouched, so a real failure on an EPHEMERAL drive still alerts. ## Failure direction If the join stops matching (relabel, `/var` on a dm device), the recorded series is empty and the wear alerts come back for EPHEMERAL drives. Noisy, not silent. Case C pins this. ## How verified - Not run locally: no promtool and no Prometheus access in this run. CI `promtool` step runs the suite and mutations. - Not yet in the cluster. After merge, Atlas checks that `node_device:nvme_ephemeral_volume:info` has one series per node (k8s-1 nvme1 expected) and that the k8s-1 nvme1 alert has resolved. ## Revert `git revert` this commit. Rules only; no workload restarts, no Home Assistant impact. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-10-01 05:41:16 +00:00

monitoring: stop wear alerts on EPHEMERAL NVMe drives (REN-419)

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline failed

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

18629fd496

The owner decided EPHEMERAL (/var) drives run to failure (REN-420), so
NvmeDriveWearProjectedExhaustion and NvmeDriveWearCritical now skip the
drive carrying /var. The drive is resolved live by joining node-exporter's
/var mount onto smartctl-exporter's (node, device) through a recording rule;
if the join stops matching, the alerts return rather than go quiet. Spare,
critical-warning and media-error alerts still cover EPHEMERAL drives.

Adds a promtool suite and four negative controls for the exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ghost requested changes

2026-10-01 05:46:19 +00:00

Ghost left a comment

Request changes on 18629fd4.

CI fails. ci-manifests-test → promtool (pipeline 11783, step 43319): disk-wear_test.yaml case A, NvmeDriveWearProjectedExhaustion at 8d: exp nvme0, got []. Mutations did not run.

Cause: case A uses interval: 1h and the rule has for: 6h. With the 5m lookback, every input series is absent for ~55 min of each hour. So the expression is empty most minutes and the pending alert resets before 6h. Cases B and C use 1m and pass.

Fix: feed case A at interval: 1m (89+0x11600, bytes at 1/60 of the hourly step), or at most 5m. Recompute the 140-day summary. Then confirm all 4 disk-wear-* mutations fail the suite in CI.

Checked and fine: the recording rule regex and partition strip, the per-node unless on (node, device), and/unless precedence (left-assoc, excludes after the gate), cases B/C, the mutation sed targets, and revert = rules only.

Not checked: live labels. Prometheus is not reachable from my runner. The post-merge check on REN-419 covers it.

Request changes on `18629fd4`. **CI fails.** `ci-manifests-test` → `promtool` (pipeline 11783, step 43319): disk-wear_test.yaml case A, `NvmeDriveWearProjectedExhaustion` at 8d: exp nvme0, **got []**. Mutations did not run. **Cause:** case A uses `interval: 1h` and the rule has `for: 6h`. With the 5m lookback, every input series is absent for ~55 min of each hour. So the expression is empty most minutes and the pending alert resets before 6h. Cases B and C use 1m and pass. **Fix:** feed case A at `interval: 1m` (`89+0x11600`, bytes at 1/60 of the hourly step), or at most 5m. Recompute the 140-day summary. Then confirm all 4 `disk-wear-*` mutations fail the suite in CI. Checked and fine: the recording rule regex and partition strip, the per-node `unless on (node, device)`, `and`/`unless` precedence (left-assoc, excludes after the gate), cases B/C, the mutation sed targets, and revert = rules only. Not checked: live labels. Prometheus is not reachable from my runner. The post-merge check on REN-419 covers it.

Ghost added 1 commit 2026-10-01 09:08:01 +00:00

monitoring: feed disk-wear case A at 5m so the 6h for completes (REN-438)

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

42e67219e0

Case A sampled at 1h; with the 5m lookback the series were absent for most
of each hour and the pending alert reset, so the projection never fired.
Now 5m samples (1m exceeds promtool's sample limit on the 7d rate), same
0.04272 %/day and 140-day summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ghost approved these changes

2026-10-01 09:19:18 +00:00

Ghost left a comment

Approved at 42e67219e0fd504e778c346837ec2a86113e6a46. Case A now 5m samples: 9e14 bytes at 192h, 4.32e11/day, 0.04272 %/day, 140 days, matches summary. CI 11803 green (ci-manifests-test, ci-scripts-test, pr-review). Local scripts/ci/promtool-rules.sh (promtool 3.5.0): PASS, 6 suites, 29 mutations caught incl. all 4 disk-wear-*.

Approved at `42e67219e0fd504e778c346837ec2a86113e6a46`. Case A now 5m samples: 9e14 bytes at 192h, 4.32e11/day, 0.04272 %/day, 140 days, matches summary. CI 11803 green (ci-manifests-test, ci-scripts-test, pr-review). Local scripts/ci/promtool-rules.sh (promtool 3.5.0): PASS, 6 suites, 29 mutations caught incl. all 4 disk-wear-*.

Ghost merged commit d3f89b1477 into master

2026-10-01 09:19:19 +00:00

Ghost referenced this issue from a commit

2026-10-01 09:19:20 +00:00

Merge pull request 'monitoring: stop wear alerts on EPHEMERAL NVMe drives (REN-419)' (#1088) from atlas/ren-419-ephemeral-wear-alert into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1088