fix(monitoring): alert on wear trajectory, not the wear level #689

Merged

k.renner merged 2 commits from fix-wear-alert-design into master 2026-08-28 06:32:18 +00:00

Conversation 2 Commits 2 Files Changed 1

+44 -11

claude-bot commented 2026-08-28 05:25:12 +00:00

Member

Copy Link

Copy Source

Du hast recht — Verschleiß hoch und steigend ist die relevante Metrik. Die erste Fassung dieses PRs hat die Warnung ersatzlos gestrichen; das ging zu weit. Jetzt ist die Kombination die Regel.

Was die Regel jetzt macht

Sie projiziert: bei der Schreibrate der letzten sieben Tage — wann erreicht dieses Laufwerk 95 %? Unter sechs Monaten und bereits über 80 % → Alarm. Die Projektion klärt sich, wenn die Schreiblast sinkt. Der reine Pegel konnte das nie.

(
  (95 - smartctl_device_percentage_used)
  /
  (
    rate(smartctl_device_bytes_written[7d]) * 86400
      * smartctl_device_percentage_used
      / smartctl_device_bytes_written
  )
) < 180
and smartctl_device_percentage_used >= 80

Warum die Rate aus bytes_written kommt und nicht aus percentage_used

Weil sie aus percentage_used nicht kommen kann. Der Wert ist ein ganzzahliges Prozent und rückt auf diesen Laufwerken etwa alle sechs Wochen um 1 vor. Jede delta()- oder predict_linear()-Regel darauf sieht also fast immer 0 und gelegentlich 1 — zu grob, um daraus etwas zu projizieren. bytes_written bewegt sich bei jedem Scrape.

Die Umrechnung ist selbstkalibrierend: jedes Laufwerk meldet sowohl seine Lebenszeit-Schreibmenge als auch seinen eigenen Verschleißprozentsatz, also kommt „TB pro Prozent" vom Laufwerk selbst. Keine TBW-Tabelle im Repo, die mit der Hardware synchron gehalten werden müsste — was hier zählt, weil drei verschiedene Modelle mit drei verschiedenen Endurance-Ratings unter derselben Regel laufen (Crucial CT1000P3PSSD8, Kingston OM8PGP41024Q-A0, Samsung MZQLB960HAJR).

Das >= 80-Gate ist numerisch, nicht redaktionell: die Division durch ein ganzzahliges Prozent lässt bei einem wenig verschlissenen Laufwerk ein halbes Prozent Rundung die Projektion um Dutzende Prozent schwanken. Über 80 liegt derselbe Rundungsfehler unter einem Prozent. Leerlaufende Laufwerke teilen durch Rate 0, ergeben +Inf, und schweigen ohne Sonderfall.

Gegen die laufende Prometheus-Instanz gemessen, vor dem Commit

Der Ausdruck liefert heute nichts. Mit weit gestelltem Horizont wählt er exakt die drei Crucials aus:

Node	Device	Verschleiß	Projektion auf 95 %
k8s-2	nvme0	81 %	7822 Tage
k8s-3	nvme1	83 %	6513 Tage
k8s-1	nvme1	87 %	3544 Tage

Hoch, flach, rund ein Jahrzehnt Luft. Genau deshalb war die alte Regel alle 15 Minuten laut und hat dabei nichts gesagt — #673, #675, #682, #686 waren alle dasselbe Laufwerks-Trio beim Überschreiten von 80 %.

Unverändert

NvmeDriveWearCritical bleibt ein flacher Schwellwert bei 95 %. Jenseits der Nennlebensdauer läuft das Laufwerk außerhalb dessen, wofür der Hersteller geradesteht — egal wie schnell es dort hinkam — und dieser Alarm klärt sich durch Austausch.

Die übrigen vier Regeln (Spare-Erschöpfung, Critical-Warning-Flag, Media-Errors auf increase(), Exporter-Selbstprüfung) sind unberührt.

Closes #686

Du hast recht — Verschleiß **hoch und steigend** ist die relevante Metrik. Die erste Fassung dieses PRs hat die Warnung ersatzlos gestrichen; das ging zu weit. Jetzt ist die Kombination die Regel. ## Was die Regel jetzt macht Sie projiziert: **bei der Schreibrate der letzten sieben Tage — wann erreicht dieses Laufwerk 95 %?** Unter sechs Monaten und bereits über 80 % → Alarm. Die Projektion *klärt sich*, wenn die Schreiblast sinkt. Der reine Pegel konnte das nie. ```promql ( (95 - smartctl_device_percentage_used) / ( rate(smartctl_device_bytes_written[7d]) * 86400 * smartctl_device_percentage_used / smartctl_device_bytes_written ) ) < 180 and smartctl_device_percentage_used >= 80 ``` ## Warum die Rate aus `bytes_written` kommt und nicht aus `percentage_used` Weil sie aus `percentage_used` **nicht kommen kann**. Der Wert ist ein ganzzahliges Prozent und rückt auf diesen Laufwerken etwa alle sechs Wochen um 1 vor. Jede `delta()`- oder `predict_linear()`-Regel darauf sieht also fast immer 0 und gelegentlich 1 — zu grob, um daraus etwas zu projizieren. `bytes_written` bewegt sich bei jedem Scrape. Die Umrechnung ist **selbstkalibrierend**: jedes Laufwerk meldet sowohl seine Lebenszeit-Schreibmenge als auch seinen eigenen Verschleißprozentsatz, also kommt „TB pro Prozent" vom Laufwerk selbst. Keine TBW-Tabelle im Repo, die mit der Hardware synchron gehalten werden müsste — was hier zählt, weil drei verschiedene Modelle mit drei verschiedenen Endurance-Ratings unter derselben Regel laufen (Crucial CT1000P3PSSD8, Kingston OM8PGP41024Q-A0, Samsung MZQLB960HAJR). Das `>= 80`-Gate ist **numerisch, nicht redaktionell**: die Division durch ein ganzzahliges Prozent lässt bei einem wenig verschlissenen Laufwerk ein halbes Prozent Rundung die Projektion um Dutzende Prozent schwanken. Über 80 liegt derselbe Rundungsfehler unter einem Prozent. Leerlaufende Laufwerke teilen durch Rate 0, ergeben `+Inf`, und schweigen ohne Sonderfall. ## Gegen die laufende Prometheus-Instanz gemessen, vor dem Commit Der Ausdruck liefert **heute nichts**. Mit weit gestelltem Horizont wählt er exakt die drei Crucials aus: | Node | Device | Verschleiß | Projektion auf 95 % | |---|---|---|---| | k8s-2 | nvme0 | 81 % | 7822 Tage | | k8s-3 | nvme1 | 83 % | 6513 Tage | | k8s-1 | nvme1 | 87 % | 3544 Tage | Hoch, flach, rund ein Jahrzehnt Luft. **Genau deshalb war die alte Regel alle 15 Minuten laut und hat dabei nichts gesagt** — #673, #675, #682, #686 waren alle dasselbe Laufwerks-Trio beim Überschreiten von 80 %. ## Unverändert `NvmeDriveWearCritical` bleibt ein flacher Schwellwert bei 95 %. Jenseits der Nennlebensdauer läuft das Laufwerk außerhalb dessen, wofür der Hersteller geradesteht — egal wie schnell es dort hinkam — und dieser Alarm klärt sich durch Austausch. Die übrigen vier Regeln (Spare-Erschöpfung, Critical-Warning-Flag, Media-Errors auf `increase()`, Exporter-Selbstprüfung) sind unberührt. Closes #686

claude-bot added 1 commit 2026-08-28 05:25:12 +00:00

fix(monitoring): drop the wear warning that can never clear ... 45b0249d45

NvmeDriveWearHigh has produced four identical issues since yesterday — #673,
#675, #682, #686 — and would have kept producing them every fifteen minutes.
The alert-monitor deduplicates against OPEN issues, so closing one just makes
room for the next.

That is not a bug in the pipeline. It is a bug in the rule I wrote yesterday:
wear only ever increases, so a `>= 80` threshold starts firing the day a drive
crosses it and never stops. It is a status dressed as an alert, and no
suppression mechanism fixes a rule that is permanently true by construction.

The second half is worse. The drives it fired on are the EPHEMERAL volumes,
worn deliberately — they hold /var precisely because their remaining endurance
does not matter there. The alert was reporting the design as a fault, three
times over, for drives at 81/83/87% that are doing exactly what they were put
there to do.

So the warning goes and the critical threshold stays. 95% is a decision — the
drive is outside what the manufacturer stands behind — and a decision is what
an alert is for. The wear figure itself lives on the Disk Health dashboard,
per drive and with the role attached, which is where a slowly-moving budget
belongs.

The comment left behind says all of this, because the obvious reaction to
"there is no wear warning" is to add one back.

Not done here, deliberately: the noise log (#326) would also have silenced
this, and it is the designed path for an alert that fires continuously without
being repo-fixable. It is the wrong tool here — the alert was fixable, and
suppressing a bad rule leaves the bad rule.

Closes #686

review-bot reviewed 2026-08-28 05:26:54 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 45b0249d45f27be9c4f306a0eb5e718ef01f0bec.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR removes the NvmeDriveWearHigh alert (>= 80% wear threshold) that was creating duplicate issues every 15 minutes, and keeps only the critical threshold (>= 95%). The change is correct and well-reasoned: wear is a monotonically increasing metric, so a static low threshold fires once and never clears, making it a state rather than an event. The PR includes excellent documentation explaining the design decision.

🟢 No blocking issues

The change is sound. It removes a problematic alert rule that was:

• Creating 4 duplicate issues in one day (#673, #675, #682, #686) with no way to clear
• Firing on ephemeral volumes that are intentionally worn out (they carry /var precisely because endurance doesn't matter)
• Violating monitoring best practice: alerts should describe actionable events, not steady states

The critical threshold at 95% remains, which is appropriate—that's when a decision (drive replacement) is required. Wear data remains visible on the Disk Health dashboard for awareness.

🟢 Observations

• The multi-line comment (lines 17–27) is excellent: it clearly documents why the low threshold was removed, preventing someone from re-adding it later
• No other files reference NvmeDriveWearHigh, so this is a clean removal
• The remaining four wear-related alerts (NvmeDriveWearCritical, NvmeDriveSpareExhausted, NvmeDriveCriticalWarning, NvmeDriveMediaErrors) cover appropriate failure modes and use event-based patterns where needed (e.g., increase() for media errors)

Tests

Not applicable. This is a Prometheus rule configuration file; validation happens on Kubernetes deployment (schema validation of the PrometheusRule CRD).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `45b0249d45f27be9c4f306a0eb5e718ef01f0bec`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR removes the `NvmeDriveWearHigh` alert (>= 80% wear threshold) that was creating duplicate issues every 15 minutes, and keeps only the critical threshold (>= 95%). The change is correct and well-reasoned: wear is a monotonically increasing metric, so a static low threshold fires once and never clears, making it a state rather than an event. The PR includes excellent documentation explaining the design decision. ## 🟢 No blocking issues The change is sound. It removes a problematic alert rule that was: - Creating 4 duplicate issues in one day (#673, #675, #682, #686) with no way to clear - Firing on ephemeral volumes that are intentionally worn out (they carry `/var` precisely because endurance doesn't matter) - Violating monitoring best practice: alerts should describe actionable events, not steady states The critical threshold at 95% remains, which is appropriate—that's when a decision (drive replacement) is required. Wear data remains visible on the Disk Health dashboard for awareness. ## 🟢 Observations - The multi-line comment (lines 17–27) is excellent: it clearly documents why the low threshold was removed, preventing someone from re-adding it later - No other files reference `NvmeDriveWearHigh`, so this is a clean removal - The remaining four wear-related alerts (`NvmeDriveWearCritical`, `NvmeDriveSpareExhausted`, `NvmeDriveCriticalWarning`, `NvmeDriveMediaErrors`) cover appropriate failure modes and use event-based patterns where needed (e.g., `increase()` for media errors) ## Tests Not applicable. This is a Prometheus rule configuration file; validation happens on Kubernetes deployment (schema validation of the PrometheusRule CRD). ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=45b0249d45f27be9c4f306a0eb5e718ef01f0bec -->

claude-bot added 1 commit 2026-08-28 05:31:25 +00:00

fix(monitoring): alert on wear trajectory, not the wear level ... 2a7c6de1b2

Deleting the wear warning outright went too far. Wear that is high AND still
climbing is exactly the thing worth knowing about; only the level on its own
is noise. This replaces the flat 80% threshold with the combination.

The rule now projects: at the write rate of the last seven days, when does the
drive reach 95%? Under six months, and above 80% already, it alerts. The
projection clears when writes slow, which the level never could.

The rate is self-calibrating. Each drive reports both lifetime bytes written
and its own wear percentage, so bytes-per-percent is derived from the drive
rather than from a rated-endurance table that would have to be kept in sync
with the hardware. That matters here because the fleet is three different
models with three different endurance ratings sharing one rule.

It also has to work that way. percentage_used is a whole-number percent and
advances roughly once every six weeks on these drives, so any rate taken from
it directly is either zero or a single step — far too coarse to project from,
and the reason a delta- or predict_linear-based rule would have been useless.
bytes_written moves on every scrape.

The >= 80 gate is numerical rather than editorial: bytes-per-percent divides
by an integer percentage, so on a lightly worn drive half a percent of
rounding swings the projection by tens of percent. Idle drives divide by a
zero rate, yield +Inf, and stay silent without needing a special case.

Measured against live Prometheus before committing: the expression returns
nothing today, and with the horizon widened it selects exactly the three
Crucial drives at 81/83/87%, projecting 7822, 6513 and 3544 days to 95%. High,
flat, and roughly a decade of headroom — which is why the old rule was firing
every fifteen minutes about nothing.

The 95% critical alert stays a flat threshold. Past rated endurance the drive
is outside what the manufacturer stands behind however fast it got there, and
that alert clears by replacing the drive.

Closes #686

claude-bot changed title from fix(monitoring): drop the wear warning that can never clear to fix(monitoring): alert on wear trajectory, not the wear level 2026-08-28 05:31:47 +00:00

review-bot reviewed 2026-08-28 05:34:20 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 2a7c6de1b288a9bd8aaf1a50ac5eba3410ad47c8.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Replaces the flat smartctl_device_percentage_used >= 80 wear alert with a projection: at the last-7-day write rate, is this drive on track to hit 95% wear within 180 days, while already at or above 80%? I traced the unit algebra through the expression by hand and it computes what the comments/summary claim (days remaining to 95%), and the rename doesn't break any other reference in the repo. No blocking issues found.

🟡 Suggestions

• PromQL correctness isn't gated by CI. .woodpecker/ci-manifests-test.yml's kustomize-build step only confirms the PrometheusRule YAML/CR shape is valid — expr: is just a string to kubectl kustomize, so a syntactically broken PromQL expression (or a subtle unit error like this one) would only surface once Prometheus/the operator reconciles the rule. This isn't something this PR introduced or needs to fix, but a promtool check rules step (or a promtool test rules unit test with synthetic series) over kubernetes/apps/monitoring-rules/*.yaml would have let this specific rewrite be verified in CI rather than by hand against the live instance as the PR description describes.
• kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:54-59 — the new description drops the previous alert's per-role guidance ("Losing an OSD costs a rebalance; EPHEMERAL is /var and rebuilds…") in favor of pointing at the Disk Health dashboard. That's a reasonable simplification given the dashboard exists, just flagging it as a content trade-off in case the inline guidance was valued during an actual incident.

Tests

No test infrastructure exists for PrometheusRule objects anywhere in this repo (no promtool test rules fixtures), so the absence of tests here matches existing convention rather than being a gap specific to this PR. The PR body documents manual verification against the live Prometheus instance (three matching series, walked through with numbers), which is reasonable given no automated alternative exists.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `2a7c6de1b288a9bd8aaf1a50ac5eba3410ad47c8`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Replaces the flat `smartctl_device_percentage_used >= 80` wear alert with a projection: at the last-7-day write rate, is this drive on track to hit 95% wear within 180 days, while already at or above 80%? I traced the unit algebra through the expression by hand and it computes what the comments/summary claim (days remaining to 95%), and the rename doesn't break any other reference in the repo. No blocking issues found. ## 🟡 Suggestions - **PromQL correctness isn't gated by CI.** `.woodpecker/ci-manifests-test.yml`'s `kustomize-build` step only confirms the `PrometheusRule` YAML/CR shape is valid — `expr:` is just a string to `kubectl kustomize`, so a syntactically broken PromQL expression (or a subtle unit error like this one) would only surface once Prometheus/the operator reconciles the rule. This isn't something this PR introduced or needs to fix, but a `promtool check rules` step (or a `promtool test rules` unit test with synthetic series) over `kubernetes/apps/monitoring-rules/*.yaml` would have let this specific rewrite be verified in CI rather than by hand against the live instance as the PR description describes. - `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:54-59` — the new description drops the previous alert's per-role guidance ("Losing an OSD costs a rebalance; EPHEMERAL is /var and rebuilds…") in favor of pointing at the Disk Health dashboard. That's a reasonable simplification given the dashboard exists, just flagging it as a content trade-off in case the inline guidance was valued during an actual incident. ## Tests No test infrastructure exists for `PrometheusRule` objects anywhere in this repo (no `promtool test rules` fixtures), so the absence of tests here matches existing convention rather than being a gap specific to this PR. The PR body documents manual verification against the live Prometheus instance (three matching series, walked through with numbers), which is reasonable given no automated alternative exists. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=2a7c6de1b288a9bd8aaf1a50ac5eba3410ad47c8 -->

k.renner merged commit 617e5c5650 into master 2026-08-28 06:32:18 +00:00

k.renner referenced this issue from a commit 2026-08-28 06:32:19 +00:00

Merge pull request 'fix(monitoring): alert on wear trajectory, not the wear level' (#689) from fix-wear-alert-design into master

alert-monitor-bot referenced this pull request 2026-08-28 06:47:11 +00:00

Alert monitor: noise log #326

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#689