konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] NvmeDriveWearHigh (monitoring) #682

New Issue
Closed
opened 2026-08-28 04:35:11 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-28 04:35:11 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: NvmeDriveWearHigh, severity warning, namespace monitoring (job smartctl-exporter)
  • Subject: device nvme1 on node k8s-1 (10.244.0.79:9633), 87% of rated write endurance used
  • Since: 2026-08-27 14:21 UTC, continuously active — not silenced, not inhibited, routed to the email receiver

This alert can never resolve on its own: wear is monotonic, and the rule has a plain >= 80 threshold with no per-drive exception.

Evidence

Prometheus instant query via the apiserver proxy (kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=smartctl_device_percentage_used") — three of the nine drives in the fleet are over the threshold, the rest are at 13% or below:

node device wear
k8s-1 nvme1 87%
k8s-3 nvme1 83%
k8s-2 nvme0 81%

Alertmanager (/api/v2/alerts?filter=alertname="NvmeDriveWearHigh") shows all three firing since 2026-08-27 ~14:21 UTC. They share one issue_title, so this issue covers the k8s-1 instance as the worst case and the other two as the same condition.

Same query for smartctl_device_media_errors: k8s-1/nvme1 = 18, every other drive in the fleet = 0.

context.txt shows nothing else implicated: all three nodes Ready, no Flux kustomization or helmrelease unready, and the only non-running pods / warning events are unrelated (woodpecker-ci and velero pods with unbound PVCs). Nothing is failing because of this drive right now.

Root cause

The wear number itself is expected and deliberate, not a regression. kubernetes/talos/patches/k8s-1.yaml documents that EPHEMERAL (/var) was deliberately placed on the retired Crucial P3 — the drive that used to be this node's Ceph OSD — precisely because its contents are reconstructible. k8s-2.yaml and k8s-3.yaml record the same arrangement at 81% and 83%. So the alert is reporting reality accurately, and the reality is a design decision that was already made.

Two things still need a human, though:

  1. The media errors, on this node specifically. k8s-1's P3 is the only drive in the fleet with a non-zero smartctl_device_media_errors (18), and on this node /var/lib/etcd lives on EPHEMERAL — i.e. on that drive. Whether those 18 are healed-over damage from its OSD career or an ongoing trend is still not established. Prometheus cannot settle it: count_over_time(smartctl_device_media_errors{node="k8s-1",device="nvme1"}[7d]) returns 912 samples at the ServiceMonitor's 60s interval ≈ 15h of history, matching the current exporter pod's age (kubectl get pods -n monitoring -l app.kubernetes.io/name=smartctl-exporter, AGE 15h). Over that window max_over_time - min_over_time is 0 for both the error count and the wear percentage — flat, but 15h of flat is not evidence of stability for a counter that only moves on media faults.
    • Not fixable in the repo. Settle it on the node: talosctl -n k8s-1 read /dev/… is not it — run smartctl -a /dev/nvme1n1 on k8s-1 and compare the error-log entry timestamps against power-on hours. If the entries are recent, EPHEMERAL belongs on the system disk instead, which per the same file means a wipe and a reinstall from install media (Talos applies VolumeConfig only at first provision).
  2. The alert will re-file this issue indefinitely. Because wear only goes up, closing this issue simply produces an identical one on the next monitor run — the previous round (#675 / merged PR #679) documented the arrangement but naturally did not change the metric. A decision is needed on what NvmeDriveWearHigh should mean now that three drives are knowingly parked above the threshold.

Where to fix it

  • kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml — the NvmeDriveWearHigh rule (smartctl_device_percentage_used >= 80, for: 1h). Options, in the repo's own terms: raise the warning threshold above the known-parked drives so it becomes a fresh-news alert again; or scope it to exclude the three EPHEMERAL P3s by device/serial; or accept that NvmeDriveWearCritical (>= 95, unchanged) is the real actionable line and demote/suppress the warning tier. NvmeDriveSpareExhausted, NvmeDriveCriticalWarning and NvmeDriveMediaErrors stay as-is either way — they are the rules that would catch actual degradation.
  • kubernetes/apps/monitoring/helmrelease-prometheus.yaml (route at lines 292–321) — if the decision is "known and accepted", this is where a receiver: 'null' sub-route would go, alongside the existing Watchdog and KubeCPUOvercommit entries. Note the cost: that also hides new drives crossing 80%, so prefer a threshold/scoping change in the rule over a blanket route drop.
  • kubernetes/talos/patches/k8s-1.yaml (EPHEMERAL VolumeConfig, and the media-error note above it) — only if the smartctl -a check comes back showing recent errors and EPHEMERAL has to move off this drive.

No change is needed to reduce the wear number; the drive is consumable in its current role by design. What this issue is asking for is (a) the on-node media-error verdict for k8s-1, and (b) a rule/route decision so this alert stops being permanently true.

Filed automatically by the alert-monitor pipeline (2026-08-28). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `NvmeDriveWearHigh`, severity `warning`, namespace `monitoring` (job `smartctl-exporter`) - **Subject:** device `nvme1` on node **k8s-1** (`10.244.0.79:9633`), 87% of rated write endurance used - **Since:** 2026-08-27 14:21 UTC, continuously active — not silenced, not inhibited, routed to the `email` receiver This alert can never resolve on its own: wear is monotonic, and the rule has a plain `>= 80` threshold with no per-drive exception. ## Evidence Prometheus instant query via the apiserver proxy (`kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=smartctl_device_percentage_used"`) — three of the nine drives in the fleet are over the threshold, the rest are at 13% or below: | node | device | wear | |---|---|---| | k8s-1 | nvme1 | **87%** | | k8s-3 | nvme1 | 83% | | k8s-2 | nvme0 | 81% | Alertmanager (`/api/v2/alerts?filter=alertname="NvmeDriveWearHigh"`) shows all three firing since 2026-08-27 ~14:21 UTC. They share one `issue_title`, so this issue covers the k8s-1 instance as the worst case and the other two as the same condition. Same query for `smartctl_device_media_errors`: **k8s-1/nvme1 = 18, every other drive in the fleet = 0.** `context.txt` shows nothing else implicated: all three nodes `Ready`, no Flux kustomization or helmrelease unready, and the only non-running pods / warning events are unrelated (woodpecker-ci and velero pods with unbound PVCs). Nothing is failing because of this drive right now. ## Root cause The wear number itself is **expected and deliberate**, not a regression. `kubernetes/talos/patches/k8s-1.yaml` documents that EPHEMERAL (`/var`) was deliberately placed on the retired Crucial P3 — the drive that used to be this node's Ceph OSD — precisely because its contents are reconstructible. `k8s-2.yaml` and `k8s-3.yaml` record the same arrangement at 81% and 83%. So the alert is reporting reality accurately, and the reality is a design decision that was already made. Two things still need a human, though: 1. **The media errors, on this node specifically.** k8s-1's P3 is the only drive in the fleet with a non-zero `smartctl_device_media_errors` (18), and on this node `/var/lib/etcd` lives on EPHEMERAL — i.e. on that drive. Whether those 18 are healed-over damage from its OSD career or an ongoing trend is **still not established**. Prometheus cannot settle it: `count_over_time(smartctl_device_media_errors{node="k8s-1",device="nvme1"}[7d])` returns 912 samples at the ServiceMonitor's 60s interval ≈ 15h of history, matching the current exporter pod's age (`kubectl get pods -n monitoring -l app.kubernetes.io/name=smartctl-exporter`, AGE 15h). Over that window `max_over_time - min_over_time` is `0` for both the error count and the wear percentage — flat, but 15h of flat is not evidence of stability for a counter that only moves on media faults. - **Not fixable in the repo.** Settle it on the node: `talosctl -n k8s-1 read /dev/…` is not it — run `smartctl -a /dev/nvme1n1` on k8s-1 and compare the error-log entry timestamps against power-on hours. If the entries are recent, EPHEMERAL belongs on the system disk instead, which per the same file means a wipe and a reinstall from install media (Talos applies `VolumeConfig` only at first provision). 2. **The alert will re-file this issue indefinitely.** Because wear only goes up, closing this issue simply produces an identical one on the next monitor run — the previous round (#675 / merged PR #679) documented the arrangement but naturally did not change the metric. A decision is needed on what `NvmeDriveWearHigh` should mean now that three drives are knowingly parked above the threshold. ## Where to fix it - `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml` — the `NvmeDriveWearHigh` rule (`smartctl_device_percentage_used >= 80`, `for: 1h`). Options, in the repo's own terms: raise the warning threshold above the known-parked drives so it becomes a fresh-news alert again; or scope it to exclude the three EPHEMERAL P3s by device/serial; or accept that `NvmeDriveWearCritical` (`>= 95`, unchanged) is the real actionable line and demote/suppress the warning tier. `NvmeDriveSpareExhausted`, `NvmeDriveCriticalWarning` and `NvmeDriveMediaErrors` stay as-is either way — they are the rules that would catch actual degradation. - `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` (route at lines 292–321) — if the decision is "known and accepted", this is where a `receiver: 'null'` sub-route would go, alongside the existing `Watchdog` and `KubeCPUOvercommit` entries. Note the cost: that also hides *new* drives crossing 80%, so prefer a threshold/scoping change in the rule over a blanket route drop. - `kubernetes/talos/patches/k8s-1.yaml` (EPHEMERAL `VolumeConfig`, and the media-error note above it) — only if the `smartctl -a` check comes back showing recent errors and EPHEMERAL has to move off this drive. No change is needed to reduce the wear number; the drive is consumable in its current role by design. What this issue is asking for is (a) the on-node media-error verdict for k8s-1, and (b) a rule/route decision so this alert stops being permanently true. --- <!-- alert-monitor:fingerprint=c292db90b2c94fc8 --> _Filed automatically by the alert-monitor pipeline (2026-08-28). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-28 04:45:54 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change made

NvmeDriveWearHigh is reporting accurately, and everything it reports is
pre-existing, already-documented, deliberately-accepted state. The two things
issue #682 actually asks for are (a) an on-node smartctl -a verdict on k8s-1's
media errors, which the issue itself correctly calls "not fixable in the repo",
and (b) a monitoring-policy decision about what the warning tier should mean
now that three drives are knowingly parked above 80%. The issue lays out three
mutually exclusive options and explicitly says "a decision is needed" — it does
not make that decision, and there is no Verified owner guidance section on
this issue authorizing any of them. Picking one myself would mean raising a
threshold or adding a route drop on my own authority, which is exactly what the
fix agent's rules prohibit absent the issue establishing the rule is wrong. So I
made no edits. Below is what I verified live, one fact the issue did not
establish, and the decision laid out so it can be settled with a one-line reply.

What I verified live this run

All via ./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=...".

  • The issue's wear table is exactly right. smartctl_device_percentage_used:
    k8s-1/nvme1 87, k8s-3/nvme1 83, k8s-2/nvme0 81; every other drive ≤ 13.
  • The media-error claim is exactly right. smartctl_device_media_errors:
    k8s-1/nvme1 18, all eight other drives 0.
  • The drive is not in active failure. smartctl_device_available_spare = 100
    against a threshold of 5, and smartctl_device_critical_warning = 0. So
    NvmeDriveSpareExhausted and NvmeDriveCriticalWarning are correctly quiet —
    the 18 errors are not currently accompanied by block remapping or a
    drive-raised fault flag.
  • k8s-1/nvme1 has 20181 power-on hours (smartctl_device_power_on_seconds / 3600),
    ~2.3 years, consistent with the retired-OSD history in
    kubernetes/talos/patches/k8s-1.yaml.

One thing the issue did not establish, and it matters

The issue attributed the 15h history limit to the exporter pod's age. I checked
the DaemonSet, not the pod:

./.ci-diag/kubectl get daemonset -n monitoring
smartctl-exporter   created 2026-08-27T12:57:55Z   3/3 ready

Compare the other three DaemonSets in the namespace, all created 2026-05-09.
The smartctl-exporter is ~16 hours old — this is the first time these drives
have ever been monitored. The alert began firing 2026-08-27 ~14:21 UTC, roughly
1h23m after the DaemonSet was created, which is just the rule's for: 1h elapsing.

I also confirmed the missing history is not a label-cardinality artifact of the
pod restarting. Collapsing the pod and instance labels
(count_over_time(max without(pod,instance)(smartctl_device_media_errors{node="k8s-1",device="nvme1"})[30d:1m]))
still returns 922 samples ≈ 15.4h, while the same query against
up{job="node-exporter"} returns 35563 ≈ 24.7 days. Prometheus retention is
fine; the smartctl series simply does not exist before yesterday.

Consequence: this is new monitoring surfacing old state, not a degradation
event — which supports treating the numbers as accepted rather than urgent. It
also means the max_over_time - min_over_time = 0 result (which I reproduced, it
is 0) carries even less weight than the issue suggested: 18 errors accumulated
over 20181 power-on hours cannot be dated from ~16h of observation, by any query.
Prometheus will not settle this in a week either — only the drive's own error log
has the timestamps.

Why no edit, option by option

The issue's three options are not equivalent, and each has a cost I cannot verify
away:

  1. Raise the warning threshold above the parked drives. Buys only 3 points of
    headroom on k8s-1 (87 → 90). Wear is monotonic, so this re-fires permanently in
    a few months and the same decision comes back. It is also the textbook
    "silence by threshold" move.
  2. Scope the three EPHEMERAL P3s out by device/serial. This permanently blinds
    the wear tier on the one drive in the fleet with non-zero media errors — the
    drive that carries /var/lib/etcd on k8s-1 — while its health is still
    unresolved. I would not recommend this one.
  3. Accept NvmeDriveWearCritical (>= 95) as the actionable line and drop the
    warning tier.     Cleanest of the three, and it keeps SpareExhausted,
    CriticalWarning and MediaErrors — the rules that catch real degradation —
    fully intact on all nine drives.

The receiver: 'null' sub-route in helmrelease-prometheus.yaml is the weakest
option, for the reason the issue already gives: it also hides new drives
crossing 80%. I agree and did not pursue it.

If you want my recommendation: option 3, expressed as narrowing the warning
rule rather than deleting it — e.g. keeping a warning tier but starting it above
the accepted band, so a fresh drive entering wear-out still produces news. But
that is your call on what the alert should mean, and it should be made
deliberately rather than inferred by an agent from an alert firing.

Recommended manual action

  1. Settle the media errors on k8s-1 (the only genuinely open technical
    question). On the node, run smartctl -a /dev/nvme1n1 and read the Error
    Information Log     entries, comparing each entry's power-on-hour stamp against
    the current 20181 hours. Entries clustered in the drive's OSD career are healed
    history and the drive stays where it is. Entries within the last few hundred
    hours mean EPHEMERAL should move to the system disk — which, per the comment in
    kubernetes/talos/patches/k8s-1.yaml, requires wiping and reinstalling from
    install media, because Talos applies VolumeConfig only at first provision.
  2. Decide the warning-tier question and say which option you want. Any of the
    three is a ≤10-line change to
    kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml; reply on the
    issue and it can be implemented directly.
  3. Interim, if the email noise is the pressing part: an Alertmanager silence
    scoped to alertname="NvmeDriveWearHigh" with an explicit expiry is reversible
    and leaves the rule intact, unlike any of the repo edits above.

Until (2) is answered, this alert will keep re-filing — that is expected, and
closing this issue without a rule decision will produce an identical one.

Notes

  • Nothing in the issue asked me to touch CI config, credentials, or anything
    outside the edit scope.
  • Every runtime number above came from a read-only query I ran this run; I made no
    live changes and RBAC would have rejected any attempt.
  • The previous round (#675 / PR #679) documented the drive arrangement in the
    Talos patches. That documentation is present and accurate — I re-read
    kubernetes/talos/patches/k8s-1.yaml and it matches the live serial-matched
    EPHEMERAL VolumeConfig. There is nothing left to document there, which is why
    a documentation-only PR would be cosmetic.
  • Local git history is a depth-1 CI clone, so I could not inspect PR #679's diff
    directly — I verified the current file contents instead.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

# Assessment: no repo change made `NvmeDriveWearHigh` is reporting accurately, and everything it reports is pre-existing, already-documented, deliberately-accepted state. The two things issue #682 actually asks for are (a) an on-node `smartctl -a` verdict on k8s-1's media errors, which the issue itself correctly calls "not fixable in the repo", and (b) a **monitoring-policy decision** about what the warning tier should mean now that three drives are knowingly parked above 80%. The issue lays out three mutually exclusive options and explicitly says "a decision is needed" — it does not make that decision, and there is no **Verified owner guidance** section on this issue authorizing any of them. Picking one myself would mean raising a threshold or adding a route drop on my own authority, which is exactly what the fix agent's rules prohibit absent the issue establishing the rule is wrong. So I made no edits. Below is what I verified live, one fact the issue did not establish, and the decision laid out so it can be settled with a one-line reply. ## What I verified live this run All via `./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=..."`. - **The issue's wear table is exactly right.** `smartctl_device_percentage_used`: k8s-1/nvme1 `87`, k8s-3/nvme1 `83`, k8s-2/nvme0 `81`; every other drive ≤ `13`. - **The media-error claim is exactly right.** `smartctl_device_media_errors`: k8s-1/nvme1 `18`, all eight other drives `0`. - **The drive is not in active failure.** `smartctl_device_available_spare` = `100` against a threshold of `5`, and `smartctl_device_critical_warning` = `0`. So `NvmeDriveSpareExhausted` and `NvmeDriveCriticalWarning` are correctly quiet — the 18 errors are not currently accompanied by block remapping or a drive-raised fault flag. - **k8s-1/nvme1 has 20181 power-on hours** (`smartctl_device_power_on_seconds / 3600`), ~2.3 years, consistent with the retired-OSD history in `kubernetes/talos/patches/k8s-1.yaml`. ### One thing the issue did not establish, and it matters The issue attributed the 15h history limit to the exporter *pod's* age. I checked the DaemonSet, not the pod: ``` ./.ci-diag/kubectl get daemonset -n monitoring smartctl-exporter created 2026-08-27T12:57:55Z 3/3 ready ``` Compare the other three DaemonSets in the namespace, all created `2026-05-09`. **The smartctl-exporter is ~16 hours old — this is the first time these drives have ever been monitored.** The alert began firing 2026-08-27 ~14:21 UTC, roughly 1h23m after the DaemonSet was created, which is just the rule's `for: 1h` elapsing. I also confirmed the missing history is not a label-cardinality artifact of the pod restarting. Collapsing the `pod` and `instance` labels (`count_over_time(max without(pod,instance)(smartctl_device_media_errors{node="k8s-1",device="nvme1"})[30d:1m])`) still returns `922` samples ≈ 15.4h, while the same query against `up{job="node-exporter"}` returns `35563` ≈ 24.7 days. Prometheus retention is fine; the smartctl series simply does not exist before yesterday. **Consequence:** this is new monitoring surfacing old state, not a degradation event — which supports treating the numbers as accepted rather than urgent. It also means the `max_over_time - min_over_time = 0` result (which I reproduced, it is `0`) carries even less weight than the issue suggested: 18 errors accumulated over 20181 power-on hours cannot be dated from ~16h of observation, by any query. Prometheus will not settle this in a week either — only the drive's own error log has the timestamps. ## Why no edit, option by option The issue's three options are not equivalent, and each has a cost I cannot verify away: 1. **Raise the warning threshold above the parked drives.** Buys only 3 points of headroom on k8s-1 (87 → 90). Wear is monotonic, so this re-fires permanently in a few months and the same decision comes back. It is also the textbook "silence by threshold" move. 2. **Scope the three EPHEMERAL P3s out by device/serial.** This permanently blinds the wear tier on the one drive in the fleet with non-zero media errors — the drive that carries `/var/lib/etcd` on k8s-1 — while its health is still unresolved. I would not recommend this one. 3. **Accept `NvmeDriveWearCritical` (>= 95) as the actionable line and drop the warning tier.** Cleanest of the three, and it keeps `SpareExhausted`, `CriticalWarning` and `MediaErrors` — the rules that catch real degradation — fully intact on all nine drives. The `receiver: 'null'` sub-route in `helmrelease-prometheus.yaml` is the weakest option, for the reason the issue already gives: it also hides *new* drives crossing 80%. I agree and did not pursue it. **If you want my recommendation:** option 3, expressed as narrowing the warning rule rather than deleting it — e.g. keeping a warning tier but starting it above the accepted band, so a fresh drive entering wear-out still produces news. But that is your call on what the alert should mean, and it should be made deliberately rather than inferred by an agent from an alert firing. ## Recommended manual action 1. **Settle the media errors on k8s-1** (the only genuinely open technical question). On the node, run `smartctl -a /dev/nvme1n1` and read the *Error Information Log* entries, comparing each entry's power-on-hour stamp against the current 20181 hours. Entries clustered in the drive's OSD career are healed history and the drive stays where it is. Entries within the last few hundred hours mean EPHEMERAL should move to the system disk — which, per the comment in `kubernetes/talos/patches/k8s-1.yaml`, requires wiping and reinstalling from install media, because Talos applies `VolumeConfig` only at first provision. 2. **Decide the warning-tier question** and say which option you want. Any of the three is a ≤10-line change to `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml`; reply on the issue and it can be implemented directly. 3. **Interim, if the email noise is the pressing part:** an Alertmanager silence scoped to `alertname="NvmeDriveWearHigh"` with an explicit expiry is reversible and leaves the rule intact, unlike any of the repo edits above. Until (2) is answered, this alert will keep re-filing — that is expected, and closing this issue without a rule decision will produce an identical one. ## Notes - Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit scope. - Every runtime number above came from a read-only query I ran this run; I made no live changes and RBAC would have rejected any attempt. - The previous round (#675 / PR #679) documented the drive arrangement in the Talos patches. That documentation is present and accurate — I re-read `kubernetes/talos/patches/k8s-1.yaml` and it matches the live serial-matched EPHEMERAL `VolumeConfig`. There is nothing left to document there, which is why a documentation-only PR would be cosmetic. - Local git history is a depth-1 CI clone, so I could not inspect PR #679's diff directly — I verified the current file contents instead. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#682