konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] NvmeDriveWearHigh (monitoring) #686

New Issue
Closed
opened 2026-08-28 04:49:23 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-28 04:49:23 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: NvmeDriveWearHigh (severity warning, namespace monitoring)
  • Target: node k8s-1, device nvme1 (instance 10.244.0.79:9633, smartctl-exporter)
  • Since: 2026-08-27T14:21:35Z, receiver email, still active
  • Rule: smartctl_device_percentage_used >= 80 for 1h — kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:16

Evidence

context.txt shows nothing wrong: all three nodes Ready, no pods outside Running/Succeeded, no Flux resource unready. This alert is about a hardware budget, not a broken workload, so that is expected and not a contradiction.

Verified live via the Prometheus API through the apiserver proxy (kubectl get --raw .../api/v1/query?query=smartctl_device_percentage_used) — wear across the whole fleet:

node device model wear
k8s-1 nvme1 CT1000P3PSSD8 87%
k8s-3 nvme1 CT1000P3PSSD8 83%
k8s-2 nvme0 CT1000P3PSSD8 81%
k8s-1/2/3 Kingston OM8PGP4 system disk 10–13%
k8s-1/2/3 Samsung MZQLB960 Ceph OSD 2–3%

ALERTS{alertname=~"Nvme.*"} confirms all three NvmeDriveWearHigh instances are firing. Only k8s-1 reached this batch because the other two dedup to the same issue title, so this one issue stands for all three.

Root cause

The worn drives are the EPHEMERAL volumes, and this is deliberate. Confirmed two independent ways:

  1. kubernetes/talos/patches/k8s-1.yaml:50 pins EPHEMERAL with disk.serial == '2332E86B749A'; node_nvme_info reports that exact serial as k8s-1 nvme1 (CT1000P3PSSD8).
  2. node_filesystem_*{mountpoint="/var"} on k8s-1 reports device /dev/nvme1n1p1 — the same drive.

The drive is not a Ceph OSD: ceph_disk_occupation puts k8s-1's osd.2 on /dev/nvme2n1 (the Samsung, 3% wear). The repo comment at kubernetes/talos/patches/k8s-1.yaml:32 states the intent — EPHEMERAL goes on "the retired Ceph OSD drive" because its contents rebuild — and flags the exception itself: /var/lib/etcd lives there and "is the one thing on the volume that is not free to lose."

So the wear itself is working as designed. Three things make it worth acting on rather than closing:

1. The three EPHEMERAL drives will hit critical together. They are the same model, at 81/83/87%, carrying comparable write volumes (132–154 TB written). NvmeDriveWearCritical (>=95%) will fire on all three within a similar window — a correlated, not independent, failure of the volume that holds every etcd member.

2. k8s-1 nvme1 has 18 media errors; every other drive in the fleet has 0. From smartctl_device_media_errors{node="k8s-1"}. This is the "failure now" signal, distinct from the endurance budget — and it is currently invisible: NvmeDriveMediaErrors alerts on increase(...[6h]) > 0, so a standing count never fires. That is the documented intent (prometheusrule-disk-wear.yaml:66), but the consequence is that the one drive holding etcd carries unrecovered data-integrity errors that no alert reports. The other failure-now signals are clean: available spare 100 (threshold 5), critical warning 0.

3. No trend is available yet. min_over_time(timestamp(...)) puts the earliest smartctl sample at 2026-08-27T13:24:32Z — roughly 15 hours of history. Wear and media errors are both flat across all of it. Hypothesis (unverified): the 18 media errors predate the drive's EPHEMERAL role and date from its earlier life as a Ceph OSD; power-on time is 72,651,600 s (~841 days). This cannot be settled from Prometheus — Ceph's devicehealth registry purges a drive's history when it leaves the cluster, and Talos kernel logs do not reach Loki. talosctl -n 10.2.40.169 dmesg is the only source and it is a ring buffer emptied by reboot.

Also unverified: etcd health could not be checked from monitoring — up{job=~".*etcd.*"} returns no series, so etcd is not scraped by this Prometheus.

Where to fix it

This is not fixable by a repo change alone. It is a hardware-lifetime problem, and the relevant Talos setting cannot be re-applied to a running node.

  • kubernetes/talos/patches/k8s-1.yaml:45-51 (and k8s-2.yaml:39-43, k8s-3.yaml:35-39) — the VolumeConfig / EPHEMERAL diskSelector. The comment at k8s-1.yaml:36 is the constraint: volume configuration only takes effect on first provision; applying a changed selector to an existing node is silently ignored, and relocating EPHEMERAL requires wiping the disk and booting install media. Moving etcd off these drives therefore means a node rebuild, not a config push. The node object ages in context.txt (k8s-1 2d9h, k8s-2 2d6h vs k8s-3 110d) suggest two of the three were rebuilt very recently, i.e. that window just closed.
  • kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml — worth reconsidering the media-errors rule so a standing non-zero count on an etcd-bearing drive is surfaced rather than dashboard-only (e.g. a separate low-severity alert on the absolute value), or recording explicitly that standing counts are accepted.

Suggested human actions (read-only, off-cluster):

  1. talosctl -n 10.2.40.169 dmesg — look for NVMe errors on nvme1 before the buffer is lost to a reboot.
  2. Verify etcd member health on k8s-1 directly (not observable via Prometheus here).
  3. Decide the replacement plan for the three CT1000P3PSSD8 drives together, since they age in lockstep. Replacing the k8s-1 drive first is the natural order: highest wear plus the only media errors.

Nothing is failing right now — spare is untouched and no critical warning is raised. The action this asks for is scheduling, not an emergency.

Filed automatically by the alert-monitor pipeline (2026-08-28). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `NvmeDriveWearHigh` (severity `warning`, namespace `monitoring`) - **Target:** node `k8s-1`, device `nvme1` (instance `10.244.0.79:9633`, `smartctl-exporter`) - **Since:** 2026-08-27T14:21:35Z, receiver `email`, still active - **Rule:** `smartctl_device_percentage_used >= 80 for 1h` — `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml:16` ## Evidence `context.txt` shows nothing wrong: all three nodes `Ready`, no pods outside Running/Succeeded, no Flux resource unready. This alert is about a hardware budget, not a broken workload, so that is expected and not a contradiction. Verified live via the Prometheus API through the apiserver proxy (`kubectl get --raw .../api/v1/query?query=smartctl_device_percentage_used`) — wear across the whole fleet: | node | device | model | wear | |---|---|---|---| | k8s-1 | nvme1 | CT1000P3PSSD8 | **87%** | | k8s-3 | nvme1 | CT1000P3PSSD8 | 83% | | k8s-2 | nvme0 | CT1000P3PSSD8 | 81% | | k8s-1/2/3 | Kingston OM8PGP4 | system disk | 10–13% | | k8s-1/2/3 | Samsung MZQLB960 | Ceph OSD | 2–3% | `ALERTS{alertname=~"Nvme.*"}` confirms **all three** `NvmeDriveWearHigh` instances are firing. Only k8s-1 reached this batch because the other two dedup to the same issue title, so this one issue stands for all three. ## Root cause **The worn drives are the EPHEMERAL volumes, and this is deliberate.** Confirmed two independent ways: 1. `kubernetes/talos/patches/k8s-1.yaml:50` pins EPHEMERAL with `disk.serial == '2332E86B749A'`; `node_nvme_info` reports that exact serial as k8s-1 `nvme1` (CT1000P3PSSD8). 2. `node_filesystem_*{mountpoint="/var"}` on k8s-1 reports device `/dev/nvme1n1p1` — the same drive. The drive is **not** a Ceph OSD: `ceph_disk_occupation` puts k8s-1's `osd.2` on `/dev/nvme2n1` (the Samsung, 3% wear). The repo comment at `kubernetes/talos/patches/k8s-1.yaml:32` states the intent — EPHEMERAL goes on "the retired Ceph OSD drive" because its contents rebuild — and flags the exception itself: `/var/lib/etcd` lives there and "is the one thing on the volume that is not free to lose." So the wear itself is working as designed. Three things make it worth acting on rather than closing: **1. The three EPHEMERAL drives will hit critical together.** They are the same model, at 81/83/87%, carrying comparable write volumes (132–154 TB written). `NvmeDriveWearCritical` (>=95%) will fire on all three within a similar window — a correlated, not independent, failure of the volume that holds every etcd member. **2. k8s-1 nvme1 has 18 media errors; every other drive in the fleet has 0.** From `smartctl_device_media_errors{node="k8s-1"}`. This is the "failure now" signal, distinct from the endurance budget — and it is currently invisible: `NvmeDriveMediaErrors` alerts on `increase(...[6h]) > 0`, so a standing count never fires. That is the documented intent (`prometheusrule-disk-wear.yaml:66`), but the consequence is that the one drive holding etcd carries unrecovered data-integrity errors that no alert reports. The other failure-now signals are clean: available spare 100 (threshold 5), critical warning 0. **3. No trend is available yet.** `min_over_time(timestamp(...))` puts the earliest smartctl sample at 2026-08-27T13:24:32Z — roughly 15 hours of history. Wear and media errors are both flat across all of it. **Hypothesis (unverified):** the 18 media errors predate the drive's EPHEMERAL role and date from its earlier life as a Ceph OSD; power-on time is 72,651,600 s (~841 days). This cannot be settled from Prometheus — Ceph's `devicehealth` registry purges a drive's history when it leaves the cluster, and Talos kernel logs do not reach Loki. `talosctl -n 10.2.40.169 dmesg` is the only source and it is a ring buffer emptied by reboot. **Also unverified:** etcd health could not be checked from monitoring — `up{job=~".*etcd.*"}` returns no series, so etcd is not scraped by this Prometheus. ## Where to fix it **This is not fixable by a repo change alone.** It is a hardware-lifetime problem, and the relevant Talos setting cannot be re-applied to a running node. - `kubernetes/talos/patches/k8s-1.yaml:45-51` (and `k8s-2.yaml:39-43`, `k8s-3.yaml:35-39`) — the `VolumeConfig` / EPHEMERAL `diskSelector`. The comment at `k8s-1.yaml:36` is the constraint: **volume configuration only takes effect on first provision**; applying a changed selector to an existing node is silently ignored, and relocating EPHEMERAL requires wiping the disk and booting install media. Moving etcd off these drives therefore means a node rebuild, not a config push. The node object ages in `context.txt` (k8s-1 2d9h, k8s-2 2d6h vs k8s-3 110d) suggest two of the three were rebuilt very recently, i.e. that window just closed. - `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml` — worth reconsidering the media-errors rule so a *standing* non-zero count on an etcd-bearing drive is surfaced rather than dashboard-only (e.g. a separate low-severity alert on the absolute value), or recording explicitly that standing counts are accepted. **Suggested human actions (read-only, off-cluster):** 1. `talosctl -n 10.2.40.169 dmesg` — look for NVMe errors on `nvme1` before the buffer is lost to a reboot. 2. Verify etcd member health on k8s-1 directly (not observable via Prometheus here). 3. Decide the replacement plan for the three CT1000P3PSSD8 drives together, since they age in lockstep. Replacing the k8s-1 drive first is the natural order: highest wear plus the only media errors. Nothing is failing right now — spare is untouched and no critical warning is raised. The action this asks for is scheduling, not an emergency. --- <!-- alert-monitor:fingerprint=c292db90b2c94fc8 --> _Filed automatically by the alert-monitor pipeline (2026-08-28). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-28 05:45:50 +00:00
Member
Copy Link
Copy Source

Assessment — no repo change is appropriate

NvmeDriveWearHigh is reporting a hardware endurance budget, not a
misconfiguration. I verified live that the alert is correct, that its target is
exactly the drive the repo deliberately put /var on, and that no edit inside
the allowed scope (ansible/, opentofu/, kubernetes/, ci/images/) can
lower a drive's wear counter. Per decision rule 6 I made no edits; the
action this needs is hardware scheduling, described step by step below.

I also declined the one repo change the issue floated — see Why I did not
change the media-errors rule.

What I verified live this run

All queries via ./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?..."
at 2026-08-28T05:44Z unless noted.

1. The alert is accurate and fires on all three nodes.
smartctl_device_percentage_used → k8s-1 nvme1 87, k8s-3 nvme1 83,
k8s-2 nvme0 81; every other drive 2–13.
ALERTS{alertname=~"Nvme.*"} → three NvmeDriveWearHigh series, all
alertstate="firing". The issue's fleet table is confirmed.

2. The worn drives are the EPHEMERAL / /var volumes — confirmed three
independent ways, so the issue's root cause holds:

  • node_nvme_info serials: k8s-1 nvme1 = 2332E86B749A, k8s-2 nvme0 =
    2332E86B74BC, k8s-3 nvme1 = 2326E6E8E67D, all model CT1000P3PSSD8.
    These are character-for-character the three VolumeConfig/EPHEMERAL
    diskSelector values in kubernetes/talos/patches/k8s-1.yaml:50,
    k8s-2.yaml:43, k8s-3.yaml:39.
  • node_filesystem_size_bytes{mountpoint="/var"} → /dev/nvme1n1p1 on
    10.2.40.169 (k8s-1), /dev/nvme0n1p1 on 10.2.40.211 (k8s-2),
    /dev/nvme1n1p1 on 10.2.40.138 (k8s-3) — the same three devices, each
    999714680832 B (the 1 TB Crucial).
  • ceph_disk_occupation → osd.2/osd.0/osd.3 all sit on /dev/nvme2n1,
    the Samsung MZQLB960HAJR on each node (2–3% wear). None of the worn
    drives is an OSD.     The install.diskSelector serials in the same patches
    are the Kingston OM8PGP4 system disks (10–13% wear).

3. Nothing is failing yet. smartctl_device_available_spare = 100 on
all nine drives (threshold 5); smartctl_device_critical_warning = 0 on
all nine. This is wear, not damage.

4. The three drives are aging in lockstep.
smartctl_device_bytes_written → 153.8 TB (k8s-1 nvme1), 143.5 TB (k8s-2
nvme0), 142.7 TB (k8s-3 nvme1). Same model, same workload, ~8% apart. The
issue's core concern — that NvmeDriveWearCritical (≥95%) will fire on all
three in a similar window, on the volume holding every etcd member — is
supported by the data.

5. k8s-1 nvme1 carries 18 standing media errors; every other drive has 0.
smartctl_device_media_errors → {node="k8s-1",device="nvme1"} 18, all eight
others 0. Same drive as the highest wear, and the one holding this node's
etcd member.

6. No usable trend exists.
min_over_time(timestamp(smartctl_device_percentage_used)[30d:1h]) → earliest
sample 1787839172 ≈ 2026-08-27T13:59Z, i.e. ~15.7 h of history despite a
30 d lookback. No wear rate can be computed, so no honest "weeks until 95%"
estimate is possible yet. Re-run this query in a few weeks to get one.

7. Two of the issue's "unverified" items are now settled:

  • etcd is not scraped. count(up) by (job) returns 43 jobs; none matches
    etcd. etcd member health genuinely cannot be checked from this Prometheus —
    the issue was right.
  • Talos kernel logs do not reach Loki. Loki's job label has exactly two
    values, loki.source.kubernetes.pods and systemd-journal, and the host
    label has exactly one value, edge-1.iot.renner.uno (not a Talos node). So
    talosctl dmesg really is the only source for the NVMe errors, and it is a
    ring buffer — capture it before the next reboot of k8s-1.

Why no repo change fixes this

  • Wear is physical. No manifest, role or Tofu resource reduces a drive's
    percentage_used. Bumping the 80% threshold would be silencing the alert,
    which decision rule 4 forbids and which the issue does not establish as
    wrong — 87% on the etcd-bearing volume is exactly what this rule exists to
    say.
  • EPHEMERAL cannot be relocated by config. The comment at
    kubernetes/talos/patches/k8s-1.yaml:36-40 states Talos applies volume
    configuration only when the volume does not yet exist; a changed selector on
    a provisioned node is silently ignored. Editing those three files would
    produce a PR that changes nothing on the running cluster while making the
    repo disagree with reality — strictly worse than no change.
  • There is nowhere to relocate it to. node_nvme_info shows exactly three
    drives per node, and all three are spoken for: Kingston = system disk
    (install.diskSelector), Crucial = EPHEMERAL, Samsung = Ceph OSD. Moving
    /var off the Crucial requires new hardware, not a different selector.
  • Image tags and pinned-component bumps are out of scope (rules 3 and 5),
    and none would help anyway.

Why I did not change the media-errors rule

The issue suggests adding an alert on the absolute value of
smartctl_device_media_errors so the standing 18 on k8s-1 nvme1 is surfaced.
I verified the current behaviour and deliberately left it alone:

  • prometheusrule-disk-wear.yaml:68 alerts on increase(...[6h]) > 0, so a
    flat counter never fires. Confirmed by inspection; the count has been flat
    across all available history.
  • The comment directly above it (lines 66-67) records this as an intentional
    trade-off — "pre-existing errors would otherwise alert forever. Standing
    counts are visible on the dashboard only."
  • I checked that second half and it is true: disk-health.json:205 plots
    smartctl_device_media_errors in the drive table. The issue's phrase "no
    alert reports it" is accurate, but its stronger claim that the count is
    "currently invisible" is not — it is on the Disk Health dashboard,
    exactly where the rule says it is.

Adding an absolute-value alert would fire permanently until the drive is
physically replaced — precisely the outcome the existing comment says was
rejected. Reversing a documented owner decision is not a call this agent
should make unsupervised, and there is no "Verified owner guidance" section
on this issue. If the owner wants that trade-off revisited, it is a ~10-line
addition to prometheusrule-disk-wear.yaml and I can do it on request.

Recommended manual actions

Nothing is failing; this is scheduling work, not an emergency. In priority
order:

  1. Capture the kernel evidence before it is lost. talosctl -n 10.2.40.169 dmesg on k8s-1, grep for nvme1. This is the only place the
    18 media errors are explained (when they happened, whether they were
    read/write/scrub) and a reboot empties it. Note the drive reports 841 days
    of power-on time, so the errors plausibly predate its EPHEMERAL role — but
    that stays a hypothesis until dmesg is read; do not treat it as
    established.
  2. Check etcd member health on k8s-1 directly — talosctl -n 10.2.40.169 etcd status. Prometheus cannot see it (item 6 above), and this is the one
    dataset on the affected volume that is not reconstructible.
  3. Order replacements for all three CT1000P3PSSD8 together. They wear in
    lockstep (item 4), so treating them as one batch is correct. k8s-1's drive
    goes first: highest wear and the only media errors in the fleet.
  4. When replacing, do it as a node rebuild, one node at a time, waiting for
    etcd quorum and Ceph to return healthy in between. The EPHEMERAL selector is
    pinned by serial, so each new drive's serial must be written into that
    node's kubernetes/talos/patches/k8s-N.yaml before reprovisioning     — this
    is the moment a repo change becomes both necessary and effective, and it is
    the only window in which EPHEMERAL's placement can be changed at all.
  5. Optional, low cost: re-run the min_over_time(timestamp(...)) query in
    ~4 weeks. Once there is real history, predict_linear on
    smartctl_device_percentage_used gives a defensible date for the 95%
    crossing instead of a guess.

Notes

  • Ceph is HEALTH_WARN, and it is unrelated. ceph_health_detail > 0
    returns only AUTH_INSECURE_CLIENT_KEY_TYPE,
    AUTH_INSECURE_KEYS_ALLOWED and AUTH_INSECURE_KEYS_CREATABLE — no PG or
    OSD problem. Storage redundancy is intact for the drive swaps above. Flagged
    only so a reviewer does not mistake it for fallout from this alert.
  • One issue claim I could not reproduce as written: the issue cites
    node_nvme_info with a node label. That metric comes from node-exporter,
    which carries instance (node IP) but no node label here, unlike the
    smartctl-exporter metrics. node_nvme_info{node="k8s-1"} returns empty. The
    serial mapping is still correct — it just has to be joined on instance.
    This does not affect SmartctlExporterIncomplete
    (prometheusrule-disk-wear.yaml:81), which compares bare count() on both
    sides: both are 9, so the rule is healthy and correctly silent.
  • The issue's inference that k8s-1/k8s-2 were "rebuilt very recently" rests on
    Node object ages, which I confirmed (kubectl get nodes -o wide: k8s-1
    2d10h, k8s-2 2d7h, k8s-3 110d). Node age resets on cluster rejoin as well as
    on reprovision, so this is suggestive, not proof. If those were true
    rebuilds, note the serial pins put /var straight back on the worn drives —
    which is the documented intent, and unavoidable given there is no fourth
    drive.
  • Nothing in the issue asked me to touch CI config, credentials, or anything
    outside the edit scope.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment — no repo change is appropriate `NvmeDriveWearHigh` is reporting a **hardware endurance budget**, not a misconfiguration. I verified live that the alert is correct, that its target is exactly the drive the repo deliberately put `/var` on, and that no edit inside the allowed scope (`ansible/`, `opentofu/`, `kubernetes/`, `ci/images/`) can lower a drive's wear counter. Per decision rule 6 I made **no edits**; the action this needs is hardware scheduling, described step by step below. I also declined the one repo change the issue floated — see *Why I did not change the media-errors rule*. ### What I verified live this run All queries via `./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?..."` at 2026-08-28T05:44Z unless noted. **1. The alert is accurate and fires on all three nodes.** `smartctl_device_percentage_used` → k8s-1 `nvme1` **87**, k8s-3 `nvme1` **83**, k8s-2 `nvme0` **81**; every other drive 2–13. `ALERTS{alertname=~"Nvme.*"}` → three `NvmeDriveWearHigh` series, all `alertstate="firing"`. The issue's fleet table is confirmed. **2. The worn drives are the EPHEMERAL / `/var` volumes — confirmed three independent ways**, so the issue's root cause holds: - `node_nvme_info` serials: k8s-1 `nvme1` = `2332E86B749A`, k8s-2 `nvme0` = `2332E86B74BC`, k8s-3 `nvme1` = `2326E6E8E67D`, all model `CT1000P3PSSD8`. These are character-for-character the three `VolumeConfig`/EPHEMERAL `diskSelector` values in `kubernetes/talos/patches/k8s-1.yaml:50`, `k8s-2.yaml:43`, `k8s-3.yaml:39`. - `node_filesystem_size_bytes{mountpoint="/var"}` → `/dev/nvme1n1p1` on 10.2.40.169 (k8s-1), `/dev/nvme0n1p1` on 10.2.40.211 (k8s-2), `/dev/nvme1n1p1` on 10.2.40.138 (k8s-3) — the same three devices, each 999714680832 B (the 1 TB Crucial). - `ceph_disk_occupation` → `osd.2`/`osd.0`/`osd.3` all sit on `/dev/nvme2n1`, the Samsung `MZQLB960HAJR` on each node (2–3% wear). **None of the worn drives is an OSD.** The `install.diskSelector` serials in the same patches are the Kingston `OM8PGP4` system disks (10–13% wear). **3. Nothing is failing yet.** `smartctl_device_available_spare` = **100** on all nine drives (threshold 5); `smartctl_device_critical_warning` = **0** on all nine. This is wear, not damage. **4. The three drives are aging in lockstep.** `smartctl_device_bytes_written` → 153.8 TB (k8s-1 `nvme1`), 143.5 TB (k8s-2 `nvme0`), 142.7 TB (k8s-3 `nvme1`). Same model, same workload, ~8% apart. The issue's core concern — that `NvmeDriveWearCritical` (≥95%) will fire on all three in a similar window, on the volume holding every etcd member — is supported by the data. **5. k8s-1 `nvme1` carries 18 standing media errors; every other drive has 0.** `smartctl_device_media_errors` → `{node="k8s-1",device="nvme1"} 18`, all eight others `0`. Same drive as the highest wear, and the one holding this node's etcd member. **6. No usable trend exists.** `min_over_time(timestamp(smartctl_device_percentage_used)[30d:1h])` → earliest sample **1787839172 ≈ 2026-08-27T13:59Z**, i.e. ~15.7 h of history despite a 30 d lookback. No wear rate can be computed, so **no honest "weeks until 95%" estimate is possible yet.** Re-run this query in a few weeks to get one. **7. Two of the issue's "unverified" items are now settled:** - *etcd is not scraped.* `count(up) by (job)` returns 43 jobs; none matches etcd. etcd member health genuinely cannot be checked from this Prometheus — the issue was right. - *Talos kernel logs do not reach Loki.* Loki's `job` label has exactly two values, `loki.source.kubernetes.pods` and `systemd-journal`, and the `host` label has exactly one value, `edge-1.iot.renner.uno` (not a Talos node). So `talosctl dmesg` really is the only source for the NVMe errors, and it is a ring buffer — **capture it before the next reboot of k8s-1.** ### Why no repo change fixes this - **Wear is physical.** No manifest, role or Tofu resource reduces a drive's `percentage_used`. Bumping the 80% threshold would be silencing the alert, which decision rule 4 forbids and which the issue does not establish as wrong — 87% on the etcd-bearing volume is exactly what this rule exists to say. - **EPHEMERAL cannot be relocated by config.** The comment at `kubernetes/talos/patches/k8s-1.yaml:36-40` states Talos applies volume configuration only when the volume does not yet exist; a changed selector on a provisioned node is silently ignored. Editing those three files would produce a PR that changes nothing on the running cluster while making the repo disagree with reality — strictly worse than no change. - **There is nowhere to relocate it to.** `node_nvme_info` shows exactly three drives per node, and all three are spoken for: Kingston = system disk (`install.diskSelector`), Crucial = EPHEMERAL, Samsung = Ceph OSD. Moving `/var` off the Crucial requires *new hardware*, not a different selector. - **Image tags and pinned-component bumps are out of scope** (rules 3 and 5), and none would help anyway. ### Why I did not change the media-errors rule The issue suggests adding an alert on the *absolute* value of `smartctl_device_media_errors` so the standing 18 on k8s-1 `nvme1` is surfaced. I verified the current behaviour and deliberately left it alone: - `prometheusrule-disk-wear.yaml:68` alerts on `increase(...[6h]) > 0`, so a flat counter never fires. Confirmed by inspection; the count has been flat across all available history. - The comment directly above it (lines 66-67) records this as an intentional trade-off — *"pre-existing errors would otherwise alert forever. Standing counts are visible on the dashboard only."* - I checked that second half and it is true: `disk-health.json:205` plots `smartctl_device_media_errors` in the drive table. The issue's phrase "no alert reports it" is accurate, but its stronger claim that the count is **"currently invisible" is not** — it is on the Disk Health dashboard, exactly where the rule says it is. Adding an absolute-value alert would fire permanently until the drive is physically replaced — precisely the outcome the existing comment says was rejected. Reversing a documented owner decision is not a call this agent should make unsupervised, and there is **no "Verified owner guidance" section on this issue**. If the owner wants that trade-off revisited, it is a ~10-line addition to `prometheusrule-disk-wear.yaml` and I can do it on request. ### Recommended manual actions Nothing is failing; this is scheduling work, not an emergency. In priority order: 1. **Capture the kernel evidence before it is lost.** `talosctl -n 10.2.40.169 dmesg` on k8s-1, grep for `nvme1`. This is the only place the 18 media errors are explained (when they happened, whether they were read/write/scrub) and a reboot empties it. Note the drive reports 841 days of power-on time, so the errors plausibly predate its EPHEMERAL role — but that stays a hypothesis until dmesg is read; **do not treat it as established.** 2. **Check etcd member health on k8s-1 directly** — `talosctl -n 10.2.40.169 etcd status`. Prometheus cannot see it (item 6 above), and this is the one dataset on the affected volume that is not reconstructible. 3. **Order replacements for all three CT1000P3PSSD8 together.** They wear in lockstep (item 4), so treating them as one batch is correct. k8s-1's drive goes first: highest wear *and* the only media errors in the fleet. 4. **When replacing, do it as a node rebuild, one node at a time**, waiting for etcd quorum and Ceph to return healthy in between. The EPHEMERAL selector is pinned by serial, so **each new drive's serial must be written into that node's `kubernetes/talos/patches/k8s-N.yaml` before reprovisioning** — this is the moment a repo change becomes both necessary and effective, and it is the only window in which EPHEMERAL's placement can be changed at all. 5. **Optional, low cost:** re-run the `min_over_time(timestamp(...))` query in ~4 weeks. Once there is real history, `predict_linear` on `smartctl_device_percentage_used` gives a defensible date for the 95% crossing instead of a guess. ### Notes - **Ceph is `HEALTH_WARN`, and it is unrelated.** `ceph_health_detail > 0` returns only `AUTH_INSECURE_CLIENT_KEY_TYPE`, `AUTH_INSECURE_KEYS_ALLOWED` and `AUTH_INSECURE_KEYS_CREATABLE` — no PG or OSD problem. Storage redundancy is intact for the drive swaps above. Flagged only so a reviewer does not mistake it for fallout from this alert. - **One issue claim I could not reproduce as written:** the issue cites `node_nvme_info` with a `node` label. That metric comes from node-exporter, which carries `instance` (node IP) but no `node` label here, unlike the smartctl-exporter metrics. `node_nvme_info{node="k8s-1"}` returns empty. The serial mapping is still correct — it just has to be joined on `instance`. This does **not** affect `SmartctlExporterIncomplete` (`prometheusrule-disk-wear.yaml:81`), which compares bare `count()` on both sides: both are 9, so the rule is healthy and correctly silent. - The issue's inference that k8s-1/k8s-2 were "rebuilt very recently" rests on Node object ages, which I confirmed (`kubectl get nodes -o wide`: k8s-1 2d10h, k8s-2 2d7h, k8s-3 110d). Node age resets on cluster rejoin as well as on reprovision, so this is suggestive, not proof. If those were true rebuilds, note the serial pins put `/var` straight back on the worn drives — which is the documented intent, and unavoidable given there is no fourth drive. - Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit scope. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#686