konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] NvmeDriveWearHigh (monitoring) #673

New Issue
Closed
opened 2026-08-27 14:34:47 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-27 14:34:47 +00:00
Member
Copy Link
Copy Source

What is firing

NvmeDriveWearHigh (severity: warning, namespace monitoring), firing since 2026-08-27 14:21 UTC.

Three instances of this alert fired together, one per node. They are one condition, so this issue covers all three:

Node Device Serial Wear Role
k8s-1 nvme1 2332E86B749A 87% EPHEMERAL (/var)
k8s-3 nvme1 2326E6E8E67D 83% EPHEMERAL (/var)
k8s-2 nvme0 2332E86B74BC 81% EPHEMERAL (/var)

Evidence

All figures below are from the live Prometheus API via ./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=<expr>".

The worn drives are the EPHEMERAL drives, not the OSDs. smartctl_device resolves every worn device to model CT1000P3PSSD8 (Crucial P3 Plus, QLC) with serials 2332E86B749A / 2332E86B74BC / 2326E6E8E67D — exactly the serials the Talos VolumeConfig documents select for EPHEMERAL. Confirmed independently by node-exporter, which places the /var xfs filesystem on /dev/nvme1n1p1 on k8s-1 and k8s-3 and on /dev/nvme0n1p1 on k8s-2 — the same devices the alert names.

The Ceph OSDs are fine. smartctl_device_percentage_used for the Samsung PM983 OSD drives (nvme2 on every node) reads 3% / 2% / 3%, on 164 / 87 / 81 TB written. The Kingston system disks read 13% / 10% / 13%. The PM983 migration recorded in cephcluster.yaml worked; the wear is entirely on the drives that were retired into /var duty.

Supporting counters (smartctl_device_bytes_written, smartctl_device_power_on_seconds):

  • k8s-1: 153.8 TB written over 840 power-on days
  • k8s-2: 143.4 TB over 839 days
  • k8s-3: 142.6 TB over 842 days

Near-identical age and write volume across all three — they were deployed together and are wearing together.

k8s-1's drive is additionally degraded. smartctl_device_media_errors reads 18 on k8s-1 nvme1 and 0 on all eight other drives. Media errors are unrecoverable data-integrity errors; a healthy drive never increments this counter.

Not yet urgent on two counts: smartctl_device_available_spare is 100 on all nine drives (no reserve blocks consumed), smartctl_device_critical_warning is 0, and /var is only ~8% full on every node. Nothing is failing right now.

Root cause

Not a fault — accumulated write endurance on low-endurance QLC consumer drives, now surfaced for the first time.

smartctl-exporter was deployed today: kubectl -n monitoring get daemonset smartctl-exporter shows creationTimestamp: 2026-08-27T12:57:55Z, ~1.5 h before these alerts. This is the blind spot prometheusrule-disk-wear.yaml was written to close, working as designed on its first observation.

The placement itself was a deliberate decision, documented in kubernetes/talos/patches/k8s-1.yaml: the worn Crucials were demoted from Ceph OSD to /var because containerd's image store, pod logs and the Ceph mon store are all reconstructible. That reasoning holds. What still needs action is that all three will exhaust in the same window, and the same file states the consequence: a node simply stops when its EPHEMERAL volume goes. A correlated failure across all three control-plane nodes is a different risk from any single drive wearing out.

Runway — hypothesis, not verified. cephcluster.yaml records a manual measurement from 2026-08-19: 80/82/87% advancing ~0.13 points/day, projecting exhaustion between Nov 2026 and Jan 2027. Today's 81/83/87% is consistent with that (~1 point in 8 days on two of the three). Prometheus cannot corroborate a rate — the exporter has under two hours of history, so delta(smartctl_device_percentage_used[7d]) returns 0 for every drive. The projection will be verifiable from Prometheus in about a week; until then treat it as the working estimate.

One caveat worth re-checking: because the exporter is new, there is no way to tell whether k8s-1's 18 media errors are historical or accruing now. NvmeDriveMediaErrors alerts on increase(...[6h]), so it will catch further growth on its own — but a deliberate re-read of smartctl_device_media_errors{node="k8s-1"} in 24 h would settle it sooner.

Also note the mon write amplification fix in cephcluster.yaml (mon_compact_on_trim: "false") targeted ~80 GB/day/mon of writes into /var/lib/rook — i.e. onto these exact drives. Its effect on the wear rate is unmeasured and should show up in the trend over the coming weeks.

Where to fix it

This is hardware — not fixable by a repo change alone. The action is procurement plus a scheduled rebuild per node.

  • Order three replacement drives. Given 143-154 TB written in ~840 days, a TLC or enterprise drive is the better choice than another QLC consumer unit; the PM983 OSDs at 2-3% after comparable writes are the evidence for that.
  • Replacement is one node at a time, gracefully. kubernetes/RUNBOOK-baremetal-conversion.md §7 documents the binding constraint: Talos applies VolumeConfig only when the volume does not yet exist, so EPHEMERAL cannot be relocated on a live node — the node must be reset and reinstalled. The runbook's reset step (talosctl -n <node> reset --graceful --wipe-mode system-disk, then kubectl delete node <node>) and its stage-1 config list are the procedure to follow. Its pre-flight check already includes nvme smart-log ... percentage_used | media_errors.
  • Repo changes needed after each swap, to the serial selectors:
    • kubernetes/talos/patches/k8s-1.yaml — VolumeConfig EPHEMERAL, disk.serial == '2332E86B749A'
    • kubernetes/talos/patches/k8s-2.yaml and k8s-3.yaml — same document, their own serials
    • kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml — the per-node drive inventory table in the storage: comment block (lines ~239-252)
    • kubernetes/RUNBOOK-baremetal-conversion.md — the drive table at the top
  • Suggest starting with k8s-1: highest wear and the only drive with media errors.

No change to the alerting rule is warranted — kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml reported this correctly and NvmeDriveWearCritical (>=95%) plus NvmeDriveSpareExhausted remain as escalation backstops if replacement slips.

The k8s-2 (81%) and k8s-3 (83%) alerts were classified as duplicates of this issue. Their NvmeDriveWearCritical alerts are a distinct alertname and are unaffected.

Filed automatically by the alert-monitor pipeline (2026-08-27). The issue-fix pipeline may pick this up and open a PR.

## What is firing `NvmeDriveWearHigh` (severity: warning, namespace `monitoring`), firing since 2026-08-27 14:21 UTC. Three instances of this alert fired together, one per node. They are one condition, so this issue covers all three: | Node | Device | Serial | Wear | Role | |---|---|---|---|---| | k8s-1 | nvme1 | 2332E86B749A | **87%** | EPHEMERAL (`/var`) | | k8s-3 | nvme1 | 2326E6E8E67D | 83% | EPHEMERAL (`/var`) | | k8s-2 | nvme0 | 2332E86B74BC | 81% | EPHEMERAL (`/var`) | ## Evidence All figures below are from the live Prometheus API via `./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=<expr>"`. **The worn drives are the EPHEMERAL drives, not the OSDs.** `smartctl_device` resolves every worn device to model `CT1000P3PSSD8` (Crucial P3 Plus, QLC) with serials `2332E86B749A` / `2332E86B74BC` / `2326E6E8E67D` — exactly the serials the Talos `VolumeConfig` documents select for EPHEMERAL. Confirmed independently by node-exporter, which places the `/var` xfs filesystem on `/dev/nvme1n1p1` on k8s-1 and k8s-3 and on `/dev/nvme0n1p1` on k8s-2 — the same devices the alert names. **The Ceph OSDs are fine.** `smartctl_device_percentage_used` for the Samsung PM983 OSD drives (`nvme2` on every node) reads **3% / 2% / 3%**, on 164 / 87 / 81 TB written. The Kingston system disks read 13% / 10% / 13%. The PM983 migration recorded in `cephcluster.yaml` worked; the wear is entirely on the drives that were *retired into* `/var` duty. **Supporting counters** (`smartctl_device_bytes_written`, `smartctl_device_power_on_seconds`): - k8s-1: 153.8 TB written over 840 power-on days - k8s-2: 143.4 TB over 839 days - k8s-3: 142.6 TB over 842 days Near-identical age and write volume across all three — they were deployed together and are wearing together. **k8s-1's drive is additionally degraded.** `smartctl_device_media_errors` reads **18** on k8s-1 nvme1 and **0** on all eight other drives. Media errors are unrecoverable data-integrity errors; a healthy drive never increments this counter. **Not yet urgent on two counts:** `smartctl_device_available_spare` is 100 on all nine drives (no reserve blocks consumed), `smartctl_device_critical_warning` is 0, and `/var` is only ~8% full on every node. Nothing is failing right now. ## Root cause Not a fault — accumulated write endurance on low-endurance QLC consumer drives, now surfaced for the first time. `smartctl-exporter` was deployed today: `kubectl -n monitoring get daemonset smartctl-exporter` shows `creationTimestamp: 2026-08-27T12:57:55Z`, ~1.5 h before these alerts. This is the blind spot `prometheusrule-disk-wear.yaml` was written to close, working as designed on its first observation. The placement itself was a deliberate decision, documented in `kubernetes/talos/patches/k8s-1.yaml`: the worn Crucials were demoted from Ceph OSD to `/var` because containerd's image store, pod logs and the Ceph mon store are all reconstructible. That reasoning holds. What still needs action is that **all three will exhaust in the same window**, and the same file states the consequence: a node simply stops when its EPHEMERAL volume goes. A correlated failure across all three control-plane nodes is a different risk from any single drive wearing out. **Runway — hypothesis, not verified.** `cephcluster.yaml` records a manual measurement from 2026-08-19: 80/82/87% advancing ~0.13 points/day, projecting exhaustion between Nov 2026 and Jan 2027. Today's 81/83/87% is consistent with that (~1 point in 8 days on two of the three). Prometheus cannot corroborate a rate — the exporter has under two hours of history, so `delta(smartctl_device_percentage_used[7d])` returns 0 for every drive. The projection will be verifiable from Prometheus in about a week; until then treat it as the working estimate. One caveat worth re-checking: because the exporter is new, there is no way to tell whether k8s-1's 18 media errors are historical or accruing now. `NvmeDriveMediaErrors` alerts on `increase(...[6h])`, so it will catch further growth on its own — but a deliberate re-read of `smartctl_device_media_errors{node="k8s-1"}` in 24 h would settle it sooner. Also note the mon write amplification fix in `cephcluster.yaml` (`mon_compact_on_trim: "false"`) targeted ~80 GB/day/mon of writes into `/var/lib/rook` — i.e. onto these exact drives. Its effect on the wear rate is unmeasured and should show up in the trend over the coming weeks. ## Where to fix it This is **hardware — not fixable by a repo change alone.** The action is procurement plus a scheduled rebuild per node. - Order three replacement drives. Given 143-154 TB written in ~840 days, a TLC or enterprise drive is the better choice than another QLC consumer unit; the PM983 OSDs at 2-3% after comparable writes are the evidence for that. - Replacement is **one node at a time**, gracefully. `kubernetes/RUNBOOK-baremetal-conversion.md` §7 documents the binding constraint: Talos applies `VolumeConfig` **only when the volume does not yet exist**, so EPHEMERAL cannot be relocated on a live node — the node must be reset and reinstalled. The runbook's reset step (`talosctl -n <node> reset --graceful --wipe-mode system-disk`, then `kubectl delete node <node>`) and its stage-1 config list are the procedure to follow. Its pre-flight check already includes `nvme smart-log ... percentage_used | media_errors`. - Repo changes needed **after** each swap, to the serial selectors: - `kubernetes/talos/patches/k8s-1.yaml` — `VolumeConfig EPHEMERAL`, `disk.serial == '2332E86B749A'` - `kubernetes/talos/patches/k8s-2.yaml` and `k8s-3.yaml` — same document, their own serials - `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` — the per-node drive inventory table in the `storage:` comment block (lines ~239-252) - `kubernetes/RUNBOOK-baremetal-conversion.md` — the drive table at the top - Suggest starting with **k8s-1**: highest wear and the only drive with media errors. No change to the alerting rule is warranted — `kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml` reported this correctly and `NvmeDriveWearCritical` (>=95%) plus `NvmeDriveSpareExhausted` remain as escalation backstops if replacement slips. --- *The k8s-2 (81%) and k8s-3 (83%) alerts were classified as duplicates of this issue. Their `NvmeDriveWearCritical` alerts are a distinct alertname and are unaffected.* --- <!-- alert-monitor:fingerprint=c292db90b2c94fc8 --> _Filed automatically by the alert-monitor pipeline (2026-08-27). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-27 14:44:15 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change is appropriate — this is hardware

NvmeDriveWearHigh is reporting a true physical condition: three consumer QLC
NVMe drives (Crucial P3 Plus, CT1000P3PSSD8) have consumed 81–87% of their
rated write endurance after ~840 power-on days. There is no repo change that
adds endurance back to a drive. The alert rule is correct, the drive placement
it is complaining about was a deliberate and still-sound decision, and the
serial selectors that would need editing can only be edited after the
hardware is physically swapped — changing them now would point Talos at drives
that do not exist.

I verified every substantive claim in the issue against the live cluster this
run; all of them held. Details and the recommended manual procedure follow.

Verification performed this run

All queries via ./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=...".

Claim in issue Command Observed Verdict
Wear 87 / 83 / 81% smartctl_device_percentage_used k8s-1 nvme1=87, k8s-3 nvme1=83, k8s-2 nvme0=81 confirmed
OSDs are fine same query nvme2 on all three nodes = 3 / 3 / 2 confirmed
Worn drives are the Crucials smartctl_device all three worn devices model_name="CT1000P3PSSD8", serials 2332E86B749A (k8s-1), 2326E6E8E67D (k8s-3), 2332E86B74BC (k8s-2) confirmed
k8s-1 has 18 media errors smartctl_device_media_errors k8s-1 nvme1 = 18; all eight other drives = 0 confirmed
Spare untouched smartctl_device_available_spare 100 on all nine drives confirmed
No critical warning max(smartctl_device_critical_warning) 0 confirmed
/var not full node_filesystem_*{mountpoint="/var"} 8.35 / 8.01 / 8.34 % used, on /dev/nvme1n1p1, /dev/nvme0n1p1, /dev/nvme1n1p1 confirmed
Exporter is new kubectl -n monitoring get daemonset smartctl-exporter creationTimestamp: 2026-08-27T12:57:55Z, 3/3 ready confirmed

Repo-side cross-check: the three worn serials are exactly the VolumeConfig EPHEMERAL diskSelector.match values in kubernetes/talos/patches/k8s-1.yaml:85,
k8s-2.yaml:51 and k8s-3.yaml:50. So the worn drives are unambiguously the
/var drives, not the Ceph OSDs — the issue's central finding is correct, and
the node-exporter device paths agree independently.

Why no edit

  • The alert is right. prometheusrule-disk-wear.yaml fired on its first
    observation of a condition that was previously invisible. Raising the 80%
    threshold or dropping the rule would be silencing a true positive.
  • The placement is deliberate. kubernetes/talos/patches/k8s-1.yaml:62-75
    already documents why the worn Crucials carry /var: containerd's image
    store, pod logs and the Ceph mon store are all reconstructible, so the drive
    is consumable in that role in a way it was not as an OSD. That reasoning
    still holds at 87%.
  • The serial selectors cannot be pre-changed. RUNBOOK-baremetal-conversion.md:302
    and the same note at k8s-1.yaml:72-75: Talos applies volume configuration
    only when the volume does not yet exist. Editing the serials before the
    swap would be silently ignored on the running nodes and would leave the repo
    describing hardware that is not installed.
  • Image tags, chart versions and thresholds are all out of scope for this
    alert, and none of them are implicated anyway.

Recommended manual action

Procurement. Order three replacement drives. The evidence for choosing TLC
or enterprise over another QLC consumer unit is in the numbers above: the
Samsung PM983 OSDs sit at 2–3% after comparable lifetime writes, while the
Crucials reached 81–87%.

Replacement — one node at a time, starting with k8s-1 (highest wear, and
the only drive with media errors). Because EPHEMERAL placement is one-shot,
each swap is a reset-and-reinstall, following the existing runbook:

  1. Evacuate the OSD — RUNBOOK-baremetal-conversion.md §1.
  2. Drain the node — §2. Single-replica workloads with minAvailable: 1 PDBs
    (influxdb-influxdb2-0, woodpecker-server, the CNPG primary) will refuse
    eviction by design; move each with kubectl delete pod.
  3. talosctl -n <node> reset --graceful --wipe-mode system-disk, then
    kubectl delete node <node> — §2, lines 74-75.
  4. Physically swap the Crucial for the new drive — §3. The pre-flight check at
    line 149 (nvme smart-log ... percentage_used | media_errors | critical)
    already covers validating the replacement before commitment; note that line
    155 is the last revert point.
  5. Reinstall with the new serial in the first apply-config — §7. Verify
    with talosctl -n <node> get volumestatus EPHEMERAL -o yaml.
  6. Bring the mon back (§8), re-add the OSD (§9), verify (§10).

Repo changes belong after each swap, not now — the new serial must be
written into kubernetes/talos/patches/k8s-<n>.yaml (VolumeConfig EPHEMERAL,
diskSelector.match) as part of step 5, and the drive inventory tables in
kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml and
RUNBOOK-baremetal-conversion.md:16 updated to match.

Notes

  • Runway is unverified, and I could not verify it either. The issue is
    appropriately careful here and I confirmed why: the exporter has under two
    hours of history, so Prometheus cannot yield a wear rate yet. The projection
    (exhaustion Nov 2026 – Jan 2027) comes from a manual measurement recorded in
    cephcluster.yaml:140-143 on 2026-08-19 — 80/82/87% advancing ~0.13
    points/day. Today's 81/83/87% is consistent with that but does not confirm
    it: percentage_used is integer-quantized, so ±1 point over 8 days cannot
    distinguish 0.10/day from 0.15/day. Treat it as a planning estimate. It
    becomes checkable from Prometheus with delta(smartctl_device_percentage_used[7d])
    in about a week.
  • The correlated-failure risk is the real finding, and it is not something
    a repo change addresses. All three drives were deployed together, have near
    identical writes (143–154 TB) and age (~840 days), and will exhaust in the
    same window — across all three control-plane nodes. Staggering the
    replacements is what breaks the correlation; that is a scheduling decision
    for a human.
  • k8s-1's 18 media errors: age unknown. Because the exporter is new there
    is no history to say whether these are historical or accruing. Nothing to
    fix in the repo, and NvmeDriveMediaErrors (on increase(...[6h])) will
    catch further growth on its own. A deliberate re-read of
    smartctl_device_media_errors{node="k8s-1"} in 24 h settles it sooner.
  • Stale comment spotted, deliberately left alone.
    cephcluster.yaml:140 reads "all three OSD drives at 80/82/87% endurance
    used". That was true when the Crucials were the OSDs; today the OSDs are
    PM983s at 2–3% and the 80–87% drives are the EPHEMERAL Crucials. The comment
    now reads as if the current OSDs are nearly worn out, which is the opposite
    of the truth and could mislead someone diagnosing this very alert. I did not
    fix it: it is documentation, not the root cause, and that whole block gets
    rewritten as part of the post-swap update above — better done in one pass
    than half-corrected now. Flagging it so it is not missed.
  • Nothing is at risk right now. Spare area is untouched (100 on all nine
    drives), no critical warnings, /var ~8% full. This is a plan-ahead alert,
    which is what the 80% threshold was chosen for
    (prometheusrule-disk-wear.yaml:28-30).
  • The issue contained no instructions to change CI config, credentials, or
    anything outside the edit scope, and no "Verified owner guidance" section.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment: no repo change is appropriate — this is hardware `NvmeDriveWearHigh` is reporting a true physical condition: three consumer QLC NVMe drives (Crucial P3 Plus, `CT1000P3PSSD8`) have consumed 81–87% of their rated write endurance after ~840 power-on days. There is no repo change that adds endurance back to a drive. The alert rule is correct, the drive placement it is complaining about was a deliberate and still-sound decision, and the serial selectors that would need editing can only be edited *after* the hardware is physically swapped — changing them now would point Talos at drives that do not exist. I verified every substantive claim in the issue against the live cluster this run; all of them held. Details and the recommended manual procedure follow. ## Verification performed this run All queries via `./.ci-diag/kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=..."`. | Claim in issue | Command | Observed | Verdict | |---|---|---|---| | Wear 87 / 83 / 81% | `smartctl_device_percentage_used` | k8s-1 `nvme1`=**87**, k8s-3 `nvme1`=**83**, k8s-2 `nvme0`=**81** | confirmed | | OSDs are fine | same query | `nvme2` on all three nodes = **3 / 3 / 2** | confirmed | | Worn drives are the Crucials | `smartctl_device` | all three worn devices `model_name="CT1000P3PSSD8"`, serials `2332E86B749A` (k8s-1), `2326E6E8E67D` (k8s-3), `2332E86B74BC` (k8s-2) | confirmed | | k8s-1 has 18 media errors | `smartctl_device_media_errors` | k8s-1 `nvme1` = **18**; all eight other drives = **0** | confirmed | | Spare untouched | `smartctl_device_available_spare` | **100** on all nine drives | confirmed | | No critical warning | `max(smartctl_device_critical_warning)` | **0** | confirmed | | `/var` not full | `node_filesystem_*{mountpoint="/var"}` | **8.35 / 8.01 / 8.34 %** used, on `/dev/nvme1n1p1`, `/dev/nvme0n1p1`, `/dev/nvme1n1p1` | confirmed | | Exporter is new | `kubectl -n monitoring get daemonset smartctl-exporter` | `creationTimestamp: 2026-08-27T12:57:55Z`, **3/3** ready | confirmed | Repo-side cross-check: the three worn serials are exactly the `VolumeConfig EPHEMERAL` `diskSelector.match` values in `kubernetes/talos/patches/k8s-1.yaml:85`, `k8s-2.yaml:51` and `k8s-3.yaml:50`. So the worn drives are unambiguously the `/var` drives, not the Ceph OSDs — the issue's central finding is correct, and the node-exporter device paths agree independently. ## Why no edit - **The alert is right.** `prometheusrule-disk-wear.yaml` fired on its first observation of a condition that was previously invisible. Raising the 80% threshold or dropping the rule would be silencing a true positive. - **The placement is deliberate.** `kubernetes/talos/patches/k8s-1.yaml:62-75` already documents why the worn Crucials carry `/var`: containerd's image store, pod logs and the Ceph mon store are all reconstructible, so the drive is consumable in that role in a way it was not as an OSD. That reasoning still holds at 87%. - **The serial selectors cannot be pre-changed.** `RUNBOOK-baremetal-conversion.md:302` and the same note at `k8s-1.yaml:72-75`: Talos applies volume configuration **only when the volume does not yet exist**. Editing the serials before the swap would be silently ignored on the running nodes and would leave the repo describing hardware that is not installed. - **Image tags, chart versions and thresholds are all out of scope** for this alert, and none of them are implicated anyway. ## Recommended manual action **Procurement.** Order three replacement drives. The evidence for choosing TLC or enterprise over another QLC consumer unit is in the numbers above: the Samsung PM983 OSDs sit at 2–3% after comparable lifetime writes, while the Crucials reached 81–87%. **Replacement — one node at a time, starting with k8s-1** (highest wear, and the only drive with media errors). Because EPHEMERAL placement is one-shot, each swap is a reset-and-reinstall, following the existing runbook: 1. Evacuate the OSD — `RUNBOOK-baremetal-conversion.md` §1. 2. Drain the node — §2. Single-replica workloads with `minAvailable: 1` PDBs (`influxdb-influxdb2-0`, `woodpecker-server`, the CNPG primary) will refuse eviction by design; move each with `kubectl delete pod`. 3. `talosctl -n <node> reset --graceful --wipe-mode system-disk`, then `kubectl delete node <node>` — §2, lines 74-75. 4. Physically swap the Crucial for the new drive — §3. The pre-flight check at line 149 (`nvme smart-log ... percentage_used | media_errors | critical`) already covers validating the replacement before commitment; note that line 155 is the last revert point. 5. Reinstall with the new serial in the **first** `apply-config` — §7. Verify with `talosctl -n <node> get volumestatus EPHEMERAL -o yaml`. 6. Bring the mon back (§8), re-add the OSD (§9), verify (§10). **Repo changes belong after each swap**, not now — the new serial must be written into `kubernetes/talos/patches/k8s-<n>.yaml` (`VolumeConfig EPHEMERAL`, `diskSelector.match`) as part of step 5, and the drive inventory tables in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` and `RUNBOOK-baremetal-conversion.md:16` updated to match. ## Notes - **Runway is unverified, and I could not verify it either.** The issue is appropriately careful here and I confirmed why: the exporter has under two hours of history, so Prometheus cannot yield a wear rate yet. The projection (exhaustion Nov 2026 – Jan 2027) comes from a manual measurement recorded in `cephcluster.yaml:140-143` on 2026-08-19 — 80/82/87% advancing ~0.13 points/day. Today's 81/83/87% is *consistent* with that but does not confirm it: `percentage_used` is integer-quantized, so ±1 point over 8 days cannot distinguish 0.10/day from 0.15/day. Treat it as a planning estimate. It becomes checkable from Prometheus with `delta(smartctl_device_percentage_used[7d])` in about a week. - **The correlated-failure risk is the real finding**, and it is not something a repo change addresses. All three drives were deployed together, have near identical writes (143–154 TB) and age (~840 days), and will exhaust in the same window — across all three control-plane nodes. Staggering the replacements is what breaks the correlation; that is a scheduling decision for a human. - **k8s-1's 18 media errors: age unknown.** Because the exporter is new there is no history to say whether these are historical or accruing. Nothing to fix in the repo, and `NvmeDriveMediaErrors` (on `increase(...[6h])`) will catch further growth on its own. A deliberate re-read of `smartctl_device_media_errors{node="k8s-1"}` in 24 h settles it sooner. - **Stale comment spotted, deliberately left alone.** `cephcluster.yaml:140` reads "all three OSD drives at 80/82/87% endurance used". That was true when the Crucials *were* the OSDs; today the OSDs are PM983s at 2–3% and the 80–87% drives are the EPHEMERAL Crucials. The comment now reads as if the current OSDs are nearly worn out, which is the opposite of the truth and could mislead someone diagnosing this very alert. I did not fix it: it is documentation, not the root cause, and that whole block gets rewritten as part of the post-swap update above — better done in one pass than half-corrected now. Flagging it so it is not missed. - **Nothing is at risk right now.** Spare area is untouched (100 on all nine drives), no critical warnings, `/var` ~8% full. This is a plan-ahead alert, which is what the 80% threshold was chosen for (`prometheusrule-disk-wear.yaml:28-30`). - The issue contained no instructions to change CI config, credentials, or anything outside the edit scope, and no "Verified owner guidance" section. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#673