NvmeDriveWearHigh is firing correctly for a real condition — k8s-1's nvme1 is at 87% rated wear — and no repo change can lower that number, so this PR does not stop the alert firing. What it does fix is a documentation defect the alert exposed. kubernetes/talos/patches/k8s-3.yaml:37 makes "available spare is still 100% and it reports no media errors" part of the stated acceptance criterion for putting EPHEMERAL on a worn-out Crucial P3. The k8s-1 comment argues the same placement but never mentions media errors — and k8s-1's P3 is the only drive in the fleet with a non-zero media-error counter (18). Since /var/lib/etcd lives on EPHEMERAL, that omission leaves the next reader believing k8s-1 meets a criterion it does not, on the one node where the consequence is etcd member data. The change records the count, states plainly that its origin is unestablished, and gives the on-node command that settles it.
Changes
kubernetes/talos/patches/k8s-1.yaml — added a comment block to the EPHEMERALVolumeConfig recording the 18 media errors, why NvmeDriveMediaErrors does not fire for them (it alerts on increase(...), not the total), that historical-vs-ongoing is not established, and the etcd consequence if the count turns out to be climbing. Comment-only: 19 added lines, zero non-comment changes, no behavioural effect.
Verification
Everything below was observed this run via ./.ci-diag/kubectl get --raw against the in-cluster Prometheus. The issue's diagnosis held up on every point I could check:
query=smartctl_device_media_errors — {node="k8s-1",device="nvme1"} = 18; all eight other drives across k8s-1/2/3 read 0. Confirms it is the only non-zero counter in the fleet.
query=min_over_time(smartctl_device_media_errors{node="k8s-1",device="nvme1"}[30d]) → 18, and the same instant query at time=1787773400 (24h ago) returns an empty result. So the counter is flat at 18 across the whole series, but the series is only hours old — confirming the issue's point that the series being flat is weak evidence, which is why the comment says the origin is unestablished rather than asserting the errors are historical.
query=smartctl_device{node="k8s-1"} — nvme1 = CT1000P3PSSD8, serial 2332E86B749A, matching the disk.serial selector at k8s-1.yaml:85. The device-name-to-serial resolution in the issue is correct.
query=smartctl_device_percentage_used — k8s-1/nvme1 87, k8s-2/nvme0 81, k8s-3/nvme1 83; all three OSD Samsungs at 2–3.
query=ceph_disk_occupation — osd.2 on k8s-1 is /dev/nvme2n1 = SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426 (3% wear). The Ceph OSD is not on the worn drive; the issue was right about this.
query=smartctl_device_available_spare{node="k8s-1"} → 100 on all three; query=smartctl_device_critical_warning → 0 fleet-wide.
For the reviewer: nothing should change after Flux reconciles — this is a comment inside a Talos machine-config patch, and per the file's own warning VolumeConfig only takes effect on first provision, so it is inert on running nodes either way. NvmeDriveWearHigh will keep firing for k8s-1/nvme1 (and for k8s-2/nvme0 and k8s-3/nvme1 at 81% and 83%) until either the drives are replaced or the rule is deliberately rescoped.
Notes
What still needs a human — this is the actual open question. Run on the node, out of band:
Compare the error-log entry timestamps against the drive's power-on hours and against when it was repurposed from OSD to EPHEMERAL (k8s-1 was converted to bare metal ~2 days before the issue was filed).
If the 18 errors are historical and static, the existing placement rationale holds and this PR is the whole fix — the comment now records it honestly.
If the count is climbing, this node's etcd member data is on failing media and EPHEMERAL should move to the system disk. That is not a config-only change: VolumeConfig applies only at first provision, so it requires wiping the disk and booting install media. Note that etcd quorum survives losing one of three members, but the member must be explicitly removed and re-added — it is not automatic.
What I did not do, and why. The issue floats rescoping NvmeDriveWearHigh so EPHEMERAL-role drives alert at a higher threshold. I left that alone: it is raising a threshold to stop a truthful alert, the wear is real, and there is no role label on the smartctl metrics to scope by — it would mean hardcoding the three P3 serials into the PrometheusRule, which then rots silently the moment a drive is swapped. That is an owner decision about accepted risk, not a fix, and this issue does not establish the rule is wrong. NvmeDriveWearCritical (>= 95%) remains separate and will still fire independently.
I also did not touch k8s-2.yaml, whose comment likewise omits media errors — but both its and k8s-3's P3s read 0, so there is no discrepancy to record there.
Scope check. The issue contained no "Verified owner guidance" section, and nothing in it asked for changes outside the edit allowlist. The issue's own assessment that this is "not fixable by a repo change alone" is correct, and I want to be explicit that this PR does not resolve the alert — it corrects the record so the hardware decision can be made against accurate information. If the reviewer would rather carry that information in the issue than in a PR, closing this without merging costs nothing.
## What this PR does
`NvmeDriveWearHigh` is firing correctly for a real condition — k8s-1's `nvme1` is at 87% rated wear — and no repo change can lower that number, so **this PR does not stop the alert firing**. What it does fix is a documentation defect the alert exposed. `kubernetes/talos/patches/k8s-3.yaml:37` makes *"available spare is still 100% and it reports no media errors"* part of the stated acceptance criterion for putting EPHEMERAL on a worn-out Crucial P3. The k8s-1 comment argues the same placement but never mentions media errors — and k8s-1's P3 is the **only drive in the fleet with a non-zero media-error counter (18)**. Since `/var/lib/etcd` lives on EPHEMERAL, that omission leaves the next reader believing k8s-1 meets a criterion it does not, on the one node where the consequence is etcd member data. The change records the count, states plainly that its origin is unestablished, and gives the on-node command that settles it.
## Changes
- `kubernetes/talos/patches/k8s-1.yaml` — added a comment block to the `EPHEMERAL` `VolumeConfig` recording the 18 media errors, why `NvmeDriveMediaErrors` does not fire for them (it alerts on `increase(...)`, not the total), that historical-vs-ongoing is **not** established, and the etcd consequence if the count turns out to be climbing. Comment-only: 19 added lines, zero non-comment changes, no behavioural effect.
## Verification
Everything below was observed this run via `./.ci-diag/kubectl get --raw` against the in-cluster Prometheus. The issue's diagnosis held up on every point I could check:
- `query=smartctl_device_media_errors` — `{node="k8s-1",device="nvme1"} = 18`; all eight other drives across k8s-1/2/3 read `0`. Confirms it is the only non-zero counter in the fleet.
- `query=min_over_time(smartctl_device_media_errors{node="k8s-1",device="nvme1"}[30d])` → `18`, and the same instant query at `time=1787773400` (24h ago) returns an **empty** result. So the counter is flat at 18 across the whole series, but the series is only hours old — confirming the issue's point that the series being flat is weak evidence, which is why the comment says the origin is unestablished rather than asserting the errors are historical.
- `query=smartctl_device{node="k8s-1"}` — `nvme1` = `CT1000P3PSSD8`, serial `2332E86B749A`, matching the `disk.serial` selector at `k8s-1.yaml:85`. The device-name-to-serial resolution in the issue is correct.
- `query=smartctl_device_percentage_used` — k8s-1/nvme1 `87`, k8s-2/nvme0 `81`, k8s-3/nvme1 `83`; all three OSD Samsungs at `2`–`3`.
- `query=ceph_disk_occupation` — `osd.2` on k8s-1 is `/dev/nvme2n1` = `SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426` (3% wear). **The Ceph OSD is not on the worn drive**; the issue was right about this.
- `query=smartctl_device_available_spare{node="k8s-1"}` → `100` on all three; `query=smartctl_device_critical_warning` → `0` fleet-wide.
For the reviewer: nothing should change after Flux reconciles — this is a comment inside a Talos machine-config patch, and per the file's own warning `VolumeConfig` only takes effect on first provision, so it is inert on running nodes either way. `NvmeDriveWearHigh` will keep firing for k8s-1/nvme1 (and for k8s-2/nvme0 and k8s-3/nvme1 at 81% and 83%) until either the drives are replaced or the rule is deliberately rescoped.
## Notes
**What still needs a human — this is the actual open question.** Run on the node, out of band:
```
talosctl -n k8s-1 -- smartctl -a /dev/nvme1n1 # or: nvme error-log /dev/nvme1n1
```
Compare the error-log entry timestamps against the drive's power-on hours and against when it was repurposed from OSD to EPHEMERAL (k8s-1 was converted to bare metal ~2 days before the issue was filed).
- **If the 18 errors are historical and static**, the existing placement rationale holds and this PR is the whole fix — the comment now records it honestly.
- **If the count is climbing**, this node's etcd member data is on failing media and EPHEMERAL should move to the system disk. That is not a config-only change: `VolumeConfig` applies only at first provision, so it requires wiping the disk and booting install media. Note that etcd quorum survives losing one of three members, but the member must be explicitly removed and re-added — it is not automatic.
**What I did not do, and why.** The issue floats rescoping `NvmeDriveWearHigh` so EPHEMERAL-role drives alert at a higher threshold. I left that alone: it is raising a threshold to stop a truthful alert, the wear is real, and there is no role label on the smartctl metrics to scope by — it would mean hardcoding the three P3 serials into the PrometheusRule, which then rots silently the moment a drive is swapped. That is an owner decision about accepted risk, not a fix, and this issue does not establish the rule is wrong. `NvmeDriveWearCritical` (>= 95%) remains separate and will still fire independently.
I also did not touch `k8s-2.yaml`, whose comment likewise omits media errors — but both its and k8s-3's P3s read `0`, so there is no discrepancy to record there.
**Scope check.** The issue contained no "Verified owner guidance" section, and nothing in it asked for changes outside the edit allowlist. The issue's own assessment that this is "not fixable by a repo change alone" is correct, and I want to be explicit that this PR does not resolve the alert — it corrects the record so the hardware decision can be made against accurate information. If the reviewer would rather carry that information in the issue than in a PR, closing this without merging costs nothing.
Closes #675
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit bdffe1804b27839fc02adae8c9c58e82066ebe4e.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR adds a 19-line comment block to kubernetes/talos/patches/k8s-1.yaml documenting a caveat specific to k8s-1's EPHEMERAL volume: the nvme1 drive reports 18 media errors (the only non-zero counter in the fleet), and whether they are historical or ongoing is unestablished. The comment explains why the related NvmeDriveMediaErrors alert does not fire, provides an actionable investigation path, and notes the consequence for etcd member data. No behavioral changes; comment-only. The PR is well-executed and transparent about its scope.
🟢 Nits
Line 76: "the total" — the comment explains this clearly (NvmeDriveMediaErrors alerts on increase() not absolute value), but "the total" could be slightly clearer as "the total count" if being pedantic. Not blocking.
Tests
No tests needed; this is a comment in a configuration file. Configuration comments do not require test coverage.
Recommendation
Looks reasonable — no blocking issues; comment is accurate, well-integrated, and helpful for future maintenance.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `bdffe1804b27839fc02adae8c9c58e82066ebe4e`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR adds a 19-line comment block to `kubernetes/talos/patches/k8s-1.yaml` documenting a caveat specific to k8s-1's EPHEMERAL volume: the nvme1 drive reports 18 media errors (the only non-zero counter in the fleet), and whether they are historical or ongoing is unestablished. The comment explains why the related `NvmeDriveMediaErrors` alert does not fire, provides an actionable investigation path, and notes the consequence for etcd member data. No behavioral changes; comment-only. The PR is well-executed and transparent about its scope.
## 🟢 Nits
- Line 76: "the total" — the comment explains this clearly (NvmeDriveMediaErrors alerts on `increase()` not absolute value), but "the total" could be slightly clearer as "the total count" if being pedantic. Not blocking.
## Tests
No tests needed; this is a comment in a configuration file. Configuration comments do not require test coverage.
## Recommendation
Looks reasonable — no blocking issues; comment is accurate, well-integrated, and helpful for future maintenance.
<!-- pr-review:sha=bdffe1804b27839fc02adae8c9c58e82066ebe4e -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
NvmeDriveWearHighis firing correctly for a real condition — k8s-1'snvme1is at 87% rated wear — and no repo change can lower that number, so this PR does not stop the alert firing. What it does fix is a documentation defect the alert exposed.kubernetes/talos/patches/k8s-3.yaml:37makes "available spare is still 100% and it reports no media errors" part of the stated acceptance criterion for putting EPHEMERAL on a worn-out Crucial P3. The k8s-1 comment argues the same placement but never mentions media errors — and k8s-1's P3 is the only drive in the fleet with a non-zero media-error counter (18). Since/var/lib/etcdlives on EPHEMERAL, that omission leaves the next reader believing k8s-1 meets a criterion it does not, on the one node where the consequence is etcd member data. The change records the count, states plainly that its origin is unestablished, and gives the on-node command that settles it.Changes
kubernetes/talos/patches/k8s-1.yaml— added a comment block to theEPHEMERALVolumeConfigrecording the 18 media errors, whyNvmeDriveMediaErrorsdoes not fire for them (it alerts onincrease(...), not the total), that historical-vs-ongoing is not established, and the etcd consequence if the count turns out to be climbing. Comment-only: 19 added lines, zero non-comment changes, no behavioural effect.Verification
Everything below was observed this run via
./.ci-diag/kubectl get --rawagainst the in-cluster Prometheus. The issue's diagnosis held up on every point I could check:query=smartctl_device_media_errors—{node="k8s-1",device="nvme1"} = 18; all eight other drives across k8s-1/2/3 read0. Confirms it is the only non-zero counter in the fleet.query=min_over_time(smartctl_device_media_errors{node="k8s-1",device="nvme1"}[30d])→18, and the same instant query attime=1787773400(24h ago) returns an empty result. So the counter is flat at 18 across the whole series, but the series is only hours old — confirming the issue's point that the series being flat is weak evidence, which is why the comment says the origin is unestablished rather than asserting the errors are historical.query=smartctl_device{node="k8s-1"}—nvme1=CT1000P3PSSD8, serial2332E86B749A, matching thedisk.serialselector atk8s-1.yaml:85. The device-name-to-serial resolution in the issue is correct.query=smartctl_device_percentage_used— k8s-1/nvme187, k8s-2/nvme081, k8s-3/nvme183; all three OSD Samsungs at2–3.query=ceph_disk_occupation—osd.2on k8s-1 is/dev/nvme2n1=SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426(3% wear). The Ceph OSD is not on the worn drive; the issue was right about this.query=smartctl_device_available_spare{node="k8s-1"}→100on all three;query=smartctl_device_critical_warning→0fleet-wide.For the reviewer: nothing should change after Flux reconciles — this is a comment inside a Talos machine-config patch, and per the file's own warning
VolumeConfigonly takes effect on first provision, so it is inert on running nodes either way.NvmeDriveWearHighwill keep firing for k8s-1/nvme1 (and for k8s-2/nvme0 and k8s-3/nvme1 at 81% and 83%) until either the drives are replaced or the rule is deliberately rescoped.Notes
What still needs a human — this is the actual open question. Run on the node, out of band:
Compare the error-log entry timestamps against the drive's power-on hours and against when it was repurposed from OSD to EPHEMERAL (k8s-1 was converted to bare metal ~2 days before the issue was filed).
VolumeConfigapplies only at first provision, so it requires wiping the disk and booting install media. Note that etcd quorum survives losing one of three members, but the member must be explicitly removed and re-added — it is not automatic.What I did not do, and why. The issue floats rescoping
NvmeDriveWearHighso EPHEMERAL-role drives alert at a higher threshold. I left that alone: it is raising a threshold to stop a truthful alert, the wear is real, and there is no role label on the smartctl metrics to scope by — it would mean hardcoding the three P3 serials into the PrometheusRule, which then rots silently the moment a drive is swapped. That is an owner decision about accepted risk, not a fix, and this issue does not establish the rule is wrong.NvmeDriveWearCritical(>= 95%) remains separate and will still fire independently.I also did not touch
k8s-2.yaml, whose comment likewise omits media errors — but both its and k8s-3's P3s read0, so there is no discrepancy to record there.Scope check. The issue contained no "Verified owner guidance" section, and nothing in it asked for changes outside the edit allowlist. The issue's own assessment that this is "not fixable by a repo change alone" is correct, and I want to be explicit that this PR does not resolve the alert — it corrects the record so the hardware decision can be made against accurate information. If the reviewer would rather carry that information in the issue than in a PR, closing this without merging costs nothing.
Closes #675
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitbdffe1804b27839fc02adae8c9c58e82066ebe4e.Summary
This PR adds a 19-line comment block to
kubernetes/talos/patches/k8s-1.yamldocumenting a caveat specific to k8s-1's EPHEMERAL volume: the nvme1 drive reports 18 media errors (the only non-zero counter in the fleet), and whether they are historical or ongoing is unestablished. The comment explains why the relatedNvmeDriveMediaErrorsalert does not fire, provides an actionable investigation path, and notes the consequence for etcd member data. No behavioral changes; comment-only. The PR is well-executed and transparent about its scope.🟢 Nits
increase()not absolute value), but "the total" could be slightly clearer as "the total count" if being pedantic. Not blocking.Tests
No tests needed; this is a comment in a configuration file. Configuration comments do not require test coverage.
Recommendation
Looks reasonable — no blocking issues; comment is accurate, well-integrated, and helpful for future maintenance.