feat(monitoring): put the SMART values on the dashboard #672

2026-08-27T13:26:39Z

claude-bot commented

2026-08-27 13:26:39 +00:00

Der Exporter meldet jetzt alle neun Laufwerke, also zeigt das Dashboard, was er misst: eine Worst-Wear-Kachel in der Statuszeile, Verschleiß pro Laufwerk über die Zeit, und eine Tabelle der Ausfall-jetzt-Signale — Available Spare, Media-Errors, das Critical-Warning-Bitfeld, Betriebszeit und geschriebene Bytes.

Alle Abfragen gegen die laufende TSDB geprüft, Label-Namen an echten Serien abgelesen statt aus einer README übernommen. Das war nötig: der Exporter labelt Geräte als nvme0 und trägt Modell und Seriennummer auf einer separaten smartctl_device-Info-Metrik. Jede Abfrage joint deshalb auf (node, device), um das Modell dazuzuholen — eine Legende auf dem Gerätenamen allein würde wieder unterschiedliche Hardware vergleichen.

Verschleiß und Schaden stehen getrennt, weil es zwei Fragen sind. Der Prozentsatz ist ein Budget, das ausgegeben wird, und sagt nichts über morgen. Erschöpfter Spare-Bereich, Media-Errors und das Critical-Warning-Flag bedeuten, dass jetzt etwas kaputt ist. Das Caveats-Panel sagt das auch nicht mehr falsch herum — es behauptete noch, Endurance fehle, was beim Schreiben stimmte und jetzt nicht mehr.

Mitgenommen: die Dublette in der OSD-Kachel. Der Commit, der sie behoben hat, ist nicht mehr in #668 gelandet, bevor der PR gemerged wurde — die Kachel ging also genau in dem Zustand live, den du bemängelt hattest. Sie zeigt jetzt nur „OSDs down".

Der erste Befund aus den neuen Daten, und er ist unangenehm:

Laufwerk	Rolle	Verschleiß	Media-Errors
CT1000P3PSSD8 k8s-1	EPHEMERAL	87 %	18
CT1000P3PSSD8 k8s-2	EPHEMERAL	81 %	0
CT1000P3PSSD8 k8s-3	EPHEMERAL	(siehe Dashboard)	0
KINGSTON	System	10–13 %	0
SAMSUNG PM983	Ceph-OSD	2–3 %	0

Media-Errors sind nicht behebbare Datenintegritätsfehler; ein gesundes Laufwerk zählt hier nie hoch. Das betroffene ist ausgerechnet das mit 87 % Verschleiß, und es trägt den Container-State.

Wichtig zur Alarmierung: NvmeDriveMediaErrors alarmiert bewusst auf den Zuwachs über 6 h, nicht auf den Absolutwert — damit ein Laufwerk mit alten Fehlern nicht dauerhaft alarmiert. Diese 18 sind Bestand und lösen daher keinen Alarm aus. Sie sind im Dashboard rot, aber sie werden dich nicht wecken. Falls du das anders willst, ist das eine Zeile in prometheusrule-disk-wear.yaml — sag Bescheid, ich ändere es nicht eigenmächtig.

Refs #549

Der Exporter meldet jetzt alle neun Laufwerke, also zeigt das Dashboard, was er misst: eine Worst-Wear-Kachel in der Statuszeile, Verschleiß pro Laufwerk über die Zeit, und eine Tabelle der Ausfall-jetzt-Signale — Available Spare, Media-Errors, das Critical-Warning-Bitfeld, Betriebszeit und geschriebene Bytes. **Alle Abfragen gegen die laufende TSDB geprüft, Label-Namen an echten Serien abgelesen statt aus einer README übernommen.** Das war nötig: der Exporter labelt Geräte als `nvme0` und trägt Modell und Seriennummer auf einer separaten `smartctl_device`-Info-Metrik. Jede Abfrage joint deshalb auf `(node, device)`, um das Modell dazuzuholen — eine Legende auf dem Gerätenamen allein würde wieder unterschiedliche Hardware vergleichen. **Verschleiß und Schaden stehen getrennt, weil es zwei Fragen sind.** Der Prozentsatz ist ein Budget, das ausgegeben wird, und sagt nichts über morgen. Erschöpfter Spare-Bereich, Media-Errors und das Critical-Warning-Flag bedeuten, dass jetzt etwas kaputt ist. Das Caveats-Panel sagt das auch nicht mehr falsch herum — es behauptete noch, Endurance fehle, was beim Schreiben stimmte und jetzt nicht mehr. **Mitgenommen: die Dublette in der OSD-Kachel.** Der Commit, der sie behoben hat, ist nicht mehr in #668 gelandet, bevor der PR gemerged wurde — die Kachel ging also genau in dem Zustand live, den du bemängelt hattest. Sie zeigt jetzt nur „OSDs down". --- **Der erste Befund aus den neuen Daten, und er ist unangenehm:** | Laufwerk | Rolle | Verschleiß | Media-Errors | |---|---|---|---| | CT1000P3PSSD8 k8s-1 | EPHEMERAL | **87 %** | **18** | | CT1000P3PSSD8 k8s-2 | EPHEMERAL | **81 %** | 0 | | CT1000P3PSSD8 k8s-3 | EPHEMERAL | (siehe Dashboard) | 0 | | KINGSTON | System | 10–13 % | 0 | | SAMSUNG PM983 | Ceph-OSD | 2–3 % | 0 | Media-Errors sind nicht behebbare Datenintegritätsfehler; ein gesundes Laufwerk zählt hier nie hoch. Das betroffene ist ausgerechnet das mit 87 % Verschleiß, und es trägt den Container-State. **Wichtig zur Alarmierung:** `NvmeDriveMediaErrors` alarmiert bewusst auf den *Zuwachs* über 6 h, nicht auf den Absolutwert — damit ein Laufwerk mit alten Fehlern nicht dauerhaft alarmiert. Diese 18 sind Bestand und lösen daher **keinen** Alarm aus. Sie sind im Dashboard rot, aber sie werden dich nicht wecken. Falls du das anders willst, ist das eine Zeile in `prometheusrule-disk-wear.yaml` — sag Bescheid, ich ändere es nicht eigenmächtig. Refs #549

claude-bot added 1 commit 2026-08-27 13:26:40 +00:00

feat(monitoring): put the SMART values on the dashboard

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

24b2a0178d

The exporter reports all nine drives now, so the dashboard shows what it
measures: a worst-wear stat in the status row, wear per drive over time, and a
table of the failure-now signals — available spare, media errors, the critical
warning bitfield, power-on time and written bytes.

Every expression was run against the live TSDB and the label names read off real
series rather than a README. That mattered: the exporter labels devices `nvme0`
and carries model and serial on a separate `smartctl_device` info metric, so
each query joins on (node, device) to pull the model in. A legend on the device
name alone would compare different hardware across machines, which is the same
trap the block-layer panels already avoid.

Wear and damage are presented as different things because they are. The wear
percentage is a budget the drive spends and says nothing about tomorrow; spare
exhaustion, media errors and the critical warning mean something is wrong now.
The caveats panel says so, and no longer claims endurance is missing — it was
missing when that text was written and is not any more.

Also fixes the OSD stat, which still rendered two identical numbers side by
side. The commit that fixed it did not make it into #668 before that PR merged,
so the panel went out in the state that prompted the complaint. It now shows
OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about
it separately.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit c9a10ab015 into master

2026-08-27 13:27:36 +00:00

k.renner referenced this issue from a commit

2026-08-27 13:27:38 +00:00

Merge pull request 'feat(monitoring): put the SMART values on the dashboard' (#672) from disk-dashboard-smart into master

review-bot reviewed

2026-08-27 13:30:41 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 24b2a0178de9411415bfbfb09a3f7458402177f4.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR extends kubernetes/apps/monitoring/dashboards/disk-health.json with SMART/endurance panels (worst-wear stat, per-drive wear-over-time, and a failure-now table for spare/media-errors/critical-warning/power-on/written-bytes), backed by the already-deployed smartctl-exporter DaemonSet, its ServiceMonitor relabeling, and the disk-wear PrometheusRule (none of which are touched by this diff — only the dashboard JSON changes). It also tightens the top-row layout to fit a new "Worst NVMe wear" tile, drops the "OSDs out" series from the OSD stat, and rewrites the caveats panel to reflect that endurance is no longer a blind spot. The change is well-reasoned and the gridPos layout math checks out cleanly across all 16 panels (verified by hand: every row's start y matches the previous row's y + h, and each row's widths sum to 24). No blocking issues found, but one thing is worth confirming before merge.

🟡 Suggestions

smartctl_device_power_on_seconds and smartctl_device_bytes_written — verify these metric names/units against the running exporter (kubernetes/apps/monitoring/dashboards/disk-health.json:207-208, also used in the "SMART health" table's renameByName at line 218 and the unit overrides at lines 195-196). The NVMe SMART/Health Information log natively reports "Power On Hours" (hours, not seconds) and "Data Units Written" (units of 512,000 bytes, not raw bytes) — the prometheus-community smartctl_exporter has historically named these fields smartctl_device_power_on_hours and smartctl_device_data_units_written rather than the _seconds/_bytes names used here. If the deployed v0.14.0 image (kubernetes/apps/monitoring/smartctl-exporter.yaml:59) still uses those upstream names, these two queries return no series and the "powered on" and "written" columns in the SMART health table render permanently empty — silently losing the "useful for projecting when a drive will reach it" capability the panel description promises. The PR body states all queries were checked against the live TSDB, which would already cover this, but since I can't reach the cluster's Prometheus from this checkout I couldn't independently confirm it. One query settles it: curl -s http://<smartctl-exporter-pod>:9633/metrics | grep -E 'power_on|written' or {__name__=~"smartctl_device.*"} against Prometheus.
"An OSD that is up but out shows as a Ceph health warning instead" (kubernetes/apps/monitoring/dashboards/disk-health.json:31) — worth double-checking against actual Ceph behavior. An OSD marked out while still up (e.g., during a deliberate ceph osd out for maintenance) is a normal administrative state and doesn't unconditionally raise a HEALTH_WARN on its own in my understanding; it's only degraded/misplaced PGs (transient, until rebalance completes) that would. If that's the case the sentence overstates what "OSDs down" now omits. Low impact since it's descriptive text only, not a query, but it could send someone chasing a warning that won't actually appear.

Tests

No automated tests exist or are expected for Grafana dashboard JSON in this repo — verification is manual (against a live TSDB, per the PR body) and via the "SmartctlExporterIncomplete" alert already in prometheusrule-disk-wear.yaml, which is a reasonable substitute (it fires if the SMART series fleet-wide coverage drops below node_nvme_info's inventory). That alert would not, however, catch the narrower case of one or two specific fields (power-on, written) being wrong/absent while the rest of smartctl_device_* reports fine, which is exactly the scenario the suggestion above is about.

Recommendation

Looks reasonable — no confirmed blocking issues; the metric-name suggestion above is worth a final check before merge given it's unverifiable from this checkout.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `24b2a0178de9411415bfbfb09a3f7458402177f4`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR extends `kubernetes/apps/monitoring/dashboards/disk-health.json` with SMART/endurance panels (worst-wear stat, per-drive wear-over-time, and a failure-now table for spare/media-errors/critical-warning/power-on/written-bytes), backed by the already-deployed `smartctl-exporter` DaemonSet, its `ServiceMonitor` relabeling, and the `disk-wear` `PrometheusRule` (none of which are touched by this diff — only the dashboard JSON changes). It also tightens the top-row layout to fit a new "Worst NVMe wear" tile, drops the "OSDs out" series from the OSD stat, and rewrites the caveats panel to reflect that endurance is no longer a blind spot. The change is well-reasoned and the gridPos layout math checks out cleanly across all 16 panels (verified by hand: every row's start `y` matches the previous row's `y + h`, and each row's widths sum to 24). No blocking issues found, but one thing is worth confirming before merge. ## 🟡 Suggestions - **`smartctl_device_power_on_seconds` and `smartctl_device_bytes_written` — verify these metric names/units against the running exporter** (`kubernetes/apps/monitoring/dashboards/disk-health.json:207-208`, also used in the "SMART health" table's `renameByName` at line 218 and the unit overrides at lines 195-196). The NVMe SMART/Health Information log natively reports "Power On Hours" (hours, not seconds) and "Data Units Written" (units of 512,000 bytes, not raw bytes) — the prometheus-community `smartctl_exporter` has historically named these fields `smartctl_device_power_on_hours` and `smartctl_device_data_units_written` rather than the `_seconds`/`_bytes` names used here. If the deployed `v0.14.0` image (`kubernetes/apps/monitoring/smartctl-exporter.yaml:59`) still uses those upstream names, these two queries return no series and the "powered on" and "written" columns in the SMART health table render permanently empty — silently losing the "useful for projecting when a drive will reach it" capability the panel description promises. The PR body states all queries were checked against the live TSDB, which would already cover this, but since I can't reach the cluster's Prometheus from this checkout I couldn't independently confirm it. One query settles it: `curl -s http://<smartctl-exporter-pod>:9633/metrics | grep -E 'power_on|written'` or `{__name__=~"smartctl_device.*"}` against Prometheus. - **"An OSD that is up but out shows as a Ceph health warning instead"** (`kubernetes/apps/monitoring/dashboards/disk-health.json:31`) — worth double-checking against actual Ceph behavior. An OSD marked `out` while still `up` (e.g., during a deliberate `ceph osd out` for maintenance) is a normal administrative state and doesn't unconditionally raise a `HEALTH_WARN` on its own in my understanding; it's only degraded/misplaced PGs (transient, until rebalance completes) that would. If that's the case the sentence overstates what "OSDs down" now omits. Low impact since it's descriptive text only, not a query, but it could send someone chasing a warning that won't actually appear. ## Tests No automated tests exist or are expected for Grafana dashboard JSON in this repo — verification is manual (against a live TSDB, per the PR body) and via the "SmartctlExporterIncomplete" alert already in `prometheusrule-disk-wear.yaml`, which is a reasonable substitute (it fires if the SMART series fleet-wide coverage drops below `node_nvme_info`'s inventory). That alert would not, however, catch the narrower case of one or two specific fields (power-on, written) being wrong/absent while the rest of `smartctl_device_*` reports fine, which is exactly the scenario the suggestion above is about. ## Recommendation Looks reasonable — no confirmed blocking issues; the metric-name suggestion above is worth a final check before merge given it's unverifiable from this checkout.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#672