feat(monitoring): put the disk health verdict on the disk health dashboard #668

Merged

k.renner merged 2 commits from disk-dashboard-status into master 2026-08-27 12:25:40 +00:00

Conversation 2 Commits 2 Files Changed 1

+127 -15

claude-bot commented 2026-08-27 12:18:36 +00:00

Member

Copy Link

Copy Source

Berechtigter Einwand — die erste Fassung zeichnete Durchsatz, Latenz und Temperatur, beantwortete aber nicht die Frage „ist etwas kaputt". Ich habe den Zustand von Hand aus Prometheus und Ceph zusammengesucht und im Chat berichtet. Genau diese Arbeit soll ein Dashboard abnehmen, und der Nächste hätte sie wiederholen müssen.

Neue Statuszeile ganz oben: Ceph-Health, OSDs up/in, belegte Ceph-Kapazität, und das vollste EPHEMERAL-Volume der drei Nodes. Jede Abfrage gegen die laufende TSDB geprüft, bevor sie ins JSON kam — aktuell HEALTH_WARN (die Auth-Key-Warnung), 3/3 OSDs, 55 % Kapazität, 8,3 % EPHEMERAL.

EPHEMERAL bekommt zusätzlich einen eigenen Graphen. Das war der eine handgeprüfte Wert ohne Platz im Dashboard, und es ist der Ausfall ohne Netz: Ceph verteilt von einem vollen OSD weg, von einem vollen /var verteilt nichts — der Node bleibt einfach stehen.

Das Caveats-Panel bekommt drei Befunde, die beim Beantworten der Frage angefallen sind:

Ceph hat Verschleißwerte für die OSDs, im devicehealth-Modul und nicht in Prometheus. Das Panel nennt den Befehl statt des aktuellen Werts — eine Zahl, die man in ein Dashboard schreibt, ist im Moment des Schreibens veraltet. Ebenfalls festgehalten: die ausgemusterten Crucial sind aus Cephs Geräteregister gelöscht, ihre Historie ist nicht mehr rekonstruierbar. Verifiziert, indem ich sie abgefragt habe und ENOENT bekam.

Talos bootet mit nvme_core.io_timeout=4294967295 — NVMe-Kommandos laufen nie in einen Timeout. Ein sterbendes Laufwerk hängt, statt Fehler zu liefern, und das Latenz-Panel wird flach statt auszuschlagen. Das kehrt um, was ein Leser sonst aus einem ruhigen Graphen schließen würde.

Talos-Kernel-Logs erreichen Loki nicht; die einzigen Job-Labels sind Kubernetes-Pods und ein systemd-journal von woanders. Gefunden, indem ich Loki nach NVMe-Fehlern durchsucht, nichts bekommen und dann geprüft habe, ob der Collector überhaupt existiert — er existiert nicht. talosctl dmesg ist die einzige Quelle, ein Ringpuffer, und die Reboots von heute früh haben ihn geleert.

Refs #549

Berechtigter Einwand — die erste Fassung zeichnete Durchsatz, Latenz und Temperatur, beantwortete aber nicht die Frage „ist etwas kaputt". Ich habe den Zustand von Hand aus Prometheus und Ceph zusammengesucht und im Chat berichtet. Genau diese Arbeit soll ein Dashboard abnehmen, und der Nächste hätte sie wiederholen müssen. **Neue Statuszeile ganz oben:** Ceph-Health, OSDs up/in, belegte Ceph-Kapazität, und das vollste EPHEMERAL-Volume der drei Nodes. Jede Abfrage gegen die laufende TSDB geprüft, bevor sie ins JSON kam — aktuell `HEALTH_WARN` (die Auth-Key-Warnung), 3/3 OSDs, 55 % Kapazität, 8,3 % EPHEMERAL. **EPHEMERAL bekommt zusätzlich einen eigenen Graphen.** Das war der eine handgeprüfte Wert ohne Platz im Dashboard, und es ist der Ausfall ohne Netz: Ceph verteilt von einem vollen OSD weg, von einem vollen `/var` verteilt nichts — der Node bleibt einfach stehen. **Das Caveats-Panel bekommt drei Befunde, die beim Beantworten der Frage angefallen sind:** Ceph *hat* Verschleißwerte für die OSDs, im devicehealth-Modul und nicht in Prometheus. Das Panel nennt den Befehl statt des aktuellen Werts — eine Zahl, die man in ein Dashboard schreibt, ist im Moment des Schreibens veraltet. Ebenfalls festgehalten: die ausgemusterten Crucial sind aus Cephs Geräteregister gelöscht, ihre Historie ist nicht mehr rekonstruierbar. Verifiziert, indem ich sie abgefragt habe und `ENOENT` bekam. Talos bootet mit `nvme_core.io_timeout=4294967295` — NVMe-Kommandos laufen nie in einen Timeout. Ein sterbendes Laufwerk **hängt**, statt Fehler zu liefern, und das Latenz-Panel wird flach statt auszuschlagen. Das kehrt um, was ein Leser sonst aus einem ruhigen Graphen schließen würde. Talos-Kernel-Logs erreichen Loki nicht; die einzigen Job-Labels sind Kubernetes-Pods und ein systemd-journal von woanders. Gefunden, indem ich Loki nach NVMe-Fehlern durchsucht, nichts bekommen und dann geprüft habe, ob der Collector überhaupt existiert — er existiert nicht. `talosctl dmesg` ist die einzige Quelle, ein Ringpuffer, und die Reboots von heute früh haben ihn geleert. Refs #549

claude-bot added 1 commit 2026-08-27 12:18:36 +00:00

feat(monitoring): put the disk health verdict on the disk health dashboard ... afbb6c3a95

The first version graphed throughput, latency and temperature but never answered
"is anything wrong". Establishing that still meant querying Prometheus and Ceph
by hand — which is exactly the work a dashboard exists to remove, and which the
next person would have had to repeat.

A status row now leads: Ceph health, OSDs up and in, Ceph capacity used, and the
fullest EPHEMERAL volume across the three nodes. Each expression was run against
the live TSDB before being written in.

EPHEMERAL fill level gets its own graph too. It was the one number checked by
hand that had no home here, and it is the failure that has no safety net: Ceph
rebalances away from a full OSD, nothing rebalances away from a full /var — the
node simply stops.

The caveats panel gains three things found while answering the question:

Ceph does hold wear for the OSDs, in its devicehealth module and not in
Prometheus. The panel names the command rather than the current value, because a
number written into a dashboard is stale the moment it is written. It also
records that the retired Crucial drives were purged from Ceph's device registry,
so their history is not recoverable either — verified by asking for it and
getting ENOENT.

Talos boots with nvme_core.io_timeout=4294967295, so NVMe commands never time
out. A dying drive stalls instead of returning errors, and the latency panel
goes flat rather than spiking. That inverts what a reader would otherwise
conclude from a quiet graph.

Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and
a systemd-journal from elsewhere. I found this by searching Loki for NVMe errors,
getting nothing, and checking whether the collector existed before believing the
zero; it did not. talosctl dmesg is the only source, it is a ring buffer, and
this morning's reboots emptied it.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed 2026-08-27 12:21:18 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit afbb6c3a9519c50c1db924b6c61edea7954cc342.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR adds a "top row" status strip (Ceph health, OSDs up/in, Ceph capacity used, worst-node EPHEMERAL usage) to kubernetes/apps/monitoring/dashboards/disk-health.json, gives EPHEMERAL (/var) its own timeseries panel, and expands the "Gaps and caveats" panel with three new findings (Ceph devicehealth wear data, the NVMe timeout override, missing Talos kernel logs in Loki). It's a single-file, dashboard-only change already wired into configMapGenerator in kustomization.yaml:41-43 (no new registration needed). I found no correctness or security issues — panel IDs, gridPos layout, and PromQL all check out against the rest of the file and against how Ceph's Prometheus exporter reports these metrics.

🟡 Suggestions

• disk-health.json:36 and the "OSDs up / in" description (disk-health.json:31) hardcode the cluster at 3 nodes/OSDs (green only at exactly 3, red below 2). This matches the existing dashboard's assumption of a fixed 3-node cluster (see the pre-existing "Drive inventory" and per-role descriptions), so it's consistent with current convention rather than a new problem — just flagging that these thresholds, like the rest of the dashboard, will need a manual update if the cluster is ever resized.
• disk-health.json:89 (EPHEMERAL used (worst node)) computes max(...) with no node label attached, so hovering the stat gives you the worst percentage but not which node it's on — you'd have to cross-reference the "EPHEMERAL usage" timeseries panel below (id: 9) to find the offender. Minor, and arguably intentional for a compact top-row tile, but worth a one-line note in the description if it's not obvious to future readers.

Tests

Not applicable — this is a Grafana dashboard JSON file with no dedicated test suite in the repo. The closest safety net is the kustomize-build step in .woodpecker/ci-manifests-test.yml, which will build the dashboard-disk-health ConfigMap and catch a missing/misnamed file, but it does not validate dashboard JSON semantics (panel ID collisions, gridPos overlaps, PromQL syntax). I checked those by hand: panel IDs 10-13 are new and don't collide with the existing 1-9, brace count is balanced (294/294), and the new top row's gridPos (y:0, h:4) lines up cleanly with the shifted-down panels below it (each existing panel's y moved by exactly +4, all four x=0/x=12 pairs share y, no overlaps). The Ceph health value mapping (0/1/2 → OK/WARN/ERR) matches the standard ceph-mgr Prometheus module convention. I could not independently query the live Prometheus/Ceph exporter to confirm the metrics return data as described (the PR body claims each query was checked against the running TSDB) — if that matters, the way to settle it is loading the dashboard in Grafana and confirming the top row renders non-empty values.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `afbb6c3a9519c50c1db924b6c61edea7954cc342`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR adds a "top row" status strip (Ceph health, OSDs up/in, Ceph capacity used, worst-node EPHEMERAL usage) to `kubernetes/apps/monitoring/dashboards/disk-health.json`, gives EPHEMERAL (`/var`) its own timeseries panel, and expands the "Gaps and caveats" panel with three new findings (Ceph devicehealth wear data, the NVMe timeout override, missing Talos kernel logs in Loki). It's a single-file, dashboard-only change already wired into `configMapGenerator` in `kustomization.yaml:41-43` (no new registration needed). I found no correctness or security issues — panel IDs, `gridPos` layout, and PromQL all check out against the rest of the file and against how Ceph's Prometheus exporter reports these metrics. ## 🟡 Suggestions - `disk-health.json:36` and the "OSDs up / in" description (`disk-health.json:31`) hardcode the cluster at 3 nodes/OSDs (`green` only at exactly 3, `red` below 2). This matches the existing dashboard's assumption of a fixed 3-node cluster (see the pre-existing "Drive inventory" and per-role descriptions), so it's consistent with current convention rather than a new problem — just flagging that these thresholds, like the rest of the dashboard, will need a manual update if the cluster is ever resized. - `disk-health.json:89` (`EPHEMERAL used (worst node)`) computes `max(...)` with no node label attached, so hovering the stat gives you the worst percentage but not which node it's on — you'd have to cross-reference the "EPHEMERAL usage" timeseries panel below (`id: 9`) to find the offender. Minor, and arguably intentional for a compact top-row tile, but worth a one-line note in the description if it's not obvious to future readers. ## Tests Not applicable — this is a Grafana dashboard JSON file with no dedicated test suite in the repo. The closest safety net is the `kustomize-build` step in `.woodpecker/ci-manifests-test.yml`, which will build the `dashboard-disk-health` ConfigMap and catch a missing/misnamed file, but it does not validate dashboard JSON semantics (panel ID collisions, `gridPos` overlaps, PromQL syntax). I checked those by hand: panel IDs 10-13 are new and don't collide with the existing 1-9, brace count is balanced (294/294), and the new top row's `gridPos` (`y:0, h:4`) lines up cleanly with the shifted-down panels below it (each existing panel's `y` moved by exactly +4, all four x=0/x=12 pairs share `y`, no overlaps). The Ceph health value mapping (`0`/`1`/`2` → OK/WARN/ERR) matches the standard ceph-mgr Prometheus module convention. I could not independently query the live Prometheus/Ceph exporter to confirm the metrics return data as described (the PR body claims each query was checked against the running TSDB) — if that matters, the way to settle it is loading the dashboard in Grafana and confirming the top row renders non-empty values. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=afbb6c3a9519c50c1db924b6c61edea7954cc342 -->

claude-bot added 1 commit 2026-08-27 12:23:44 +00:00

fix(monitoring): state mechanisms in the panel text, not the current hardware ... 1213c771ca

The descriptions I wrote listed the drives fitted today: which model holds which
role, that nvme0 is the system disk on two named nodes and something else on the
third, that a count below three means a drive is gone. All true this afternoon,
none of it true after the next swap — and prose does not get re-read when
hardware changes. That is the same failure this session already found in the
velero schedule comment, in the ci-claude consumer list, and in an instruction
pointing at a deleted file. I built a fresh instance of it while writing about
the others.

The text now describes mechanisms that outlive any particular set of drives, and
points at the inventory table for anything specific. Roles are assigned by serial
in Talos and by by-id path in Rook; a broad `disk.transport == 'nvme'` selector
matches every drive in the box; a drive retired from Ceph is not necessarily out
of service. None of that changes when the hardware does.

The OSD-count panel had the same problem in a worse form, since it was a
threshold rather than a sentence: green at exactly 3. It now reads OSDs down and
out as a difference against ceph_osd_metadata, so growing or shrinking the
cluster does not turn the panel red.

Fixes a claim that was simply false: the inventory description said the OSD
column came from ceph_disk_occupation, and the query never joined it. There was
no such column. It exists now, as a PromQL outer join — the matched half via
group_left, the unmatched half via `unless`, unioned — so drives with no OSD
still appear. Verified against the live TSDB: nine rows, three carrying a
ceph_daemon.

A Grafana `merge` transformation would have been the shorter route and would not
have worked. Merge only combines rows whose shared fields all agree, and the two
queries disagree on `instance` and `job` — node-exporter reports an IP and port
where ceph-mgr reports a node name — so it would have produced twelve rows
instead of nine, silently. Doing the join in PromQL also made it testable before
commit rather than after deploy.

Temperature thresholds are now labelled as generic NVMe limits rather than any
drive's rating, since they are not derived from a datasheet.

Refs #549

k.renner merged commit 8c03a6e310 into master 2026-08-27 12:25:40 +00:00

k.renner referenced this issue from a commit 2026-08-27 12:25:41 +00:00

Merge pull request 'feat(monitoring): put the disk health verdict on the disk health dashboard' (#668) from disk-dashboard-status into master

review-bot reviewed 2026-08-27 12:26:27 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-4-6 (complexity: normal), commit 1213c771ca577bb369eb6637428b7c69c5a5cf03.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Adds a four-panel status row (Ceph health, OSDs down/out, Ceph capacity, EPHEMERAL worst node) above the existing performance graphs, a full-width EPHEMERAL timeseries panel, an enhanced drive-inventory join that surfaces the Ceph OSD assignment, and a substantially expanded "Gaps and caveats" text panel. The change directly addresses the identified gap: the previous dashboard showed throughput/latency/temperature but not "is anything broken right now." No blocking issues.

🔴 Blocking

None.

🟡 Suggestions

ceph_health_status without an aggregation (disk-health.json:25).
If the Ceph exporter produces more than one time series for this metric (e.g., two replicas or two clusters), the stat panel renders multiple tiles. Adding max(ceph_health_status) or a job-label filter {job="ceph"} would make the panel unambiguous. In a single-cluster homelab this is theoretical, but cheap to guard against.

EPHEMERAL stat max() matches any job exposing mountpoint="/var" (disk-health.json:89).
The expression max(1 - node_filesystem_avail_bytes{mountpoint="/var"} / node_filesystem_size_bytes{mountpoint="/var"}) does not restrict to the node-exporter job. If any other exporter or a pod inadvertently exposes a /var filesystem series, the max could be wrong. Adding {job="node-exporter"} (or whatever the actual job label is in this cluster) removes the ambiguity. Same consideration applies to the EPHEMERAL timeseries panel at disk-health.json:349.

OSD down/out stat shows two values in one panel (disk-health.json:44-47).
Two targets ("down" and "out") in a single stat panel produce two separate stat tiles with their own backgrounds. This works, but a reader glancing quickly could interpret it as one number. Consider splitting into two stat panels (each w=3) or adding a brief label in the panel title (e.g., "OSDs: down / out") and relying on the individual legendFormat labels — but this is cosmetic.

🟢 Nits

• The drive-inventory join query (disk-health.json:111) is correct and already well-tested per the PR description. The comment in the panel description ("so that no panel description has to be") gives good rationale for why this is the canonical source of truth — that prose could move into the panel description field rather than the dashboard-level description, but this is just a placement preference.
• The nvmedev helper label created by label_replace is correctly excluded from the table display via "nvmedev": true in excludeByName (disk-health.json:123). Good catch.

Tests

This is a Grafana dashboard JSON; there are no unit tests to write. The PR description documents the author's approach of running every new PromQL expression against the live TSDB before committing (verifying current values of HEALTH_WARN, OSD counts, capacity, and EPHEMERAL percentage). That is the appropriate test for this change.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-4-6` (complexity: normal), commit `1213c771ca577bb369eb6637428b7c69c5a5cf03`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a four-panel status row (Ceph health, OSDs down/out, Ceph capacity, EPHEMERAL worst node) above the existing performance graphs, a full-width EPHEMERAL timeseries panel, an enhanced drive-inventory join that surfaces the Ceph OSD assignment, and a substantially expanded "Gaps and caveats" text panel. The change directly addresses the identified gap: the previous dashboard showed throughput/latency/temperature but not "is anything broken right now." No blocking issues. ## 🔴 Blocking None. ## 🟡 Suggestions **`ceph_health_status` without an aggregation (`disk-health.json:25`).** If the Ceph exporter produces more than one time series for this metric (e.g., two replicas or two clusters), the stat panel renders multiple tiles. Adding `max(ceph_health_status)` or a job-label filter `{job="ceph"}` would make the panel unambiguous. In a single-cluster homelab this is theoretical, but cheap to guard against. **EPHEMERAL stat `max()` matches any job exposing `mountpoint="/var"` (`disk-health.json:89`).** The expression `max(1 - node_filesystem_avail_bytes{mountpoint="/var"} / node_filesystem_size_bytes{mountpoint="/var"})` does not restrict to the node-exporter job. If any other exporter or a pod inadvertently exposes a `/var` filesystem series, the max could be wrong. Adding `{job="node-exporter"}` (or whatever the actual job label is in this cluster) removes the ambiguity. Same consideration applies to the EPHEMERAL timeseries panel at `disk-health.json:349`. **OSD down/out stat shows two values in one panel (`disk-health.json:44-47`).** Two targets ("down" and "out") in a single stat panel produce two separate stat tiles with their own backgrounds. This works, but a reader glancing quickly could interpret it as one number. Consider splitting into two stat panels (each w=3) or adding a brief label in the panel title (e.g., "OSDs: down / out") and relying on the individual `legendFormat` labels — but this is cosmetic. ## 🟢 Nits - The drive-inventory join query (`disk-health.json:111`) is correct and already well-tested per the PR description. The comment in the panel description ("so that no panel description has to be") gives good rationale for why this is the canonical source of truth — that prose could move into the panel `description` field rather than the dashboard-level `description`, but this is just a placement preference. - The `nvmedev` helper label created by `label_replace` is correctly excluded from the table display via `"nvmedev": true` in `excludeByName` (`disk-health.json:123`). Good catch. ## Tests This is a Grafana dashboard JSON; there are no unit tests to write. The PR description documents the author's approach of running every new PromQL expression against the live TSDB before committing (verifying current values of `HEALTH_WARN`, OSD counts, capacity, and EPHEMERAL percentage). That is the appropriate test for this change. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=1213c771ca577bb369eb6637428b7c69c5a5cf03 -->

claude-bot referenced this issue from a commit 2026-08-27 13:26:17 +00:00

feat(monitoring): put the SMART values on the dashboard

claude-bot referenced this pull request 2026-08-27 13:26:39 +00:00

feat(monitoring): put the SMART values on the dashboard #672

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#668