From afbb6c3a9519c50c1db924b6c61edea7954cc342 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 27 Aug 2026 14:18:11 +0200 Subject: [PATCH 1/2] feat(monitoring): put the disk health verdict on the disk health dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first version graphed throughput, latency and temperature but never answered "is anything wrong". Establishing that still meant querying Prometheus and Ceph by hand — which is exactly the work a dashboard exists to remove, and which the next person would have had to repeat. A status row now leads: Ceph health, OSDs up and in, Ceph capacity used, and the fullest EPHEMERAL volume across the three nodes. Each expression was run against the live TSDB before being written in. EPHEMERAL fill level gets its own graph too. It was the one number checked by hand that had no home here, and it is the failure that has no safety net: Ceph rebalances away from a full OSD, nothing rebalances away from a full /var — the node simply stops. The caveats panel gains three things found while answering the question: Ceph does hold wear for the OSDs, in its devicehealth module and not in Prometheus. The panel names the command rather than the current value, because a number written into a dashboard is stale the moment it is written. It also records that the retired Crucial drives were purged from Ceph's device registry, so their history is not recoverable either — verified by asking for it and getting ENOENT. Talos boots with nvme_core.io_timeout=4294967295, so NVMe commands never time out. A dying drive stalls instead of returning errors, and the latency panel goes flat rather than spiking. That inverts what a reader would otherwise conclude from a quiet graph. Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. I found this by searching Loki for NVMe errors, getting nothing, and checking whether the collector existed before believing the zero; it did not. talosctl dmesg is the only source, it is a ring buffer, and this morning's reboots emptied it. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) --- .../monitoring/dashboards/disk-health.json | 132 ++++++++++++++++-- 1 file changed, 122 insertions(+), 10 deletions(-) diff --git a/kubernetes/apps/monitoring/dashboards/disk-health.json b/kubernetes/apps/monitoring/dashboards/disk-health.json index 6bc35ea9..262a7420 100644 --- a/kubernetes/apps/monitoring/dashboards/disk-health.json +++ b/kubernetes/apps/monitoring/dashboards/disk-health.json @@ -1,12 +1,95 @@ { "annotations": {"list": []}, - "description": "NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.", + "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.", "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 1, "id": null, "links": [], "panels": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "description": "Ceph's overall verdict. A HEALTH_WARN here is frequently about something other than the drives — insecure auth key types, for one — so read the detail before treating it as a storage problem.", + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}], + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0}, + "id": 10, + "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, + "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}], + "title": "Ceph health", + "type": "stat" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "description": "OSDs up and in — one Samsung PM983 per node. Anything below 3 means a drive or a node is gone.", + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "mappings": [], + "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "orange", "value": 2}, {"color": "green", "value": 3}]}, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0}, + "id": 11, + "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, + "targets": [ + {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_up)", "legendFormat": "up", "refId": "A"}, + {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_in)", "legendFormat": "in", "refId": "B"} + ], + "title": "OSDs up / in", + "type": "stat" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.", + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]}, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0}, + "id": 12, + "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, + "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}], + "title": "Ceph capacity used", + "type": "stat" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "description": "Fullest EPHEMERAL volume across the three nodes — the Crucial P3 in each machine, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.", + "fieldConfig": { + "defaults": { + "color": {"mode": "thresholds"}, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]}, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0}, + "id": 13, + "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, + "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}], + "title": "EPHEMERAL used (worst node)", + "type": "stat" + }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, "description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.", @@ -19,7 +102,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 24, "x": 0, "y": 0}, + "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4}, "id": 1, "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]}, "targets": [ @@ -58,7 +141,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 12, "x": 0, "y": 9}, + "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13}, "id": 2, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -91,7 +174,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 12, "x": 12, "y": 9}, + "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13}, "id": 3, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -124,7 +207,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 12, "x": 0, "y": 18}, + "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22}, "id": 4, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -162,7 +245,7 @@ } ] }, - "gridPos": {"h": 9, "w": 12, "x": 12, "y": 18}, + "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22}, "id": 5, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -195,7 +278,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 12, "x": 0, "y": 27}, + "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31}, "id": 6, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -222,7 +305,7 @@ }, "overrides": [] }, - "gridPos": {"h": 9, "w": 12, "x": 12, "y": 27}, + "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31}, "id": 7, "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, "targets": [ @@ -244,11 +327,40 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "gridPos": {"h": 8, "w": 24, "x": 0, "y": 36}, + "description": "EPHEMERAL fill level per node — the volume on the retired Crucial drives. Growth here is usually container images and pod scratch space rather than anything Ceph does.", + "fieldConfig": { + "defaults": { + "color": {"mode": "palette-classic"}, + "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}}, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]}, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40}, + "id": 9, + "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info", + "legendFormat": "{{nodename}} /var", + "refId": "A" + } + ], + "title": "EPHEMERAL usage", + "type": "timeseries" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48}, "id": 8, "options": { "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false}, - "content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.", + "content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear for the OSDs, but only inside its own devicehealth module and not in Prometheus. To see it: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers the Samsung OSDs only — the retired Crucial drives were purged from Ceph's device registry, so even their history is gone.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries EPHEMERAL. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing watches.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and systemd-journal from elsewhere. `talosctl -n dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error that happened before the last reboot left no trace anywhere.", "mode": "markdown" }, "title": "Gaps and caveats", -- 2.54.0 From 1213c771ca577bb369eb6637428b7c69c5a5cf03 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 27 Aug 2026 14:23:36 +0200 Subject: [PATCH 2/2] fix(monitoring): state mechanisms in the panel text, not the current hardware MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The descriptions I wrote listed the drives fitted today: which model holds which role, that nvme0 is the system disk on two named nodes and something else on the third, that a count below three means a drive is gone. All true this afternoon, none of it true after the next swap — and prose does not get re-read when hardware changes. That is the same failure this session already found in the velero schedule comment, in the ci-claude consumer list, and in an instruction pointing at a deleted file. I built a fresh instance of it while writing about the others. The text now describes mechanisms that outlive any particular set of drives, and points at the inventory table for anything specific. Roles are assigned by serial in Talos and by by-id path in Rook; a broad `disk.transport == 'nvme'` selector matches every drive in the box; a drive retired from Ceph is not necessarily out of service. None of that changes when the hardware does. The OSD-count panel had the same problem in a worse form, since it was a threshold rather than a sentence: green at exactly 3. It now reads OSDs down and out as a difference against ceph_osd_metadata, so growing or shrinking the cluster does not turn the panel red. Fixes a claim that was simply false: the inventory description said the OSD column came from ceph_disk_occupation, and the query never joined it. There was no such column. It exists now, as a PromQL outer join — the matched half via group_left, the unmatched half via `unless`, unioned — so drives with no OSD still appear. Verified against the live TSDB: nine rows, three carrying a ceph_daemon. A Grafana `merge` transformation would have been the shorter route and would not have worked. Merge only combines rows whose shared fields all agree, and the two queries disagree on `instance` and `job` — node-exporter reports an IP and port where ceph-mgr reports a node name — so it would have produced twelve rows instead of nine, silently. Doing the join in PromQL also made it testable before commit rather than after deploy. Temperature thresholds are now labelled as generic NVMe limits rather than any drive's rating, since they are not derived from a datasheet. Refs #549 --- .../monitoring/dashboards/disk-health.json | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/kubernetes/apps/monitoring/dashboards/disk-health.json b/kubernetes/apps/monitoring/dashboards/disk-health.json index 262a7420..81233b67 100644 --- a/kubernetes/apps/monitoring/dashboards/disk-health.json +++ b/kubernetes/apps/monitoring/dashboards/disk-health.json @@ -1,6 +1,6 @@ { "annotations": {"list": []}, - "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.", + "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.", "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 1, @@ -9,7 +9,7 @@ "panels": [ { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "Ceph's overall verdict. A HEALTH_WARN here is frequently about something other than the drives — insecure auth key types, for one — so read the detail before treating it as a storage problem.", + "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.", "fieldConfig": { "defaults": { "color": {"mode": "thresholds"}, @@ -28,12 +28,12 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "OSDs up and in — one Samsung PM983 per node. Anything below 3 means a drive or a node is gone.", + "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.", "fieldConfig": { "defaults": { "color": {"mode": "thresholds"}, "mappings": [], - "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "orange", "value": 2}, {"color": "green", "value": 3}]}, + "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}, "unit": "short" }, "overrides": [] @@ -42,10 +42,10 @@ "id": 11, "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, "targets": [ - {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_up)", "legendFormat": "up", "refId": "A"}, - {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_in)", "legendFormat": "in", "refId": "B"} + {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"}, + {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"} ], - "title": "OSDs up / in", + "title": "OSDs down / out", "type": "stat" }, { @@ -71,7 +71,7 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "Fullest EPHEMERAL volume across the three nodes — the Crucial P3 in each machine, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.", + "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.", "fieldConfig": { "defaults": { "color": {"mode": "thresholds"}, @@ -92,7 +92,7 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.", + "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.", "fieldConfig": { "defaults": { "color": {"mode": "thresholds"}, @@ -108,7 +108,7 @@ "targets": [ { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "expr": "node_nvme_info * on (instance) group_left(nodename) node_uname_info", + "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))", "format": "table", "instant": true, "legendFormat": "", @@ -120,9 +120,9 @@ { "id": "organize", "options": { - "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true}, + "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true}, "indexByName": {}, - "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware"} + "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"} } } ], @@ -267,7 +267,7 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.", + "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.", "fieldConfig": { "defaults": { "color": {"mode": "palette-classic"}, @@ -327,7 +327,7 @@ }, { "datasource": {"type": "prometheus", "uid": "prometheus"}, - "description": "EPHEMERAL fill level per node — the volume on the retired Crucial drives. Growth here is usually container images and pod scratch space rather than anything Ceph does.", + "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.", "fieldConfig": { "defaults": { "color": {"mode": "palette-classic"}, @@ -360,7 +360,7 @@ "id": 8, "options": { "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false}, - "content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear for the OSDs, but only inside its own devicehealth module and not in Prometheus. To see it: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers the Samsung OSDs only — the retired Crucial drives were purged from Ceph's device registry, so even their history is gone.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries EPHEMERAL. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing watches.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and systemd-journal from elsewhere. `talosctl -n dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error that happened before the last reboot left no trace anywhere.", + "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.", "mode": "markdown" }, "title": "Gaps and caveats", -- 2.54.0