claude-botandClaude Opus 5
53887dd88f
I described the worn drives carrying EPHEMERAL as the dangerous placement, repeatedly, in the alert text and in the dashboard's caveats. That reads the volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be disposable — container images, kubelet state, pod scratch, spegel and Rook's local state all rebuild. Putting the drives with the least endurance left there is a sensible destination, not a risk someone should be warned about. The alert said "a node simply stops when it goes", which is simply wrong. A node losing /var rebuilds it. What survives the correction is narrower and checked rather than recalled: /var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node costs that member's data. Quorum holds at two of three and the member is removed and re-added. That is a rebuild, not data loss, and the alert now says so at that weight. The fill-level panel keeps its warning but for the right reason: a full /var stops a node being useful long before anything is lost, and nothing rebalances it the way Ceph rebalances a full OSD. That is a capacity argument, not an endurance one. None of this changes the monitoring — measuring every drive is still right, and the wear figures are unchanged. It changes what a reader is told the numbers mean, which is the part that was misleading. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
476 lines
34 KiB
JSON
476 lines
34 KiB
JSON
{
|
|
"annotations": {"list": []},
|
|
"description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
|
|
"editable": true,
|
|
"fiscalYearStartMonth": 0,
|
|
"graphTooltip": 1,
|
|
"id": null,
|
|
"links": [],
|
|
"panels": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
|
|
"unit": "short"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
|
|
"id": 10,
|
|
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
|
|
"title": "Ceph health",
|
|
"type": "stat"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
|
|
"unit": "short"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
|
|
"id": 11,
|
|
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
"targets": [
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
|
|
],
|
|
"title": "OSDs down",
|
|
"type": "stat"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"mappings": [],
|
|
"max": 1,
|
|
"min": 0,
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
|
|
"unit": "percentunit"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
|
|
"id": 12,
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
|
|
"title": "Ceph capacity used",
|
|
"type": "stat"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"mappings": [],
|
|
"max": 1,
|
|
"min": 0,
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
|
|
"unit": "percentunit"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
|
|
"id": 13,
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
|
|
"title": "EPHEMERAL used (worst node)",
|
|
"type": "stat"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
|
|
"id": 1,
|
|
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
|
|
"format": "table",
|
|
"instant": true,
|
|
"legendFormat": "",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"title": "Drive inventory (identity and role)",
|
|
"transformations": [
|
|
{
|
|
"id": "organize",
|
|
"options": {
|
|
"excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
|
|
"indexByName": {},
|
|
"renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
|
|
}
|
|
}
|
|
],
|
|
"type": "table"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"mappings": [],
|
|
"max": 100,
|
|
"min": 0,
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
|
"unit": "percent"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
|
|
"id": 14,
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
|
|
"title": "Worst NVMe wear",
|
|
"type": "stat"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
|
"mappings": [],
|
|
"max": 100,
|
|
"min": 0,
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
|
"unit": "percent"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
|
|
"id": 15,
|
|
"options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
|
|
"legendFormat": "{{node}} {{model_name}}",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"title": "NVMe wear",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "thresholds"},
|
|
"custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
|
|
},
|
|
"overrides": [
|
|
{"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
|
|
{"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
|
|
{"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
|
{"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
|
{"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
|
|
{"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
|
|
]
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
|
|
"id": 16,
|
|
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
|
|
"targets": [
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
|
|
],
|
|
"title": "SMART health",
|
|
"transformations": [
|
|
{"id": "merge", "options": {}},
|
|
{
|
|
"id": "organize",
|
|
"options": {
|
|
"excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
|
|
"indexByName": {},
|
|
"renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
|
|
}
|
|
}
|
|
],
|
|
"type": "table"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Read and write throughput per drive.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
|
|
"unit": "Bps"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
|
|
"id": 2,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "read {{nodename}} {{model}}",
|
|
"refId": "A"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "write {{nodename}} {{model}}",
|
|
"refId": "B"
|
|
}
|
|
],
|
|
"title": "Throughput",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
|
|
"unit": "iops"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
|
|
"id": 3,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "read {{nodename}} {{model}}",
|
|
"refId": "A"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "write {{nodename}} {{model}}",
|
|
"refId": "B"
|
|
}
|
|
],
|
|
"title": "IOPS",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
|
|
"unit": "s"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
|
|
"id": 4,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "read {{nodename}} {{model}}",
|
|
"refId": "A"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "write {{nodename}} {{model}}",
|
|
"refId": "B"
|
|
}
|
|
],
|
|
"title": "Mean operation latency",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
|
|
"unit": "percentunit"
|
|
},
|
|
"overrides": [
|
|
{
|
|
"matcher": {"id": "byRegexp", "options": "queue .*"},
|
|
"properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
|
|
}
|
|
]
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
|
|
"id": 5,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "busy {{nodename}} {{model}}",
|
|
"refId": "A"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "queue {{nodename}} {{model}}",
|
|
"refId": "B"
|
|
}
|
|
],
|
|
"title": "Utilisation and queue depth",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
|
|
"unit": "celsius"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
|
|
"id": 6,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "{{nodename}} {{model}}",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"title": "Controller temperature",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
|
|
"mappings": [],
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
|
|
"unit": "ms"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
|
|
"id": 7,
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "ceph_osd_apply_latency_ms",
|
|
"legendFormat": "apply {{ceph_daemon}}",
|
|
"refId": "A"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "ceph_osd_commit_latency_ms",
|
|
"legendFormat": "commit {{ceph_daemon}}",
|
|
"refId": "B"
|
|
}
|
|
],
|
|
"title": "Ceph OSD latency",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"color": {"mode": "palette-classic"},
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
|
"mappings": [],
|
|
"max": 1,
|
|
"min": 0,
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
|
|
"unit": "percentunit"
|
|
},
|
|
"overrides": []
|
|
},
|
|
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
|
|
"id": 9,
|
|
"options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
"targets": [
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
|
|
"legendFormat": "{{nodename}} /var",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"title": "EPHEMERAL usage",
|
|
"type": "timeseries"
|
|
},
|
|
{
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
"gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
|
|
"id": 8,
|
|
"options": {
|
|
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
|
|
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
|
|
"mode": "markdown"
|
|
},
|
|
"title": "Gaps and caveats",
|
|
"type": "text"
|
|
}
|
|
],
|
|
"preload": false,
|
|
"refresh": "1m",
|
|
"schemaVersion": 39,
|
|
"tags": ["storage", "nvme", "ceph"],
|
|
"templating": {"list": []},
|
|
"time": {"from": "now-6h", "to": "now"},
|
|
"timepicker": {},
|
|
"timezone": "",
|
|
"title": "Disk Health",
|
|
"uid": "disk-health",
|
|
"version": 1,
|
|
"weekStart": ""
|
|
}
|