|
|
|
@@ -1,6 +1,6 @@
|
|
|
|
|
{
|
|
|
|
|
"annotations": {"list": []},
|
|
|
|
|
"description": "NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.",
|
|
|
|
|
"description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
|
|
|
|
|
"editable": true,
|
|
|
|
|
"fiscalYearStartMonth": 0,
|
|
|
|
|
"graphTooltip": 1,
|
|
|
|
@@ -9,7 +9,89 @@
|
|
|
|
|
"panels": [
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.",
|
|
|
|
|
"description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
|
|
|
|
|
"unit": "short"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
|
|
|
|
|
"id": 10,
|
|
|
|
|
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
|
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
|
|
|
|
|
"title": "Ceph health",
|
|
|
|
|
"type": "stat"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
|
|
|
|
|
"unit": "short"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
|
|
|
|
|
"id": 11,
|
|
|
|
|
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
|
|
|
"targets": [
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
|
|
|
|
|
],
|
|
|
|
|
"title": "OSDs down",
|
|
|
|
|
"type": "stat"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"max": 1,
|
|
|
|
|
"min": 0,
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
|
|
|
|
|
"unit": "percentunit"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
|
|
|
|
|
"id": 12,
|
|
|
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
|
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
|
|
|
|
|
"title": "Ceph capacity used",
|
|
|
|
|
"type": "stat"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"max": 1,
|
|
|
|
|
"min": 0,
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
|
|
|
|
|
"unit": "percentunit"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
|
|
|
|
|
"id": 13,
|
|
|
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
|
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
|
|
|
|
|
"title": "EPHEMERAL used (worst node)",
|
|
|
|
|
"type": "stat"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
@@ -19,27 +101,121 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 24, "x": 0, "y": 0},
|
|
|
|
|
"gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
|
|
|
|
|
"id": 1,
|
|
|
|
|
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
|
|
|
|
|
"targets": [
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"expr": "node_nvme_info * on (instance) group_left(nodename) node_uname_info",
|
|
|
|
|
"expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
|
|
|
|
|
"format": "table",
|
|
|
|
|
"instant": true,
|
|
|
|
|
"legendFormat": "",
|
|
|
|
|
"refId": "A"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"title": "Drive inventory",
|
|
|
|
|
"title": "Drive inventory (identity and role)",
|
|
|
|
|
"transformations": [
|
|
|
|
|
{
|
|
|
|
|
"id": "organize",
|
|
|
|
|
"options": {
|
|
|
|
|
"excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true},
|
|
|
|
|
"excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
|
|
|
|
|
"indexByName": {},
|
|
|
|
|
"renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware"}
|
|
|
|
|
"renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"type": "table"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"max": 100,
|
|
|
|
|
"min": 0,
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
|
|
|
|
"unit": "percent"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
|
|
|
|
|
"id": 14,
|
|
|
|
|
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
|
|
|
|
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
|
|
|
|
|
"title": "Worst NVMe wear",
|
|
|
|
|
"type": "stat"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "palette-classic"},
|
|
|
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"max": 100,
|
|
|
|
|
"min": 0,
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
|
|
|
|
"unit": "percent"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
|
|
|
|
|
"id": 15,
|
|
|
|
|
"options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
|
|
|
|
|
"legendFormat": "{{node}} {{model_name}}",
|
|
|
|
|
"refId": "A"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"title": "NVMe wear",
|
|
|
|
|
"type": "timeseries"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "thresholds"},
|
|
|
|
|
"custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
|
|
|
|
|
},
|
|
|
|
|
"overrides": [
|
|
|
|
|
{"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
|
|
|
|
|
{"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
|
|
|
|
|
{"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
|
|
|
|
{"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
|
|
|
|
{"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
|
|
|
|
|
{"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
|
|
|
|
|
"id": 16,
|
|
|
|
|
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
|
|
|
|
|
"targets": [
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
|
|
|
|
|
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
|
|
|
|
|
],
|
|
|
|
|
"title": "SMART health",
|
|
|
|
|
"transformations": [
|
|
|
|
|
{"id": "merge", "options": {}},
|
|
|
|
|
{
|
|
|
|
|
"id": "organize",
|
|
|
|
|
"options": {
|
|
|
|
|
"excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
|
|
|
|
|
"indexByName": {},
|
|
|
|
|
"renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
@@ -58,7 +234,7 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 9},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
|
|
|
|
|
"id": 2,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -91,7 +267,7 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 9},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
|
|
|
|
|
"id": 3,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -124,7 +300,7 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 18},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
|
|
|
|
|
"id": 4,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -162,7 +338,7 @@
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 18},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
|
|
|
|
|
"id": 5,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -184,7 +360,7 @@
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.",
|
|
|
|
|
"description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "palette-classic"},
|
|
|
|
@@ -195,7 +371,7 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 27},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
|
|
|
|
|
"id": 6,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -222,7 +398,7 @@
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 27},
|
|
|
|
|
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
|
|
|
|
|
"id": 7,
|
|
|
|
|
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
@@ -244,11 +420,40 @@
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 36},
|
|
|
|
|
"description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
|
|
|
|
|
"fieldConfig": {
|
|
|
|
|
"defaults": {
|
|
|
|
|
"color": {"mode": "palette-classic"},
|
|
|
|
|
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
|
|
|
|
"mappings": [],
|
|
|
|
|
"max": 1,
|
|
|
|
|
"min": 0,
|
|
|
|
|
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
|
|
|
|
|
"unit": "percentunit"
|
|
|
|
|
},
|
|
|
|
|
"overrides": []
|
|
|
|
|
},
|
|
|
|
|
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
|
|
|
|
|
"id": 9,
|
|
|
|
|
"options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
|
|
|
|
"targets": [
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
|
|
|
|
|
"legendFormat": "{{nodename}} /var",
|
|
|
|
|
"refId": "A"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"title": "EPHEMERAL usage",
|
|
|
|
|
"type": "timeseries"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
|
|
|
|
"gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
|
|
|
|
|
"id": 8,
|
|
|
|
|
"options": {
|
|
|
|
|
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
|
|
|
|
|
"content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.",
|
|
|
|
|
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
|
|
|
|
|
"mode": "markdown"
|
|
|
|
|
},
|
|
|
|
|
"title": "Gaps and caveats",
|
|
|
|
|
claude-botandClaude Opus 5
k.renner