feat(monitoring): put the SMART values on the dashboard

The exporter reports all nine drives now, so the dashboard shows what it measures: a worst-wear stat in the status row, wear per drive over time, and a table of the failure-now signals — available spare, media errors, the critical warning bitfield, power-on time and written bytes. Every expression was run against the live TSDB and the label names read off real series rather than a README. That mattered: the exporter labels devices `nvme0` and carries model and serial on a separate `smartctl_device` info metric, so each query joins on (node, device) to pull the model in. A legend on the device name alone would compare different hardware across machines, which is the same trap the block-layer panels already avoid. Wear and damage are presented as different things because they are. The wear percentage is a budget the drive spends and says nothing about tomorrow; spare exhaustion, media errors and the critical warning mean something is wrong now. The caveats panel says so, and no longer claims endurance is missing — it was missing when that text was written and is not any more. Also fixes the OSD stat, which still rendered two identical numbers side by side. The commit that fixed it did not make it into #668 before that PR merged, so the panel went out in the state that prompted the complaint. It now shows OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about it separately. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix(monitoring): privileged is the mechanism, not the shortcut' (#671 ) from smartctl-privileged into master
2026-08-27 15:26:09 +02:00 · 2026-08-27 13:19:38 +00:00 · 2026-08-27 15:14:32 +02:00 · 2026-08-27 13:05:08 +00:00
2 changed files with 164 additions and 55 deletions
@@ -1,6 +1,6 @@
 {
  "annotations": {"list": []},
-  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
@@ -19,7 +19,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
      "id": 10,
      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
@@ -28,7 +28,7 @@
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
+      "description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
@@ -38,14 +38,13 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
      "id": 11,
      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
      "targets": [
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
      ],
-      "title": "OSDs down / out",
+      "title": "OSDs down",
      "type": "stat"
    },
    {
@@ -62,7 +61,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
      "id": 12,
      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
@@ -83,7 +82,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
      "id": 13,
      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
@@ -115,7 +114,7 @@
          "refId": "A"
        }
      ],
-      "title": "Drive inventory",
+      "title": "Drive inventory (identity and role)",
      "transformations": [
        {
          "id": "organize",
@@ -128,6 +127,100 @@
      ],
      "type": "table"
    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
+      "id": 14,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
+      "title": "Worst NVMe wear",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 15,
+      "options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
+          "legendFormat": "{{node}} {{model_name}}",
+          "refId": "A"
+        }
+      ],
+      "title": "NVMe wear",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": [
+          {"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
+          {"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
+          {"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
+          {"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 16,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
+      ],
+      "title": "SMART health",
+      "transformations": [
+        {"id": "merge", "options": {}},
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
+            "indexByName": {},
+            "renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
+          }
+        }
+      ],
+      "type": "table"
+    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "description": "Read and write throughput per drive.",
@@ -141,7 +234,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
      "id": 2,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -174,7 +267,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
      "id": 3,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -207,7 +300,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
      "id": 4,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -245,7 +338,7 @@
          }
        ]
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
      "id": 5,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -278,7 +371,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
      "id": 6,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -305,7 +398,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
      "id": 7,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -340,7 +433,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
      "id": 9,
      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -356,11 +449,11 @@
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
      "id": 8,
      "options": {
        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
        "mode": "markdown"
      },
      "title": "Gaps and caveats",
@@ -16,17 +16,19 @@
 # healthy drives and not the worn ones was judged not worth a new service; the
 # owner chose fleet-wide coverage with the privilege exception instead.
 #
-# The exception is kept as small as the kernel allows: no `privileged: true`,
-# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are
-# added, everything else is dropped, and the only host mount is /dev — which
-# is required because the exporter must open the NVMe character devices and
-# Kubernetes has no way to pass individual character devices to a pod without
-# a device plugin.
+# The exception is `privileged: true`, and narrower settings were tried and
+# measured to fail — see the securityContext below for what was ruled out and
+# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
+# read-only root filesystem, and /dev as the only host mount.
 #
-# If this ever needs revisiting: the thing to check is whether the kernel has
-# grown a sysfs interface for NVMe health, not whether some other exporter
-# needs fewer privileges. The requirement comes from the ioctl gate, not from
-# the tool.
+# If this ever needs revisiting, there are exactly two things worth checking,
+# and neither is "does another exporter ask for less":
+#   - whether the kernel has grown a sysfs interface for NVMe health, which
+#     would remove the need for device access altogether
+#   - whether enough workloads need raw device access to justify a device
+#     plugin, which is the only way Kubernetes grants individual devices
+# The requirement comes from the device cgroup and the ioctl gate, not from
+# the choice of tool.
 #
 # Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
 # The repository is not archived; the pin is simply its newest tag.
@@ -66,38 +68,52 @@ spec:
              containerPort: 9633
              protocol: TCP
          securityContext:
-            # Root plus two capabilities, NOT privileged. See the header.
+            # PRIVILEGED, and there is no narrower setting that works. This
+            # was established by measurement after two narrower attempts
+            # failed, so the reasoning is recorded here rather than rediscovered:
+            #
+            #   Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
+            #   dropped. The exporter found all three drives per node and
+            #   could open none: "Smartctl open device: /dev/nvme0 failed:
+            #   Operation not permitted".
+            #
+            #   Attempt 2 — the same plus seccompProfile: Unconfined, on the
+            #   theory that EPERM is the default seccomp action for an
+            #   unlisted syscall and the kubelet runs seccompDefault: true.
+            #   /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
+            #   capabilities were effective and a filter was loaded. Removing
+            #   the filter changed nothing.
+            #
+            #   What actually settled it: `head -c 512 /dev/nvme0n1` inside
+            #   the container fails with EPERM too. A plain read of a block
+            #   device involves no ioctl, no capability and no syscall a
+            #   seccomp profile would filter. The block is the device cgroup —
+            #   containerd denies non-privileged containers access to host
+            #   device nodes whether or not /dev is mounted.
+            #
+            # Kubernetes exposes no per-device allowlist in a Pod spec; there
+            # is no equivalent of `docker --device`. The alternatives are
+            # `privileged: true` or a device plugin advertising the NVMe
+            # drives as a resource. A device plugin is the genuinely narrower
+            # answer and is worth building if more workloads ever need raw
+            # device access; for a single read-only metrics exporter it is a
+            # component to write, deploy and maintain in exchange for the same
+            # access.
            runAsUser: 0
            runAsNonRoot: false
-            privileged: false
-            allowPrivilegeEscalation: false
+            privileged: true
+            # Stated rather than omitted: the API rejects `privileged: true`
+            # together with `allowPrivilegeEscalation: false`, so leaving it
+            # out would only hide that privileged already implies it.
+            allowPrivilegeEscalation: true
            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["SYS_ADMIN", "SYS_RAWIO"]
-            # The capabilities alone are not enough, and this was measured
-            # rather than assumed. With the default profile the exporter found
-            # all three drives and could open none of them:
-            #   "Smartctl open device: /dev/nvme0 failed: Operation not
-            #   permitted"
-            # EPERM is the default seccomp action for a syscall the profile
-            # does not list. Reading /proc/1/status in the running container
-            # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN
-            # and SYS_RAWIO were both effective — while Seccomp was 2 with one
-            # filter loaded. The kubelet sets seccompDefault: true, so every
-            # pod that does not say otherwise gets RuntimeDefault.
-            #
-            # This is still narrower than `privileged: true`, which would also
-            # grant every capability, all devices and write access to /sys.
-            # Narrowing further would mean shipping a custom seccomp profile
-            # that allows exactly the ioctl smartctl needs — worth doing if
-            # this list of exceptions ever grows, not worth it for one pod.
-            seccompProfile:
-              type: Unconfined
          volumeMounts:
+            # Not read-only. smartmontools opens NVMe devices read-write on
+            # some paths, and once the container is privileged a read-only
+            # flag on this mount is not a boundary anyone should rely on —
+            # keeping it would suggest a protection that is not there.
            - name: dev
              mountPath: /dev
-              readOnly: true
            # smartctl is not known to need scratch space, but the root
            # filesystem is read-only and a tool that turns out to want /tmp
            # would crashloop instead of degrading. Cheaper to provide it than
Author	SHA1	Message	Date
claude-botandClaude Opus 5	24b2a0178d	feat(monitoring): put the SMART values on the dashboard ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details The exporter reports all nine drives now, so the dashboard shows what it measures: a worst-wear stat in the status row, wear per drive over time, and a table of the failure-now signals — available spare, media errors, the critical warning bitfield, power-on time and written bytes. Every expression was run against the live TSDB and the label names read off real series rather than a README. That mattered: the exporter labels devices `nvme0` and carries model and serial on a separate `smartctl_device` info metric, so each query joins on (node, device) to pull the model in. A legend on the device name alone would compare different hardware across machines, which is the same trap the block-layer panels already avoid. Wear and damage are presented as different things because they are. The wear percentage is a budget the drive spends and says nothing about tomorrow; spare exhaustion, media errors and the critical warning mean something is wrong now. The caveats panel says so, and no longer claims endurance is missing — it was missing when that text was written and is not any more. Also fixes the OSD stat, which still rendered two identical numbers side by side. The commit that fixed it did not make it into #668 before that PR merged, so the panel went out in the state that prompted the complaint. It now shows OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about it separately. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-27 15:26:09 +02:00
k.renner	b8061ea05d	Merge pull request 'fix(monitoring): privileged is the mechanism, not the shortcut' (#671 ) from smartctl-privileged into master Reviewed-on: k.renner/homelab#671	2026-08-27 13:19:38 +00:00
claude-botandClaude Opus 5	d0bea0dc15	fix(monitoring): privileged is the mechanism, not the shortcut ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Third attempt, and the first two were wrong for the same reason: I kept looking for the gate in the capability and syscall layers, and it was never there. Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything else dropped. It found all three drives per node and could open none of them. Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the default action for a syscall an unlisted profile blocks and that the kubelet runs seccompDefault: true. /proc/1/status confirmed both capabilities effective and a filter loaded, so the theory was at least testable — and removing the filter changed nothing. What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1` inside the container also fails with EPERM. Reading a block device involves no ioctl, no capability and nothing a seccomp profile filters. The denial is the device cgroup, which containerd applies to every non-privileged container regardless of whether /dev is mounted. Mounting /dev makes the nodes visible; it does not make them usable. Kubernetes has no per-device allowlist in a Pod spec — no equivalent of `docker --device`. So the honest position is that `privileged: true` here is not a lazy shortcut standing in for a narrower setting; it is the only mechanism the API offers, short of writing a device plugin. The manifest now says that, and records what was ruled out and how, so the next reader does not spend the same two rounds in the capability layer. The /dev mount loses its readOnly flag with it. smartmontools opens NVMe devices read-write on some paths, and once the container is privileged that flag suggests a boundary that is not there. A device plugin advertising the drives as a schedulable resource is the genuinely narrower answer and is noted as such. It is worth building when more than one workload needs raw device access; for a single read-only metrics exporter it is a component to write, deploy and maintain in exchange for the same access. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-27 15:14:32 +02:00
k.renner	8ccf5ac8e5	Merge pull request 'fix(monitoring): unconfine seccomp so smartctl can open the devices' (#670 ) from smartctl-seccomp into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#670	2026-08-27 13:05:08 +00:00