fix(monitoring): privileged is the mechanism, not the shortcut

Third attempt, and the first two were wrong for the same reason: I kept looking for the gate in the capability and syscall layers, and it was never there. Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything else dropped. It found all three drives per node and could open none of them. Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the default action for a syscall an unlisted profile blocks and that the kubelet runs seccompDefault: true. /proc/1/status confirmed both capabilities effective and a filter loaded, so the theory was at least testable — and removing the filter changed nothing. What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1` inside the container also fails with EPERM. Reading a block device involves no ioctl, no capability and nothing a seccomp profile filters. The denial is the device cgroup, which containerd applies to every non-privileged container regardless of whether /dev is mounted. Mounting /dev makes the nodes visible; it does not make them usable. Kubernetes has no per-device allowlist in a Pod spec — no equivalent of `docker --device`. So the honest position is that `privileged: true` here is not a lazy shortcut standing in for a narrower setting; it is the only mechanism the API offers, short of writing a device plugin. The manifest now says that, and records what was ruled out and how, so the next reader does not spend the same two rounds in the capability layer. The /dev mount loses its readOnly flag with it. smartmontools opens NVMe devices read-write on some paths, and once the container is privileged that flag suggests a boundary that is not there. A device plugin advertising the drives as a schedulable resource is the genuinely narrower answer and is noted as such. It is worth building when more than one workload needs raw device access; for a single read-only metrics exporter it is a component to write, deploy and maintain in exchange for the same access. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix(monitoring): unconfine seccomp so smartctl can open the devices' (#670 ) from smartctl-seccomp into master
2026-08-27 15:14:32 +02:00 · 2026-08-27 13:05:08 +00:00 · 2026-08-27 15:03:34 +02:00 · 2026-08-27 12:56:00 +00:00 · 2026-08-27 14:42:18 +02:00 · 2026-08-27 12:25:39 +00:00
17 changed files with 756 additions and 52 deletions
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:

 steps:
  - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/ansible
    depends_on: []
    # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-ansible:latest

  - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/k8s
    depends_on: []
    when:
@@ -77,7 +88,7 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest

  - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/tofu
    depends_on: []
    when:
@@ -108,7 +119,7 @@ steps:
  # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
  # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
  - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: secret-rotator
    depends_on: []
    when:
@@ -136,21 +147,6 @@ steps:
  # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
-  # First step moved to the maintained kaniko fork (#552). Google archived
-  # GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
-  # it to chainguard-dev/kaniko and kept releasing: this is v1.25.18, well past
-  # the v1.24.0 the Google repo ended on. The fork publishes no binaries of its
-  # own, so the image comes from gitlab-ci-utils, which builds it for amd64 and
-  # arm64 with versioned tags.
-  #
-  # Same /kaniko/executor path, same flags, and /bin/sh, base64, tr, mkdir and
-  # date are all present — verified by pulling the image and running it, not by
-  # reading the README.
-  #
-  # Deliberately only this step for now. A change to this file rebuilds every
-  # step anyway, so one run gives the new executor on alpine-jq beside seven
-  # unchanged kaniko builds — an A/B in a single pipeline. The other seven and
-  # ci-claude-refresh.yml follow once this one is green.
  - name: build-alpine-jq
    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-jq
@@ -181,7 +177,7 @@ steps:
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/cni-plugins
    depends_on: []
    when:
@@ -210,7 +206,7 @@ steps:
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-openssl
    depends_on: []
    when:
@@ -239,7 +235,7 @@ steps:
  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
  # builds the same way and lands in the same registry so the audit sees it.
  - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/heating-can
    depends_on: []
    when:
@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
      - |
        set -eu
        mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
        NEW=$$(npm view @anthropic-ai/claude-code version)
        case "$$NEW" in
          [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
  #    --build-arg to the resolved version (not "latest") guarantees the tag and
  #    the image content agree even if a new release lands mid-run.
  - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/claude
    depends_on: [decide]
    environment:
@@ -176,13 +186,10 @@ steps:
          echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
          exit 0
        fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
        # Exact-string swap of the current tag for the new one, consumers only.
        for f in $$CONSUMERS; do
          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
        {
          printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
          printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
          printf 'run the image so CI stops running a stale Claude Code.\n\n'
          printf 'Merging is a human decision — review, then merge to roll it out.\n'
        } > /tmp/pr-body.md
@@ -17,6 +17,6 @@ when:

 steps:
  - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh
@@ -25,7 +25,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -65,7 +65,7 @@ when:

 steps:
  - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -42,7 +42,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -30,7 +30,7 @@ when:

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -6,6 +6,7 @@ kind: Kustomization
 resources:
  - prometheusrule-ceph.yaml
  - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
  - prometheusrule-mariadb-backup.yaml
@@ -24,4 +25,5 @@ resources:
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
@@ -0,0 +1,121 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # THE GAP THESE RULES CLOSE
+        #
+        # Drive wear is invisible in every other signal this cluster has.
+        # `ceph -s` reads active+clean at any wear level, no Ceph health check
+        # fires until a device actually fails, and the block-layer metrics show
+        # throughput and latency but never endurance. That blind spot is how a
+        # set of OSD drives reached 81-87% before anyone looked (#549, #434).
+        #
+        # The numbers below are wear percentages reported by the drive itself:
+        # 100 means the manufacturer's rated endurance is used up. Drives do
+        # not stop at 100 — they typically keep working and start consuming
+        # spare blocks — so treat these as planning thresholds, not failure
+        # predictions.
+        - alert: NvmeDriveWearHigh
+          # 80% with months of runway left is the point where ordering a
+          # replacement is still a calm decision rather than an outage.
+          # `for` is long because wear is monotonic and slow: it cannot
+          # flap, and a short window would only make the alert noisier
+          # after a restart.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive has used {{ $value }}% of its rated write endurance.
+              Nothing is failing and no data is at risk yet. Check what the
+              drive is carrying before deciding urgency: an OSD is replaceable
+              with a rebalance, whereas the EPHEMERAL volume has no such
+              escape — a node simply stops when it goes. The Disk Health
+              dashboard resolves which drive holds which role.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # A drive that has consumed its spare area is no longer masking media
+        # defects, which is a materially different state from being worn:
+        # wear is a budget, spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          # The NVMe critical warning byte is a bitfield: spare below
+          # threshold, temperature out of range, degraded reliability,
+          # read-only mode, volatile-memory backup failed. Any non-zero value
+          # is the drive itself asking for attention.
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # Unlike wear, this counter should never move. Alert on the increase
+          # rather than the absolute value, so a drive that recorded errors
+          # long ago does not alert forever while a drive that starts failing
+          # today does.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # Without this the whole group is silently useless: no series means no
+        # alert fires, and a dashboard of empty panels reads much like a
+        # healthy one. This is the check that the checks are running.
+        - alert: SmartctlExporterIncomplete
+          # Compared against node-exporter's own NVMe inventory rather than a
+          # written-down fleet size. Adding or removing a drive then moves both
+          # sides together, so this cannot rot into a permanent false alarm the
+          # way a hardcoded count would.
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.
@@ -0,0 +1,29 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl itself polls the drives every 60s, so scraping faster than that
+    # returns the same numbers with a new timestamp. Wear moves on a scale of
+    # weeks; 60s is already far finer than the signal.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this every series is labelled with the exporter pod's IP and
+        # nothing else. A DaemonSet pod's IP says nothing about which machine
+        # the drive is in, and it changes on every reschedule — so the wear
+        # figure would be unattributable, which is most of its value gone.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
@@ -0,0 +1,382 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
+      ],
+      "title": "OSDs down / out",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}
@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
  - name: dashboard-blackbox-overview
    files:
      - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
  - name: dashboard-gitea-overview
    files:
      - gitea-overview.json
@@ -15,6 +15,7 @@ resources:
  - prometheus-gate-service.yaml
  - prometheus-gate-ingress.yaml
  - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
  - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -0,0 +1,158 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# WHY THIS HOLDS ELEVATED CAPABILITIES, and why that is a deliberate exception
+# rather than an oversight:
+#
+# NVMe reports health over an admin ioctl on the character device
+# (NVME_IOCTL_ADMIN_CMD), which the kernel gates on CAP_SYS_ADMIN. There is no
+# sysfs path for wear, media errors or written-bytes totals — hwmon publishes
+# the controller temperature and nothing else. Upstream smartctl_exporter
+# documents `privileged: true` and `user: root` for exactly this reason.
+#
+# The alternative that needs no privileges — reading Ceph's devicehealth
+# archive, which does hold full smartctl output — covers OSD drives only. In
+# this cluster that is three of nine, and the six it misses include the drives
+# that were retired from Ceph for wear and now carry EPHEMERAL. Covering the
+# healthy drives and not the worn ones was judged not worth a new service; the
+# owner chose fleet-wide coverage with the privilege exception instead.
+#
+# The exception is `privileged: true`, and narrower settings were tried and
+# measured to fail — see the securityContext below for what was ruled out and
+# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
+# read-only root filesystem, and /dev as the only host mount.
+#
+# If this ever needs revisiting, there are exactly two things worth checking,
+# and neither is "does another exporter ask for less":
+#   - whether the kernel has grown a sysfs interface for NVMe health, which
+#     would remove the need for device access altogether
+#   - whether enough workloads need raw device access to justify a device
+#     plugin, which is the only way Kubernetes grants individual devices
+# The requirement comes from the device cgroup and the ioctl gate, not from
+# the choice of tool.
+#
+# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
+# The repository is not archived; the pin is simply its newest tag.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device on purpose: the exporter scans for devices
+          # itself and rescans every 10m by default. Device names are not
+          # stable across these machines, so a hardcoded list would quietly
+          # stop covering a drive the next time one is replaced.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            # PRIVILEGED, and there is no narrower setting that works. This
+            # was established by measurement after two narrower attempts
+            # failed, so the reasoning is recorded here rather than rediscovered:
+            #
+            #   Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
+            #   dropped. The exporter found all three drives per node and
+            #   could open none: "Smartctl open device: /dev/nvme0 failed:
+            #   Operation not permitted".
+            #
+            #   Attempt 2 — the same plus seccompProfile: Unconfined, on the
+            #   theory that EPERM is the default seccomp action for an
+            #   unlisted syscall and the kubelet runs seccompDefault: true.
+            #   /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
+            #   capabilities were effective and a filter was loaded. Removing
+            #   the filter changed nothing.
+            #
+            #   What actually settled it: `head -c 512 /dev/nvme0n1` inside
+            #   the container fails with EPERM too. A plain read of a block
+            #   device involves no ioctl, no capability and no syscall a
+            #   seccomp profile would filter. The block is the device cgroup —
+            #   containerd denies non-privileged containers access to host
+            #   device nodes whether or not /dev is mounted.
+            #
+            # Kubernetes exposes no per-device allowlist in a Pod spec; there
+            # is no equivalent of `docker --device`. The alternatives are
+            # `privileged: true` or a device plugin advertising the NVMe
+            # drives as a resource. A device plugin is the genuinely narrower
+            # answer and is worth building if more workloads ever need raw
+            # device access; for a single read-only metrics exporter it is a
+            # component to write, deploy and maintain in exchange for the same
+            # access.
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: true
+            # Stated rather than omitted: the API rejects `privileged: true`
+            # together with `allowPrivilegeEscalation: false`, so leaving it
+            # out would only hide that privileged already implies it.
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # Not read-only. smartmontools opens NVMe devices read-write on
+            # some paths, and once the container is privileged a read-only
+            # flag on this mount is not a boundary anyone should rely on —
+            # keeping it would suggest a protection that is not there.
+            - name: dev
+              mountPath: /dev
+            # smartctl is not known to need scratch space, but the root
+            # filesystem is read-only and a tool that turns out to want /tmp
+            # would crashloop instead of degrading. Cheaper to provide it than
+            # to find out from a restart counter.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP
@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.