feat(monitoring): add a disk health dashboard

Throughput, IOPS, mean operation latency, utilisation, queue depth and
controller temperature for all nine NVMe drives, plus Ceph's own OSD latency for
comparison against the block layer.

Every panel joins node_disk_* to node_nvme_info and node_uname_info rather than
plotting the device name. That is not decoration: nvme0 is the Kingston system
disk on k8s-1 and k8s-3 and the Crucial on k8s-2, so a legend keyed on the
device compares different hardware across nodes and does so silently. The
inventory table resolves the OSD column through ceph_disk_occupation for the
same reason — it follows Rook instead of restating a device list that goes stale
the next time a drive is swapped.

Every expression was run against Prometheus before being written into the JSON,
which is also how the caveats in the panel descriptions were found. Latency is
accumulated-time over completed-operations, so it is a mean with no tail and it
vanishes while a device is idle: absence of traffic, not absence of a problem.
Utilisation near 100% on NVMe does not mean saturation the way it does on a
spindle, which is why queue depth shares that panel.

Temperature is the only SMART-adjacent value present, because it is the only one
the kernel publishes through sysfs. Wear, written-bytes totals, power-on hours
and media errors need an NVMe admin ioctl and no exporter here produces them
(#549). The dashboard says so in a panel rather than letting a green screen imply
a healthy drive.

That gap is worse than #549 describes, and the note records why. The Crucial P3
drives were retired from Ceph because they were worn, and now carry the
EPHEMERAL volume — so the drives with the least endurance left are the ones
holding container state, and they are the ones nothing watches. The measured
write latency already separates them from the Samsung OSDs by more than an order
of magnitude.

Also fixes the add-a-dashboard instructions above the generator, whose third
step told the reader to tick a row off ../DASHBOARD-MIGRATION.md. That file does
not exist anywhere in the repo, so the step could not be followed.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/alert-monitor.yml

@@ -101,7 +101,7 @@ steps:
         bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/build-ci-images.yml

@@ -1,6 +1,17 @@
 # Builds the CI tooling images (and the secret-rotator runtime image) with
 # kaniko and pushes them to the Gitea container registry.
 #
+# The kaniko image is NOT Google's. Google archived
+# GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
+# it to chainguard-dev/kaniko and kept releasing — the pin here is well past
+# the v1.24.0 the Google repo ended on. The fork ships no binaries or images
+# of its own, so these come from gitlab-ci-utils, which builds that fork for
+# amd64 and arm64 under versioned tags. A generic `:debug` tag exists there
+# and is deliberately unused; everything else in this repo is pinned too.
+#
+# The executor path, flags and busybox layout are unchanged from Google's
+# image, which is why the swap touched nothing but the `image:` lines (#552).
+#
 # Credential: ci_images_token — a PAT scoped to `package: read+write` and
 # NOTHING else (verified: 403 on /user/repos, /admin/users and the issues
 # API; 200 on the /v2/ registry endpoints kaniko uses). It replaces the
@@ -25,7 +36,7 @@ when:
 
 steps:
   - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/ansible
     depends_on: []
     # Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -53,7 +64,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
 
   - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/k8s
     depends_on: []
     when:
@@ -77,7 +88,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
     depends_on: []
     when:
@@ -108,7 +119,7 @@ steps:
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
   - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: secret-rotator
     depends_on: []
     when:
@@ -136,21 +147,6 @@ steps:
   # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
-  # First step moved to the maintained kaniko fork (#552). Google archived
-  # GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked
-  # it to chainguard-dev/kaniko and kept releasing: this is v1.25.18, well past
-  # the v1.24.0 the Google repo ended on. The fork publishes no binaries of its
-  # own, so the image comes from gitlab-ci-utils, which builds it for amd64 and
-  # arm64 with versioned tags.
-  #
-  # Same /kaniko/executor path, same flags, and /bin/sh, base64, tr, mkdir and
-  # date are all present — verified by pulling the image and running it, not by
-  # reading the README.
-  #
-  # Deliberately only this step for now. A change to this file rebuilds every
-  # step anyway, so one run gives the new executor on alpine-jq beside seven
-  # unchanged kaniko builds — an A/B in a single pipeline. The other seven and
-  # ci-claude-refresh.yml follow once this one is green.
   - name: build-alpine-jq
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
@@ -181,7 +177,7 @@ steps:
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
   - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
     depends_on: []
     when:
@@ -210,7 +206,7 @@ steps:
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
   - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
     depends_on: []
     when:
@@ -239,7 +235,7 @@ steps:
   # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
   # builds the same way and lands in the same registry so the audit sees it.
   - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/heating-can
     depends_on: []
     when:

.woodpecker/ci-claude-refresh.yml

@@ -18,7 +18,7 @@
 #                 consumer pins to  ci-claude:<version>  and open a PR. A human
 #                 reviews and merges it; nothing auto-merges.
 #
-# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
 # tags the other ci-* images use. The tag encodes the thing that changes here
 # (the Claude version), which makes "did it change?" a trivial, race-free
 # string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
       - |
         set -eu
         mkdir -p "$$CI_WORKSPACE/.ci-refresh"
-        # Files that pin the image. Keep this list in sync with the consumers.
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Files that pin the image, DISCOVERED rather than listed. This used to
+        # be a hardcoded list carrying the instruction "keep this in sync with
+        # the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
+        # missing, so they sat on a version no bump could ever reach. A list
+        # that must be maintained by hand is the bug; deriving it removes the
+        # failure mode instead of correcting one instance of it.
+        #
+        # Matches a version-pinned reference only, which is what excludes this
+        # file: its own steps run ci-claude:latest, and :latest never needs
+        # rewriting. That exclusion is load-bearing — do not write a literal
+        # ci-claude version anywhere in THIS file, not even in prose, or it
+        # adds itself to the consumer set and the sed below rewrites it. The
+        # tag-scheme note at the top says <claude-version> for that reason.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        # An empty list would make the grep below read stdin and hang forever
+        # rather than fail, so it is a hard stop.
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
+        echo "consumers: $$CONSUMERS"
         NEW=$$(npm view @anthropic-ai/claude-code version)
         case "$$NEW" in
           [0-9]*.[0-9]*.[0-9]*) : ;;
@@ -130,7 +140,7 @@ steps:
   #    --build-arg to the resolved version (not "latest") guarantees the tag and
   #    the image content agree even if a new release lands mid-run.
   - name: build
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/claude
     depends_on: [decide]
     environment:
@@ -176,13 +186,10 @@ steps:
           echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
           exit 0
         fi
-        CONSUMERS="
-          .woodpecker/security-scan.yml
-          .woodpecker/alert-monitor.yml
-          .woodpecker/update-audit.yml
-          .woodpecker/pr-review.yml
-          .woodpecker/issue-fix.yml
-        "
+        # Same discovery as the decide step — see the comment there for why this
+        # is not a maintained list.
+        CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
+        [ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
         # Exact-string swap of the current tag for the new one, consumers only.
         for f in $$CONSUMERS; do
           sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
         {
           printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
           printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
-          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
           printf 'run the image so CI stops running a stale Claude Code.\n\n'
           printf 'Merging is a human decision — review, then merge to roll it out.\n'
         } > /tmp/pr-body.md

.woodpecker/ci-scripts-test.yml

@@ -17,6 +17,6 @@ when:
 
 steps:
   - name: model-selection-tests
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh

.woodpecker/e2e.yml

@@ -25,7 +25,7 @@ steps:
     # fetch-models.sh response mapping. No cluster, no secrets, no network — the
     # ci-claude image just provides sh + jq. Keeps the flag-injection regression
     # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     commands:
       - sh scripts/ci/test-model-selection.sh
 

.woodpecker/issue-fix.yml

@@ -62,7 +62,7 @@ steps:
       - bash scripts/ci/make-diag-kubeconfig.sh
 
   - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/pr-review.yml

@@ -65,7 +65,7 @@ when:
 
 steps:
   - name: pr-review-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/security-scan.yml

@@ -42,7 +42,7 @@ steps:
     # this step is skipped — otherwise we'd fork-bomb PRs.
     when:
       - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

.woodpecker/update-audit.yml

@@ -30,7 +30,7 @@ when:
 
 steps:
   - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
     environment:
       CLAUDE_CODE_OAUTH_TOKEN:
         from_secret: claude_code_oauth_token

kubernetes/apps/monitoring/dashboards/disk-health.json

@@ -0,0 +1,270 @@
+{
+  "annotations": {"list": []},
+  "description": "NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 0},
+      "id": 1,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "node_nvme_info * on (instance) group_left(nodename) node_uname_info",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Drive inventory",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true},
+            "indexByName": {},
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Read and write throughput per drive.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 9},
+      "id": 2,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Throughput",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "iops"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 9},
+      "id": 3,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "IOPS",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 18},
+      "id": 4,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "read {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "write {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Mean operation latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "percentunit"
+        },
+        "overrides": [
+          {
+            "matcher": {"id": "byRegexp", "options": "queue .*"},
+            "properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
+          }
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 18},
+      "id": 5,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "busy {{nodename}} {{model}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "queue {{nodename}} {{model}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Utilisation and queue depth",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
+          "unit": "celsius"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 27},
+      "id": 6,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} {{model}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Controller temperature",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 27},
+      "id": 7,
+      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_apply_latency_ms",
+          "legendFormat": "apply {{ceph_daemon}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "ceph_osd_commit_latency_ms",
+          "legendFormat": "commit {{ceph_daemon}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Ceph OSD latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 36},
+      "id": 8,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.",
+        "mode": "markdown"
+      },
+      "title": "Gaps and caveats",
+      "type": "text"
+    }
+  ],
+  "preload": false,
+  "refresh": "1m",
+  "schemaVersion": 39,
+  "tags": ["storage", "nvme", "ceph"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "Disk Health",
+  "uid": "disk-health",
+  "version": 1,
+  "weekStart": ""
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -7,9 +7,12 @@ namespace: monitoring
 # the kube-prometheus-stack Grafana sidecar watches for.
 #
 # To add a dashboard:
-#   1. Copy the ported JSON to this directory as <name>.json
+#   1. Copy the JSON to this directory as <name>.json
 #   2. Add a configMapGenerator entry below referencing the file
-#   3. Tick off the row in ../DASHBOARD-MIGRATION.md
+#
+# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
+# That file no longer exists anywhere in the repo — the port it tracked is
+# finished — so following the instruction was impossible.
 #
 # Dashboard datasource UIDs must match K8s Grafana datasources:
 #   - prometheus  (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
+  - name: dashboard-disk-health
+    files:
+      - disk-health.json
   - name: dashboard-gitea-overview
     files:
       - gitea-overview.json

kubernetes/infrastructure/admission-policies/woodpecker-ci-restrict.yaml

@@ -12,9 +12,11 @@
 #   * host namespaces + hostPath volumes: denied for every step pod (neither
 #     kaniko nor the buildx plugin needs them).
 #   * privileged containers: denied EXCEPT the allow-listed buildx plugin image.
-# Verified non-breaking: this repo's own builds use rootless kaniko
-# (gcr.io/kaniko-project/executor) which needs none of the above; only the
-# external buildx plugin needs privileged, and it is allow-listed by image ref.
+# Verified non-breaking: this repo's own builds use rootless kaniko, which needs
+# none of the above; only the external buildx plugin needs privileged, and it is
+# allow-listed by image ref. The kaniko image moved off gcr.io to the maintained
+# chainguard-dev fork in #552 — no policy change was needed, because nothing
+# here matches on the kaniko image, only on the buildx one.
 #
 # ValidatingAdmissionPolicy is GA / in-tree on the cluster's Kubernetes v1.35 —
 # no external admission controller (Kyverno/Gatekeeper) is introduced.