Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 d04b09310f feat(monitoring): add a disk health dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Throughput, IOPS, mean operation latency, utilisation, queue depth and
controller temperature for all nine NVMe drives, plus Ceph's own OSD latency for
comparison against the block layer.

Every panel joins node_disk_* to node_nvme_info and node_uname_info rather than
plotting the device name. That is not decoration: nvme0 is the Kingston system
disk on k8s-1 and k8s-3 and the Crucial on k8s-2, so a legend keyed on the
device compares different hardware across nodes and does so silently. The
inventory table resolves the OSD column through ceph_disk_occupation for the
same reason — it follows Rook instead of restating a device list that goes stale
the next time a drive is swapped.

Every expression was run against Prometheus before being written into the JSON,
which is also how the caveats in the panel descriptions were found. Latency is
accumulated-time over completed-operations, so it is a mean with no tail and it
vanishes while a device is idle: absence of traffic, not absence of a problem.
Utilisation near 100% on NVMe does not mean saturation the way it does on a
spindle, which is why queue depth shares that panel.

Temperature is the only SMART-adjacent value present, because it is the only one
the kernel publishes through sysfs. Wear, written-bytes totals, power-on hours
and media errors need an NVMe admin ioctl and no exporter here produces them
(#549). The dashboard says so in a panel rather than letting a green screen imply
a healthy drive.

That gap is worse than #549 describes, and the note records why. The Crucial P3
drives were retired from Ceph because they were worn, and now carry the
EPHEMERAL volume — so the drives with the least endurance left are the ones
holding container state, and they are the ones nothing watches. The measured
write latency already separates them from the Samsung OSDs by more than an order
of magnitude.

Also fixes the add-a-dashboard instructions above the generator, whose third
step told the reader to tick a row off ../DASHBOARD-MIGRATION.md. That file does
not exist anywhere in the repo, so the step could not be followed.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 14:02:11 +02:00
k.renner dd050d811d Merge pull request 'chore(ci): bump ci-claude image to 2.1.247' (#666) from ci-image/claude-2.1.247 into master
ci/woodpecker/push/ci-claude-refresh Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/e2e Pipeline was successful
Reviewed-on: k.renner/homelab#666
2026-08-27 11:51:32 +00:00
claude-bot c4bb35173d fix(ci): derive the ci-claude consumer list instead of maintaining it
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
review-bot found that this bump reached five of seven consumers: e2e.yml and
ci-scripts-test.yml stayed on the old version. Verified — both still pinned the
previous tag on the bump branch while the other five moved.

The cause is worth more than the symptom. ci-claude-refresh.yml carried the
consumer list twice, hardcoded, under the instruction "keep this list in sync
with the consumers". It was not in sync, and nothing could ever notice: the two
missing files were invisible to the bump that would have raised them and to the
guard that checks the pins agree. They were not drifting slowly, they were
unreachable — no future run would have touched them either.

So both copies are now derived by grepping .woodpecker for a version-pinned
reference. A list that must be maintained by hand is the defect; deriving it
removes the failure mode rather than correcting one instance of it.

The pattern deliberately matches a digit after the tag, which is what keeps this
file out of its own consumer set — its steps run ci-claude:latest, which never
needs rewriting. That makes the exclusion load-bearing, so it is spelled out in
a comment: a literal version anywhere in this file, prose included, would add it
to the set and let the sed rewrite it. The tag-scheme note at the top now says
<claude-version> for exactly that reason.

An empty result is a hard stop. Without it the following grep would be invoked
with no file arguments, read stdin, and hang rather than fail — a pipeline that
never finishes is worse than one that exits non-zero.

The generated PR body no longer claims a fixed "5 pipelines"; it counts.

Both files are also bumped here rather than left for a follow-up, and the order
matters: adding them to the consumer set without bumping them would make the
next refresh see two disagreeing tags and abort with "refusing to guess". That
guard is well built — it simply never saw these two files.

Refs #666
2026-08-27 13:19:41 +02:00
claude-bot a9df3ea9ce chore(ci): bump ci-claude image to 2.1.247
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
2026-08-27 11:12:42 +00:00
k.renner 2549535961 Merge pull request 'ci(kaniko): move the remaining eight builds to the maintained fork' (#665) from kaniko-fork-rest into master
ci/woodpecker/push/ci-claude-refresh Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#665
2026-08-27 11:11:22 +00:00
10 changed files with 309 additions and 26 deletions
+1 -1
View File
@@ -101,7 +101,7 @@ steps:
bash scripts/ci/make-diag-kubeconfig.sh
- name: alert-triage-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+24 -17
View File
@@ -18,7 +18,7 @@
# consumer pins to ci-claude:<version> and open a PR. A human
# reviews and merges it; nothing auto-merges.
#
# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
# tags the other ci-* images use. The tag encodes the thing that changes here
# (the Claude version), which makes "did it change?" a trivial, race-free
# string compare and keeps the weekly cron from opening no-op PRs. The other
@@ -84,14 +84,24 @@ steps:
- |
set -eu
mkdir -p "$$CI_WORKSPACE/.ci-refresh"
# Files that pin the image. Keep this list in sync with the consumers.
CONSUMERS="
.woodpecker/security-scan.yml
.woodpecker/alert-monitor.yml
.woodpecker/update-audit.yml
.woodpecker/pr-review.yml
.woodpecker/issue-fix.yml
"
# Files that pin the image, DISCOVERED rather than listed. This used to
# be a hardcoded list carrying the instruction "keep this in sync with
# the consumers" — and it was not: e2e.yml and ci-scripts-test.yml were
# missing, so they sat on a version no bump could ever reach. A list
# that must be maintained by hand is the bug; deriving it removes the
# failure mode instead of correcting one instance of it.
#
# Matches a version-pinned reference only, which is what excludes this
# file: its own steps run ci-claude:latest, and :latest never needs
# rewriting. That exclusion is load-bearing — do not write a literal
# ci-claude version anywhere in THIS file, not even in prose, or it
# adds itself to the consumer set and the sed below rewrites it. The
# tag-scheme note at the top says <claude-version> for that reason.
CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
# An empty list would make the grep below read stdin and hang forever
# rather than fail, so it is a hard stop.
[ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
echo "consumers: $$CONSUMERS"
NEW=$$(npm view @anthropic-ai/claude-code version)
case "$$NEW" in
[0-9]*.[0-9]*.[0-9]*) : ;;
@@ -176,13 +186,10 @@ steps:
echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
exit 0
fi
CONSUMERS="
.woodpecker/security-scan.yml
.woodpecker/alert-monitor.yml
.woodpecker/update-audit.yml
.woodpecker/pr-review.yml
.woodpecker/issue-fix.yml
"
# Same discovery as the decide step — see the comment there for why this
# is not a maintained list.
CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
[ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
# Exact-string swap of the current tag for the new one, consumers only.
for f in $$CONSUMERS; do
sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
@@ -198,7 +205,7 @@ steps:
{
printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
printf 'latest published `@anthropic-ai/claude-code`. This rewires the %s pipelines that\n' "$$(printf '%s\n' $$CONSUMERS | grep -c .)"
printf 'run the image so CI stops running a stale Claude Code.\n\n'
printf 'Merging is a human decision — review, then merge to roll it out.\n'
} > /tmp/pr-body.md
+1 -1
View File
@@ -17,6 +17,6 @@ when:
steps:
- name: model-selection-tests
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
commands:
- sh scripts/ci/test-model-selection.sh
+1 -1
View File
@@ -25,7 +25,7 @@ steps:
# fetch-models.sh response mapping. No cluster, no secrets, no network — the
# ci-claude image just provides sh + jq. Keeps the flag-injection regression
# test from rotting (it exercises the exact injection shapes).
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
commands:
- sh scripts/ci/test-model-selection.sh
+1 -1
View File
@@ -62,7 +62,7 @@ steps:
- bash scripts/ci/make-diag-kubeconfig.sh
- name: issue-fix-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -65,7 +65,7 @@ when:
steps:
- name: pr-review-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -42,7 +42,7 @@ steps:
# this step is skipped — otherwise we'd fork-bomb PRs.
when:
- branch: master
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -30,7 +30,7 @@ when:
steps:
- name: update-audit-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.218
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.247
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
@@ -0,0 +1,270 @@
{
"annotations": {"list": []},
"description": "NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.",
"fieldConfig": {
"defaults": {
"color": {"mode": "thresholds"},
"custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
},
"overrides": []
},
"gridPos": {"h": 9, "w": 24, "x": 0, "y": 0},
"id": 1,
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "node_nvme_info * on (instance) group_left(nodename) node_uname_info",
"format": "table",
"instant": true,
"legendFormat": "",
"refId": "A"
}
],
"title": "Drive inventory",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true},
"indexByName": {},
"renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware"}
}
}
],
"type": "table"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Read and write throughput per drive.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
"unit": "Bps"
},
"overrides": []
},
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 9},
"id": 2,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_read_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "read {{nodename}} {{model}}",
"refId": "A"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_written_bytes_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "write {{nodename}} {{model}}",
"refId": "B"
}
],
"title": "Throughput",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Completed operations per second. Read together with throughput this separates many small operations from few large ones.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
"unit": "iops"
},
"overrides": []
},
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 9},
"id": 3,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "read {{nodename}} {{model}}",
"refId": "A"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "write {{nodename}} {{model}}",
"refId": "B"
}
],
"title": "IOPS",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Mean service time per operation, derived as accumulated time divided by completed operations. This is an average and will not show a tail — a drive can look healthy here while a small fraction of operations stall. The line disappears while a device is idle, because the denominator is zero; that is absence of traffic, not absence of a problem.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
"unit": "s"
},
"overrides": []
},
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 18},
"id": 4,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_read_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_reads_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "read {{nodename}} {{model}}",
"refId": "A"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_write_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]) / rate(node_disk_writes_completed_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "write {{nodename}} {{model}}",
"refId": "B"
}
],
"title": "Mean operation latency",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Fraction of wall-clock time the device had at least one operation in flight, and the number of operations queued. Utilisation near 100% on an NVMe drive does not mean saturation the way it does on a spindle — these devices serve many operations in parallel — so read it alongside the queue depth.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
"unit": "percentunit"
},
"overrides": [
{
"matcher": {"id": "byRegexp", "options": "queue .*"},
"properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]
}
]
},
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 18},
"id": 5,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(rate(node_disk_io_time_seconds_total{device=~\"nvme[0-9]+n[0-9]+\"}[$__rate_interval]), \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "busy {{nodename}} {{model}}",
"refId": "A"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(node_disk_io_now{device=~\"nvme[0-9]+n[0-9]+\"}, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)n[0-9]+\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "queue {{nodename}} {{model}}",
"refId": "B"
}
],
"title": "Utilisation and queue depth",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 70}, {"color": "red", "value": 80}]},
"unit": "celsius"
},
"overrides": []
},
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 27},
"id": 6,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "label_replace(node_hwmon_temp_celsius{chip=~\"nvme_nvme[0-9]+\", sensor=\"temp1\"}, \"nvmedev\", \"$1\", \"chip\", \"nvme_(nvme[0-9]+)\") * on (instance, nvmedev) group_left(model) label_replace(node_nvme_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (instance) group_left(nodename) node_uname_info",
"legendFormat": "{{nodename}} {{model}}",
"refId": "A"
}
],
"title": "Controller temperature",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Ceph's own view of the OSDs, for comparison with the block-layer numbers above. A divergence between these and the device latency is the interesting case: it points at Ceph rather than at the hardware.",
"fieldConfig": {
"defaults": {
"color": {"mode": "palette-classic"},
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}},
"mappings": [],
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]},
"unit": "ms"
},
"overrides": []
},
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 27},
"id": 7,
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "ceph_osd_apply_latency_ms",
"legendFormat": "apply {{ceph_daemon}}",
"refId": "A"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"expr": "ceph_osd_commit_latency_ms",
"legendFormat": "commit {{ceph_daemon}}",
"refId": "B"
}
],
"title": "Ceph OSD latency",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 36},
"id": 8,
"options": {
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
"content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.",
"mode": "markdown"
},
"title": "Gaps and caveats",
"type": "text"
}
],
"preload": false,
"refresh": "1m",
"schemaVersion": 39,
"tags": ["storage", "nvme", "ceph"],
"templating": {"list": []},
"time": {"from": "now-6h", "to": "now"},
"timepicker": {},
"timezone": "",
"title": "Disk Health",
"uid": "disk-health",
"version": 1,
"weekStart": ""
}
@@ -7,9 +7,12 @@ namespace: monitoring
# the kube-prometheus-stack Grafana sidecar watches for.
#
# To add a dashboard:
# 1. Copy the ported JSON to this directory as <name>.json
# 1. Copy the JSON to this directory as <name>.json
# 2. Add a configMapGenerator entry below referencing the file
# 3. Tick off the row in ../DASHBOARD-MIGRATION.md
#
# There used to be a third step, ticking a row off ../DASHBOARD-MIGRATION.md.
# That file no longer exists anywhere in the repo — the port it tracked is
# finished — so following the instruction was impossible.
#
# Dashboard datasource UIDs must match K8s Grafana datasources:
# - prometheus (kube-prom-stack default, in-cluster)
@@ -35,6 +38,9 @@ configMapGenerator:
- name: dashboard-blackbox-overview
files:
- blackbox-overview.json
- name: dashboard-disk-health
files:
- disk-health.json
- name: dashboard-gitea-overview
files:
- gitea-overview.json