Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 claude-bot
|
c59f0050b3 | ||
 k.renner
|
48c46610a6 | ||
 claude-botandClaude Opus 5
|
53887dd88f | ||
|
k.renner
|
c9a10ab015 | ||
|
claude-botandClaude Opus 5
|
24b2a0178d | ||
|
k.renner
|
b8061ea05d | ||
|
claude-botandClaude Opus 5
|
d0bea0dc15 | ||
|
k.renner
|
8ccf5ac8e5 |
@@ -15,4 +15,4 @@ spec:
|
||||
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/authentik/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -1,21 +1,14 @@
|
||||
---
|
||||
# Barman S3 credentials for ha-pg.
|
||||
#
|
||||
# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
|
||||
# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
|
||||
# problem and the same note. The values are the same today, so this is correct
|
||||
# but coupled: rotating paperless's Hetzner credentials would silently change
|
||||
# the backup credentials for Home Assistant and Woodpecker too.
|
||||
# One shared path for one shared credential — see the note in
|
||||
# ../paperless/hetzner-creds-externalsecret.yaml.
|
||||
#
|
||||
# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
|
||||
# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
|
||||
#
|
||||
# bao kv put homelab/seed/home-assistant/hetzner-creds \
|
||||
# @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
|
||||
#
|
||||
# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
|
||||
# 403 on `bao kv list seed`, which was a wrong-path error rather than a
|
||||
# permissions problem.
|
||||
# The earlier note here proposed a dedicated seed/home-assistant path. That was
|
||||
# the wrong direction: it would have made a fifth copy of the same key look like
|
||||
# a fourth independent credential. It did get one thing right, and it is worth
|
||||
# keeping: the 403 recorded in #397 was a wrong-path error, not a permissions
|
||||
# problem, because the mount is `homelab` and the path is homelab/seed/...
|
||||
#
|
||||
# The namespace allowlist on the openbao ClusterSecretStore
|
||||
# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
|
||||
@@ -35,4 +28,4 @@ spec:
|
||||
name: hetzner-s3-creds
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/paperless/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -15,4 +15,4 @@ spec:
|
||||
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/immich/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -38,11 +38,15 @@ spec:
|
||||
summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
|
||||
description: >-
|
||||
The drive has used {{ $value }}% of its rated write endurance.
|
||||
Nothing is failing and no data is at risk yet. Check what the
|
||||
drive is carrying before deciding urgency: an OSD is replaceable
|
||||
with a rebalance, whereas the EPHEMERAL volume has no such
|
||||
escape — a node simply stops when it goes. The Disk Health
|
||||
dashboard resolves which drive holds which role.
|
||||
Nothing is failing and no data is at risk yet. What the drive
|
||||
carries decides the urgency, and the Disk Health dashboard
|
||||
resolves which role it holds. An OSD losing a drive costs a
|
||||
rebalance. EPHEMERAL is Talos's /var and is meant to be
|
||||
disposable — container images, kubelet state and pod scratch all
|
||||
rebuild — so a worn drive there is a reasonable place for it. The
|
||||
one thing on that volume which is not free to lose is the node's
|
||||
etcd member data: quorum survives losing one of three, but the
|
||||
member has to be removed and re-added.
|
||||
- alert: NvmeDriveWearCritical
|
||||
expr: smartctl_device_percentage_used >= 95
|
||||
for: 1h
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"annotations": {"list": []},
|
||||
"description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance/wear is absent; see the note at the bottom and issue #549.",
|
||||
"description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -19,7 +19,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
|
||||
"gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
|
||||
"id": 10,
|
||||
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
||||
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
|
||||
@@ -28,7 +28,7 @@
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "OSDs the cluster knows about but which are not up, and not in. Both should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm.",
|
||||
"description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {"mode": "thresholds"},
|
||||
@@ -38,14 +38,13 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
|
||||
"gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
|
||||
"id": 11,
|
||||
"options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
||||
"targets": [
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "down", "refId": "A"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_in)", "legendFormat": "out", "refId": "B"}
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
|
||||
],
|
||||
"title": "OSDs down / out",
|
||||
"title": "OSDs down",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
@@ -62,7 +61,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
|
||||
"gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
|
||||
"id": 12,
|
||||
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
||||
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
|
||||
@@ -71,7 +70,7 @@
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
|
||||
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {"mode": "thresholds"},
|
||||
@@ -83,7 +82,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
|
||||
"gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
|
||||
"id": 13,
|
||||
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
||||
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
|
||||
@@ -115,7 +114,7 @@
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Drive inventory",
|
||||
"title": "Drive inventory (identity and role)",
|
||||
"transformations": [
|
||||
{
|
||||
"id": "organize",
|
||||
@@ -128,6 +127,100 @@
|
||||
],
|
||||
"type": "table"
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {"mode": "thresholds"},
|
||||
"mappings": [],
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
|
||||
"id": 14,
|
||||
"options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
|
||||
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
|
||||
"title": "Worst NVMe wear",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {"mode": "palette-classic"},
|
||||
"custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
|
||||
"mappings": [],
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
|
||||
"id": 15,
|
||||
"options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
|
||||
"legendFormat": "{{node}} {{model_name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "NVMe wear",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {"mode": "thresholds"},
|
||||
"custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
|
||||
"mappings": [],
|
||||
"thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
|
||||
},
|
||||
"overrides": [
|
||||
{"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
|
||||
{"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
|
||||
{"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
||||
{"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
|
||||
{"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
|
||||
{"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
|
||||
]
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
|
||||
"id": 16,
|
||||
"options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
|
||||
"targets": [
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
|
||||
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
|
||||
],
|
||||
"title": "SMART health",
|
||||
"transformations": [
|
||||
{"id": "merge", "options": {}},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
|
||||
"indexByName": {},
|
||||
"renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
|
||||
}
|
||||
}
|
||||
],
|
||||
"type": "table"
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"description": "Read and write throughput per drive.",
|
||||
@@ -141,7 +234,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
|
||||
"id": 2,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -174,7 +267,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
|
||||
"id": 3,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -207,7 +300,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
|
||||
"id": 4,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -245,7 +338,7 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
|
||||
"id": 5,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -278,7 +371,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
|
||||
"id": 6,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -305,7 +398,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
|
||||
"gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
|
||||
"id": 7,
|
||||
"options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -340,7 +433,7 @@
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 40},
|
||||
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
|
||||
"id": 9,
|
||||
"options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
|
||||
"targets": [
|
||||
@@ -356,11 +449,11 @@
|
||||
},
|
||||
{
|
||||
"datasource": {"type": "prometheus", "uid": "prometheus"},
|
||||
"gridPos": {"h": 11, "w": 24, "x": 0, "y": 48},
|
||||
"gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
|
||||
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Endurance is missing.** No wear, no written-bytes total, no power-on hours, no media-error count — no exporter here produces them. NVMe reports those over an admin ioctl that needs elevated privileges; temperature is the only health value the kernel publishes through sysfs. Tracked as **#549**. Do not read a green top row as a healthy drive.\n\nCeph does record wear, but only inside its own devicehealth module and not in Prometheus: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. It covers OSDs and nothing else, and a drive removed from Ceph is purged from that registry — its history goes with it, so the wear figure of a retired drive is not recoverable after the fact.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
|
||||
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
|
||||
"mode": "markdown"
|
||||
},
|
||||
"title": "Gaps and caveats",
|
||||
|
||||
@@ -16,17 +16,19 @@
|
||||
# healthy drives and not the worn ones was judged not worth a new service; the
|
||||
# owner chose fleet-wide coverage with the privilege exception instead.
|
||||
#
|
||||
# The exception is kept as small as the kernel allows: no `privileged: true`,
|
||||
# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are
|
||||
# added, everything else is dropped, and the only host mount is /dev — which
|
||||
# is required because the exporter must open the NVMe character devices and
|
||||
# Kubernetes has no way to pass individual character devices to a pod without
|
||||
# a device plugin.
|
||||
# The exception is `privileged: true`, and narrower settings were tried and
|
||||
# measured to fail — see the securityContext below for what was ruled out and
|
||||
# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
|
||||
# read-only root filesystem, and /dev as the only host mount.
|
||||
#
|
||||
# If this ever needs revisiting: the thing to check is whether the kernel has
|
||||
# grown a sysfs interface for NVMe health, not whether some other exporter
|
||||
# needs fewer privileges. The requirement comes from the ioctl gate, not from
|
||||
# the tool.
|
||||
# If this ever needs revisiting, there are exactly two things worth checking,
|
||||
# and neither is "does another exporter ask for less":
|
||||
# - whether the kernel has grown a sysfs interface for NVMe health, which
|
||||
# would remove the need for device access altogether
|
||||
# - whether enough workloads need raw device access to justify a device
|
||||
# plugin, which is the only way Kubernetes grants individual devices
|
||||
# The requirement comes from the device cgroup and the ioctl gate, not from
|
||||
# the choice of tool.
|
||||
#
|
||||
# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
|
||||
# The repository is not archived; the pin is simply its newest tag.
|
||||
@@ -66,38 +68,52 @@ spec:
|
||||
containerPort: 9633
|
||||
protocol: TCP
|
||||
securityContext:
|
||||
# Root plus two capabilities, NOT privileged. See the header.
|
||||
# PRIVILEGED, and there is no narrower setting that works. This
|
||||
# was established by measurement after two narrower attempts
|
||||
# failed, so the reasoning is recorded here rather than rediscovered:
|
||||
#
|
||||
# Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
|
||||
# dropped. The exporter found all three drives per node and
|
||||
# could open none: "Smartctl open device: /dev/nvme0 failed:
|
||||
# Operation not permitted".
|
||||
#
|
||||
# Attempt 2 — the same plus seccompProfile: Unconfined, on the
|
||||
# theory that EPERM is the default seccomp action for an
|
||||
# unlisted syscall and the kubelet runs seccompDefault: true.
|
||||
# /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
|
||||
# capabilities were effective and a filter was loaded. Removing
|
||||
# the filter changed nothing.
|
||||
#
|
||||
# What actually settled it: `head -c 512 /dev/nvme0n1` inside
|
||||
# the container fails with EPERM too. A plain read of a block
|
||||
# device involves no ioctl, no capability and no syscall a
|
||||
# seccomp profile would filter. The block is the device cgroup —
|
||||
# containerd denies non-privileged containers access to host
|
||||
# device nodes whether or not /dev is mounted.
|
||||
#
|
||||
# Kubernetes exposes no per-device allowlist in a Pod spec; there
|
||||
# is no equivalent of `docker --device`. The alternatives are
|
||||
# `privileged: true` or a device plugin advertising the NVMe
|
||||
# drives as a resource. A device plugin is the genuinely narrower
|
||||
# answer and is worth building if more workloads ever need raw
|
||||
# device access; for a single read-only metrics exporter it is a
|
||||
# component to write, deploy and maintain in exchange for the same
|
||||
# access.
|
||||
runAsUser: 0
|
||||
runAsNonRoot: false
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
privileged: true
|
||||
# Stated rather than omitted: the API rejects `privileged: true`
|
||||
# together with `allowPrivilegeEscalation: false`, so leaving it
|
||||
# out would only hide that privileged already implies it.
|
||||
allowPrivilegeEscalation: true
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
add: ["SYS_ADMIN", "SYS_RAWIO"]
|
||||
# The capabilities alone are not enough, and this was measured
|
||||
# rather than assumed. With the default profile the exporter found
|
||||
# all three drives and could open none of them:
|
||||
# "Smartctl open device: /dev/nvme0 failed: Operation not
|
||||
# permitted"
|
||||
# EPERM is the default seccomp action for a syscall the profile
|
||||
# does not list. Reading /proc/1/status in the running container
|
||||
# separated the two candidates: CapEff was 0x220000, so SYS_ADMIN
|
||||
# and SYS_RAWIO were both effective — while Seccomp was 2 with one
|
||||
# filter loaded. The kubelet sets seccompDefault: true, so every
|
||||
# pod that does not say otherwise gets RuntimeDefault.
|
||||
#
|
||||
# This is still narrower than `privileged: true`, which would also
|
||||
# grant every capability, all devices and write access to /sys.
|
||||
# Narrowing further would mean shipping a custom seccomp profile
|
||||
# that allows exactly the ioctl smartctl needs — worth doing if
|
||||
# this list of exceptions ever grows, not worth it for one pod.
|
||||
seccompProfile:
|
||||
type: Unconfined
|
||||
volumeMounts:
|
||||
# Not read-only. smartmontools opens NVMe devices read-write on
|
||||
# some paths, and once the container is privileged a read-only
|
||||
# flag on this mount is not a boundary anyone should rely on —
|
||||
# keeping it would suggest a protection that is not there.
|
||||
- name: dev
|
||||
mountPath: /dev
|
||||
readOnly: true
|
||||
# smartctl is not known to need scratch space, but the root
|
||||
# filesystem is read-only and a tool that turns out to want /tmp
|
||||
# would crashloop instead of degrading. Cheaper to provide it than
|
||||
|
||||
@@ -1,3 +1,27 @@
|
||||
# CANONICAL NOTE on the Hetzner S3 credentials. The other consumers point here
|
||||
# rather than repeating it.
|
||||
#
|
||||
# There is exactly ONE Hetzner S3 key, and every barman ObjectStore in this
|
||||
# cluster uses it. It used to live under four paths — seed/paperless,
|
||||
# seed/authentik, seed/immich, seed/trmnl — with byte-identical contents,
|
||||
# verified by comparing SHA-256 fingerprints of the synced Secrets rather than
|
||||
# by reading them. The same key also sits in infrastructure/velero-hetzner, in
|
||||
# a different shape (a single `cloud` field holding an AWS credentials file);
|
||||
# its access key id fingerprints the same.
|
||||
#
|
||||
# Per-app paths implied a separation that did not exist. Rotating the key at
|
||||
# Hetzner changes it for everyone regardless of how many paths hold a copy, so
|
||||
# the separate paths bought nothing and cost a standing invitation to believe
|
||||
# otherwise — #397 was opened on exactly that misreading, proposing to give
|
||||
# woodpecker a fifth copy.
|
||||
#
|
||||
# Real separation needs one Hetzner key per consumer, and Hetzner keys can only
|
||||
# be created by hand in their console. Until someone wants that, one path is the
|
||||
# honest representation.
|
||||
#
|
||||
# Not yet folded in: velero. It needs the credentials file form, so moving it
|
||||
# here means templating the INI from these fields — worth doing, not worth
|
||||
# bundling with a path swap that touches every database backup.
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
@@ -15,4 +39,4 @@ spec:
|
||||
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/paperless/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -12,4 +12,4 @@ spec:
|
||||
name: hetzner-s3-creds
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/trmnl/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -1,11 +1,8 @@
|
||||
---
|
||||
# Barman S3 credentials for wp-pg.
|
||||
#
|
||||
# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
|
||||
# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
|
||||
# that path needs OpenBao write access (`bao kv list seed` → 403 for the
|
||||
# identities available here), so it stays open. The values are identical, so it
|
||||
# is a one-line swap of the `key:` below once the path exists.
|
||||
# One shared path for one shared credential — see the note in
|
||||
# ../paperless/hetzner-creds-externalsecret.yaml.
|
||||
#
|
||||
# The namespace allowlist on the openbao ClusterSecretStore
|
||||
# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
|
||||
@@ -24,4 +21,4 @@ spec:
|
||||
name: hetzner-s3-creds
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: seed/paperless/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
@@ -36,11 +36,9 @@ spec:
|
||||
target:
|
||||
name: hetzner-s3-creds
|
||||
dataFrom:
|
||||
# Reuses the paperless copy of the shared Hetzner S3 credentials (identical
|
||||
# ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
|
||||
# path for this migration — same approach woodpecker took. FOLLOW-UP: give
|
||||
# wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
|
||||
# written, and repoint this here (the value is the same, so it's a drop-in
|
||||
# swap).
|
||||
# One shared path for one shared credential. There is a single Hetzner S3
|
||||
# key behind all of these — see the note in
|
||||
# ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
|
||||
# the wrong shape and what would have to change for them to be right.
|
||||
- extract:
|
||||
key: seed/paperless/hetzner-creds
|
||||
key: infrastructure/hetzner-s3
|
||||
|
||||
Reference in New Issue
Block a user