Files
homelab/kubernetes/apps/monitoring/dashboards
claude-botandClaude Opus 5 53887dd88f
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
docs(monitoring): stop treating a worn EPHEMERAL drive as a hazard
I described the worn drives carrying EPHEMERAL as the dangerous placement,
repeatedly, in the alert text and in the dashboard's caveats. That reads the
volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be
disposable — container images, kubelet state, pod scratch, spegel and Rook's
local state all rebuild. Putting the drives with the least endurance left there
is a sensible destination, not a risk someone should be warned about.

The alert said "a node simply stops when it goes", which is simply wrong. A node
losing /var rebuilds it.

What survives the correction is narrower and checked rather than recalled:
/var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node
costs that member's data. Quorum holds at two of three and the member is removed
and re-added. That is a rebuild, not data loss, and the alert now says so at
that weight.

The fill-level panel keeps its warning but for the right reason: a full /var
stops a node being useful long before anything is lost, and nothing rebalances
it the way Ceph rebalances a full OSD. That is a capacity argument, not an
endurance one.

None of this changes the monitoring — measuring every drive is still right, and
the wear figures are unchanged. It changes what a reader is told the numbers
mean, which is the part that was misleading.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 20:35:11 +02:00
..