claude-botandClaude Opus 5
e8b217e8a1
Drive wear has been invisible in every signal this cluster has. `ceph -s` reads active+clean at any wear level, no Ceph health check fires until a device actually fails, and the block-layer metrics show throughput and latency but never endurance. That blind spot is how a set of OSD drives reached 81-87% before anyone looked (#549, #434). This closes it for every drive, and it costs a deliberate exception to the no-privileged-containers rule. The owner made that call explicitly after being shown the alternative. The alternative was real and is worth recording. Ceph's devicehealth module already archives complete smartctl output daily — percentage_used, power_on_hours, media_errors, critical_warning, available_spare — collected without any elevated privileges, because the OSD pods hold the device access already. Exporting that needs no exception at all. It also covers OSDs and nothing else: three of nine drives here, and the six it misses include the ones retired from Ceph for wear that now carry EPHEMERAL. Monitoring the healthy drives while leaving the worn ones dark was not worth a new service. The requirement comes from the kernel, not from the tool. NVMe health is read through an admin ioctl on the character device, which is gated on CAP_SYS_ADMIN; sysfs publishes the controller temperature and nothing more. Swapping smartctl_exporter for another exporter would not change that, so the header says so — the thing to re-check one day is whether the kernel grew a sysfs interface, not whether some other tool asks for less. The exception is kept to the smallest form that works: no `privileged: true`, no host namespaces, no hostPID. Root plus CAP_SYS_ADMIN and CAP_SYS_RAWIO, all other capabilities dropped, a read-only root filesystem, and /dev as the only host mount — required because Kubernetes cannot hand individual character devices to a pod without a device plugin. The admission policy that forbids privileged pods is bound to the woodpecker-ci namespace only, so nothing here circumvents it; monitoring already enforces pod-security `privileged`. Two details that would otherwise have made the metrics useless: The ServiceMonitor relabels the pod's node onto every series. Without it each sample carries only the DaemonSet pod's IP, which says nothing about which machine the drive is in and changes on every reschedule — an unattributable wear figure is most of the value gone. No --smartctl.device list. The exporter scans and rescans on its own, and a written-down device list would quietly stop covering a drive the next time one is replaced. Device names are not stable across these machines in the first place. The alert rules deliberately separate wear from damage: a wear percentage is a budget being spent and warns at 80, while spare exhaustion, a critical-warning flag or new media errors mean the drive is failing now. Media errors alert on the increase rather than the total, so a drive that recorded errors long ago does not alert forever while one that starts failing today does. The completeness check compares the number of reporting drives against node-exporter's own NVMe inventory rather than a written-down fleet size, so adding or removing a drive moves both sides together. A rule that alerts on nothing looks exactly like a healthy cluster, which is the failure mode worth guarding. Dashboard panels follow separately, once the metrics are live and the label names can be read off the real series rather than guessed from a README. Closes #549 Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>