konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

Ceph observability gaps: no toolbox / no admin path, plus osd.2 endurance clock to confirm #434

New Issue
Closed
opened 2026-07-27 19:33:28 +00:00 by claude-bot · 9 comments
claude-bot commented 2026-07-27 19:33:28 +00:00
Member
Copy Link
Copy Source

Found during the 2026-07-27 issue round. The triggering warning is gone; what remains are two durable gaps and one hardware clock worth tracking.

The triggering event — resolved, benign

CephCluster reported HEALTH_WARN / BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore earlier today. It has since cleared on its own:

phase=Ready  health=HEALTH_OK  details={}   (lastChecked 2026-07-27T19:32:24Z)

Investigation concluded it was a self-limiting tail-latency event driven by the Sunday restore-test sweep, which cleared at 12:04:47Z. Redundancy was never at risk and the affected OSD's hardware is fine. No action needed for the warning itself — this issue is about what the investigation could not do.

Gap 1 — Ceph has never been scraped (fix already up as PR #432)

Verified: zero ceph_* series in Prometheus, and zero active rook-ceph scrape targets — for all 79 days of the cluster's life.

Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack}
rook-ceph-mgr / rook-ceph-exporter: labels      {team: rook}

Rook creates those ServiceMonitors at runtime with its own label, so nothing in this repo can relabel them. 12 of 30 ServiceMonitors were being dropped cluster-wide (the 2 rook ones plus 10 authentik outposts).

PR #432 fixes the scrape side. Two follow-ups belong here rather than there:

  • Once ceph_* series actually appear, add a Ceph PrometheusRule (carrying release: kube-prometheus-stack) alerting on ceph_health_status == 1 for: 6h, plus a device-wear rule. Adding one before #432 lands would silently never fire.
  • Audit whether any other operator-managed ServiceMonitors were being dropped the same way — Rook is unlikely to be the only offender.

Gap 2 — there is no read-only path to Ceph internals at all

There is no rook-ceph-tools deployment, and no other route works. Verified by attempting each:

Attempt Result
kubectl -n rook-ceph get deploy rook-ceph-tools not found
exec into rook-ceph-osd-2, run ceph device ls no keyring — auth: unable to find a keyring on /etc/ceph/..., cephx disabled, handle_auth_bad_method
exec into rook-ceph-mon-a no usable output
exec into rook-ceph-operator /etc/ceph/ empty; ceph -s → ObjectNotFound('RADOS object not found (error calling conf_read_file)')
exec into rook-ceph-mgr RADOS permission denied (no admin keyring mounted)

So ceph status, ceph health detail, ceph osd df tree, ceph osd perf and ceph device ls are all unreachable during an incident. Combined with Gap 1, the storage layer had neither live introspection nor metric history — the two independent ways you would normally diagnose it.

Deploying the toolbox (or mounting a read-only client keyring somewhere reachable) would close this cheaply. If it is deliberately omitted, that reasoning should be written down where the next investigation will find it — I could not locate any comment explaining its absence.

Item 3 — drive endurance clock (needs confirmation)

The investigation reported osd.2's drive at 83% endurance used, projecting replacement around Dec 2026, sourced from Ceph's devicehealth module (which stores SMART history to RADOS — which is why that series survived despite Gap 1).

⚠️ I could not independently confirm this figure, precisely because of Gap 2. Flagging it rather than asserting it. Confirm once a toolbox exists:

ceph device ls
ceph device get-health-metrics <devid>

or read SMART directly on the Proxmox host backing osd.2. If 83% is accurate this is not urgent but it is a real, dated hardware clock and deserves tracking rather than rediscovery. (Note: SSH to homelab hosts must be serial — parallel connections trip UDM rate-limiting and produce spurious "host down" faults.)

Found during the 2026-07-27 issue round. The triggering warning is gone; what remains are two durable gaps and one hardware clock worth tracking. ## The triggering event — resolved, benign `CephCluster` reported `HEALTH_WARN / BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore` earlier today. It has since cleared on its own: ``` phase=Ready health=HEALTH_OK details={} (lastChecked 2026-07-27T19:32:24Z) ``` Investigation concluded it was a self-limiting tail-latency event driven by the Sunday `restore-test` sweep, which cleared at 12:04:47Z. Redundancy was never at risk and the affected OSD's hardware is fine. **No action needed for the warning itself** — this issue is about what the investigation could not do. ## Gap 1 — Ceph has never been scraped (fix already up as PR #432) Verified: **zero `ceph_*` series in Prometheus, and zero active `rook-ceph` scrape targets** — for all 79 days of the cluster's life. ``` Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack} rook-ceph-mgr / rook-ceph-exporter: labels {team: rook} ``` Rook creates those ServiceMonitors at runtime with its own label, so nothing in this repo can relabel them. 12 of 30 ServiceMonitors were being dropped cluster-wide (the 2 rook ones plus 10 authentik outposts). **PR #432** fixes the scrape side. Two follow-ups belong here rather than there: - Once `ceph_*` series actually appear, add a Ceph `PrometheusRule` (carrying `release: kube-prometheus-stack`) alerting on `ceph_health_status == 1 for: 6h`, plus a device-wear rule. Adding one **before** #432 lands would silently never fire. - Audit whether any other operator-managed ServiceMonitors were being dropped the same way — Rook is unlikely to be the only offender. ## Gap 2 — there is no read-only path to Ceph internals at all There is **no `rook-ceph-tools` deployment**, and no other route works. Verified by attempting each: | Attempt | Result | |---|---| | `kubectl -n rook-ceph get deploy rook-ceph-tools` | not found | | `exec` into `rook-ceph-osd-2`, run `ceph device ls` | no keyring — `auth: unable to find a keyring on /etc/ceph/...`, cephx disabled, `handle_auth_bad_method` | | `exec` into `rook-ceph-mon-a` | no usable output | | `exec` into `rook-ceph-operator` | `/etc/ceph/` empty; `ceph -s` → `ObjectNotFound('RADOS object not found (error calling conf_read_file)')` | | `exec` into `rook-ceph-mgr` | RADOS permission denied (no admin keyring mounted) | So `ceph status`, `ceph health detail`, `ceph osd df tree`, `ceph osd perf` and `ceph device ls` are all **unreachable** during an incident. Combined with Gap 1, the storage layer had neither live introspection nor metric history — the two independent ways you would normally diagnose it. Deploying the toolbox (or mounting a read-only client keyring somewhere reachable) would close this cheaply. If it is deliberately omitted, that reasoning should be written down where the next investigation will find it — I could not locate any comment explaining its absence. ## Item 3 — drive endurance clock (needs confirmation) The investigation reported **osd.2's drive at 83% endurance used, projecting replacement around Dec 2026**, sourced from Ceph's `devicehealth` module (which stores SMART history to RADOS — which is why that series survived despite Gap 1). ⚠️ **I could not independently confirm this figure**, precisely because of Gap 2. Flagging it rather than asserting it. Confirm once a toolbox exists: ``` ceph device ls ceph device get-health-metrics <devid> ``` or read SMART directly on the Proxmox host backing osd.2. If 83% is accurate this is not urgent but it is a real, dated hardware clock and deserves tracking rather than rediscovery. (Note: SSH to homelab hosts must be **serial** — parallel connections trip UDM rate-limiting and produce spurious "host down" faults.)
claude-bot commented 2026-08-16 21:28:02 +00:00
Author
Member
Copy Link
Copy Source

Status check on both gaps, plus a live HEALTH_WARN that turns out to be structural.

Gap 1 — closed. Ceph is scraped now.

PR #432 landed and worked. ceph_health_status returns a value from Prometheus today, against "zero ceph_* series for all 79 days" when this was filed.

Gap 2 — still open, unchanged.

kubectl get deploy -n rook-ceph lists 19 deployments; rook-ceph-tools is not among them. No admin path exists.

New: the cluster is in HEALTH_WARN right now, and the threshold collides with kubelet's by construction

health:      HEALTH_WARN
details:     MON_DISK_LOW — "mon b is low on available space"
lastChecked: 2026-08-16T21:25:11Z

mon b runs on k8s-2. Mons are on hostPath (no PVCs exist in rook-ceph), so this is the node's /var:

node /var free % free 7d change
k8s-1 28.97 GiB 48.34% −1.42 GiB
k8s-2 (mon b) 17.86 GiB 29.81% −5.71 GiB
k8s-3 21.62 GiB 36.08% +0.27 GiB

k8s-2's /var is a single 59.93 GiB /dev/vda5. 29.81% free is 70.19% used.

Now the two relevant thresholds:

  • Ceph mon_data_avail_warn — 30% free (upstream default; warn), mon_data_avail_crit 5% (err).
  • kubelet imageGCHighThresholdPercent — 70% used, set in kubernetes/talos/patches/cluster.yaml:40 and confirmed live via /api/v1/nodes/k8s-2/proxy/configz. imageGCLowThresholdPercent: 60.

70% used is 30% free. kubelet is configured to let the image cache grow to precisely the point where Ceph starts warning. This is not a runaway — it is the steady-state behaviour of the current configuration, and it will recur on whichever node currently holds the most images. That also makes MON_DISK_LOW useless as a signal, because it fires as a matter of routine.

Not urgent: mon_data_avail_crit is 5% free, and kubelet reclaims at 70% used, so the disk should not approach the error threshold.

Proposed fix

Lower the kubelet image-GC thresholds in kubernetes/talos/patches/cluster.yaml so reclaim happens before Ceph's warn point — e.g. imageGCHighThresholdPercent: 60 / imageGCLowThresholdPercent: 50, leaving ≥40% free at worst against Ceph's 30% floor. Cost is a smaller image cache and more re-pulls; this cluster runs Spegel, which blunts that.

Two caveats before anyone acts:

  1. That file is a bootstrap patch. Editing it alone changes nothing on running nodes — the live value needs talosctl patch machineconfig. Same trap as the Cilium inline manifest in #495.
  2. I could not confirm the live mon_data_avail_warn is still the 30% default, because doing so needs ceph config get mon mon_data_avail_warn — which requires exactly the admin path Gap 2 says does not exist. The gap this issue was filed about is what blocks closing out the finding.

Leaving open for Gap 2 and the threshold decision.

Status check on both gaps, plus a live HEALTH_WARN that turns out to be structural. ## Gap 1 — closed. Ceph is scraped now. PR #432 landed and worked. `ceph_health_status` returns a value from Prometheus today, against "zero `ceph_*` series for all 79 days" when this was filed. ## Gap 2 — still open, unchanged. `kubectl get deploy -n rook-ceph` lists 19 deployments; **`rook-ceph-tools` is not among them**. No admin path exists. ## New: the cluster is in HEALTH_WARN right now, and the threshold collides with kubelet's by construction ``` health: HEALTH_WARN details: MON_DISK_LOW — "mon b is low on available space" lastChecked: 2026-08-16T21:25:11Z ``` mon b runs on **k8s-2**. Mons are on `hostPath` (no PVCs exist in `rook-ceph`), so this is the node's `/var`: | node | /var free | % free | 7d change | |---|---|---|---| | k8s-1 | 28.97 GiB | 48.34% | −1.42 GiB | | k8s-2 (**mon b**) | 17.86 GiB | **29.81%** | **−5.71 GiB** | | k8s-3 | 21.62 GiB | 36.08% | +0.27 GiB | k8s-2's `/var` is a single 59.93 GiB `/dev/vda5`. 29.81% free is **70.19% used**. Now the two relevant thresholds: - Ceph `mon_data_avail_warn` — **30%** free (upstream default; warn), `mon_data_avail_crit` 5% (err). - kubelet `imageGCHighThresholdPercent` — **70%** used, set in `kubernetes/talos/patches/cluster.yaml:40` and confirmed live via `/api/v1/nodes/k8s-2/proxy/configz`. `imageGCLowThresholdPercent: 60`. **70% used is 30% free.** kubelet is configured to let the image cache grow to precisely the point where Ceph starts warning. This is not a runaway — it is the steady-state behaviour of the current configuration, and it will recur on whichever node currently holds the most images. That also makes MON_DISK_LOW useless as a signal, because it fires as a matter of routine. Not urgent: `mon_data_avail_crit` is 5% free, and kubelet reclaims at 70% used, so the disk should not approach the error threshold. ### Proposed fix Lower the kubelet image-GC thresholds in `kubernetes/talos/patches/cluster.yaml` so reclaim happens before Ceph's warn point — e.g. `imageGCHighThresholdPercent: 60` / `imageGCLowThresholdPercent: 50`, leaving ≥40% free at worst against Ceph's 30% floor. Cost is a smaller image cache and more re-pulls; this cluster runs Spegel, which blunts that. Two caveats before anyone acts: 1. That file is a **bootstrap** patch. Editing it alone changes nothing on running nodes — the live value needs `talosctl patch machineconfig`. Same trap as the Cilium inline manifest in #495. 2. **I could not confirm the live `mon_data_avail_warn` is still the 30% default**, because doing so needs `ceph config get mon mon_data_avail_warn` — which requires exactly the admin path Gap 2 says does not exist. The gap this issue was filed about is what blocks closing out the finding. Leaving open for Gap 2 and the threshold decision.
claude-bot commented 2026-08-18 19:31:25 +00:00
Author
Member
Copy Link
Copy Source

Applied the kubelet threshold change to k8s-2 only. It did not work, and the reasoning in #500 was wrong. Recording the measurement.

What happened

talosctl patch machineconfig applied cleanly without a reboot; kubelet picked up imageGCHigh=60 imageGCLow=50; node stayed Ready, 50 pods running, no ImagePull or CrashLoop fallout. GC then fired as intended:

before after
unique cached images ~89 61
containerd content store 9.93 GB 8.62 GB
/var free 31.49% 31.47%
containerd total 29.09 GB 29.18 GB

GC deleted 28 images and freed ~1.3 GB of blobs — and free space did not move at all.

Why

/var/lib/containerd breaks down as:

io.containerd.snapshotter.v1.overlayfs   20.53 GB
io.containerd.content.v1.content          8.62 GB

The dominant consumer is overlayfs snapshots, not image blobs. Those are unpacked layers belonging to in-use images plus container writable layers, and image GC cannot reclaim them. #500 assumed the ~19 GB of snapshots would be freed alongside their images at roughly 2.9x the compressed size; that is wrong, because unused images share most of their layers with in-use ones. Deleting the unused image references freed only their unshared blobs.

Remaining headroom for GC is now ~0.7 GB (8.62 GB content minus 7.92 GB in use). Further sweeps will free essentially nothing. k8s-2 will keep hovering at ~31% free and dip under Ceph's 30% line on any growth.

The actual shape of the problem

Confirmed with the toolbox this PR added — its first useful act:

$ ceph config get mon mon_data_avail_warn
30

So the threshold really is the 30% default, which I could not verify before. And the mon's own data is 106 MB (/var/lib/rook).

A 106 MB database is being judged against a 60 GB filesystem it shares with 29 GB of container storage. No amount of image-cache tuning fixes that; the mon's free-space check is measuring something that has nothing to do with Ceph.

Options, best first

  1. Give mons their own volume — Rook supports mon.volumeClaimTemplate. The mon then measures its own filesystem, and MON_DISK_LOW becomes a real signal again instead of a proxy for image churn. Structural fix; needs a mon migration, so it wants care.
  2. Grow /var. Blunt, and cluster.yaml notes the VM disk was already raised once.
  3. Raise mon_data_avail_warn. Now settable via the toolbox, but it is silencing a check rather than fixing it, and it is cluster-global.
  4. Reduce pod density on k8s-2 (75 pods against 42 on k8s-1) — fewer containers, fewer writable layers.

On the merged change

60/50 is harmless but no longer justified: it keeps the image cache near minimum and buys more re-pulls for ~1.3 GB. Worth reverting to 70/60 unless someone wants the leaner cache for its own sake. k8s-1 and k8s-3 are untouched — I stopped after one node, which is what surfaced this.

Applied the kubelet threshold change to k8s-2 only. **It did not work, and the reasoning in #500 was wrong.** Recording the measurement. ## What happened `talosctl patch machineconfig` applied cleanly without a reboot; kubelet picked up `imageGCHigh=60 imageGCLow=50`; node stayed Ready, 50 pods running, no ImagePull or CrashLoop fallout. GC then fired as intended: | | before | after | |---|---|---| | unique cached images | ~89 | **61** | | containerd content store | 9.93 GB | **8.62 GB** | | /var free | 31.49% | **31.47%** | | containerd total | 29.09 GB | **29.18 GB** | GC deleted 28 images and freed ~1.3 GB of blobs — and **free space did not move at all**. ## Why `/var/lib/containerd` breaks down as: ``` io.containerd.snapshotter.v1.overlayfs 20.53 GB io.containerd.content.v1.content 8.62 GB ``` The dominant consumer is **overlayfs snapshots**, not image blobs. Those are unpacked layers belonging to in-use images plus container writable layers, and image GC cannot reclaim them. #500 assumed the ~19 GB of snapshots would be freed alongside their images at roughly 2.9x the compressed size; that is wrong, because unused images share most of their layers with in-use ones. Deleting the unused image references freed only their unshared blobs. Remaining headroom for GC is now ~0.7 GB (8.62 GB content minus 7.92 GB in use). **Further sweeps will free essentially nothing.** k8s-2 will keep hovering at ~31% free and dip under Ceph's 30% line on any growth. ## The actual shape of the problem Confirmed with the toolbox this PR added — its first useful act: ``` $ ceph config get mon mon_data_avail_warn 30 ``` So the threshold really is the 30% default, which I could not verify before. And the mon's own data is **106 MB** (`/var/lib/rook`). **A 106 MB database is being judged against a 60 GB filesystem it shares with 29 GB of container storage.** No amount of image-cache tuning fixes that; the mon's free-space check is measuring something that has nothing to do with Ceph. ## Options, best first 1. **Give mons their own volume** — Rook supports `mon.volumeClaimTemplate`. The mon then measures its own filesystem, and MON_DISK_LOW becomes a real signal again instead of a proxy for image churn. Structural fix; needs a mon migration, so it wants care. 2. **Grow /var.** Blunt, and `cluster.yaml` notes the VM disk was already raised once. 3. **Raise `mon_data_avail_warn`.** Now settable via the toolbox, but it is silencing a check rather than fixing it, and it is cluster-global. 4. **Reduce pod density on k8s-2** (75 pods against 42 on k8s-1) — fewer containers, fewer writable layers. ## On the merged change 60/50 is harmless but no longer justified: it keeps the image cache near minimum and buys more re-pulls for ~1.3 GB. Worth reverting to 70/60 unless someone wants the leaner cache for its own sake. k8s-1 and k8s-3 are untouched — I stopped after one node, which is what surfaced this.
claude-bot commented 2026-08-18 19:48:00 +00:00
Author
Member
Copy Link
Copy Source

Threshold change is live and verified end to end.

$ ceph config get mon mon_data_avail_warn
20
$ ceph config get mon mon_data_avail_crit
5
$ ceph health
HEALTH_OK

Propagation went CR spec -> Rook -> Ceph as expected; spec.cephConfig.mon landed first via Flux, then the operator pushed it into the mon config store. Reconciled from git, so no live ceph config set drift.

Margin it buys, at the moment of writing:

node /var free headroom over the 20% line
k8s-1 44.61% 24.6 pts
k8s-2 (mon b) 31.37% 11.4 pts
k8s-3 33.56% 13.6 pts

k8s-2 previously sat ~1.5 points above the old 30% line and crossed it on ordinary image-cache growth. It now has ~11 points, which is roughly 7 GB of /var — against a mon database of ~106 MB.

mon_data_avail_crit is untouched at 5%, and CephMonDiskLow still watches /var directly at the 30% mark, so the node-filling risk is still surfaced independently of what Ceph thinks. The change narrows a misfiring alert; it does not remove the signal.

Status of this issue

  • Gap 1 (Ceph not scraped) — closed by #432; ceph_* series confirmed present.
  • Gap 2 (no admin path) — closed by #500. The toolbox has already paid for itself: it is how mon_data_avail_warn, mon_data_avail_crit and the 106 MB mon size were established, none of which were knowable before.
  • osd.2 endurance clock — not revisited in this pass; still open.
  • ServiceMonitor audit (whether operator-managed ServiceMonitors besides Rook's are dropped by the release: kube-prometheus-stack selector) — untouched, still open.
  • Mon storage layout — carried to #503 for the bare-metal migration, where the 20% override should be dropped in favour of a dedicated mon volume.

Leaving open for the endurance clock and the ServiceMonitor audit.

Threshold change is live and verified end to end. ``` $ ceph config get mon mon_data_avail_warn 20 $ ceph config get mon mon_data_avail_crit 5 $ ceph health HEALTH_OK ``` Propagation went CR spec -> Rook -> Ceph as expected; `spec.cephConfig.mon` landed first via Flux, then the operator pushed it into the mon config store. Reconciled from git, so no live `ceph config set` drift. Margin it buys, at the moment of writing: | node | /var free | headroom over the 20% line | |---|---|---| | k8s-1 | 44.61% | 24.6 pts | | k8s-2 (mon b) | **31.37%** | **11.4 pts** | | k8s-3 | 33.56% | 13.6 pts | k8s-2 previously sat ~1.5 points above the old 30% line and crossed it on ordinary image-cache growth. It now has ~11 points, which is roughly 7 GB of `/var` — against a mon database of ~106 MB. `mon_data_avail_crit` is untouched at 5%, and `CephMonDiskLow` still watches `/var` directly at the 30% mark, so the node-filling risk is still surfaced independently of what Ceph thinks. The change narrows a misfiring alert; it does not remove the signal. ## Status of this issue - **Gap 1 (Ceph not scraped)** — closed by #432; `ceph_*` series confirmed present. - **Gap 2 (no admin path)** — closed by #500. The toolbox has already paid for itself: it is how `mon_data_avail_warn`, `mon_data_avail_crit` and the 106 MB mon size were established, none of which were knowable before. - **osd.2 endurance clock** — not revisited in this pass; still open. - **ServiceMonitor audit** (whether operator-managed ServiceMonitors besides Rook's are dropped by the `release: kube-prometheus-stack` selector) — untouched, still open. - **Mon storage layout** — carried to #503 for the bare-metal migration, where the 20% override should be dropped in favour of a dedicated mon volume. Leaving open for the endurance clock and the ServiceMonitor audit.
claude-bot commented 2026-08-19 11:33:47 +00:00
Author
Member
Copy Link
Copy Source

Status update after the toolbox landed — all gaps except the hardware clock are now closed, and the endurance figure needed correcting.

Gap 1 — scraping: CLOSED

  • 820 ceph_* series present, including ceph_health_status.
  • Root cause was fixed globally rather than per-operator: serviceMonitorSelector is now {}, so all 32 ServiceMonitors are selected, not just the 18 carrying release: kube-prometheus-stack. Nothing can be label-dropped any more.
  • ServiceMonitor audit (the second follow-up): done. Active targets with health != up: 0. So Rook was not the only offender, but the empty-selector fix covers every operator-managed ServiceMonitor at once, present and future.

Gap 1 — Ceph PrometheusRule: DONE, except the wear rule

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml exists and carries the right label: CephHealthWarn (ceph_health_status == 1 for 6h, exactly as specified), CephHealthError (== 2 for 5m), CephMonDiskLow.

The device-wear rule is NOT possible as written. There is no wear metric exported:

  • No ceph_device_* / SMART / percentage-used series exists.
  • ceph_health_detail exposes named checks, but DEVICE_HEALTH is not among them.
  • ceph device ls shows an EMPTY LIFE EXPECTANCY column — Ceph is not making predictions, so the health check that would drive such an alert never fires.

Writing a wear rule today would silently never fire, which is the exact failure mode this issue warned about. It needs either Ceph devicehealth prediction enabled, or a SMART exporter (node-exporter textfile collector). Flagging rather than shipping a dead rule.

Gap 2 — toolbox: CLOSED

rook-ceph-tools is deployed and 1/1.

Item 3 — endurance clock: CONFIRMED, and the numbers were wrong

The reported 83% for osd.2 is now 87%, and more importantly this is not one drive. All three OSDs are the same model wearing in lockstep:

OSD Host Device 2026-05-09 2026-08-19 Rate Projected 100%
osd.2 k8s-1 CT1000P3PSSD8_2332E86B749A 73% 87% 0.137 pts/day ~22 Nov 2026
osd.1 k8s-3 CT1000P3PSSD8_2326E6E8E67D 68% 82% 0.137 pts/day ~28 Dec 2026
osd.0 k8s-2 CT1000P3PSSD8_2332E86B74BC 67% 80% 0.128 pts/day ~23 Jan 2027

Derived from Ceph devicehealth SMART history (ceph device get-health-metrics), using the measured 102-day trend rather than a lifetime average.

osd.2 detail: percentage_used 87, available_spare 100% (threshold 5%), data_units_written 298,713,850 (~153 TB), media_errors 18, power_on_hours 19,967 (~2.3 y).

Why this matters more than a single drive

The wear rates are near-identical because Ceph replicates every write to all three OSDs. They therefore wear in lockstep by design, and all three exhaust rated endurance within roughly two months of each other. Replication protects against independent failure; it does not help when three identical drives of the same age and same write load reach end-of-rated-life together.

Calibration

percentage_used 100% means the manufacturer TBW rating is consumed, not that the drive dies — and available_spare is still 100%, which is the better near-term health signal. So this is a procurement clock, not an imminent outage. But it is dated, it is correlated across the whole storage tier, and the first date is ~3 months out.

Suggest staggering replacement (e.g. osd.2 well before late November) so the three do not age back into lockstep afterwards.

🤖 Generated with Claude Code

Status update after the toolbox landed — all gaps except the hardware clock are now closed, and the endurance figure needed correcting. ## Gap 1 — scraping: CLOSED - **820** `ceph_*` series present, including `ceph_health_status`. - Root cause was fixed globally rather than per-operator: `serviceMonitorSelector` is now `{}`, so all **32** ServiceMonitors are selected, not just the 18 carrying `release: kube-prometheus-stack`. Nothing can be label-dropped any more. - **ServiceMonitor audit (the second follow-up): done.** Active targets with health != up: **0**. So Rook was not the only offender, but the empty-selector fix covers every operator-managed ServiceMonitor at once, present and future. ## Gap 1 — Ceph PrometheusRule: DONE, except the wear rule `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml` exists and carries the right label: `CephHealthWarn` (`ceph_health_status == 1` for 6h, exactly as specified), `CephHealthError` (`== 2` for 5m), `CephMonDiskLow`. **The device-wear rule is NOT possible as written.** There is no wear metric exported: - No `ceph_device_*` / SMART / percentage-used series exists. - `ceph_health_detail` exposes named checks, but `DEVICE_HEALTH` is not among them. - `ceph device ls` shows an EMPTY `LIFE EXPECTANCY` column — Ceph is not making predictions, so the health check that would drive such an alert never fires. Writing a wear rule today would silently never fire, which is the exact failure mode this issue warned about. It needs either Ceph devicehealth prediction enabled, or a SMART exporter (node-exporter textfile collector). Flagging rather than shipping a dead rule. ## Gap 2 — toolbox: CLOSED `rook-ceph-tools` is deployed and 1/1. ## Item 3 — endurance clock: CONFIRMED, and the numbers were wrong The reported **83% for osd.2 is now 87%**, and more importantly this is **not one drive**. All three OSDs are the same model wearing in lockstep: | OSD | Host | Device | 2026-05-09 | 2026-08-19 | Rate | Projected 100% | |---|---|---|---|---|---|---| | osd.2 | k8s-1 | CT1000P3PSSD8_2332E86B749A | 73% | **87%** | 0.137 pts/day | **~22 Nov 2026** | | osd.1 | k8s-3 | CT1000P3PSSD8_2326E6E8E67D | 68% | 82% | 0.137 pts/day | ~28 Dec 2026 | | osd.0 | k8s-2 | CT1000P3PSSD8_2332E86B74BC | 67% | 80% | 0.128 pts/day | ~23 Jan 2027 | Derived from Ceph devicehealth SMART history (`ceph device get-health-metrics`), using the **measured 102-day trend** rather than a lifetime average. osd.2 detail: `percentage_used` 87, `available_spare` **100%** (threshold 5%), `data_units_written` 298,713,850 (~153 TB), `media_errors` **18**, `power_on_hours` 19,967 (~2.3 y). ### Why this matters more than a single drive The wear rates are near-identical because Ceph replicates every write to all three OSDs. They therefore wear in lockstep **by design**, and all three exhaust rated endurance within roughly two months of each other. Replication protects against *independent* failure; it does not help when three identical drives of the same age and same write load reach end-of-rated-life together. ### Calibration `percentage_used` 100% means the manufacturer TBW rating is consumed, not that the drive dies — and `available_spare` is still 100%, which is the better near-term health signal. So this is a **procurement clock, not an imminent outage**. But it is dated, it is correlated across the whole storage tier, and the first date is ~3 months out. Suggest staggering replacement (e.g. osd.2 well before late November) so the three do not age back into lockstep afterwards. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-19 13:17:51 +00:00
Author
Member
Copy Link
Copy Source

Root cause of the drive wear found and fixed. It was not client traffic.

The finding: the monitors were a quarter of all cluster writes

The three Ceph mons were writing 240 GB/day between them — 80 GB/day each — for a ~106 MB database. That was ~62% of all OS-disk writes and roughly a quarter of everything the cluster wrote, more than every client workload combined.

paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
paxos logical data       811 MB/day  (21.9 GB over 27 d mon uptime)
ACTUAL mon disk writes    80 GB/day  per mon  (cgroup io.stat: 2.15 TB / 27 d)
=> ~98x write amplification

Mechanism. mon_compact_on_trim (upstream default true) compacts the RocksDB prefix including paxos on every trim. With paxos_trim_min at 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store.

The churn is not cluster activity. 99% of paxos versions are cluster log and mgr stats:

paxos service versions share
logm (cluster log) 6,706,070 60%
mgrstat 4,414,921 39%
osdmap 9,585 0.09%

Every Rook-operator poll (mgr stat, versions, status, report, osd dump — 23 audit entries/min/mon) becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons.

Fixes and MEASURED results

#520 — mon_compact_on_trim: false, plus mon_compact_on_start: true as a compensating control.

mon.a write rate:  80 GB/day  ->  4 GB/day     (600 s window after apply)
                   20x reduction
mon store size:    125 MB      (stable; the growth risk did not materialise)
cluster health:    HEALTH_OK

#519 — replicapool and cephfs-data0 to size 2 / min_size 1; cephfs metadata deliberately left at 3.

raw used:  1.4 TiB -> 951 GiB   (50.92% -> 34.03%)

Combined, at the device layer:

before now change
vda (OS disks) 389 GB/day 130 GB/day -67%
nvme (Ceph OSDs) 630 GB/day 309 GB/day -51%
total 1,019 GB/day 439 GB/day -57%

The vda drop (-259 GB/day) independently matches the predicted mon saving (~228 GB/day).

Effect on the endurance clock

per drive:  210 GB/day -> 103 GB/day
wear rate:  ~0.137 pts/day -> ~0.067 pts/day
osd.2:      13 points remaining -> ~194 days
            22 Nov 2026  ->  roughly March 2027

Remaining life roughly doubles, and with size 2 the three drives no longer wear in perfect lockstep.

Correction

#518 (bluestore_prefer_deferred_size_ssd 65536 -> 0) produced no measurable improvement. BlueStore counters show write_small is only 0.4% of write volume (4 GB vs 1,015 GB write_big), so that setting could never have mattered at this scale. It remains correct — HDD tuning on SSD — but it was not the cause and should not be credited as the fix.

Also: cAdvisor container_fs_writes_bytes_total roughly double-counts here (425 GB/day for OSDs where nvme0n1 shows 212; 155 GB/day for mons where cgroup io.stat shows 80). Useful for ranking writers, not for absolute figures. Every number above comes from cgroup io.stat, kernel diskstats, or SMART.

Pending verification (24 h)

Device figures above are a 10-minute window. Baselines for a clean re-check:

  • SMART samples daily at ~00:08. osd.2 at 20260819-000812: percentage_used=87, data_units_written=298713850. Compare 20260820 -> 20260821 for a pure post-change 24 h delta (the change landed 2026-08-19T13:00:23Z, so the 08-20 sample straddles it).
  • Prometheus: after 2026-08-20T13:00Z, rate(node_disk_written_bytes_total[24h]) is entirely post-change.

Remaining

  • Complementary fix, optional: slow the Rook operator polling (ROOK_CEPH_STATUS_CHECK_INTERVAL et al., unset in repo so defaults apply). #520 removed the amplification; this would remove the churn. Much smaller win now that mons are at 4 GB/day.
  • Bare-metal migration: size=3 on exactly 3 OSDs was the structural error — every drive absorbed 100% of cluster writes, so wear was correlated by construction. Build the new cluster with enough OSDs that replication actually spreads load.

🤖 Generated with Claude Code

Root cause of the drive wear found and fixed. It was not client traffic. ## The finding: the monitors were a quarter of all cluster writes The three Ceph mons were writing **240 GB/day** between them — 80 GB/day each — for a **~106 MB** database. That was ~62% of all OS-disk writes and roughly a quarter of everything the cluster wrote, more than every client workload combined. ``` paxos transactions ~99,000/day (1.15/s, sampled over 60 s) paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime) ACTUAL mon disk writes 80 GB/day per mon (cgroup io.stat: 2.15 TB / 27 d) => ~98x write amplification ``` **Mechanism.** `mon_compact_on_trim` (upstream default `true`) compacts the RocksDB prefix including paxos on every trim. With `paxos_trim_min` at 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store. **The churn is not cluster activity.** 99% of paxos versions are cluster log and mgr stats: | paxos service | versions | share | |---|---|---| | logm (cluster log) | 6,706,070 | 60% | | mgrstat | 4,414,921 | 39% | | osdmap | 9,585 | 0.09% | Every Rook-operator poll (`mgr stat`, `versions`, `status`, `report`, `osd dump` — 23 audit entries/min/mon) becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons. ## Fixes and MEASURED results **#520** — `mon_compact_on_trim: false`, plus `mon_compact_on_start: true` as a compensating control. ``` mon.a write rate: 80 GB/day -> 4 GB/day (600 s window after apply) 20x reduction mon store size: 125 MB (stable; the growth risk did not materialise) cluster health: HEALTH_OK ``` **#519** — `replicapool` and `cephfs-data0` to `size 2 / min_size 1`; cephfs metadata deliberately left at 3. ``` raw used: 1.4 TiB -> 951 GiB (50.92% -> 34.03%) ``` **Combined, at the device layer:** | | before | now | change | |---|---|---|---| | `vda` (OS disks) | 389 GB/day | 130 GB/day | **-67%** | | `nvme` (Ceph OSDs) | 630 GB/day | 309 GB/day | **-51%** | | **total** | **1,019 GB/day** | **439 GB/day** | **-57%** | The `vda` drop (-259 GB/day) independently matches the predicted mon saving (~228 GB/day). ## Effect on the endurance clock ``` per drive: 210 GB/day -> 103 GB/day wear rate: ~0.137 pts/day -> ~0.067 pts/day osd.2: 13 points remaining -> ~194 days 22 Nov 2026 -> roughly March 2027 ``` Remaining life roughly doubles, and with `size 2` the three drives no longer wear in perfect lockstep. ## Correction **#518** (`bluestore_prefer_deferred_size_ssd` 65536 -> 0) produced **no measurable improvement**. BlueStore counters show `write_small` is only 0.4% of write volume (4 GB vs 1,015 GB `write_big`), so that setting could never have mattered at this scale. It remains correct — HDD tuning on SSD — but it was not the cause and should not be credited as the fix. Also: cAdvisor `container_fs_writes_bytes_total` roughly **double-counts** here (425 GB/day for OSDs where `nvme0n1` shows 212; 155 GB/day for mons where cgroup `io.stat` shows 80). Useful for ranking writers, not for absolute figures. Every number above comes from cgroup `io.stat`, kernel diskstats, or SMART. ## Pending verification (24 h) Device figures above are a 10-minute window. Baselines for a clean re-check: - **SMART** samples daily at ~00:08. osd.2 at `20260819-000812`: `percentage_used=87`, `data_units_written=298713850`. Compare `20260820` -> `20260821` for a pure post-change 24 h delta (the change landed 2026-08-19T13:00:23Z, so the 08-20 sample straddles it). - **Prometheus**: after 2026-08-20T13:00Z, `rate(node_disk_written_bytes_total[24h])` is entirely post-change. ## Remaining - **Complementary fix, optional:** slow the Rook operator polling (`ROOK_CEPH_STATUS_CHECK_INTERVAL` et al., unset in repo so defaults apply). #520 removed the amplification; this would remove the churn. Much smaller win now that mons are at 4 GB/day. - **Bare-metal migration:** `size=3` on exactly 3 OSDs was the structural error — every drive absorbed 100% of cluster writes, so wear was correlated by construction. Build the new cluster with enough OSDs that replication actually spreads load. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-20 04:39:55 +00:00
Author
Member
Copy Link
Copy Source

Verified result of the trmnl renderer fix (#529) and the emptyDir survey (#530).

trmnl-renderer: 34 GB/day -> ~18 MB/day

before:  253:0 (node disk)  249,092,697,088 bytes over 7.3 days  = ~34 GB/day
after:   253:0 (node disk)          44,544 bytes over 3m32s      = ~18 MB/day

/tmp is now tmpfs 512M (27M used) instead of /dev/vda5. Renderer confirmed still working: Completed: Entrance e-ink panel in 8788ms | webhook: 200 OK.

Cluster-wide OS-disk writes: 215 GB/day -> 123 GB/day (-43%) for a one-line change.

What it was

Not volume — cadence. schedules.template.json renders on "* * * * *", so Chrome launched 1440x/day and each launch created, filled and deleted a /tmp/puppeteer_dev_chrome_profile-*. At rest /tmp held under 8 MB. Pure churn onto the node filesystem, for data that never outlived the render.

The trap: emptyDir: {} without medium: Memory is node storage. Easy to read as "ephemeral, therefore free".

Survey (#530)

23 emptyDir declarations in the repo, only 2 RAM-backed. 12 more converted — but honestly, only trmnl mattered. Ranking every pod by OS-disk writes shows nothing emptyDir-driven after it; each converted volume measured 8.0K in the running pod. The remaining value is correctness plus keeping mosquitto/auth (generated passwd/acl) off the node filesystem.

Running total for the day

start of day        ~1,019 GB/day   (nvme 630 + vda 389)
after #518/#519/#520   ~500 GB/day
after #529             ~432 GB/day   (nvme ~309 + vda 123)

Roughly a 58% reduction in total cluster disk writes.

Correction on the recorder migration

The Home Assistant SQLite -> Postgres move (#521/#524) did NOT deliver a cluster-level reduction, contrary to what I projected when proposing it. Measured over 8h+:

home-assistant container:  40,000 -> 9,077 MB/day   (real, -77%)
ha-pg postgres:                   -> 26,292 MB/day  (inflated by the one-time bulk migration)
combined:                  40,000 -> 35,369 MB/day  (~12% net)

HA own writes genuinely fell 4.4x, and 2.4 years of statistics survived intact, so the migration stands on its own merits. But it was not the lever I claimed. The real levers were the Ceph mon compaction fix (240 GB/day) and this renderer (34 GB/day) — both found by looking at where writes actually landed rather than where they were assumed to.

Endurance clock

Worth re-deriving once these settle. At ~432 GB/day across 3 drives with size=2, per-drive load is well down from the 210 GB/day that produced the ~0.137 points/day wear rate behind the Nov 2026 projection.

🤖 Generated with Claude Code

Verified result of the trmnl renderer fix (#529) and the emptyDir survey (#530). ## trmnl-renderer: 34 GB/day -> ~18 MB/day ``` before: 253:0 (node disk) 249,092,697,088 bytes over 7.3 days = ~34 GB/day after: 253:0 (node disk) 44,544 bytes over 3m32s = ~18 MB/day ``` `/tmp` is now `tmpfs 512M` (27M used) instead of `/dev/vda5`. Renderer confirmed still working: `Completed: Entrance e-ink panel in 8788ms | webhook: 200 OK`. Cluster-wide OS-disk writes: **215 GB/day -> 123 GB/day (-43%)** for a one-line change. ## What it was Not volume — cadence. `schedules.template.json` renders on `"* * * * *"`, so Chrome launched 1440x/day and each launch created, filled and deleted a `/tmp/puppeteer_dev_chrome_profile-*`. At rest `/tmp` held under 8 MB. Pure churn onto the node filesystem, for data that never outlived the render. The trap: `emptyDir: {}` **without** `medium: Memory` is node storage. Easy to read as "ephemeral, therefore free". ## Survey (#530) 23 emptyDir declarations in the repo, only 2 RAM-backed. 12 more converted — but honestly, **only trmnl mattered**. Ranking every pod by OS-disk writes shows nothing emptyDir-driven after it; each converted volume measured 8.0K in the running pod. The remaining value is correctness plus keeping `mosquitto/auth` (generated passwd/acl) off the node filesystem. ## Running total for the day ``` start of day ~1,019 GB/day (nvme 630 + vda 389) after #518/#519/#520 ~500 GB/day after #529 ~432 GB/day (nvme ~309 + vda 123) ``` Roughly a 58% reduction in total cluster disk writes. ## Correction on the recorder migration The Home Assistant SQLite -> Postgres move (#521/#524) did NOT deliver a cluster-level reduction, contrary to what I projected when proposing it. Measured over 8h+: ``` home-assistant container: 40,000 -> 9,077 MB/day (real, -77%) ha-pg postgres: -> 26,292 MB/day (inflated by the one-time bulk migration) combined: 40,000 -> 35,369 MB/day (~12% net) ``` HA own writes genuinely fell 4.4x, and 2.4 years of statistics survived intact, so the migration stands on its own merits. But it was not the lever I claimed. The real levers were the Ceph mon compaction fix (240 GB/day) and this renderer (34 GB/day) — both found by looking at where writes actually landed rather than where they were assumed to. ## Endurance clock Worth re-deriving once these settle. At ~432 GB/day across 3 drives with size=2, per-drive load is well down from the 210 GB/day that produced the ~0.137 points/day wear rate behind the Nov 2026 projection. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-20 13:11:12 +00:00
Author
Member
Copy Link
Copy Source

24h verification of #519 (pool size 2) and #520 (mon_compact_on_trim=false)

Both landed 2026-08-19T13:00:23Z. Measured 2026-08-20T13:07Z, so every [24h] window below sits entirely after the change.

Device write rates — held

metric baseline now change
vda 389 GB/day 179.5 −54%
nvme (3 OSD drives) 630 GB/day 328 −48%
total 1019 GB/day 507.5 −50%

Mon writes — held, but my earlier estimate was optimistic

Baseline 80 GB/day/mon. The 10-minute-window estimate posted earlier said 4 GB/day. The sustained figure over the mon container lifetime (started 2026-08-19T15:45:34Z, i.e. entirely post-change) is:

7.152 GB over 21h22m = 8.03 GB/day/mon

A 90% reduction — and a correction: the real rate is 2x the short-window estimate. The 600s sample undercounted.

Mon store size — no regression

145 MB, against a ~125 MB baseline and the 1 GB investigation threshold. Disabling compact_on_trim has not produced unbounded store growth after 24h. Worth re-checking at 7d, since that is the failure mode this parameter guards against.

Health and settings

  • mon_compact_on_trim = false — confirmed applied
  • replicapool size 2, cephfs-data0 size 2, cephfs-metadata size 3 — confirmed
  • ceph health: HEALTH_WARN, but only the CVE-2025-30156 insecure-key-type warnings tracked in #527 (mute expires 2026-09-02). No PG, data, or redundancy warnings arising from size 2.

SMART wear — methodology correction

The 20260819 to 20260820 SMART interval is not a clean post-change window. Ceph samples device health at ~00:08 daily and the change landed at 13:00, so that interval is ~13h pre-change and ~11h post-change. It reads 326,586 data units = 167 GB/day, which is a blend rather than a result.

The clean measure for the same drive is the device counter: osd.2 (k8s-1 nvme0n1) sits at 99 GB/day over a full post-change 24h.

Historical baseline from the 104 retained samples:

2026-05-09  used=73%  duw=257276933
2026-08-20  used=87%  duw=299040436
103 days, 14 wear points, 21.4 TB written
=> 0.136 pts/day at 207.6 GB/day, approx 1.53 TB per wear point

Projection for the 13 points osd.2 has remaining:

rate days 100% reached
207.6 GB/day (pre-change) 96 2026-11-24
99 GB/day (measured now) 201 2027-03-09

This corrects an earlier estimate of roughly May 2027, which assumed a larger reduction than was actually measured. March 2027 is the defensible figure.

percentage_used is still 87% on osd.2, unchanged since 2026-08-18. The first fully clean SMART interval will be 20260820 to 20260821; pts/day should be recomputed then to confirm the projection against observed wear rather than against host writes.

Summary

The improvement held: roughly half the cluster write volume is gone, and the mon fix delivered 90%. Two figures need correcting from the earlier short-window estimates — mon writes are 8 GB/day rather than 4, and the endurance projection is March 2027 rather than May 2027.

## 24h verification of #519 (pool size 2) and #520 (mon_compact_on_trim=false) Both landed 2026-08-19T13:00:23Z. Measured 2026-08-20T13:07Z, so every `[24h]` window below sits entirely after the change. ### Device write rates — held | metric | baseline | now | change | |---|---|---|---| | vda | 389 GB/day | **179.5** | −54% | | nvme (3 OSD drives) | 630 GB/day | **328** | −48% | | **total** | **1019 GB/day** | **507.5** | **−50%** | ### Mon writes — held, but my earlier estimate was optimistic Baseline 80 GB/day/mon. The 10-minute-window estimate posted earlier said 4 GB/day. The sustained figure over the mon container lifetime (started 2026-08-19T15:45:34Z, i.e. entirely post-change) is: 7.152 GB over 21h22m = 8.03 GB/day/mon A **90% reduction** — and a correction: the real rate is 2x the short-window estimate. The 600s sample undercounted. ### Mon store size — no regression **145 MB**, against a ~125 MB baseline and the 1 GB investigation threshold. Disabling compact_on_trim has not produced unbounded store growth after 24h. Worth re-checking at 7d, since that is the failure mode this parameter guards against. ### Health and settings - `mon_compact_on_trim = false` — confirmed applied - `replicapool` size 2, `cephfs-data0` size 2, `cephfs-metadata` size 3 — confirmed - `ceph health`: HEALTH_WARN, but only the CVE-2025-30156 insecure-key-type warnings tracked in #527 (mute expires 2026-09-02). No PG, data, or redundancy warnings arising from size 2. ### SMART wear — methodology correction **The 20260819 to 20260820 SMART interval is not a clean post-change window.** Ceph samples device health at ~00:08 daily and the change landed at 13:00, so that interval is ~13h pre-change and ~11h post-change. It reads 326,586 data units = 167 GB/day, which is a blend rather than a result. The clean measure for the same drive is the device counter: osd.2 (k8s-1 nvme0n1) sits at **99 GB/day** over a full post-change 24h. Historical baseline from the 104 retained samples: 2026-05-09 used=73% duw=257276933 2026-08-20 used=87% duw=299040436 103 days, 14 wear points, 21.4 TB written => 0.136 pts/day at 207.6 GB/day, approx 1.53 TB per wear point Projection for the 13 points osd.2 has remaining: | rate | days | 100% reached | |---|---|---| | 207.6 GB/day (pre-change) | 96 | 2026-11-24 | | 99 GB/day (measured now) | 201 | **2027-03-09** | This **corrects an earlier estimate of roughly May 2027**, which assumed a larger reduction than was actually measured. March 2027 is the defensible figure. `percentage_used` is still 87% on osd.2, unchanged since 2026-08-18. The first fully clean SMART interval will be 20260820 to 20260821; pts/day should be recomputed then to confirm the projection against observed wear rather than against host writes. ### Summary The improvement held: roughly half the cluster write volume is gone, and the mon fix delivered 90%. Two figures need correcting from the earlier short-window estimates — mon writes are 8 GB/day rather than 4, and the endurance projection is March 2027 rather than May 2027.
claude-bot commented 2026-08-21 20:37:54 +00:00
Author
Member
Copy Link
Copy Source

Verified all three items against the live cluster. Two closed, one confirmed, one follow-up split out.

Gap 1 — Ceph scraping: CLOSED

PR #432 landed. ceph_* series are present and in active use — ceph_health_detail, ceph_pool_bytes_used, ceph_pool_stored, ceph_pool_metadata all back live alert rules today. The root cause is structurally gone: Prometheus serviceMonitorSelector is now {} (selects all), so operator-created ServiceMonitors carrying their own labels are no longer dropped. 32 ServiceMonitors are selected. That also answers the follow-up asking whether other operators were affected — nothing can be dropped by label mismatch now.

The requested PrometheusRule exists (kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml): CephHealthWarn, CephHealthError, CephPoolLostRedundancy, CephMonDiskLow.

The device-wear rule from that same follow-up was never written, and cannot be as specified. Split out below.

Gap 2 — no read-only path to Ceph: CLOSED

rook-ceph-tools is deployed (kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml). ceph -s, ceph health detail, ceph osd df, ceph device ls, ceph osd pool ls detail and ceph osd ok-to-stop all work. It carried the entire #545/#547 investigation.

One operational caveat found while using it: CephX key rotation invalidates a running toolbox's keyring — the pod holds the pre-rotation admin key and returns [errno 13] RADOS permission denied until restarted. The toolbox is defined in this repo, not managed by Rook, so nothing repairs it automatically.

Item 3 — endurance clock: CONFIRMED, and worse than the estimate

The 83% figure was right in kind but the projection was optimistic. Measured from ceph device get-health-metrics:

osd.2 (k8s-1)  87% used  153.2 TB written  power_on 20,014 h
osd.1 (k8s-3)  82%
osd.0 (k8s-2)  81%

All three are Crucial CT1000P3PSSD8. Current write rate on osd.2 is 152 GB/day (867,429 data units over 70 h). Remaining 13% projects two ways: ~150 days linear-on-bytes, or ~75 days on the observed wear cadence (a steady ~140 h per 1%; 86->87% took 140 h). Wear is accelerating per byte written, normal for aging QLC — available_spare is still 100%, so this is not spare-block exhaustion.

So 2.5-5 months from 2026-08-21, not December. Tracked with a replacement plan (Samsung PM983, slot shuffle, one-node-at-a-time procedure). Note the wiki recorded these as "Crucial P2" — lspci mislabels them because the P3 reuses the P2's PCI ID c0a9:540a; the model string is CT1000P3PSSD8. Corrected.

Closing. The device-wear alerting gap continues separately.

Verified all three items against the live cluster. Two closed, one confirmed, one follow-up split out. ## Gap 1 — Ceph scraping: CLOSED PR #432 landed. `ceph_*` series are present and in active use — `ceph_health_detail`, `ceph_pool_bytes_used`, `ceph_pool_stored`, `ceph_pool_metadata` all back live alert rules today. The root cause is structurally gone: Prometheus `serviceMonitorSelector` is now `{}` (selects all), so operator-created ServiceMonitors carrying their own labels are no longer dropped. 32 ServiceMonitors are selected. That also answers the follow-up asking whether other operators were affected — nothing can be dropped by label mismatch now. The requested PrometheusRule exists (`kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml`): `CephHealthWarn`, `CephHealthError`, `CephPoolLostRedundancy`, `CephMonDiskLow`. **The device-wear rule from that same follow-up was never written, and cannot be as specified.** Split out below. ## Gap 2 — no read-only path to Ceph: CLOSED `rook-ceph-tools` is deployed (`kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml`). `ceph -s`, `ceph health detail`, `ceph osd df`, `ceph device ls`, `ceph osd pool ls detail` and `ceph osd ok-to-stop` all work. It carried the entire #545/#547 investigation. One operational caveat found while using it: CephX key rotation invalidates a running toolbox's keyring — the pod holds the pre-rotation admin key and returns `[errno 13] RADOS permission denied` until restarted. The toolbox is defined in this repo, not managed by Rook, so nothing repairs it automatically. ## Item 3 — endurance clock: CONFIRMED, and worse than the estimate The 83% figure was right in kind but the projection was optimistic. Measured from `ceph device get-health-metrics`: ``` osd.2 (k8s-1) 87% used 153.2 TB written power_on 20,014 h osd.1 (k8s-3) 82% osd.0 (k8s-2) 81% ``` All three are Crucial CT1000P3PSSD8. Current write rate on osd.2 is 152 GB/day (867,429 data units over 70 h). Remaining 13% projects two ways: ~150 days linear-on-bytes, or ~75 days on the observed wear cadence (a steady ~140 h per 1%; 86->87% took 140 h). Wear is accelerating per byte written, normal for aging QLC — `available_spare` is still 100%, so this is not spare-block exhaustion. **So 2.5-5 months from 2026-08-21, not December.** Tracked with a replacement plan (Samsung PM983, slot shuffle, one-node-at-a-time procedure). Note the wiki recorded these as "Crucial P2" — `lspci` mislabels them because the P3 reuses the P2's PCI ID `c0a9:540a`; the model string is CT1000P3PSSD8. Corrected. Closing. The device-wear alerting gap continues separately.
claude-bot commented 2026-08-21 20:41:03 +00:00
Author
Member
Copy Link
Copy Source

Correction to the endurance projection in my previous comment. I wrote "2.5-5 months". The 2.5 was derived wrongly and the range needs a stated denominator.

Method error. I took the observed calendar cadence of ~140 h per 1% of wear — measured while the drives were writing 400+ GB/day — and projected it onto a future writing less, after the write-reduction work. Calendar cadence is not portable across a change in write rate. Recomputed from bytes:

total written   299,186,405 units x 512 KB  =  153.2 TB

Two denominators, ~3x apart:

drive percentage_used 87%   -> full life ~176 TB  ->  22.9 TB remaining
spec sheet 220 TBW rated    -> 69.6% used         ->  66.8 TB remaining

percentage_used is the controller's own estimate and includes the drive's internal write amplification; TBW is a host-bytes warranty figure. They do not measure the same thing.

Write rate is not a single number either — 152 GB/day over the last 70 h, 231 GB/day over the last 210 h, with one 24 h window alone at 576 GB.

                    at 231 GB/day    at 152 GB/day
drive indicator      99 days          151 days     -> 3-5 months
spec sheet          289 days          439 days     -> 9.5-14 months

Plan on 3-5 months (drive indicator, conservative). Wear is also accelerating per byte written — 1% cost 2.22 TB at the 73->74% step versus 1.58 TB at 86->87% — so the remaining 13% likely costs under 20.5 TB, favouring the low end.

This does not change the conclusion that replacement is needed well before the drives are a year older, but the earlier "projecting replacement around Dec 2026" in the issue body and my "2.5 months" were both wrong, in opposite directions and for different reasons.

Correction to the endurance projection in my previous comment. I wrote "2.5-5 months". The 2.5 was derived wrongly and the range needs a stated denominator. **Method error.** I took the observed calendar cadence of ~140 h per 1% of wear — measured while the drives were writing 400+ GB/day — and projected it onto a future writing less, after the write-reduction work. Calendar cadence is not portable across a change in write rate. Recomputed from bytes: ``` total written 299,186,405 units x 512 KB = 153.2 TB ``` **Two denominators, ~3x apart:** ``` drive percentage_used 87% -> full life ~176 TB -> 22.9 TB remaining spec sheet 220 TBW rated -> 69.6% used -> 66.8 TB remaining ``` `percentage_used` is the controller's own estimate and includes the drive's internal write amplification; TBW is a host-bytes warranty figure. They do not measure the same thing. **Write rate is not a single number** either — 152 GB/day over the last 70 h, 231 GB/day over the last 210 h, with one 24 h window alone at 576 GB. ``` at 231 GB/day at 152 GB/day drive indicator 99 days 151 days -> 3-5 months spec sheet 289 days 439 days -> 9.5-14 months ``` **Plan on 3-5 months** (drive indicator, conservative). Wear is also accelerating per byte written — 1% cost 2.22 TB at the 73->74% step versus 1.58 TB at 86->87% — so the remaining 13% likely costs under 20.5 TB, favouring the low end. This does not change the conclusion that replacement is needed well before the drives are a year older, but the earlier "projecting replacement around Dec 2026" in the issue body and my "2.5 months" were both wrong, in opposite directions and for different reasons.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#434