fix(ceph): stop mon compacting on every paxos trim (240 GB/day of writes) #520
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#520
Reference in New Issue
Block a user
Refs #434.
The finding
The three Ceph monitors write 240 GB/day between them — 80 GB/day each — for a database the repo documents as ~106 MB.
That is ~62% of all OS-disk writes on these nodes, and roughly a quarter of everything the cluster writes. More than every client workload combined.
Measured 2026-08-19:
Cause
mon_compact_on_trim, upstream defaulttrue. Per the Ceph docs: "Compact a certain prefix (including paxos) when we trim its old states."With
paxos_trim_minat 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store.The churn is not cluster activity
99% of paxos versions are cluster log and mgr stats:
Every Rook-operator poll —
mgr stat,versions,status,report,osd dump, at 23 audit entries per minute per mon — becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons.Reducing that polling is the complementary fix and is tracked separately. This change removes the amplification regardless of the rate.
This does not disable compaction
RocksDB levelled compaction still runs and still reclaims trimmed keys. What stops is the extra forced
compact_rangeon every trim.Two backstops if that assumption is wrong:
mon_data_size_warn(15 GB default) raisesHEALTH_WARN→ already alerted byCephHealthWarnin monitoring-rules.mon_compact_on_start: trueis set here (upstream defaultfalse), so every mon restart — upgrade, node reboot, reschedule — does a full compaction. Cheap on a ~106 MB store.What to watch:
du -sh /var/lib/rook/mon-*/store.db. If it grows past ~1 GB and stays there, revisit rather than leaving it.Verification baseline
mon.a cgroup
io.statat2026-08-19T12:56:29Z:wbytes=2154257645056,wios=22509683, uptime since2026-07-23T13:20:44Z. I will re-read this after it applies and report the actual delta rather than claim an improvement.Methodology note
cAdvisor
container_fs_writes_bytes_totalroughly double-counts here — it reported 425 GB/day for OSDs wherenvme0n1shows 212, and 155 GB/day for mons where cgroupio.statshows 80. It was used only to rank writers. Every figure above comes from cgroupio.stator kernel diskstats, which reconcile with each other and with the drives SMART counters.🤖 Generated with Claude Code
The three Ceph monitors were writing 240 GB/day between them for a ~106 MB database. That was ~62% of all OS-disk writes on the nodes and roughly a quarter of everything the cluster writes -- more than every client workload combined. Measured 2026-08-19: paxos transactions ~99,000/day (1.15/s, sampled over 60 s) paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime) ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three (cgroup io.stat: 2.15 TB / 27 d) => ~98x write amplification Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix (including paxos) when we trim its old states". With paxos_trim_min at 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store. The churn driving it is not cluster activity. 99% of paxos versions are cluster log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook operator poll (mgr stat / versions / status / report / osd dump; 23 audit entries per minute per mon) becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons. Reducing that polling is the complementary fix and is tracked separately; this change removes the amplification regardless of the rate. This does NOT disable compaction. RocksDB's own levelled compaction still runs and still reclaims trimmed keys; what stops is the extra forced compact_range on every trim. Two backstops if that assumption is wrong: - mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn in monitoring-rules already alerts on. - mon_compact_on_start is set to true here (upstream default false), so every mon restart -- upgrade, node reboot, reschedule -- does a full compaction. Cheap on a ~106 MB store. Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z: wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z. Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212, and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank writers; every figure above comes from cgroup io.stat or kernel diskstats, which reconcile with each other and with the drives' SMART counters. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit63f25c15112ad64f268b46c9d7de968508e20adb.Summary
Adds two Ceph mon config overrides in
kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:mon_compact_on_trim: "false"(disables the forced RocksDBcompact_rangeon every paxos trim) paired withmon_compact_on_start: "true"(compensating full compaction on mon restart) as a backstop. This is a pure config-and-comment change — no code, no new resources — using a pattern (cephConfig.mon.<key>: "<value>") already established elsewhere in this same file formon_data_avail_warn. I found no blocking issues.🟡 Suggestions
mon_compact_on_trim(kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:46) is applied via Rook'scephConfigmechanism, which maps toceph config set mon <key> <value>— a live/central config change. I could not confirm from the checkout alone whether the mon daemon re-reads this particular option on every trim (likely, since it's a simple boolean gate) or only picks it up at process start likemon_compact_on_startdoes. If it's the latter, the write-amplification fix won't actually take effect on the three already-running mons until they next restart, even though the config value is set immediately. The PR body already commits to re-checking cgroupio.statafter applying, which is the right way to catch this — just flagging that this verification step matters more than usual here, since it's the only thing that would catch a "set but not yet effective" gap. Worth confirming (or capturing the mon restart /ceph daemon mon.<id> config show | grep mon_compact_on_trimoutput) in the follow-up rather than assuming the metric drop alone proves it.mon_data_avail_warnand the OSD deferred-write settings just below in this same file, so it's consistent with the repo's established convention of over-documenting non-obvious infra tuning — not something to change, just noting it's intentional and in line with precedent rather than a one-off.Tests
No automated tests apply to this change (a Flux-managed Ceph config value) — that's consistent with the rest of this file, which also carries no tests for
mon_data_avail_warnor the OSD discard/deferred-write settings. Validation here is operational: the PR description states a baseline cgroupio.statreading was captured before applying and will be re-read after, which is the appropriate check for a change like this given the "unclear if it will be caught in the current review environment" risk noted in the suggestion above.Recommendation
Looks reasonable — no blocking issues; comments above are optional.