From 57acf5d8f21ccc1c7f42ad7e178069cfee30c2bb Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Tue, 18 Aug 2026 21:41:02 +0200 Subject: [PATCH] fix(ceph): set mon_data_avail_warn to 20%, matching the shared /var MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses #434. Ceph's 30% default assumes the mon owns its filesystem, where 30% free is a real reserve. Our mons sit on dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB of containerd storage — and the mon DB is ~106 MB. The default therefore measures container churn rather than Ceph state, and k8s-2 trips MON_DISK_LOW whenever its image cache fills. Tuning image GC does not help: measured on k8s-2, dropping the GC high threshold to 60 deleted 28 images and moved free space by 0.02 percentage points, because most of /var/lib/containerd is overlayfs snapshots held by running containers, which image GC cannot reclaim. 20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database. mon_data_avail_crit is untouched at 5%, and CephMonDiskLow watches /var directly for the real risk — the node filling, not the mon growing. Set through cephConfig so it is reconciled from git rather than applied as a live `ceph config set`. The comment records that this override should be revisited, and dropped, when mons get their own volume at bare-metal migration. That is not possible on the current cluster: every StorageClass is Ceph-backed, so a mon PVC would be circular, and Rook applies mon.volumeClaimTemplate only to newly created mons. Co-Authored-By: Claude Bot --- .../rook-ceph-cluster/cephcluster.yaml | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml index 795daaa1..693d160d 100644 --- a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml +++ b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml @@ -24,6 +24,27 @@ spec: # gives headroom for parallel restores during cluster rebuild too # (paperless + audiobookshelf + immich + ... all clone at once). cephConfig: + mon: + # Ceph's 30% default assumes the mon has a filesystem to itself, where + # 30% free is a meaningful reserve. Here the mons live on + # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB + # of containerd image and snapshot storage — and the mon DB itself is + # ~106 MB. So the default measures container churn, not Ceph: k8s-2 + # trips MON_DISK_LOW whenever the image cache fills, and image GC does + # not clear it (most of /var/lib/containerd is overlayfs snapshots held + # by running containers, which GC cannot reclaim — see #434). + # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database. + # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in + # monitoring-rules watches /var directly for the real risk, which is the + # node filling rather than the mon growing. + # + # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and + # the default 30% becomes meaningful again, at which point this override + # should be dropped rather than carried over. It cannot be done on the + # current cluster — every StorageClass is Ceph-backed, so a mon PVC + # would be circular, and Rook only applies mon.volumeClaimTemplate to + # newly created mons, never migrating existing ones. + mon_data_avail_warn: "20" mgr: mgr/volumes/max_concurrent_clones: "16" # QLC tune for the Crucial P2 NVMe per host. Baseline 2026-05-05: