konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests Packages Projects Releases 1 Wiki Activity

Bare-metal migration: give Ceph mons their own volume and drop the mon_data_avail_warn override #503

New Issue
Closed
opened 2026-08-18 19:42:10 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-08-18 19:42:10 +00:00
Member
Copy Link
Copy Source

Parked for the bare-metal migration. Nothing to do until then — filing it so the decision surfaces at planning time rather than being discovered in a manifest comment afterwards.

What was done, and why it is temporary

mon_data_avail_warn is set to 20% in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml (PR #502), down from Ceph's 30% default.

That default assumes the mon owns its filesystem. On this cluster the mons live on dataDirHostPath: /var/lib/rook — the node's /var, shared with ~29 GB of containerd image and snapshot storage — while the mon DB itself is ~106 MB. The check therefore measures container churn rather than Ceph state, and k8s-2 trips MON_DISK_LOW whenever its image cache fills.

The override is a workaround for that mismatch, not a fix for it.

Why the real fix is blocked today

Giving mons a dedicated volume via mon.volumeClaimTemplate is the correct answer, and it is not possible on the current cluster for two independent reasons:

  1. Every StorageClass here is Ceph-backed — ceph-block, ceph-filesystem, ceph-bucket, all rook-ceph.*.csi.ceph.com. Mons on a Ceph-backed PVC are circular: the mons are the quorum authority, so on a cold start there is no Ceph to serve the volume the mons need in order to boot. A node-local provisioner would have to exist first.
  2. Rook does not migrate existing mons. Upstream: volumeClaimTemplate "only applies to new monitors that are created when the requested number of monitors increases, or when a monitor fails and is recreated." Even with a suitable class, converting means deliberately failing and recreating mons one at a time.

What to consider at migration time

  • Give the mons a dedicated, node-local volume — separate from the container runtime's filesystem, and not backed by Ceph itself.
  • Once they have one, drop the mon_data_avail_warn: "20" override rather than carrying it over. On a dedicated volume the 30% default is meaningful again, and keeping the override would hide a real mon-disk problem.
  • Size it against reality: the mon DB is ~106 MB today. It grows with osdmap churn, so it does not need to be large — it needs to be separate.
  • If bare metal keeps the container runtime and mons on one filesystem, this whole problem comes with it and the override stays needed.

Related

  • #434 — where the MON_DISK_LOW investigation happened
  • #500 — added the Ceph toolbox (which is how the 106 MB and live threshold were confirmed) and the alerts
  • #501 — reverted the image-GC threshold change that was tried first and measurably did nothing
  • #502 — the 20% override this issue is about
Parked for the bare-metal migration. Nothing to do until then — filing it so the decision surfaces at planning time rather than being discovered in a manifest comment afterwards. ## What was done, and why it is temporary `mon_data_avail_warn` is set to **20%** in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` (PR #502), down from Ceph's 30% default. That default assumes the mon owns its filesystem. On this cluster the mons live on `dataDirHostPath: /var/lib/rook` — the node's `/var`, shared with ~29 GB of containerd image and snapshot storage — while the mon DB itself is **~106 MB**. The check therefore measures container churn rather than Ceph state, and k8s-2 trips MON_DISK_LOW whenever its image cache fills. The override is a workaround for that mismatch, not a fix for it. ## Why the real fix is blocked today Giving mons a dedicated volume via `mon.volumeClaimTemplate` is the correct answer, and it is **not possible on the current cluster** for two independent reasons: 1. **Every StorageClass here is Ceph-backed** — `ceph-block`, `ceph-filesystem`, `ceph-bucket`, all `rook-ceph.*.csi.ceph.com`. Mons on a Ceph-backed PVC are circular: the mons *are* the quorum authority, so on a cold start there is no Ceph to serve the volume the mons need in order to boot. A node-local provisioner would have to exist first. 2. **Rook does not migrate existing mons.** Upstream: `volumeClaimTemplate` "only applies to new monitors that are created when the requested number of monitors increases, or when a monitor fails and is recreated." Even with a suitable class, converting means deliberately failing and recreating mons one at a time. ## What to consider at migration time - Give the mons a **dedicated, node-local volume** — separate from the container runtime's filesystem, and not backed by Ceph itself. - Once they have one, **drop the `mon_data_avail_warn: "20"` override** rather than carrying it over. On a dedicated volume the 30% default is meaningful again, and keeping the override would hide a real mon-disk problem. - Size it against reality: the mon DB is ~106 MB today. It grows with osdmap churn, so it does not need to be large — it needs to be *separate*. - If bare metal keeps the container runtime and mons on one filesystem, this whole problem comes with it and the override stays needed. ## Related - #434 — where the MON_DISK_LOW investigation happened - #500 — added the Ceph toolbox (which is how the 106 MB and live threshold were confirmed) and the alerts - #501 — reverted the image-GC threshold change that was tried first and measurably did nothing - #502 — the 20% override this issue is about
claude-bot commented 2026-08-27 05:21:11 +00:00
Author
Member
Copy Link
Copy Source

Superseded by the bare-metal migration.

The issue rested on the mons sharing a small /var with containerd, where the 30% mon_data_avail_warn default measured container churn rather than Ceph — a 106 MB mon database against ~60 GiB of shared filesystem. Giving the mons their own volume was the way out, and it was blocked because every StorageClass is Ceph-backed, so a mon PVC would be circular.

On bare metal that premise is gone. /var is now 931 GiB per node at 7-8% used, so 30% free is a meaningful reserve again and the mon database is irrelevant against it. The mons do not need their own volume for this reason any more.

What remains is smaller and is not this issue: the mon_data_avail_warn: "20" override in cephcluster.yaml is now unnecessary and could go back to the Ceph default, at which point the comment above it — which still describes the VM-era 60 GiB /var — should go with it. Worth a separate change rather than carrying this issue for it.

Superseded by the bare-metal migration. The issue rested on the mons sharing a small /var with containerd, where the 30% `mon_data_avail_warn` default measured container churn rather than Ceph — a 106 MB mon database against ~60 GiB of shared filesystem. Giving the mons their own volume was the way out, and it was blocked because every StorageClass is Ceph-backed, so a mon PVC would be circular. On bare metal that premise is gone. /var is now **931 GiB per node at 7-8% used**, so 30% free is a meaningful reserve again and the mon database is irrelevant against it. The mons do not need their own volume for this reason any more. What remains is smaller and is not this issue: the `mon_data_avail_warn: "20"` override in cephcluster.yaml is now unnecessary and could go back to the Ceph default, at which point the comment above it — which still describes the VM-era 60 GiB /var — should go with it. Worth a separate change rather than carrying this issue for it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#503