The CP VMs are seeing Ceph `MON_DISK_LOW` on mons a,b because their
shared /var (xfs on /dev/vda5) sits at 77-79% used — past Ceph's
default `mon_data_avail_warn` line of <30% free. Mon DB itself is
118 MiB; the disk pressure is pod density (k8s-2 hosts 75 pods vs
k8s-1's 42), driving containerd image cache and /var/log/pods.
Kubelet's default image GC thresholds (85% high / 80% low) sit
above where we land, so GC never fires and unused image layers
accumulate indefinitely. Lowering to 70/60 forces eviction before
/var crosses the mon warn line, with Spegel's P2P pull-through cache
softening the re-pull cost on the rare case a GC'd image is needed
back.
Requires a node reboot to take effect (Talos `extraConfig` change).
Roll one CP at a time with `kubectl drain` + Ceph `HEALTH_OK` wait
between nodes, per the standing drain/Ceph-wait discipline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7