Compare commits

..
Author SHA1 Message Date
issue-fix-bot f6dba0eefc fix: [alert] CephMonDiskLow (monitoring) (#569)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
2026-08-25 16:44:20 +00:00
k.renner 63fc1ce63e Merge pull request 'chore(immich): take server and machine-learning down for the jupiter window' (#570) from immich-window-scaledown into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#570
2026-08-25 15:36:48 +00:00
@@ -93,5 +93,5 @@ spec:
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."
summary: "{{ $labels.instance }} /var below 30% free — early warning above Ceph's mon threshold"
description: "/var on {{ $labels.instance }} is under 30% free. This is an early-warning margin, not Ceph's threshold: rook-ceph-cluster/cephcluster.yaml overrides mon_data_avail_warn to 20, so the mon on this node reports MON_DISK_LOW only below 20% free. Check whether it actually has, with `ceph_health_detail{name=\"MON_DISK_LOW\"} == 1` in Prometheus, before treating this as a Ceph problem. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."