konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 1 Wiki Activity

[alert] CephMonDiskLow (monitoring) #569

New Issue
Closed
opened 2026-08-25 15:35:23 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-25 15:35:23 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: CephMonDiskLow, severity warning
  • Target: instance=10.2.40.168:9100, mountpoint=/var, device=/dev/vda5 — this is node k8s-3 (namespace=monitoring is only the node-exporter's namespace, not the affected workload)
  • Since: 2026-08-25T15:25:20Z. The rule uses for: 30m, so the condition has been true since ~14:55Z.
  • Still firing: yes.

Evidence

./.ci-diag/kubectl get --raw → Prometheus query on the rule expression, all three nodes:

node /var free
k8s-3 (10.2.40.168) 19.76 %
k8s-1 (10.2.40.174) 41.75 %
k8s-2 (10.2.40.136) 42.85 %

/var is 59.9 GiB on all three (node_filesystem_size_bytes = 64347963392), so k8s-3 has 11.79 GiB free.

This is past the real Ceph threshold, not just the alert's margin. Prometheus ceph_health_detail == 1 currently returns MON_DISK_LOW as an active HEALTH_WARN. kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:58 sets mon_data_avail_warn: "20", and 19.76 % is below it. (PG_DEGRADED and TOO_FEW_OSDS are also active — those belong to the k8s-1 OSD removal described below, not to this alert. AUTH_INSECURE_* is excluded by rule design, #527.)

Node events for k8s-3 (kubectl get events --field-selector involvedObject.name=k8s-3):

36m  Warning  EvictionThresholdMet   Attempting to reclaim ephemeral-storage
36m  Normal   NodeHasDiskPressure    Node k8s-3 status is now: NodeHasDiskPressure
84s  Warning  ImageGCFailed          ... (81% of 59.9 GiB used). Failed to free
                                     sufficient space by deleting unused images (freed 0 bytes).

Image GC ran and reclaimed 0 bytes — exactly the behaviour already documented in kubernetes/talos/patches/cluster.yaml:34-42 and #434: the bulk of /var/lib/containerd is overlayfs snapshots held by running containers, which GC cannot reclaim. Raising the GC thresholds will not help.

Root cause (verified by correlation, one step unverified)

query_range on node_filesystem_avail_bytes{mountpoint="/var"} for k8s-3 shows free space flat at 20.3–20.5 GiB (~34 %) for the previous three days, then two step drops today:

window (UTC) free space
until ~14:53 20.30 GiB
14:53 → 14:58 20.30 → 16.07 GiB
15:08 → 15:13 16.85 → 11.92 GiB
now 11.79 GiB

Both steps line up with waves of pods being (re)scheduled onto k8s-3 — from kubectl get pods --field-selector spec.nodeName=k8s-3 --sort-by=.metadata.creationTimestamp:

  • age ~37m (≈14:55): source-controller, helm-controller, kube-prometheus-stack-grafana, rook-ceph-operator, rook-ceph-mgr-a, home-assistant, zigbee2mqtt, openwakeword, snapshot-controller, woodpecker-agent
  • age ~23m (≈15:09): immich-server, authentik-server, ak-pg-2, matter-server, crowdsec-agent

Why they moved: the FailedScheduling events in the pipeline snapshot repeatedly say 1 node(s) were unschedulable, and the rook-ceph pods pinned to k8s-1 (rook-ceph-crashcollector-k8s-1, rook-ceph-exporter-k8s-1) could not place. So k8s-1 was cordoned. It was a cordon, not a reboot: time() - node_boot_time_seconds is ~2.86e6 s (~33 days) for all three nodes. That is consistent with the in-flight maintenance documented in cephcluster.yaml ("k8s-1 is absent while jupiter's OSD drive is swapped and the node is converted to bare metal").

k8s-1 is already back: spec.unschedulable empty and no taints on any node, rook-ceph-mon-c is Running on k8s-1 again (age 24m), and there are 0 Pending pods cluster-wide. But the evicted workloads stayed on k8s-3, and so did the ~8.5 GiB of image layers unpacked to serve them.

Unverified (hypothesis): that the 8.5 GiB is specifically containerd image/snapshot growth rather than, say, log or /var/lib/rook growth. nodes/proxy is denied to the CI ServiceAccount and there is no talosctl here. Settle it with the command the alert itself suggests:
talosctl -n 10.2.40.168 usage /var/lib --depth 1, then talosctl -n 10.2.40.168 usage /var/lib/containerd --depth 1.

Blast radius

  • rook-ceph-mon-a runs on k8s-3. All three mons are Running and quorum is intact, so there is no immediate availability risk.
  • mon_data_avail_crit is left at the 5 % default (~3.0 GiB), at which the mon stops. Current 20-minute trend is −271 MiB/h (deriv(...[20m])), i.e. roughly 33 hours of headroom if that rate holds — an extrapolation, not a prediction.
  • MON_DISK_LOW staying active for 6h will additionally trip CephHealthWarn.

Where to fix it

Primary action is on the cluster, not in the repo. k8s-1 is uncordoned and has 41.75 % free /var. Rebalancing the pods that landed on k8s-3 back onto k8s-1 both relieves the pressure and makes the snapshots reclaimable — image GC can only free layers once the holding containers stop. Deleting the k8s-3-resident replicas listed above (they will reschedule onto the now-available node) is the low-risk version; verify with the query_range above that free space recovers toward the ~34 % baseline.

Repo files that bear on this:

  • kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml — mon_data_avail_warn: "20" (line 58) and the block explaining why k8s-1 is out of the OSD list. No change proposed; this is the context that makes the drain expected.
  • kubernetes/talos/patches/cluster.yaml:43-45 — imageGCHighThresholdPercent: 70 / Low: 60. Do not tune these in response to this alert; the file already records that dropping high to 60 moved free space by 0.02 percentage points.
  • kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:97 — documentation defect worth a small PR. The description states that 30 % "is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW". That is stale: cephcluster.yaml overrides mon_data_avail_warn to 20. The 30 % rule is now an early-warning margin above the cluster's actual threshold, and the wording made this alert read as a prediction when MON_DISK_LOW was in fact already firing. Reword the description to say so.

Not fixable in the repo: /var is 59.9 GiB on all three nodes. If a single cordoned node is expected to be routine during the bare-metal conversion, the durable fix is more /var headroom (vda5 lives in the VM definitions under opentofu/) or giving the mons their own volume — which cephcluster.yaml:52-57 already flags as a bare-metal-migration decision and explicitly cannot be done on the current cluster. That is a human capacity call, not a manifest tweak.

Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `CephMonDiskLow`, severity `warning` - **Target:** `instance=10.2.40.168:9100`, `mountpoint=/var`, `device=/dev/vda5` — this is node **k8s-3** (`namespace=monitoring` is only the node-exporter's namespace, not the affected workload) - **Since:** `2026-08-25T15:25:20Z`. The rule uses `for: 30m`, so the condition has been true since ~14:55Z. - **Still firing:** yes. ## Evidence `./.ci-diag/kubectl get --raw` → Prometheus `query` on the rule expression, all three nodes: | node | /var free | |---|---| | k8s-3 (10.2.40.168) | **19.76 %** | | k8s-1 (10.2.40.174) | 41.75 % | | k8s-2 (10.2.40.136) | 42.85 % | `/var` is 59.9 GiB on all three (`node_filesystem_size_bytes` = 64347963392), so k8s-3 has **11.79 GiB free**. **This is past the real Ceph threshold, not just the alert's margin.** Prometheus `ceph_health_detail == 1` currently returns `MON_DISK_LOW` as an active `HEALTH_WARN`. `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:58` sets `mon_data_avail_warn: "20"`, and 19.76 % is below it. (`PG_DEGRADED` and `TOO_FEW_OSDS` are also active — those belong to the k8s-1 OSD removal described below, not to this alert. `AUTH_INSECURE_*` is excluded by rule design, #527.) Node events for k8s-3 (`kubectl get events --field-selector involvedObject.name=k8s-3`): ``` 36m Warning EvictionThresholdMet Attempting to reclaim ephemeral-storage 36m Normal NodeHasDiskPressure Node k8s-3 status is now: NodeHasDiskPressure 84s Warning ImageGCFailed ... (81% of 59.9 GiB used). Failed to free sufficient space by deleting unused images (freed 0 bytes). ``` Image GC ran and reclaimed **0 bytes** — exactly the behaviour already documented in `kubernetes/talos/patches/cluster.yaml:34-42` and `#434`: the bulk of `/var/lib/containerd` is overlayfs snapshots held by *running* containers, which GC cannot reclaim. Raising the GC thresholds will not help. ## Root cause (verified by correlation, one step unverified) `query_range` on `node_filesystem_avail_bytes{mountpoint="/var"}` for k8s-3 shows free space flat at **20.3–20.5 GiB (~34 %) for the previous three days**, then two step drops today: | window (UTC) | free space | |---|---| | until ~14:53 | 20.30 GiB | | 14:53 → 14:58 | 20.30 → **16.07 GiB** | | 15:08 → 15:13 | 16.85 → **11.92 GiB** | | now | 11.79 GiB | Both steps line up with waves of pods being (re)scheduled onto k8s-3 — from `kubectl get pods --field-selector spec.nodeName=k8s-3 --sort-by=.metadata.creationTimestamp`: - **age ~37m (≈14:55)**: `source-controller`, `helm-controller`, `kube-prometheus-stack-grafana`, `rook-ceph-operator`, `rook-ceph-mgr-a`, `home-assistant`, `zigbee2mqtt`, `openwakeword`, `snapshot-controller`, `woodpecker-agent` - **age ~23m (≈15:09)**: `immich-server`, `authentik-server`, `ak-pg-2`, `matter-server`, `crowdsec-agent` Why they moved: the `FailedScheduling` events in the pipeline snapshot repeatedly say `1 node(s) were unschedulable`, and the rook-ceph pods pinned to k8s-1 (`rook-ceph-crashcollector-k8s-1`, `rook-ceph-exporter-k8s-1`) could not place. So **k8s-1 was cordoned**. It was a cordon, not a reboot: `time() - node_boot_time_seconds` is ~2.86e6 s (~33 days) for all three nodes. That is consistent with the in-flight maintenance documented in `cephcluster.yaml` ("k8s-1 is absent while jupiter's OSD drive is swapped and the node is converted to bare metal"). **k8s-1 is already back**: `spec.unschedulable` empty and no taints on any node, `rook-ceph-mon-c` is Running on k8s-1 again (age 24m), and there are **0 Pending pods** cluster-wide. But the evicted workloads stayed on k8s-3, and so did the ~8.5 GiB of image layers unpacked to serve them. *Unverified (hypothesis):* that the 8.5 GiB is specifically containerd image/snapshot growth rather than, say, log or `/var/lib/rook` growth. `nodes/proxy` is denied to the CI ServiceAccount and there is no talosctl here. Settle it with the command the alert itself suggests: `talosctl -n 10.2.40.168 usage /var/lib --depth 1`, then `talosctl -n 10.2.40.168 usage /var/lib/containerd --depth 1`. ## Blast radius - `rook-ceph-mon-a` runs on k8s-3. All three mons are Running and quorum is intact, so there is no immediate availability risk. - `mon_data_avail_crit` is left at the 5 % default (~3.0 GiB), at which the mon stops. Current 20-minute trend is **−271 MiB/h** (`deriv(...[20m])`), i.e. roughly **33 hours** of headroom *if that rate holds* — an extrapolation, not a prediction. - `MON_DISK_LOW` staying active for 6h will additionally trip `CephHealthWarn`. ## Where to fix it **Primary action is on the cluster, not in the repo.** k8s-1 is uncordoned and has 41.75 % free /var. Rebalancing the pods that landed on k8s-3 back onto k8s-1 both relieves the pressure and makes the snapshots reclaimable — image GC can only free layers once the holding containers stop. Deleting the k8s-3-resident replicas listed above (they will reschedule onto the now-available node) is the low-risk version; verify with the `query_range` above that free space recovers toward the ~34 % baseline. Repo files that bear on this: - `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` — `mon_data_avail_warn: "20"` (line 58) and the block explaining why k8s-1 is out of the OSD list. No change proposed; this is the context that makes the drain expected. - `kubernetes/talos/patches/cluster.yaml:43-45` — `imageGCHighThresholdPercent: 70` / `Low: 60`. **Do not tune these** in response to this alert; the file already records that dropping high to 60 moved free space by 0.02 percentage points. - `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:97` — **documentation defect worth a small PR.** The description states that 30 % "is Ceph's `mon_data_avail_warn` threshold, so the mon on this node will report MON_DISK_LOW". That is stale: `cephcluster.yaml` overrides `mon_data_avail_warn` to `20`. The 30 % rule is now an *early-warning margin above* the cluster's actual threshold, and the wording made this alert read as a prediction when MON_DISK_LOW was in fact already firing. Reword the description to say so. **Not fixable in the repo:** `/var` is 59.9 GiB on all three nodes. If a single cordoned node is expected to be routine during the bare-metal conversion, the durable fix is more `/var` headroom (vda5 lives in the VM definitions under `opentofu/`) or giving the mons their own volume — which `cephcluster.yaml:52-57` already flags as a bare-metal-migration decision and explicitly cannot be done on the current cluster. That is a human capacity call, not a manifest tweak. --- <!-- alert-monitor:fingerprint=d512515ebe4c8b8b --> _Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-25 16:44:22 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#572 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/572 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#569