fix: [alert] CephOSDNearfullApproaching (cluster) (#1089) #1090

2026-10-01T09:44:40Z

issue-fix-bot commented

2026-10-01 09:44:40 +00:00

⚠️ Urgent manual action first (not fixed by this PR)

Today's restore-test Job velero/restore-test-29847360 is still running. When last checked it was on gitea (started 09:34:58Z). Next come home-assistant and then immich. Today's immich backup (immich-daily-20261001051507) contains both the orphaned immich-library (replicated, ~194 GiB) and immich-library-bulk (EC 2+1, ~184.5 GiB). The restore-test restores every PVC in that backup in a single Restore. The projection below puts that beyond the cluster's free raw capacity. Ceph would cross nearfull (0.85) and then hit full (0.95), which blocks writes cluster-wide.

Recommended, before the immich iteration starts:

kubectl -n velero logs job/restore-test-29847360 --tail=5. Check which service it is on.
If it has not finished immich yet, run kubectl -n velero delete job restore-test-29847360. If the immich-daily-rtest-* Restore already exists, also run kubectl -n velero delete restore.velero.io immich-daily-rtest-1790841606 and kubectl delete ns immich-restore-test.
Clean up stranded Released PVs (*-restore-test/*, velero/*-rtest-*). Either let next month's run do it (the start-of-run sweep in restore-test.sh handles this), or patch them to Delete and remove them by hand.
Re-run the restore-test after this PR has merged and a new immich backup has been taken without immich-library (the next daily run at ~05:15Z): kubectl -n velero create job --from=cronjob/restore-test restore-test-manual.

What this PR does

CephOSDNearfullApproaching fired on all three OSDs during the monthly Velero restore-test (0 8 1 * *). The issue correctly identified the trigger: the audiobookshelf iteration staged ~81 GiB on cephfs-bulk. It also correctly flagged the high baseline. That iteration has now finished and been cleaned up, and utilisation is back to ~74.7%. But the underlying risk is larger than the issue projected. The old pre-migration immich/immich-library PVC (ceph-filesystem, replicated ×3) is not mounted by any pod, and its contents have been frozen since the 2026-09-25 move to immich-library-bulk. Even so, it is still captured by the immich-daily Velero schedule every day: the schedule selects all non-CNPG PVCs in the namespace. So each monthly restore-test stages a ~194 GiB ×3 copy of dead data alongside the ~184.5 GiB bulk library. This PR labels that PVC velero.io/exclude-from-backup: "true". That is the same mechanism already used by paperless-redis and the music-assistant PVC. Future backups, and therefore future restore-tests, stop carrying it. Nothing is deleted.

Changes

kubernetes/apps/immich/pvcs.yaml — add the velero.io/exclude-from-backup: "true" label to the immich-library PVC, with a comment explaining why.

Verification

All observed live this run:

kubectl get --raw .../prometheus/api/v1/query?query=ceph_osd_stat_bytes_used/ceph_osd_stat_bytes returned osd.0 0.7471, osd.2 0.7467, osd.3 0.7475. The audiobookshelf staging has been released, and the alert should resolve on its own for now.
kubectl logs -n velero job/restore-test-29847360 showed run_id=1790841606 and 13 schedules in order apps audiobookshelf authentik gitea home-assistant immich .... Its latest line was [09:34:58] gitea: using backup gitea-daily-20261001074507, so immich is still to come. That settles the issue's open question. The log also shows audiobookshelf: PVC audiobookshelf-books-bulk file count parity ... FAIL, which is unrelated to this alert but worth a look.
kubectl get pods -n immich (volumes) showed immich-server mounting only immich-library-bulk immich-thumbs, and immich-machine-learning only immich-ml-cache. No pod mounts immich-library.
kubectl get pvc -n immich showed immich-library Bound to pvc-6c6848b0-… (300Gi, ceph-filesystem, 145d old). immich-library-bulk and immich-thumbs are 6d1h old.
kubectl get schedule.velero.io -n velero immich-daily showed includedResources: [persistentvolumeclaims] and labelSelector: cnpg.io/cluster DoesNotExist, with snapshotMoveData: true.
kubectl get dataupload -n velero (immich namespace): immich-library totalBytes is 208020538919 in every backup from 2026-09-25 to 2026-10-01 (frozen). Before that it grew daily. immich-library-bulk was 198086940178 today, and immich-thumbs 11900377349.
kubectl get pvc -n paperless paperless-redis showed velero.io/exclude-from-backup: "true" live, so Flux applies this label mechanism here.

Projection (arithmetic, not observed). Raw total is 2683 GiB, a figure from the issue that I did not re-measure. The immich iteration would add about:

immich-library: 193.7 GiB × 3 ≈ 581 GiB raw
immich-library-bulk: 184.5 GiB × 1.5 ≈ 277 GiB raw
immich-thumbs and immich-ml-cache: ≈ 36 GiB raw

That totals ≈ +33 pp, against ~25 pp free. Without the orphan it is ≈ +11.5 pp, peaking around 86%. That is still tight, see Notes.

After merge and Flux reconcile:

kubectl get pvc -n immich immich-library -o jsonpath='{.metadata.labels}' should include the label.
After the next immich-daily backup, kubectl get dataupload -n velero should show no immich-library entry for that backup, only -bulk, thumbs and ml-cache.

Notes

This does not fix today's run. The latest backup already contains immich-library, so the manual step above is required.
The orphan is not reclaimed. The PVC still holds ~194 GiB × 3 ≈ 580 GiB raw (~22 pp of the cluster), which explains most of the 74% baseline. Deleting it was not done here, because it irreversibly destroys data and the issue has no owner guidance authorising it. To reclaim the space, the owner should:
1. Confirm immich-library-bulk is a complete copy. The sizes are consistent with that: bulk 184.5 GiB + thumbs 11.1 GiB ≈ the old library's 193.7 GiB.
2. Remove the immich-library document from pvcs.yaml.
3. Run kubectl delete pvc -n immich immich-library, then patch PV pvc-6c6848b0-884c-418b-8dcd-781c49d11eeb to persistentVolumeReclaimPolicy: Delete and delete it. Both steps are needed because of prune: disabled and the Retain StorageClass.
Existing backups keep a copy until their 720h TTL expires (~2026-10-31).
Headroom is tight even after this fix. Restoring immich with only the bulk library still peaks around 86%, above the 0.85 nearfull ratio, if the baseline stays at ~74.7%. Deleting the orphan (−22 pp baseline) is what restores real headroom. A pre-flight capacity check in restore-test.sh, as the issue suggested, is a reasonable follow-up, but it is a larger change than this PR.
The alert rule is behaving correctly and was not touched.
The issue's statement that cephfs-data0 holds 234 GiB is consistent with the orphan being ~194 GiB of it. That pool figure itself was not re-measured this run, and ceph fs subvolume info / ceph df need rook-ceph-tools exec, which the CI ServiceAccount cannot do.

Closes #1089

## ⚠️ Urgent manual action first (not fixed by this PR) Today's restore-test Job `velero/restore-test-29847360` is still running. When last checked it was on `gitea` (started 09:34:58Z). Next come `home-assistant` and then **`immich`**. Today's immich backup (`immich-daily-20261001051507`) contains **both** the orphaned `immich-library` (replicated, ~194 GiB) and `immich-library-bulk` (EC 2+1, ~184.5 GiB). The restore-test restores every PVC in that backup in a single Restore. The projection below puts that **beyond the cluster's free raw capacity**. Ceph would cross nearfull (0.85) and then hit full (0.95), which blocks writes cluster-wide. **Recommended, before the immich iteration starts:** 1. `kubectl -n velero logs job/restore-test-29847360 --tail=5`. Check which service it is on. 2. If it has not finished `immich` yet, run `kubectl -n velero delete job restore-test-29847360`. If the `immich-daily-rtest-*` Restore already exists, also run `kubectl -n velero delete restore.velero.io immich-daily-rtest-1790841606` and `kubectl delete ns immich-restore-test`. 3. Clean up stranded Released PVs (`*-restore-test/*`, `velero/*-rtest-*`). Either let next month's run do it (the start-of-run sweep in `restore-test.sh` handles this), or patch them to `Delete` and remove them by hand. 4. Re-run the restore-test after this PR has merged **and** a new immich backup has been taken without `immich-library` (the next daily run at ~05:15Z): `kubectl -n velero create job --from=cronjob/restore-test restore-test-manual`. ## What this PR does `CephOSDNearfullApproaching` fired on all three OSDs during the monthly Velero restore-test (`0 8 1 * *`). The issue correctly identified the trigger: the audiobookshelf iteration staged ~81 GiB on `cephfs-bulk`. It also correctly flagged the high baseline. That iteration has now finished and been cleaned up, and utilisation is back to ~74.7%. But the underlying risk is larger than the issue projected. The old pre-migration `immich/immich-library` PVC (`ceph-filesystem`, replicated ×3) is not mounted by any pod, and its contents have been frozen since the 2026-09-25 move to `immich-library-bulk`. Even so, it is **still captured by the `immich-daily` Velero schedule every day**: the schedule selects all non-CNPG PVCs in the namespace. So each monthly restore-test stages a ~194 GiB ×3 copy of dead data alongside the ~184.5 GiB bulk library. This PR labels that PVC `velero.io/exclude-from-backup: "true"`. That is the same mechanism already used by `paperless-redis` and the music-assistant PVC. Future backups, and therefore future restore-tests, stop carrying it. Nothing is deleted. ## Changes - `kubernetes/apps/immich/pvcs.yaml` — add the `velero.io/exclude-from-backup: "true"` label to the `immich-library` PVC, with a comment explaining why. ## Verification All observed live this run: - `kubectl get --raw .../prometheus/api/v1/query?query=ceph_osd_stat_bytes_used/ceph_osd_stat_bytes` returned osd.0 0.7471, osd.2 0.7467, osd.3 0.7475. The audiobookshelf staging has been released, and the alert should resolve on its own for now. - `kubectl logs -n velero job/restore-test-29847360` showed `run_id=1790841606` and 13 schedules in order `apps audiobookshelf authentik gitea home-assistant immich ...`. Its latest line was `[09:34:58] gitea: using backup gitea-daily-20261001074507`, so **immich is still to come**. That settles the issue's open question. The log also shows `audiobookshelf: PVC audiobookshelf-books-bulk file count parity ... FAIL`, which is unrelated to this alert but worth a look. - `kubectl get pods -n immich` (volumes) showed `immich-server` mounting only `immich-library-bulk immich-thumbs`, and `immich-machine-learning` only `immich-ml-cache`. **No pod mounts `immich-library`.** - `kubectl get pvc -n immich` showed `immich-library` Bound to `pvc-6c6848b0-…` (300Gi, `ceph-filesystem`, 145d old). `immich-library-bulk` and `immich-thumbs` are 6d1h old. - `kubectl get schedule.velero.io -n velero immich-daily` showed `includedResources: [persistentvolumeclaims]` and `labelSelector: cnpg.io/cluster DoesNotExist`, with `snapshotMoveData: true`. - `kubectl get dataupload -n velero` (immich namespace): `immich-library` totalBytes is **208020538919 in every backup from 2026-09-25 to 2026-10-01** (frozen). Before that it grew daily. `immich-library-bulk` was 198086940178 today, and `immich-thumbs` 11900377349. - `kubectl get pvc -n paperless paperless-redis` showed `velero.io/exclude-from-backup: "true"` live, so Flux applies this label mechanism here. **Projection (arithmetic, not observed).** Raw total is 2683 GiB, a figure from the issue that I did not re-measure. The immich iteration would add about: - `immich-library`: 193.7 GiB × 3 ≈ 581 GiB raw - `immich-library-bulk`: 184.5 GiB × 1.5 ≈ 277 GiB raw - `immich-thumbs` and `immich-ml-cache`: ≈ 36 GiB raw That totals ≈ **+33 pp**, against ~25 pp free. Without the orphan it is ≈ +11.5 pp, peaking around 86%. That is still tight, see Notes. **After merge and Flux reconcile:** - `kubectl get pvc -n immich immich-library -o jsonpath='{.metadata.labels}'` should include the label. - After the next `immich-daily` backup, `kubectl get dataupload -n velero` should show no `immich-library` entry for that backup, only `-bulk`, `thumbs` and `ml-cache`. ## Notes - **This does not fix today's run.** The latest backup already contains `immich-library`, so the manual step above is required. - **The orphan is not reclaimed.** The PVC still holds ~194 GiB × 3 ≈ 580 GiB raw (~22 pp of the cluster), which explains most of the 74% baseline. Deleting it was not done here, because it irreversibly destroys data and the issue has no owner guidance authorising it. To reclaim the space, the owner should: 1. Confirm `immich-library-bulk` is a complete copy. The sizes are consistent with that: bulk 184.5 GiB + thumbs 11.1 GiB ≈ the old library's 193.7 GiB. 2. Remove the `immich-library` document from `pvcs.yaml`. 3. Run `kubectl delete pvc -n immich immich-library`, then patch PV `pvc-6c6848b0-884c-418b-8dcd-781c49d11eeb` to `persistentVolumeReclaimPolicy: Delete` and delete it. Both steps are needed because of `prune: disabled` and the Retain StorageClass. Existing backups keep a copy until their 720h TTL expires (~2026-10-31). - **Headroom is tight even after this fix.** Restoring immich with only the bulk library still peaks around 86%, above the 0.85 nearfull ratio, if the baseline stays at ~74.7%. Deleting the orphan (−22 pp baseline) is what restores real headroom. A pre-flight capacity check in `restore-test.sh`, as the issue suggested, is a reasonable follow-up, but it is a larger change than this PR. - The alert rule is behaving correctly and was not touched. - The issue's statement that `cephfs-data0` holds 234 GiB is consistent with the orphan being ~194 GiB of it. That pool figure itself was not re-measured this run, and `ceph fs subvolume info` / `ceph df` need `rook-ceph-tools` exec, which the CI ServiceAccount cannot do. Closes #1089

issue-fix-bot added 1 commit 2026-10-01 09:44:41 +00:00

fix: [alert] CephOSDNearfullApproaching (cluster) (#1089 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline failed

Details

41f0ebb6ce

issue-fix-bot referenced this pull request

2026-10-01 09:44:41 +00:00

[alert] CephOSDNearfullApproaching (cluster) #1089