chore: bring immich back up, keep cephfs-metadata degraded until the end #585

2026-08-25T19:53:57Z

claude-bot commented

2026-08-25 19:53:57 +00:00

jupiter is converted, Ready, and its OSD is rebuilt on the PM983 — so the capacity constraint that took immich down is gone with all three nodes back.

immich back to 1 replica. Both Deployments. The CNPG cluster was left running throughout, so this is the whole recovery.

cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be degraded again for saturn and then neptune, and each transition re-replicates the pool — toggling it per node buys nothing but churn. Its comment said to restore once k8s-1 was back, which is now actively misleading about the plan, so it points at the end of all three conversions instead.

The trade while it sits there: two copies with min_size 1 rather than three with min_size 2. A further OSD loss degrades rather than blocking IO on every CephFS mount, which is the right side to err on during maintenance — but it is less protected than steady state, and that is the reason to finish the remaining conversions rather than leave it indefinitely.

The immich comments lose their jupiter-specific framing; taking both Deployments down is now a pre-flight step for any single-node maintenance, carried by RUNBOOK-baremetal-conversion.md (#584).

Not in this PR

osd_mclock_profile is still high_recovery_ops from speeding the evacuation; back to balanced once backfill finishes (live setting, not manifest).
.mgr was set to size 2 / min_size 1 live during the window — also not manifest-managed, restore separately.

🤖 Generated with Claude Code

jupiter is converted, Ready, and its OSD is rebuilt on the PM983 — so the capacity constraint that took immich down is gone with all three nodes back. **immich back to 1 replica.** Both Deployments. The CNPG cluster was left running throughout, so this is the whole recovery. **`cephfs-metadata` deliberately stays at size 2 / min_size 1.** It has to be degraded again for saturn and then neptune, and each transition re-replicates the pool — toggling it per node buys nothing but churn. Its comment said to restore once k8s-1 was back, which is now actively misleading about the plan, so it points at the end of all three conversions instead. The trade while it sits there: two copies with min_size 1 rather than three with min_size 2. A further OSD loss degrades rather than blocking IO on every CephFS mount, which is the right side to err on during maintenance — but it is *less* protected than steady state, and that is the reason to finish the remaining conversions rather than leave it indefinitely. The immich comments lose their jupiter-specific framing; taking both Deployments down is now a pre-flight step for any single-node maintenance, carried by `RUNBOOK-baremetal-conversion.md` (#584). ## Not in this PR - `osd_mclock_profile` is still `high_recovery_ops` from speeding the evacuation; back to `balanced` once backfill finishes (live setting, not manifest). - `.mgr` was set to size 2 / min_size 1 live during the window — also not manifest-managed, restore separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-25 19:53:58 +00:00

chore: bring immich back up, keep cephfs-metadata degraded until the end

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

5b47dc3271

jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.

cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.

The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-25 19:55:52 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 5b47dc3271ba38614585f2a5f355f6b0a5529a7e.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR restores the immich deployments (machine-learning and server) from 0 to 1 replica now that Jupiter is recovered, and updates Ceph pool comments to clarify that cephfs-metadata will remain deliberately degraded (size 2 / min_size 1) through the complete maintenance window of all three node conversions rather than just until Jupiter rejoins.

🔴 Blocking

None — no blocking issues found.

🟡 Suggestions

Forward reference to unmerged PR
The immich and Ceph comments now reference RUNBOOK-baremetal-conversion.md, which does not yet exist in the repository. The PR body mentions this runbook is provided by PR #584. If these references are intended as sequential PRs (this one after #584), that's appropriate; if not, these references should be confirmed to exist before merge or the comments should defer the reference until the runbook lands.

Clarity on pool degradation window
The cephfs-metadata comment is now more accurate about the maintenance scope (all three conversions), but the phrase "size 2 / min_size 1" could be clearer about whether this is the current state or the target state. However, reading it in context with the storageclass.yaml spec (which shows size: 2 and min_size: "1"), it's correct — this is the current state that will persist.

🟢 Nits

None.

Tests

This is infrastructure-as-code (Kubernetes manifests applied via GitOps). No unit or integration tests are expected. The actual validation is operational: the deployments should scale up cleanly and the Ceph pool should remain healthy with the updated configuration.

Recommendation

Looks reasonable — no blocking issues; the forward reference to the runbook should be confirmed before merge, but this appears to be intended sequential work with PR #584.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `5b47dc3271ba38614585f2a5f355f6b0a5529a7e`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR restores the immich deployments (machine-learning and server) from 0 to 1 replica now that Jupiter is recovered, and updates Ceph pool comments to clarify that `cephfs-metadata` will remain deliberately degraded (size 2 / min_size 1) through the complete maintenance window of all three node conversions rather than just until Jupiter rejoins. ## 🔴 Blocking None — no blocking issues found. ## 🟡 Suggestions **Forward reference to unmerged PR** The immich and Ceph comments now reference `RUNBOOK-baremetal-conversion.md`, which does not yet exist in the repository. The PR body mentions this runbook is provided by PR #584. If these references are intended as sequential PRs (this one after #584), that's appropriate; if not, these references should be confirmed to exist before merge or the comments should defer the reference until the runbook lands. **Clarity on pool degradation window** The cephfs-metadata comment is now more accurate about the maintenance scope (all three conversions), but the phrase "size 2 / min_size 1" could be clearer about whether this is the *current* state or the *target* state. However, reading it in context with the storageclass.yaml spec (which shows `size: 2` and `min_size: "1"`), it's correct — this is the current state that will persist. ## 🟢 Nits None. ## Tests This is infrastructure-as-code (Kubernetes manifests applied via GitOps). No unit or integration tests are expected. The actual validation is operational: the deployments should scale up cleanly and the Ceph pool should remain healthy with the updated configuration. ## Recommendation Looks reasonable — no blocking issues; the forward reference to the runbook should be confirmed before merge, but this appears to be intended sequential work with PR #584.

k.renner merged commit 4a2839b35f into master

2026-08-25 19:56:11 +00:00

k.renner referenced this issue from a commit

2026-08-25 19:56:12 +00:00

Merge pull request 'chore: bring immich back up, keep cephfs-metadata degraded until the end' (#585) from restore-after-jupiter into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#585