Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 5b47dc3271 chore: bring immich back up, keep cephfs-metadata degraded until the end
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.

cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.

The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 21:53:33 +02:00
3 changed files with 24 additions and 24 deletions
+8 -11
View File
@@ -6,17 +6,14 @@ metadata:
labels:
app.kubernetes.io/name: immich-machine-learning
spec:
# TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED.
#
# Down for the jupiter bare-metal conversion. With one of three nodes out,
# the remaining pair have no room: cluster CPU requests exceed two nodes'
# capacity, and k8s-3 runs ~5.8 GiB above the kubelet's eviction threshold,
# so every pod that does not need to be running is one less thing competing
# for both. This one is the best candidate: 500m reserved against a measured
# 2m idle draw, and inference bursts hard enough that lowering the request
# instead would be the wrong trade. Photo processing pauses; nothing else
# depends on it.
replicas: 0
# Scale this and immich-server to 0 for the duration of any single-node
# maintenance — cluster CPU requests exceed what two nodes can hold, and this
# pair frees ~1000m against a shortfall of roughly 200m. Stopping them is the
# right lever rather than lowering their requests: 500m reserved against a
# 2m idle draw looks like waste, but inference bursts hard and the request is
# honest about what it needs when someone uploads photos.
# See RUNBOOK-baremetal-conversion.md.
replicas: 1
strategy:
type: RollingUpdate
selector:
+4 -5
View File
@@ -36,11 +36,10 @@ metadata:
labels:
app.kubernetes.io/name: immich-server
spec:
# TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED. See the note on
# immich-machine-learning; immich is offline for the jupiter conversion.
# Its Postgres cluster is deliberately left running, so nothing needs
# restoring beyond these two replica counts.
replicas: 0
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
# so recovery is just these two replica counts.
replicas: 1
strategy:
type: Recreate
selector:
@@ -87,7 +87,10 @@ metadata:
name: cephfs
namespace: rook-ceph
spec:
# TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
# TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
# NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
# three conversions rather than being toggled per node: each change
# re-replicates the pool, and the window reopens with the next node anyway.
#
# The standing trade for this pool is size 3: it holds ~253 MiB against the
# data pool's 280 GiB, so it contributes nothing measurable to drive wear —
@@ -96,13 +99,14 @@ spec:
# quarter of a gigabyte to keep that intact is the right trade in steady
# state.
#
# It does not hold while one of three hosts is out for the jupiter bare-metal
# conversion. size 3 on two remaining hosts leaves every PG undersized at two
# copies with min_size 2, so a single further OSD loss drops the pool below
# min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
# two copies are active+clean instead, and a further loss degrades rather
# than blocks — availability bought at the cost of accepting writes onto a
# single copy while degraded.
# It does not hold while one of three hosts is out. size 3 on two remaining
# hosts leaves every PG undersized at two copies with min_size 2, so a single
# further OSD loss drops the pool below min_size and blocks IO on every
# CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
# instead, and a further loss degrades rather than blocks — availability
# bought at the cost of accepting writes onto a single copy while degraded.
#
# See RUNBOOK-baremetal-conversion.md.
metadataPool:
replicated:
size: 2