chore: bring immich back up, keep cephfs-metadata degraded until the end

jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.

cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.

The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/immich/machine-learning.yaml

@@ -6,17 +6,14 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-machine-learning
 spec:
-  # TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED.
-  #
-  # Down for the jupiter bare-metal conversion. With one of three nodes out,
-  # the remaining pair have no room: cluster CPU requests exceed two nodes'
-  # capacity, and k8s-3 runs ~5.8 GiB above the kubelet's eviction threshold,
-  # so every pod that does not need to be running is one less thing competing
-  # for both. This one is the best candidate: 500m reserved against a measured
-  # 2m idle draw, and inference bursts hard enough that lowering the request
-  # instead would be the wrong trade. Photo processing pauses; nothing else
-  # depends on it.
-  replicas: 0
+  # Scale this and immich-server to 0 for the duration of any single-node
+  # maintenance — cluster CPU requests exceed what two nodes can hold, and this
+  # pair frees ~1000m against a shortfall of roughly 200m. Stopping them is the
+  # right lever rather than lowering their requests: 500m reserved against a
+  # 2m idle draw looks like waste, but inference bursts hard and the request is
+  # honest about what it needs when someone uploads photos.
+  # See RUNBOOK-baremetal-conversion.md.
+  replicas: 1
   strategy:
     type: RollingUpdate
   selector:

kubernetes/apps/immich/server.yaml

@@ -36,11 +36,10 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-server
 spec:
-  # TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED. See the note on
-  # immich-machine-learning; immich is offline for the jupiter conversion.
-  # Its Postgres cluster is deliberately left running, so nothing needs
-  # restoring beyond these two replica counts.
-  replicas: 0
+  # Scaled to 0 alongside immich-machine-learning during single-node
+  # maintenance; see the note there. The CNPG Postgres cluster stays running,
+  # so recovery is just these two replica counts.
+  replicas: 1
   strategy:
     type: Recreate
   selector:

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,17 +167,20 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # k8s-1 is absent while jupiter's OSD drive is swapped and the node is
-  # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-  # rejoined; the cluster runs on two OSDs until then.
-  #
   # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see. Bare-metal k8s-1
-  # will see two — the OSD drive in MS-01 slot 1 (PCI 01:00.0) and the Talos
-  # system disk in slot 2 (58:00.0) — so when it is re-added, name its device
-  # deliberately rather than assuming slot 1 keeps enumerating first. The system
-  # disk is separately protected: talos/patches/k8s-1.yaml pins the installer to
-  # it by serial, and Rook skips a disk carrying partitions.
+  # neptune, and is the only block device those guests can see, so a bare device
+  # name is unambiguous there.
+  #
+  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
+  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
+  # enumerates this machine's drives in a different order than Proxmox did, and
+  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
+  # carrying partitions, but that is luck rather than intent. The by-id path is
+  # derived from model and serial and does not depend on enumeration at all.
+  #
+  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
+  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   nvme2n1  S437NX0M400426    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
@@ -188,6 +191,9 @@ spec:
       - name: k8s-3
         devices:
           - name: nvme0n1
+      - name: k8s-1
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -87,7 +87,10 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
+  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
+  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
+  # three conversions rather than being toggled per node: each change
+  # re-replicates the pool, and the window reopens with the next node anyway.
   #
   # The standing trade for this pool is size 3: it holds ~253 MiB against the
   # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
@@ -96,13 +99,14 @@ spec:
   # quarter of a gigabyte to keep that intact is the right trade in steady
   # state.
   #
-  # It does not hold while one of three hosts is out for the jupiter bare-metal
-  # conversion. size 3 on two remaining hosts leaves every PG undersized at two
-  # copies with min_size 2, so a single further OSD loss drops the pool below
-  # min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
-  # two copies are active+clean instead, and a further loss degrades rather
-  # than blocks — availability bought at the cost of accepting writes onto a
-  # single copy while degraded.
+  # It does not hold while one of three hosts is out. size 3 on two remaining
+  # hosts leaves every PG undersized at two copies with min_size 2, so a single
+  # further OSD loss drops the pool below min_size and blocks IO on every
+  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
+  # instead, and a further loss degrades rather than blocks — availability
+  # bought at the cost of accepting writes onto a single copy while degraded.
+  #
+  # See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
       size: 2

kubernetes/talos/patches/k8s-1.yaml

@@ -9,9 +9,6 @@
 # every link in one schema instead of straddling them.
 machine:
   network:
-    # Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
-    # below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
-    # as k8s.renner.uno.
     hostname: k8s-1
   sysctls:
     # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
@@ -26,7 +23,7 @@ machine:
   install:
     # Three NVMe drives are ordinary block devices to Talos here: the Kingston
     # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
-    # Ceph OSD, and the retired Crucial P3 that now carries EPHEMERAL (see the
+    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
     # VolumeConfig below). Every one of them is selected by serial. Nothing in
     # this file may use a broad selector: `disk.transport == 'nvme'` matches all
     # three, and the failure mode is Talos taking the OSD drive.
@@ -51,9 +48,8 @@ cluster:
 #
 # THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; a config applied after EPHEMERAL
-# has been created on the system disk is silently ignored, and relocating it
-# then requires wiping the disk and booting install media again. So this
-# document has to be present in the very first `apply-config --insecure`.
+# has been created elsewhere is silently ignored, and relocating it then
+# requires wiping the disk and booting install media again.
 #
 # Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
 # would match the PM983 as well, and losing that coin flip means Talos formats
@@ -67,42 +63,43 @@ provisioning:
   minSize: 50GiB
   grow: true
 ---
-# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
-# driver selector k8s-2/k8s-3 use for their single passed-through port would
-# match two links here. MAC matching is unambiguous and survives udev renaming —
-# which is not hypothetical: adding the third NVMe renumbered the PCI buses and
-# moved the second I226 from enp89s0 to enp90s0.
-apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: lan0
-selector:
-  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
----
-apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: ceph0
-selector:
-  match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
----
-# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
-# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
-# before, which is why the VM saw two already-untagged virtio NICs instead.
+# X710 port 0 carries the switch trunk. It takes no addressing itself — the
+# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
+# business being on — so management and v30iot both ride it as tagged VLANs.
+#
+# Links are addressed by predictable kernel name. Those derive from PCI
+# topology, not probe order — though not immune: adding the third NVMe
+# renumbered the buses and moved the unused second I226 from enp89s0 to
+# enp90s0. Neither name used here was affected.
+#
+# LinkAliasConfig would be the better way to pin these, and is worth
+# reintroducing — but only once it is proven. Aliasing both X710 ports by
+# permanent MAC and using the aliases as VLAN parents produced a node with no
+# address on any interface: the aliases were never created, so every document
+# referencing them did nothing, including the static address on port 1. Verify
+# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
+# back on its own, before making it permanent.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: lan0
+name: enp2s0f0np0
 up: true
 ---
+# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
+# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
+# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: lan0
+parent: enp2s0f0np0
 up: true
 ---
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
 ---
+# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
+# k8s.renner.uno.
 apiVersion: v1alpha1
 kind: Layer2VIPConfig
 name: 10.2.40.60
@@ -112,21 +109,20 @@ link: mgmt0
 # `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
 # that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
 # serve both VM and bare-metal nodes for as long as the cluster is mixed.
-# Nothing else on this machine is called ens19: the physical NICs are
-# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp90s0.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: lan0
+parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
-# on X710 port 1. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. No default route.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
+# untagged on the wire. Used by Rook for OSD<->OSD replication via
+# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
+# segment, and deliberately has no default route.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph0
+name: enp2s0f1np1
 up: true
 mtu: 9000
 addresses: