feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-infrastructure.yml

@@ -131,18 +131,18 @@ steps:
       - |
         # Ensure Talos image exists on all Proxmox nodes via API.
         #
-        # jupiter is NOT in this list: it runs Talos on bare metal and is no
-        # longer a Proxmox node, so the storage API returns nothing for it and
-        # the download-wait loop below would burn its full 60x10s before
-        # giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
-        # still need the nocloud image; bare-metal nodes boot the metal-amd64
-        # image from their own installed disk instead.
+        # Only nodes still running Proxmox belong in this list. jupiter and
+        # saturn run Talos on bare metal now, so the storage API returns
+        # nothing for them and the download-wait loop below would burn its full
+        # 60x10s per node before giving up. Bare-metal nodes boot the
+        # metal-amd64 image from their own installed disk instead, and the
+        # whole step goes away once neptune is converted too.
         TALOS_VERSION="v1.12.6"
         TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
         TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
         TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
         API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in saturn neptune; do
+        for node in neptune; do
           EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
             -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
           if [ "$EXISTS" -gt 0 ]; then

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -0,0 +1,395 @@
+# Runbook — converting a control-plane node from Proxmox VM to bare metal
+
+Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
+metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
+after doing jupiter on 2026-08-25; saturn and neptune remain.
+
+Everything here that reads like an over-specific warning is something that
+actually happened. The conversion itself is straightforward — the failures were
+all in state that outlived the VM.
+
+## What the node ends up as
+
+| | |
+|---|---|
+| System disk | Kingston OM8PGP41024Q-A0, selected by serial |
+| EPHEMERAL (`/var`) | the retired Crucial P3, selected by serial |
+| Ceph OSD | Samsung PM983, selected by `/dev/disk/by-id` |
+| Management | VLAN 40 tagged on X710 port 0 |
+| v30iot | VLAN 30 on the same port, named `ens19` |
+| Ceph cluster net | X710 port 1, `10.99.0.x/29`, MTU 9000 |
+
+## Before you start
+
+- [ ] All three OSDs up, all mons in quorum, PGs `active+clean`. Do not start
+      from a degraded cluster; the window itself removes one of three.
+- [ ] Cluster fits on two nodes. Check `kubectl describe node` allocation on the
+      two survivors. Take immich down (`kubernetes/apps/immich`, both
+      Deployments to `replicas: 0`) — it frees ~1000m against a shortfall of
+      roughly 200m and takes its image pulls off the remaining nodes.
+- [ ] `cephfs-metadata` to size 2 / min_size 1 for the duration. At size 3 on
+      two hosts every PG is undersized with min_size 2, so one further OSD loss
+      blocks IO on every CephFS mount cluster-wide.
+- [ ] Talos metal ISO on a USB stick. Schematic
+      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
+      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      Verify by comparing partition offsets and sizes against the ISO with
+      `fdisk -l` — blkid labels lie after a rewrite.
+- [ ] Physical access. Secure Boot must be turned off (see below), and that is
+      the one step nothing can do remotely.
+
+## 1. Evacuate the OSD
+
+```sh
+ceph osd out <id>
+```
+
+Wait until no PGs are `backfilling` or `backfill_wait`. Do **not** wait for
+`ceph osd safe-to-destroy` — pools at size 3 keep the OSD in their acting set
+because three copies cannot be placed on two hosts, so it never returns clean.
+
+Then remove it. The node must already be out of `cephcluster.yaml`'s
+`storage.nodes` list, or the operator rebuilds the OSD within one reconcile and
+starts backfilling onto the drive you are about to pull:
+
+```sh
+kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
+ceph osd purge <id> --yes-i-really-mean-it
+kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>
+```
+
+## 2. Drain, and leave the cluster
+
+```sh
+kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
+```
+
+Single-replica workloads with `minAvailable: 1` PDBs will refuse eviction — that
+is deliberate, they exist to surface the pod rather than let it be yanked. On
+this cluster that is `influxdb-influxdb2-0`, `woodpecker-server` and the
+CNPG primary. Move each one deliberately with `kubectl delete pod`; they
+reschedule onto the survivors and their RBD volumes follow.
+
+```sh
+talosctl -n <node> reset --graceful --wipe-mode system-disk
+kubectl delete node <node>
+```
+
+`--wipe-mode system-disk` and not the default `all`: the OSD drive is being
+pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
+etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
+the drain, which the rebuilt node would otherwise inherit.
+
+## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
+
+Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
+host leaves the Proxmox cluster: afterwards the provider cannot reach the node,
+the plan errors, and the only escape is `tofu state rm`, which is off the table.
+
+Point `proxmox_api_url` at a host that will *survive* the conversion, not the one
+being converted.
+
+Manual triggers are currently broken by the Woodpecker v3.18.0 regression
+(issue #582). Until that is resolved, run it locally:
+
+```sh
+AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
+  tofu -chdir=opentofu init -backend-config=<file>
+… plan -target=module.k8s_N -var-file=<file> -out=<plan>
+… apply <plan>
+```
+
+The `AWS_*` environment variables are required *in addition to*
+`-backend-config`, or the S3 backend fails `HeadObject` with 403 even with
+credentials that can read the object fine via the aws CLI.
+
+## 2b. Proxmox quorum — matters from the second conversion onward
+
+Two nodes remain after jupiter, with `expected_votes: 2` and quorum 2. Removing
+one leaves the last node inquorate and `/etc/pve` read-only, which blocks the
+final VM destroy. On the surviving node:
+
+```sh
+pvecm expected 1
+```
+
+This is the official recovery. Proxmox documents it in the storage replication
+chapter (`/usr/share/pve-docs/chapter-pvesr.html`), verbatim:
+
+> If you have no quorum, we strongly advise to fix this first and make the node
+> operable again. Only if this is not possible at the moment, you may use the
+> following command to enforce quorum on the current node: `# pvecm expected 1`
+
+The caveat there is about restoring the missing node instead. Here the missing
+node is deliberately gone for good, so enforcing quorum is the correct branch
+rather than a workaround.
+
+Run it only once the node is actually alone — attempting it while both nodes are
+up fails with `CS_ERR_INVALID_PARAM`, since corosync will not set expected votes
+below the members it can currently see. Which is exactly why it becomes valid
+when one remains.
+
+Runtime only; it does not survive a reboot, so re-run it if the survivor
+restarts before its own conversion.
+
+## 3. Hardware
+
+Power the host off, then:
+
+1. Kingston boot drive out of slot 1, into slot 2.
+2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather
+   than setting it aside.
+3. Slot 1 switch to **U.2**. Set it before the drive goes in; there is nothing
+   to record afterwards, the installed drive type implies the position.
+4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.
+
+Boot the host's existing PVE once before touching Talos and check the drive:
+
+```sh
+nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
+nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use'      # want 512b data, 0 metadata
+wipefs /dev/nvmeYn1                                 # Crucial: clear the old bluestore sig
+ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'
+```
+
+This is the last revert point. A bad drive here means the Crucial goes back and
+the VM is recreated from tofu with nothing lost. Once Talos writes to the
+Kingston, PVE on that host is gone.
+
+Relocating the boot drive is safe: `rpool` is referenced by NVMe EUI and the
+UEFI entries by GPT PARTUUID, neither of which encodes a slot.
+
+## 4. Secure Boot
+
+MS-01 ships with Secure Boot on, and Proxmox satisfies it with a
+Microsoft-signed shim. The Image Factory `metal-amd64.iso` is an unsigned UKI
+and the firmware rejects it outright. **Turn Secure Boot off in the BIOS.**
+
+A signed variant exists per schematic, but it is signed with SideroLabs' key
+which firmware does not trust by default, and the installed system would have to
+use the secureboot installer image too. That is a separate project.
+
+## 5. Install
+
+Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
+that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
+*every* link, so one machine answers on several addresses at once — they are the
+same host, not several hosts. `get disks` on each confirms it.
+
+Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
+subcommand rather than a global, so it must come *after* `get` — placed before
+it, cobra resolves to the legacy `talosctl disks` command and reports
+`unknown flag: --insecure`:
+
+```sh
+talosctl -n <maintenance-ip> get disks --insecure
+talosctl -n <maintenance-ip> get links --insecure
+```
+
+Verify serials and MACs here, on the machine itself, against the config about to
+be applied.
+
+Build the config from the **live cluster's secrets**, not a fresh
+`talosctl gen config`, or the node forms a new cluster instead of joining:
+
+```sh
+talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
+talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
+talosctl gen config homelab https://k8s.renner.uno:6443 \
+  --with-secrets secrets.yaml --output-types controlplane \
+  --config-patch @talos/patches/cluster.yaml \
+  --config-patch @<cilium additive half> \
+  --config-patch @talos/patches/registry-mirror.yaml \
+  --config-patch @talos/patches/k8s-N.yaml -o install.yaml
+```
+
+Confirm `cluster.id` matches the running cluster before applying.
+
+Strip the generated `HostnameConfig` document — it collides with
+`machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.
+
+`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
+against a base with no `inlineManifests` (issue #581). Use only the additive
+second document.
+
+Validate:
+
+```sh
+talosctl validate --config install.yaml --mode metal
+```
+
+### Install minimal, complete on the running node
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
+and the console scrolling
+
+```
+k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
+```
+
+That message is a symptom of an install that never finished, not a Kubernetes or
+networking fault — the node holds a DHCP address and is reachable throughout. On
+saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
+`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
+config applied cleanly once the node was up. Wiping disks makes no difference —
+it is not leftover state.
+
+**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
+which is one-shot (section 7) and so cannot be deferred to stage 2:
+
+```
+machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
+LinkConfig    <uplink>
+VLANConfig    mgmt0
+DHCPv4Config  mgmt0
+VolumeConfig  EPHEMERAL
+```
+
+```sh
+talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
+```
+
+**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
+apply the full config to the running node. `--mode=try` reverts on its own if the
+network config is wrong, so it cannot strand the node; re-apply with
+`--mode=auto` to persist.
+
+```sh
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
+```
+
+Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
+link are up afterwards with `talosctl get links` and `get addresses`.
+
+## 6. The network config will strand the node if it is wrong
+
+Talos stops running default DHCP operators **for the entire machine** as soon as
+a single new-style network document exists — the gate is presence of a document,
+not success of resolution. If the configured path fails to materialise there is
+no fallback on any interface, and recovery needs a console and install media.
+
+Concretely: do not use `LinkAliasConfig` as a `VLANConfig` parent without
+verifying it. Aliases *are* supported (confirmed in v1.12.6 source and by a
+maintainer), but a selector matching zero links is skipped **silently**, and
+`permanent_addr` comes from an ethtool ioctl with no fallback — if the driver
+does not answer, the match is against an empty string and everything downstream
+that references the alias does nothing.
+
+Address links by predictable kernel name. To introduce aliases later, do it on a
+node that is already up, with `talosctl apply-config --mode=try --timeout=5m`,
+which rolls back on its own, and check `talosctl get linkaliasspec`.
+
+Interface names derive from PCI topology and are stable — but not immune.
+Adding a third NVMe to jupiter renumbered the buses and renamed the unused
+second I226 from `enp89s0` to `enp90s0`.
+
+## 7. EPHEMERAL placement is one-shot
+
+```yaml
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '<crucial serial>'
+  minSize: 50GiB
+  grow: true
+```
+
+Talos applies volume configuration **only when the volume does not yet exist**.
+Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a
+disk wipe and a fresh boot from install media. It must be in the first
+`apply-config`.
+
+Match by serial. `disk.transport == 'nvme' && !system_disk` matches the PM983 as
+well, and losing that coin flip formats the OSD drive.
+
+Verify afterwards:
+
+```sh
+talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
+talosctl -n <node> get disks
+```
+
+## 8. The mon will not come back on its own
+
+**This is the step that cost the most time on jupiter.** The rebuilt node has a
+new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old
+address in three places. The mon cannot bind an address the host no longer has,
+so it crashloops; the operator then tries to restart a *healthy* mon as its next
+step, Ceph refuses because that would drop quorum, and it retries every 60
+seconds forever without ever reaching the storage reconcile. Nothing else
+progresses behind it.
+
+Fix in this order — the order matters, because doing it the other way round has
+the operator faithfully recreate the mon from the stale ConfigMap:
+
+```sh
+# 1. ConfigMap first
+kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
+#    remove the dead mon from `data` and from `mapping`
+
+# 2. then the monmap
+ceph mon remove <id>
+
+# 3. then the deployment — check it really carries the old address
+kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
+kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>
+```
+
+Rook then creates a **new** mon with the next free ID (jupiter's `c` came back
+as `d`), on the correct address. It does not repair the old one.
+
+Note mons use host ports under `network.provider: host`, so a replacement mon
+cannot schedule while the broken one's deployment still holds 3300/6789 — the
+canary sits `Pending` with "didn't have free ports" until the old deployment is
+deleted.
+
+## 9. Re-add the OSD
+
+Add the node back to `cephcluster.yaml` under `storage.nodes`, naming the PM983
+by `/dev/disk/by-id`:
+
+```yaml
+- name: k8s-N
+  devices:
+    - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>
+```
+
+**Not** a bare device name. Talos enumerates these drives in a different order
+than Proxmox did — on jupiter `nvme0n1` is the Kingston *system* disk, and the
+PM983 is `nvme2n1`. Rook would probably decline a partitioned system disk, but
+that is luck rather than design.
+
+Get the path from `talosctl -n <node> get disk <dev> -o yaml`, under `symlinks`.
+
+## 10. Verify and restore
+
+```sh
+kubectl get nodes                                   # Ready
+talosctl -n <node> etcd members                     # 3 members
+ceph status                                         # 3 mons in quorum, 3 osds
+ceph osd tree                                       # OSD under the right zone/host
+ceph device ls                                      # PM983 backing the new OSD
+```
+
+Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
+1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
+raised to `high_recovery_ops` to speed the evacuation.
+
+The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
+`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
+
+## Known-stale things that will bite
+
+- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
+  cold rebuild is broken independently of this work.
+- Manual Woodpecker pipelines are broken (issue #582), so anything gated on
+  `PIPELINE == "..."` must be run another way.
+- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
+  converted node. Not observed on jupiter, but it is the first place to look if
+  the v30iot macvlan workloads misbehave there.

kubernetes/apps/immich/machine-learning.yaml

@@ -6,6 +6,13 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-machine-learning
 spec:
+  # Scale this and immich-server to 0 for the duration of any single-node
+  # maintenance — cluster CPU requests exceed what two nodes can hold, and this
+  # pair frees ~1000m against a shortfall of roughly 200m. Stopping them is the
+  # right lever rather than lowering their requests: 500m reserved against a
+  # 2m idle draw looks like waste, but inference bursts hard and the request is
+  # honest about what it needs when someone uploads photos.
+  # See RUNBOOK-baremetal-conversion.md.
   replicas: 1
   strategy:
     type: RollingUpdate

kubernetes/apps/immich/server.yaml

@@ -36,6 +36,9 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-server
 spec:
+  # Scaled to 0 alongside immich-machine-learning during single-node
+  # maintenance; see the note there. The CNPG Postgres cluster stays running,
+  # so recovery is just these two replica counts.
   replicas: 1
   strategy:
     type: Recreate

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -160,19 +160,51 @@ spec:
         - "10.2.40.0/24"
       cluster:
         - "10.99.0.0/29"
+  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
+  # taken out of the storage fleet by removing it here. Under useAllNodes the
+  # operator re-provisions an OSD on any node whose device still carries a
+  # BlueStore signature, which makes purging an OSD ahead of pulling its drive
+  # impossible: the OSD is recreated within a reconcile and starts backfilling
+  # onto the drive being replaced.
+  #
+  # All three nodes are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
+  # Talos enumerates these machines' drives in a different order than Proxmox
+  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
+  # depending on the boot. Rook would probably decline the system disk for
+  # carrying partitions, but that is luck rather than intent. The by-id path is
+  # derived from model and serial and does not depend on enumeration at all.
+  #
+  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
+  # and the Crucial as nvme0n1 on the very next one.
+  #
+  #   k8s-1 (jupiter)
+  #   50026B7382B48B86  Kingston  — Talos system disk
+  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M400426    PM983     — this OSD
+  #
+  #   k8s-2 (saturn)
+  #   50026B7382DA5F6E  Kingston  — Talos system disk
+  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M403161    PM983     — this OSD
+  #
+  #   k8s-3 (neptune)
+  #   50026B7382B48B90  Kingston  — Talos system disk
+  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NF0M501176    PM983     — this OSD
   storage:
-    useAllNodes: true
+    useAllNodes: false
     useAllDevices: false
-    devices:
-      # On k8s-2/k8s-3 this is the NVMe drive PCI-passed-through from saturn
-      # and neptune, and it is the only block device the guest can see. k8s-1
-      # is bare metal on jupiter and sees two: the OSD drive in MS-01 slot 1
-      # (PCI 01:00.0) and the Talos system disk in slot 2 (58:00.0). Slot 1
-      # enumerates first, so nvme0n1 is still the OSD drive there — but that
-      # is enumeration order, not a guarantee. The system disk is protected
-      # from the other direction instead: talos/patches/k8s-1.yaml pins the
-      # installer to it by serial, and Rook skips a disk carrying partitions.
-      - name: nvme0n1
+    nodes:
+      - name: k8s-1
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+      - name: k8s-2
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
+      - name: k8s-3
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
-  # every drive in the cluster: per-drive write volume equalled total cluster
-  # write volume, and all three drives wore at an identical rate and would have
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
-  # wear-out here is correlated by construction rather than independent.
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
+  # volume equalled total cluster write volume and all three wore at an
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that, because the wear was
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
+  # override existed only because Ceph does not lower min_size when size is
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
-  parameters:
-    min_size: "1"
+    size: 3
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -87,35 +80,26 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
+  # size 3 with no min_size override. This pool holds ~253 MiB against the data
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
+  # than just the objects on the lost OSD, so it is the last pool that should
+  # ever run thin.
   #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out for the jupiter bare-metal
-  # conversion. size 3 on two remaining hosts leaves every PG undersized at two
-  # copies with min_size 2, so a single further OSD loss drops the pool below
-  # min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
-  # two copies are active+clean instead, and a further loss degrades rather
-  # than blocks — availability bought at the cost of accepting writes onto a
-  # single copy while degraded.
+  # It drops to 2/1 whenever one of the three hosts is out for longer than a
+  # reboot. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss falls below min_size
+  # and blocks IO on every CephFS mount cluster-wide. At 2/1 the same two
+  # copies are active+clean instead, and a further loss degrades rather than
+  # blocks. See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
-      # the comment there for the full rationale and the accepted trade.
+      # size 3 for the same reasons as replicapool above — see the comment
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
-      parameters:
-        min_size: "1"
+        size: 3
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/talos/patches/k8s-1.yaml

@@ -1,7 +1,7 @@
 # k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
 # still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and both
-# NVMe drives are visible to Talos at once.
+# for that reason: there are no virtio NICs, no PCI-passthrough layer, and all
+# four physical NICs and all three NVMe drives are visible to Talos at once.
 #
 # Networking uses the v1.12 multi-document config rather than the deprecated
 # machine.network.interfaces. That is deliberate: the docs do not define what
@@ -9,9 +9,6 @@
 # every link in one schema instead of straddling them.
 machine:
   network:
-    # Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
-    # below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
-    # as k8s.renner.uno.
     hostname: k8s-1
   sysctls:
     # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
@@ -24,10 +21,12 @@ machine:
     # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    # Both NVMe drives are now ordinary block devices to Talos: the Kingston
-    # OM8PGP41024Q-A0 system disk and the Samsung PM983 that Rook claims as the
-    # Ceph OSD. Selecting the installer target by serial means it cannot land
-    # on the OSD drive regardless of which one the kernel enumerates first.
+    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
+    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
+    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
+    # VolumeConfig below). Every one of them is selected by serial. Nothing in
+    # this file may use a broad selector: `disk.transport == 'nvme'` matches all
+    # three, and the failure mode is Talos taking the OSD drive.
     diskSelector:
       serial: 50026B7382B48B86
   certSANs:
@@ -37,40 +36,70 @@ machine:
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
-# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
-# driver selector k8s-2/k8s-3 use for their single passed-through port would
-# match two links here. MAC matching is unambiguous and survives udev renaming.
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk.
+#
+# That drive is at 87% of its rated endurance and was this node's Ceph OSD until
+# the PM983 replaced it. /var is exactly the right role for a disk in that
+# state: containerd's image and snapshot store, pod logs and the Ceph mon store
+# are all reconstructible, so the drive is consumable here in a way it is not as
+# an OSD. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had,
+# which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
+# they absorb another node's pods.
+#
+# THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; a config applied after EPHEMERAL
+# has been created elsewhere is silently ignored, and relocating it then
+# requires wiping the disk and booting install media again.
+#
+# Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
+# would match the PM983 as well, and losing that coin flip means Talos formats
+# the OSD drive.
 apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: lan0
-selector:
-  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2332E86B749A'
+  minSize: 50GiB
+  grow: true
 ---
-apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: ceph0
-selector:
-  match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
----
-# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
-# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
-# before, which is why the VM saw two already-untagged virtio NICs instead.
+# X710 port 0 carries the switch trunk. It takes no addressing itself — the
+# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
+# business being on — so management and v30iot both ride it as tagged VLANs.
+#
+# Links are addressed by predictable kernel name. Those derive from PCI
+# topology, not probe order — though not immune: adding the third NVMe
+# renumbered the buses and moved the unused second I226 from enp89s0 to
+# enp90s0. Neither name used here was affected.
+#
+# LinkAliasConfig would be the better way to pin these, and is worth
+# reintroducing — but only once it is proven. Aliasing both X710 ports by
+# permanent MAC and using the aliases as VLAN parents produced a node with no
+# address on any interface: the aliases were never created, so every document
+# referencing them did nothing, including the static address on port 1. Verify
+# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
+# back on its own, before making it permanent.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: lan0
+name: enp2s0f0np0
 up: true
 ---
+# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
+# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
+# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: lan0
+parent: enp2s0f0np0
 up: true
 ---
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
 ---
+# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
+# k8s.renner.uno.
 apiVersion: v1alpha1
 kind: Layer2VIPConfig
 name: 10.2.40.60
@@ -80,21 +109,20 @@ link: mgmt0
 # `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
 # that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
 # serve both VM and bare-metal nodes for as long as the cluster is mixed.
-# Nothing else on this machine is called ens19: the physical NICs are
-# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp89s0.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: lan0
+parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
-# on X710 port 1. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. No default route.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
+# untagged on the wire. Used by Rook for OSD<->OSD replication via
+# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
+# segment, and deliberately has no default route.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph0
+name: enp2s0f1np1
 up: true
 mtu: 9000
 addresses:

kubernetes/talos/patches/k8s-2.yaml

@@ -1,31 +1,98 @@
+# k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials, MACs and addresses differ.
 machine:
   network:
     hostname: k8s-2
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.2/29
-        mtu: 9000
   sysctls:
-    # v30iot NIC carries no host addressing — see k8s-1.yaml.
+    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It is the
+    # macvlan parent for the v30iot pod attachments; host addresses on that L2
+    # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382DA5F6E
   certSANs:
     - k8s-2.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: saturn
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 81% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2332E86B74BC'
+  minSize: 50GiB
+  grow: true
+---
+# X710 port 0 carries the switch trunk and takes no addressing itself — the
+# port's untagged VLAN is not one this node belongs on. Management and v30iot
+# both ride it as tagged VLANs.
+#
+# Addressed by predictable kernel name, not by LinkAliasConfig. Aliases are
+# valid VLAN parents, but a selector matching zero links is skipped silently
+# and Talos disables default DHCP machine-wide as soon as any new-style network
+# document exists — so a silent miss leaves the node with no address on any
+# interface. See RUNBOOK-baremetal-conversion.md.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
+# k8s.renner.uno.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
+# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
+# name. VLANConfig.name is free-form, so one set of NADs serves both.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
+# untagged on the wire. OSD<->OSD replication only. Not routed off its own
+# segment and deliberately has no default route.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.2/29

kubernetes/talos/patches/k8s-3.yaml

@@ -1,31 +1,82 @@
+# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials and addresses differ. neptune was the last node converted, so all
+# three control-plane nodes now share this layout and no Proxmox remains.
 machine:
   network:
     hostname: k8s-3
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.3/29
-        mtu: 9000
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382B48B90
   certSANs:
     - k8s-3.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: neptune
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 83% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. Available spare is still 100% and it reports no media errors.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2326E6E8E67D'
+  minSize: 50GiB
+  grow: true
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, held by whichever node wins the etcd campaign.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# v30iot: link only, no host addressing — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.3/29

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_151_k8s_2.tf

@@ -1,50 +0,0 @@
-module "k8s_2" {
-  source = "./modules/proxmox_vm"
-
-  name        = "k8s-2.srv.renner.uno"
-  target_node = "saturn"
-  vmid        = 151
-  memory      = 49152
-  cores       = 8
-
-  template_name = null
-  cloud_init    = false
-  full_clone    = false
-  bios          = "ovmf"
-  machine       = "q35"
-  boot          = "order=virtio0"
-  agent_enabled = true
-
-  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
-  # VLAN from this node (previously only k8s-edge had v30iot, so any
-  # IoT-pool LB lease landing here was unreachable from devices).
-  extra_network_bridges = ["v30iot"]
-
-  disks = [
-    {
-      storage = "local-zfs"
-      size    = "64G"
-      format  = "raw"
-      slot    = "virtio0"
-      file_id = local.talos_image
-    }
-  ]
-
-  hostpci = [
-    {
-      mapping = "osd-ssd"
-    },
-    {
-      mapping = "x710-port1"
-    }
-  ]
-
-  tags            = "kubernetes;talos;controlplane"
-  ssh_public_keys = local.common_ssh_keys
-
-  vm_state = "running"
-
-  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
-  # non-responsive Talos guest can't hang the destroy for 30 minutes.
-  stop_on_destroy = true
-}

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}