feat(rook-ceph): re-add neptune's OSD on the PM983

neptune is converted and back in the cluster, so k8s-3 returns to the storage
fleet on its new PM983. All three nodes now name their OSD by /dev/disk/by-id,
and the storage fleet is whole again.

The comment generalises to all three nodes and records neptune's serials.
k8s-3 is the clearest evidence for by-id addressing: it enumerated the Kingston
as nvme0n1 on one boot and the Crucial as nvme0n1 on the very next one, so a
kernel-name reference had no stable meaning on that machine at all.

The old osd.1 lived on the Crucial that is now EPHEMERAL. That drive was wiped
during the conversion, so the OSD is unrecoverable and is purged from the CRUSH
map separately; this change only creates its replacement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-infrastructure.yml

@@ -131,18 +131,18 @@ steps:
       - |
         # Ensure Talos image exists on all Proxmox nodes via API.
         #
-        # jupiter is NOT in this list: it runs Talos on bare metal and is no
-        # longer a Proxmox node, so the storage API returns nothing for it and
-        # the download-wait loop below would burn its full 60x10s before
-        # giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
-        # still need the nocloud image; bare-metal nodes boot the metal-amd64
-        # image from their own installed disk instead.
+        # Only nodes still running Proxmox belong in this list. jupiter and
+        # saturn run Talos on bare metal now, so the storage API returns
+        # nothing for them and the download-wait loop below would burn its full
+        # 60x10s per node before giving up. Bare-metal nodes boot the
+        # metal-amd64 image from their own installed disk instead, and the
+        # whole step goes away once neptune is converted too.
         TALOS_VERSION="v1.12.6"
         TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
         TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
         TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
         API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in saturn neptune; do
+        for node in neptune; do
           EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
             -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
           if [ "$EXISTS" -gt 0 ]; then

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -0,0 +1,395 @@
+# Runbook — converting a control-plane node from Proxmox VM to bare metal
+
+Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
+metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
+after doing jupiter on 2026-08-25; saturn and neptune remain.
+
+Everything here that reads like an over-specific warning is something that
+actually happened. The conversion itself is straightforward — the failures were
+all in state that outlived the VM.
+
+## What the node ends up as
+
+| | |
+|---|---|
+| System disk | Kingston OM8PGP41024Q-A0, selected by serial |
+| EPHEMERAL (`/var`) | the retired Crucial P3, selected by serial |
+| Ceph OSD | Samsung PM983, selected by `/dev/disk/by-id` |
+| Management | VLAN 40 tagged on X710 port 0 |
+| v30iot | VLAN 30 on the same port, named `ens19` |
+| Ceph cluster net | X710 port 1, `10.99.0.x/29`, MTU 9000 |
+
+## Before you start
+
+- [ ] All three OSDs up, all mons in quorum, PGs `active+clean`. Do not start
+      from a degraded cluster; the window itself removes one of three.
+- [ ] Cluster fits on two nodes. Check `kubectl describe node` allocation on the
+      two survivors. Take immich down (`kubernetes/apps/immich`, both
+      Deployments to `replicas: 0`) — it frees ~1000m against a shortfall of
+      roughly 200m and takes its image pulls off the remaining nodes.
+- [ ] `cephfs-metadata` to size 2 / min_size 1 for the duration. At size 3 on
+      two hosts every PG is undersized with min_size 2, so one further OSD loss
+      blocks IO on every CephFS mount cluster-wide.
+- [ ] Talos metal ISO on a USB stick. Schematic
+      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
+      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      Verify by comparing partition offsets and sizes against the ISO with
+      `fdisk -l` — blkid labels lie after a rewrite.
+- [ ] Physical access. Secure Boot must be turned off (see below), and that is
+      the one step nothing can do remotely.
+
+## 1. Evacuate the OSD
+
+```sh
+ceph osd out <id>
+```
+
+Wait until no PGs are `backfilling` or `backfill_wait`. Do **not** wait for
+`ceph osd safe-to-destroy` — pools at size 3 keep the OSD in their acting set
+because three copies cannot be placed on two hosts, so it never returns clean.
+
+Then remove it. The node must already be out of `cephcluster.yaml`'s
+`storage.nodes` list, or the operator rebuilds the OSD within one reconcile and
+starts backfilling onto the drive you are about to pull:
+
+```sh
+kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
+ceph osd purge <id> --yes-i-really-mean-it
+kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>
+```
+
+## 2. Drain, and leave the cluster
+
+```sh
+kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
+```
+
+Single-replica workloads with `minAvailable: 1` PDBs will refuse eviction — that
+is deliberate, they exist to surface the pod rather than let it be yanked. On
+this cluster that is `influxdb-influxdb2-0`, `woodpecker-server` and the
+CNPG primary. Move each one deliberately with `kubectl delete pod`; they
+reschedule onto the survivors and their RBD volumes follow.
+
+```sh
+talosctl -n <node> reset --graceful --wipe-mode system-disk
+kubectl delete node <node>
+```
+
+`--wipe-mode system-disk` and not the default `all`: the OSD drive is being
+pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
+etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
+the drain, which the rebuilt node would otherwise inherit.
+
+## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
+
+Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
+host leaves the Proxmox cluster: afterwards the provider cannot reach the node,
+the plan errors, and the only escape is `tofu state rm`, which is off the table.
+
+Point `proxmox_api_url` at a host that will *survive* the conversion, not the one
+being converted.
+
+Manual triggers are currently broken by the Woodpecker v3.18.0 regression
+(issue #582). Until that is resolved, run it locally:
+
+```sh
+AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
+  tofu -chdir=opentofu init -backend-config=<file>
+… plan -target=module.k8s_N -var-file=<file> -out=<plan>
+… apply <plan>
+```
+
+The `AWS_*` environment variables are required *in addition to*
+`-backend-config`, or the S3 backend fails `HeadObject` with 403 even with
+credentials that can read the object fine via the aws CLI.
+
+## 2b. Proxmox quorum — matters from the second conversion onward
+
+Two nodes remain after jupiter, with `expected_votes: 2` and quorum 2. Removing
+one leaves the last node inquorate and `/etc/pve` read-only, which blocks the
+final VM destroy. On the surviving node:
+
+```sh
+pvecm expected 1
+```
+
+This is the official recovery. Proxmox documents it in the storage replication
+chapter (`/usr/share/pve-docs/chapter-pvesr.html`), verbatim:
+
+> If you have no quorum, we strongly advise to fix this first and make the node
+> operable again. Only if this is not possible at the moment, you may use the
+> following command to enforce quorum on the current node: `# pvecm expected 1`
+
+The caveat there is about restoring the missing node instead. Here the missing
+node is deliberately gone for good, so enforcing quorum is the correct branch
+rather than a workaround.
+
+Run it only once the node is actually alone — attempting it while both nodes are
+up fails with `CS_ERR_INVALID_PARAM`, since corosync will not set expected votes
+below the members it can currently see. Which is exactly why it becomes valid
+when one remains.
+
+Runtime only; it does not survive a reboot, so re-run it if the survivor
+restarts before its own conversion.
+
+## 3. Hardware
+
+Power the host off, then:
+
+1. Kingston boot drive out of slot 1, into slot 2.
+2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather
+   than setting it aside.
+3. Slot 1 switch to **U.2**. Set it before the drive goes in; there is nothing
+   to record afterwards, the installed drive type implies the position.
+4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.
+
+Boot the host's existing PVE once before touching Talos and check the drive:
+
+```sh
+nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
+nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use'      # want 512b data, 0 metadata
+wipefs /dev/nvmeYn1                                 # Crucial: clear the old bluestore sig
+ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'
+```
+
+This is the last revert point. A bad drive here means the Crucial goes back and
+the VM is recreated from tofu with nothing lost. Once Talos writes to the
+Kingston, PVE on that host is gone.
+
+Relocating the boot drive is safe: `rpool` is referenced by NVMe EUI and the
+UEFI entries by GPT PARTUUID, neither of which encodes a slot.
+
+## 4. Secure Boot
+
+MS-01 ships with Secure Boot on, and Proxmox satisfies it with a
+Microsoft-signed shim. The Image Factory `metal-amd64.iso` is an unsigned UKI
+and the firmware rejects it outright. **Turn Secure Boot off in the BIOS.**
+
+A signed variant exists per schematic, but it is signed with SideroLabs' key
+which firmware does not trust by default, and the installed system would have to
+use the secureboot installer image too. That is a separate project.
+
+## 5. Install
+
+Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
+that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
+*every* link, so one machine answers on several addresses at once — they are the
+same host, not several hosts. `get disks` on each confirms it.
+
+Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
+subcommand rather than a global, so it must come *after* `get` — placed before
+it, cobra resolves to the legacy `talosctl disks` command and reports
+`unknown flag: --insecure`:
+
+```sh
+talosctl -n <maintenance-ip> get disks --insecure
+talosctl -n <maintenance-ip> get links --insecure
+```
+
+Verify serials and MACs here, on the machine itself, against the config about to
+be applied.
+
+Build the config from the **live cluster's secrets**, not a fresh
+`talosctl gen config`, or the node forms a new cluster instead of joining:
+
+```sh
+talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
+talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
+talosctl gen config homelab https://k8s.renner.uno:6443 \
+  --with-secrets secrets.yaml --output-types controlplane \
+  --config-patch @talos/patches/cluster.yaml \
+  --config-patch @<cilium additive half> \
+  --config-patch @talos/patches/registry-mirror.yaml \
+  --config-patch @talos/patches/k8s-N.yaml -o install.yaml
+```
+
+Confirm `cluster.id` matches the running cluster before applying.
+
+Strip the generated `HostnameConfig` document — it collides with
+`machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.
+
+`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
+against a base with no `inlineManifests` (issue #581). Use only the additive
+second document.
+
+Validate:
+
+```sh
+talosctl validate --config install.yaml --mode metal
+```
+
+### Install minimal, complete on the running node
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
+and the console scrolling
+
+```
+k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
+```
+
+That message is a symptom of an install that never finished, not a Kubernetes or
+networking fault — the node holds a DHCP address and is reachable throughout. On
+saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
+`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
+config applied cleanly once the node was up. Wiping disks makes no difference —
+it is not leftover state.
+
+**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
+which is one-shot (section 7) and so cannot be deferred to stage 2:
+
+```
+machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
+LinkConfig    <uplink>
+VLANConfig    mgmt0
+DHCPv4Config  mgmt0
+VolumeConfig  EPHEMERAL
+```
+
+```sh
+talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
+```
+
+**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
+apply the full config to the running node. `--mode=try` reverts on its own if the
+network config is wrong, so it cannot strand the node; re-apply with
+`--mode=auto` to persist.
+
+```sh
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
+```
+
+Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
+link are up afterwards with `talosctl get links` and `get addresses`.
+
+## 6. The network config will strand the node if it is wrong
+
+Talos stops running default DHCP operators **for the entire machine** as soon as
+a single new-style network document exists — the gate is presence of a document,
+not success of resolution. If the configured path fails to materialise there is
+no fallback on any interface, and recovery needs a console and install media.
+
+Concretely: do not use `LinkAliasConfig` as a `VLANConfig` parent without
+verifying it. Aliases *are* supported (confirmed in v1.12.6 source and by a
+maintainer), but a selector matching zero links is skipped **silently**, and
+`permanent_addr` comes from an ethtool ioctl with no fallback — if the driver
+does not answer, the match is against an empty string and everything downstream
+that references the alias does nothing.
+
+Address links by predictable kernel name. To introduce aliases later, do it on a
+node that is already up, with `talosctl apply-config --mode=try --timeout=5m`,
+which rolls back on its own, and check `talosctl get linkaliasspec`.
+
+Interface names derive from PCI topology and are stable — but not immune.
+Adding a third NVMe to jupiter renumbered the buses and renamed the unused
+second I226 from `enp89s0` to `enp90s0`.
+
+## 7. EPHEMERAL placement is one-shot
+
+```yaml
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '<crucial serial>'
+  minSize: 50GiB
+  grow: true
+```
+
+Talos applies volume configuration **only when the volume does not yet exist**.
+Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a
+disk wipe and a fresh boot from install media. It must be in the first
+`apply-config`.
+
+Match by serial. `disk.transport == 'nvme' && !system_disk` matches the PM983 as
+well, and losing that coin flip formats the OSD drive.
+
+Verify afterwards:
+
+```sh
+talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
+talosctl -n <node> get disks
+```
+
+## 8. The mon will not come back on its own
+
+**This is the step that cost the most time on jupiter.** The rebuilt node has a
+new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old
+address in three places. The mon cannot bind an address the host no longer has,
+so it crashloops; the operator then tries to restart a *healthy* mon as its next
+step, Ceph refuses because that would drop quorum, and it retries every 60
+seconds forever without ever reaching the storage reconcile. Nothing else
+progresses behind it.
+
+Fix in this order — the order matters, because doing it the other way round has
+the operator faithfully recreate the mon from the stale ConfigMap:
+
+```sh
+# 1. ConfigMap first
+kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
+#    remove the dead mon from `data` and from `mapping`
+
+# 2. then the monmap
+ceph mon remove <id>
+
+# 3. then the deployment — check it really carries the old address
+kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
+kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>
+```
+
+Rook then creates a **new** mon with the next free ID (jupiter's `c` came back
+as `d`), on the correct address. It does not repair the old one.
+
+Note mons use host ports under `network.provider: host`, so a replacement mon
+cannot schedule while the broken one's deployment still holds 3300/6789 — the
+canary sits `Pending` with "didn't have free ports" until the old deployment is
+deleted.
+
+## 9. Re-add the OSD
+
+Add the node back to `cephcluster.yaml` under `storage.nodes`, naming the PM983
+by `/dev/disk/by-id`:
+
+```yaml
+- name: k8s-N
+  devices:
+    - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>
+```
+
+**Not** a bare device name. Talos enumerates these drives in a different order
+than Proxmox did — on jupiter `nvme0n1` is the Kingston *system* disk, and the
+PM983 is `nvme2n1`. Rook would probably decline a partitioned system disk, but
+that is luck rather than design.
+
+Get the path from `talosctl -n <node> get disk <dev> -o yaml`, under `symlinks`.
+
+## 10. Verify and restore
+
+```sh
+kubectl get nodes                                   # Ready
+talosctl -n <node> etcd members                     # 3 members
+ceph status                                         # 3 mons in quorum, 3 osds
+ceph osd tree                                       # OSD under the right zone/host
+ceph device ls                                      # PM983 backing the new OSD
+```
+
+Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
+1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
+raised to `high_recovery_ops` to speed the evacuation.
+
+The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
+`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
+
+## Known-stale things that will bite
+
+- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
+  cold rebuild is broken independently of this work.
+- Manual Woodpecker pipelines are broken (issue #582), so anything gated on
+  `PIPELINE == "..."` must be run another way.
+- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
+  converted node. Not observed on jupiter, but it is the first place to look if
+  the v30iot macvlan workloads misbehave there.

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,33 +167,44 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
-  #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
+  # All three nodes are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
+  # Talos enumerates these machines' drives in a different order than Proxmox
+  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
+  # depending on the boot. Rook would probably decline the system disk for
   # carrying partitions, but that is luck rather than intent. The by-id path is
   # derived from model and serial and does not depend on enumeration at all.
   #
-  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
+  # and the Crucial as nvme0n1 on the very next one.
+  #
+  #   k8s-1 (jupiter)
+  #   50026B7382B48B86  Kingston  — Talos system disk
+  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M400426    PM983     — this OSD
+  #
+  #   k8s-2 (saturn)
+  #   50026B7382DA5F6E  Kingston  — Talos system disk
+  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M403161    PM983     — this OSD
+  #
+  #   k8s-3 (neptune)
+  #   50026B7382B48B90  Kingston  — Talos system disk
+  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NF0M501176    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      - name: k8s-2
-        devices:
-          - name: nvme0n1
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+      - name: k8s-2
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
+      - name: k8s-3
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/talos/patches/k8s-2.yaml

@@ -1,31 +1,99 @@
+# k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
+# VM-era layout until neptune is converted.
 machine:
   network:
     hostname: k8s-2
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.2/29
-        mtu: 9000
   sysctls:
-    # v30iot NIC carries no host addressing — see k8s-1.yaml.
+    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It is the
+    # macvlan parent for the v30iot pod attachments; host addresses on that L2
+    # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382DA5F6E
   certSANs:
     - k8s-2.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: saturn
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 81% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2332E86B74BC'
+  minSize: 50GiB
+  grow: true
+---
+# X710 port 0 carries the switch trunk and takes no addressing itself — the
+# port's untagged VLAN is not one this node belongs on. Management and v30iot
+# both ride it as tagged VLANs.
+#
+# Addressed by predictable kernel name, not by LinkAliasConfig. Aliases are
+# valid VLAN parents, but a selector matching zero links is skipped silently
+# and Talos disables default DHCP machine-wide as soon as any new-style network
+# document exists — so a silent miss leaves the node with no address on any
+# interface. See RUNBOOK-baremetal-conversion.md.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
+# k8s.renner.uno.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
+# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
+# name. VLANConfig.name is free-form, so one set of NADs serves both.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
+# untagged on the wire. OSD<->OSD replication only. Not routed off its own
+# segment and deliberately has no default route.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.2/29

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_151_k8s_2.tf

@@ -1,50 +0,0 @@
-module "k8s_2" {
-  source = "./modules/proxmox_vm"
-
-  name        = "k8s-2.srv.renner.uno"
-  target_node = "saturn"
-  vmid        = 151
-  memory      = 49152
-  cores       = 8
-
-  template_name = null
-  cloud_init    = false
-  full_clone    = false
-  bios          = "ovmf"
-  machine       = "q35"
-  boot          = "order=virtio0"
-  agent_enabled = true
-
-  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
-  # VLAN from this node (previously only k8s-edge had v30iot, so any
-  # IoT-pool LB lease landing here was unreachable from devices).
-  extra_network_bridges = ["v30iot"]
-
-  disks = [
-    {
-      storage = "local-zfs"
-      size    = "64G"
-      format  = "raw"
-      slot    = "virtio0"
-      file_id = local.talos_image
-    }
-  ]
-
-  hostpci = [
-    {
-      mapping = "osd-ssd"
-    },
-    {
-      mapping = "x710-port1"
-    }
-  ]
-
-  tags            = "kubernetes;talos;controlplane"
-  ssh_public_keys = local.common_ssh_keys
-
-  vm_state = "running"
-
-  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
-  # non-responsive Talos guest can't hang the destroy for 30 minutes.
-  stop_on_destroy = true
-}

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}