fix: [alert] KubeDeploymentReplicasMismatch (rook-ceph) (#571)

.woodpecker/deploy-infrastructure.yml

@@ -131,18 +131,18 @@ steps:
       - |
         # Ensure Talos image exists on all Proxmox nodes via API.
         #
-        # Only nodes still running Proxmox belong in this list. jupiter and
-        # saturn run Talos on bare metal now, so the storage API returns
-        # nothing for them and the download-wait loop below would burn its full
-        # 60x10s per node before giving up. Bare-metal nodes boot the
-        # metal-amd64 image from their own installed disk instead, and the
-        # whole step goes away once neptune is converted too.
+        # jupiter is NOT in this list: it runs Talos on bare metal and is no
+        # longer a Proxmox node, so the storage API returns nothing for it and
+        # the download-wait loop below would burn its full 60x10s before
+        # giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
+        # still need the nocloud image; bare-metal nodes boot the metal-amd64
+        # image from their own installed disk instead.
         TALOS_VERSION="v1.12.6"
         TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
         TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
         TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
         API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in neptune; do
+        for node in saturn neptune; do
           EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
             -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
           if [ "$EXISTS" -gt 0 ]; then

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -1,340 +0,0 @@
-# Runbook — converting a control-plane node from Proxmox VM to bare metal
-
-Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
-metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25; saturn and neptune remain.
-
-Everything here that reads like an over-specific warning is something that
-actually happened. The conversion itself is straightforward — the failures were
-all in state that outlived the VM.
-
-## What the node ends up as
-
-| | |
-|---|---|
-| System disk | Kingston OM8PGP41024Q-A0, selected by serial |
-| EPHEMERAL (`/var`) | the retired Crucial P3, selected by serial |
-| Ceph OSD | Samsung PM983, selected by `/dev/disk/by-id` |
-| Management | VLAN 40 tagged on X710 port 0 |
-| v30iot | VLAN 30 on the same port, named `ens19` |
-| Ceph cluster net | X710 port 1, `10.99.0.x/29`, MTU 9000 |
-
-## Before you start
-
-- [ ] All three OSDs up, all mons in quorum, PGs `active+clean`. Do not start
-      from a degraded cluster; the window itself removes one of three.
-- [ ] Cluster fits on two nodes. Check `kubectl describe node` allocation on the
-      two survivors. Take immich down (`kubernetes/apps/immich`, both
-      Deployments to `replicas: 0`) — it frees ~1000m against a shortfall of
-      roughly 200m and takes its image pulls off the remaining nodes.
-- [ ] `cephfs-metadata` to size 2 / min_size 1 for the duration. At size 3 on
-      two hosts every PG is undersized with min_size 2, so one further OSD loss
-      blocks IO on every CephFS mount cluster-wide.
-- [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
-      Verify by comparing partition offsets and sizes against the ISO with
-      `fdisk -l` — blkid labels lie after a rewrite.
-- [ ] Physical access. Secure Boot must be turned off (see below), and that is
-      the one step nothing can do remotely.
-
-## 1. Evacuate the OSD
-
-```sh
-ceph osd out <id>
-```
-
-Wait until no PGs are `backfilling` or `backfill_wait`. Do **not** wait for
-`ceph osd safe-to-destroy` — pools at size 3 keep the OSD in their acting set
-because three copies cannot be placed on two hosts, so it never returns clean.
-
-Then remove it. The node must already be out of `cephcluster.yaml`'s
-`storage.nodes` list, or the operator rebuilds the OSD within one reconcile and
-starts backfilling onto the drive you are about to pull:
-
-```sh
-kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
-ceph osd purge <id> --yes-i-really-mean-it
-kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>
-```
-
-## 2. Drain, and leave the cluster
-
-```sh
-kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
-```
-
-Single-replica workloads with `minAvailable: 1` PDBs will refuse eviction — that
-is deliberate, they exist to surface the pod rather than let it be yanked. On
-this cluster that is `influxdb-influxdb2-0`, `woodpecker-server` and the
-CNPG primary. Move each one deliberately with `kubectl delete pod`; they
-reschedule onto the survivors and their RBD volumes follow.
-
-```sh
-talosctl -n <node> reset --graceful --wipe-mode system-disk
-kubectl delete node <node>
-```
-
-`--wipe-mode system-disk` and not the default `all`: the OSD drive is being
-pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
-etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
-the drain, which the rebuilt node would otherwise inherit.
-
-## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
-
-Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
-host leaves the Proxmox cluster: afterwards the provider cannot reach the node,
-the plan errors, and the only escape is `tofu state rm`, which is off the table.
-
-Point `proxmox_api_url` at a host that will *survive* the conversion, not the one
-being converted.
-
-Manual triggers are currently broken by the Woodpecker v3.18.0 regression
-(issue #582). Until that is resolved, run it locally:
-
-```sh
-AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
-  tofu -chdir=opentofu init -backend-config=<file>
-… plan -target=module.k8s_N -var-file=<file> -out=<plan>
-… apply <plan>
-```
-
-The `AWS_*` environment variables are required *in addition to*
-`-backend-config`, or the S3 backend fails `HeadObject` with 403 even with
-credentials that can read the object fine via the aws CLI.
-
-## 2b. Proxmox quorum — matters from the second conversion onward
-
-Two nodes remain after jupiter, with `expected_votes: 2` and quorum 2. Removing
-one leaves the last node inquorate and `/etc/pve` read-only, which blocks the
-final VM destroy. On the surviving node:
-
-```sh
-pvecm expected 1
-```
-
-This is the official recovery. Proxmox documents it in the storage replication
-chapter (`/usr/share/pve-docs/chapter-pvesr.html`), verbatim:
-
-> If you have no quorum, we strongly advise to fix this first and make the node
-> operable again. Only if this is not possible at the moment, you may use the
-> following command to enforce quorum on the current node: `# pvecm expected 1`
-
-The caveat there is about restoring the missing node instead. Here the missing
-node is deliberately gone for good, so enforcing quorum is the correct branch
-rather than a workaround.
-
-Run it only once the node is actually alone — attempting it while both nodes are
-up fails with `CS_ERR_INVALID_PARAM`, since corosync will not set expected votes
-below the members it can currently see. Which is exactly why it becomes valid
-when one remains.
-
-Runtime only; it does not survive a reboot, so re-run it if the survivor
-restarts before its own conversion.
-
-## 3. Hardware
-
-Power the host off, then:
-
-1. Kingston boot drive out of slot 1, into slot 2.
-2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather
-   than setting it aside.
-3. Slot 1 switch to **U.2**. Set it before the drive goes in; there is nothing
-   to record afterwards, the installed drive type implies the position.
-4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.
-
-Boot the host's existing PVE once before touching Talos and check the drive:
-
-```sh
-nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
-nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use'      # want 512b data, 0 metadata
-wipefs /dev/nvmeYn1                                 # Crucial: clear the old bluestore sig
-ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'
-```
-
-This is the last revert point. A bad drive here means the Crucial goes back and
-the VM is recreated from tofu with nothing lost. Once Talos writes to the
-Kingston, PVE on that host is gone.
-
-Relocating the boot drive is safe: `rpool` is referenced by NVMe EUI and the
-UEFI entries by GPT PARTUUID, neither of which encodes a slot.
-
-## 4. Secure Boot
-
-MS-01 ships with Secure Boot on, and Proxmox satisfies it with a
-Microsoft-signed shim. The Image Factory `metal-amd64.iso` is an unsigned UKI
-and the firmware rejects it outright. **Turn Secure Boot off in the BIOS.**
-
-A signed variant exists per schematic, but it is signed with SideroLabs' key
-which firmware does not trust by default, and the installed system would have to
-use the secureboot installer image too. That is a separate project.
-
-## 5. Install
-
-Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation.
-
-Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
-and `version` returns "API is not implemented in maintenance mode". You cannot
-inspect disks or links before applying. Verify serials and MACs from the host's
-PVE beforehand, in step 3.
-
-Build the config from the **live cluster's secrets**, not a fresh
-`talosctl gen config`, or the node forms a new cluster instead of joining:
-
-```sh
-talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
-talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
-talosctl gen config homelab https://k8s.renner.uno:6443 \
-  --with-secrets secrets.yaml --output-types controlplane \
-  --config-patch @talos/patches/cluster.yaml \
-  --config-patch @<cilium additive half> \
-  --config-patch @talos/patches/registry-mirror.yaml \
-  --config-patch @talos/patches/k8s-N.yaml -o install.yaml
-```
-
-Confirm `cluster.id` matches the running cluster before applying.
-
-Strip the generated `HostnameConfig` document — it collides with
-`machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.
-
-`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
-against a base with no `inlineManifests` (issue #581). Use only the additive
-second document.
-
-Validate, then apply:
-
-```sh
-talosctl validate --config install.yaml --mode metal
-talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
-```
-
-## 6. The network config will strand the node if it is wrong
-
-Talos stops running default DHCP operators **for the entire machine** as soon as
-a single new-style network document exists — the gate is presence of a document,
-not success of resolution. If the configured path fails to materialise there is
-no fallback on any interface, and recovery needs a console and install media.
-
-Concretely: do not use `LinkAliasConfig` as a `VLANConfig` parent without
-verifying it. Aliases *are* supported (confirmed in v1.12.6 source and by a
-maintainer), but a selector matching zero links is skipped **silently**, and
-`permanent_addr` comes from an ethtool ioctl with no fallback — if the driver
-does not answer, the match is against an empty string and everything downstream
-that references the alias does nothing.
-
-Address links by predictable kernel name. To introduce aliases later, do it on a
-node that is already up, with `talosctl apply-config --mode=try --timeout=5m`,
-which rolls back on its own, and check `talosctl get linkaliasspec`.
-
-Interface names derive from PCI topology and are stable — but not immune.
-Adding a third NVMe to jupiter renumbered the buses and renamed the unused
-second I226 from `enp89s0` to `enp90s0`.
-
-## 7. EPHEMERAL placement is one-shot
-
-```yaml
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '<crucial serial>'
-  minSize: 50GiB
-  grow: true
-```
-
-Talos applies volume configuration **only when the volume does not yet exist**.
-Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a
-disk wipe and a fresh boot from install media. It must be in the first
-`apply-config`.
-
-Match by serial. `disk.transport == 'nvme' && !system_disk` matches the PM983 as
-well, and losing that coin flip formats the OSD drive.
-
-Verify afterwards:
-
-```sh
-talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
-talosctl -n <node> get disks
-```
-
-## 8. The mon will not come back on its own
-
-**This is the step that cost the most time on jupiter.** The rebuilt node has a
-new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old
-address in three places. The mon cannot bind an address the host no longer has,
-so it crashloops; the operator then tries to restart a *healthy* mon as its next
-step, Ceph refuses because that would drop quorum, and it retries every 60
-seconds forever without ever reaching the storage reconcile. Nothing else
-progresses behind it.
-
-Fix in this order — the order matters, because doing it the other way round has
-the operator faithfully recreate the mon from the stale ConfigMap:
-
-```sh
-# 1. ConfigMap first
-kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
-#    remove the dead mon from `data` and from `mapping`
-
-# 2. then the monmap
-ceph mon remove <id>
-
-# 3. then the deployment — check it really carries the old address
-kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
-kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>
-```
-
-Rook then creates a **new** mon with the next free ID (jupiter's `c` came back
-as `d`), on the correct address. It does not repair the old one.
-
-Note mons use host ports under `network.provider: host`, so a replacement mon
-cannot schedule while the broken one's deployment still holds 3300/6789 — the
-canary sits `Pending` with "didn't have free ports" until the old deployment is
-deleted.
-
-## 9. Re-add the OSD
-
-Add the node back to `cephcluster.yaml` under `storage.nodes`, naming the PM983
-by `/dev/disk/by-id`:
-
-```yaml
-- name: k8s-N
-  devices:
-    - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>
-```
-
-**Not** a bare device name. Talos enumerates these drives in a different order
-than Proxmox did — on jupiter `nvme0n1` is the Kingston *system* disk, and the
-PM983 is `nvme2n1`. Rook would probably decline a partitioned system disk, but
-that is luck rather than design.
-
-Get the path from `talosctl -n <node> get disk <dev> -o yaml`, under `symlinks`.
-
-## 10. Verify and restore
-
-```sh
-kubectl get nodes                                   # Ready
-talosctl -n <node> etcd members                     # 3 members
-ceph status                                         # 3 mons in quorum, 3 osds
-ceph osd tree                                       # OSD under the right zone/host
-ceph device ls                                      # PM983 backing the new OSD
-```
-
-Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
-1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
-raised to `high_recovery_ops` to speed the evacuation.
-
-The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
-`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
-
-## Known-stale things that will bite
-
-- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
-  cold rebuild is broken independently of this work.
-- Manual Woodpecker pipelines are broken (issue #582), so anything gated on
-  `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.

kubernetes/apps/immich/machine-learning.yaml

@@ -6,14 +6,17 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-machine-learning
 spec:
-  # Scale this and immich-server to 0 for the duration of any single-node
-  # maintenance — cluster CPU requests exceed what two nodes can hold, and this
-  # pair frees ~1000m against a shortfall of roughly 200m. Stopping them is the
-  # right lever rather than lowering their requests: 500m reserved against a
-  # 2m idle draw looks like waste, but inference bursts hard and the request is
-  # honest about what it needs when someone uploads photos.
-  # See RUNBOOK-baremetal-conversion.md.
-  replicas: 1
+  # TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED.
+  #
+  # Down for the jupiter bare-metal conversion. With one of three nodes out,
+  # the remaining pair have no room: cluster CPU requests exceed two nodes'
+  # capacity, and k8s-3 runs ~5.8 GiB above the kubelet's eviction threshold,
+  # so every pod that does not need to be running is one less thing competing
+  # for both. This one is the best candidate: 500m reserved against a measured
+  # 2m idle draw, and inference bursts hard enough that lowering the request
+  # instead would be the wrong trade. Photo processing pauses; nothing else
+  # depends on it.
+  replicas: 0
   strategy:
     type: RollingUpdate
   selector:

kubernetes/apps/immich/server.yaml

@@ -36,10 +36,11 @@ metadata:
   labels:
     app.kubernetes.io/name: immich-server
 spec:
-  # Scaled to 0 alongside immich-machine-learning during single-node
-  # maintenance; see the note there. The CNPG Postgres cluster stays running,
-  # so recovery is just these two replica counts.
-  replicas: 1
+  # TEMPORARY — RESTORE TO 1 ONCE k8s-1 HAS REJOINED. See the note on
+  # immich-machine-learning; immich is offline for the jupiter conversion.
+  # Its Postgres cluster is deliberately left running, so nothing needs
+  # restoring beyond these two replica counts.
+  replicas: 0
   strategy:
     type: Recreate
   selector:

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -11,6 +11,43 @@ spec:
   mon:
     count: 3
     allowMultiplePerNode: false
+  # TEMPORARY — SET BACK TO false (or drop this block) ONCE k8s-1 HAS REJOINED.
+  #
+  # Stops Rook's mon failover loop for the jupiter window. k8s-1 is no longer a
+  # Node object at all, so the health checker takes the "assigned node not
+  # found" path and fails over *immediately* instead of after the 600s
+  # out-of-quorum timeout:
+  #
+  #   op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not
+  #           found, failing over immediately
+  #
+  # The failover can never complete. Mons run hostNetwork on 3300/6789 and
+  # allowMultiplePerNode is false, so with k8s-1 gone the replacement's canary
+  # has nowhere to land ("2 node(s) didn't have free ports for the requested
+  # pod ports"). The operator gives up after ~2.5 min, deletes the canary, and
+  # the 45s health-check interval starts the next attempt. Each cycle churns a
+  # Deployment plus pod, rewrites the mon-endpoints ConfigMap and the ceph-csi
+  # connection/clientProfile CRs, and buries everything else in the operator log.
+  #
+  # This does NOT hide the degradation: daemonHealth.status is untouched, so
+  # MON_DOWN stays in CephCluster.status.ceph and CephHealthWarn still fires,
+  # and rook-ceph-mon-c stays 0/1 Pending so KubeDeploymentReplicasMismatch
+  # keeps firing. Nothing is given up, because there is no node the failover
+  # could have succeeded on anyway.
+  #
+  # It also keeps mon "c" intact for the end of the window: with the checker
+  # live, the moment k8s-1 rejoins the operator races to stand mon-d up there
+  # and drop "c" from the monmap. Off, mon-c reschedules onto k8s-1 and rejoins
+  # with its existing store under /var/lib/rook/mon-c. If that store did not
+  # survive the reinstall, re-enable the checker first — then failover is
+  # exactly what you want.
+  #
+  # Reverts together with the `RE-ADD IT` note under storage.nodes below and
+  # the `RESTORE TO 1` note in kubernetes/apps/immich/machine-learning.yaml.
+  healthCheck:
+    daemonHealth:
+      mon:
+        disabled: true
   mgr:
     count: 2
     allowMultiplePerNode: false
@@ -167,34 +204,27 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
+  # k8s-1 is absent while jupiter's OSD drive is swapped and the node is
+  # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
+  # rejoined; the cluster runs on two OSDs until then.
+  #
   # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
-  #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
-  #
-  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  # neptune, and is the only block device those guests can see. Bare-metal k8s-1
+  # will see two — the OSD drive in MS-01 slot 1 (PCI 01:00.0) and the Talos
+  # system disk in slot 2 (58:00.0) — so when it is re-added, name its device
+  # deliberately rather than assuming slot 1 keeps enumerating first. The system
+  # disk is separately protected: talos/patches/k8s-1.yaml pins the installer to
+  # it by serial, and Rook skips a disk carrying partitions.
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
-      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
-      # metal sees three NVMe drives and the bare name is not the OSD.
+      - name: k8s-2
+        devices:
+          - name: nvme0n1
       - name: k8s-3
         devices:
           - name: nvme0n1
-      - name: k8s-1
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -87,10 +87,7 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
-  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
-  # three conversions rather than being toggled per node: each change
-  # re-replicates the pool, and the window reopens with the next node anyway.
+  # TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
   #
   # The standing trade for this pool is size 3: it holds ~253 MiB against the
   # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
@@ -99,14 +96,13 @@ spec:
   # quarter of a gigabyte to keep that intact is the right trade in steady
   # state.
   #
-  # It does not hold while one of three hosts is out. size 3 on two remaining
-  # hosts leaves every PG undersized at two copies with min_size 2, so a single
-  # further OSD loss drops the pool below min_size and blocks IO on every
-  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
-  # instead, and a further loss degrades rather than blocks — availability
-  # bought at the cost of accepting writes onto a single copy while degraded.
-  #
-  # See RUNBOOK-baremetal-conversion.md.
+  # It does not hold while one of three hosts is out for the jupiter bare-metal
+  # conversion. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss drops the pool below
+  # min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
+  # two copies are active+clean instead, and a further loss degrades rather
+  # than blocks — availability bought at the cost of accepting writes onto a
+  # single copy while degraded.
   metadataPool:
     replicated:
       size: 2

kubernetes/talos/patches/k8s-1.yaml

@@ -1,7 +1,7 @@
 # k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
 # still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and all
-# four physical NICs and all three NVMe drives are visible to Talos at once.
+# for that reason: there are no virtio NICs, no PCI-passthrough layer, and both
+# NVMe drives are visible to Talos at once.
 #
 # Networking uses the v1.12 multi-document config rather than the deprecated
 # machine.network.interfaces. That is deliberate: the docs do not define what
@@ -9,6 +9,9 @@
 # every link in one schema instead of straddling them.
 machine:
   network:
+    # Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
+    # below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
+    # as k8s.renner.uno.
     hostname: k8s-1
   sysctls:
     # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
@@ -21,12 +24,10 @@ machine:
     # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
-    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
-    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
-    # VolumeConfig below). Every one of them is selected by serial. Nothing in
-    # this file may use a broad selector: `disk.transport == 'nvme'` matches all
-    # three, and the failure mode is Talos taking the OSD drive.
+    # Both NVMe drives are now ordinary block devices to Talos: the Kingston
+    # OM8PGP41024Q-A0 system disk and the Samsung PM983 that Rook claims as the
+    # Ceph OSD. Selecting the installer target by serial means it cannot land
+    # on the OSD drive regardless of which one the kernel enumerates first.
     diskSelector:
       serial: 50026B7382B48B86
   certSANs:
@@ -36,70 +37,40 @@ machine:
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk.
-#
-# That drive is at 87% of its rated endurance and was this node's Ceph OSD until
-# the PM983 replaced it. /var is exactly the right role for a disk in that
-# state: containerd's image and snapshot store, pod logs and the Ceph mon store
-# are all reconstructible, so the drive is consumable here in a way it is not as
-# an OSD. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had,
-# which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
-# they absorb another node's pods.
-#
-# THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
-# solely when the volume does not yet exist; a config applied after EPHEMERAL
-# has been created elsewhere is silently ignored, and relocating it then
-# requires wiping the disk and booting install media again.
-#
-# Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
-# would match the PM983 as well, and losing that coin flip means Talos formats
-# the OSD drive.
+# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
+# driver selector k8s-2/k8s-3 use for their single passed-through port would
+# match two links here. MAC matching is unambiguous and survives udev renaming.
 apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '2332E86B749A'
-  minSize: 50GiB
-  grow: true
+kind: LinkAliasConfig
+name: lan0
+selector:
+  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
 ---
-# X710 port 0 carries the switch trunk. It takes no addressing itself — the
-# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
-# business being on — so management and v30iot both ride it as tagged VLANs.
-#
-# Links are addressed by predictable kernel name. Those derive from PCI
-# topology, not probe order — though not immune: adding the third NVMe
-# renumbered the buses and moved the unused second I226 from enp89s0 to
-# enp90s0. Neither name used here was affected.
-#
-# LinkAliasConfig would be the better way to pin these, and is worth
-# reintroducing — but only once it is proven. Aliasing both X710 ports by
-# permanent MAC and using the aliases as VLAN parents produced a node with no
-# address on any interface: the aliases were never created, so every document
-# referencing them did nothing, including the static address on port 1. Verify
-# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
-# back on its own, before making it permanent.
+apiVersion: v1alpha1
+kind: LinkAliasConfig
+name: ceph0
+selector:
+  match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
+---
+# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
+# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
+# before, which is why the VM saw two already-untagged virtio NICs instead.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f0np0
+name: lan0
 up: true
 ---
-# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
-# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
 vlanID: 40
-parent: enp2s0f0np0
+parent: lan0
 up: true
 ---
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
 ---
-# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
-# k8s.renner.uno.
 apiVersion: v1alpha1
 kind: Layer2VIPConfig
 name: 10.2.40.60
@@ -109,20 +80,21 @@ link: mgmt0
 # `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
 # that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
 # serve both VM and bare-metal nodes for as long as the cluster is mixed.
+# Nothing else on this machine is called ens19: the physical NICs are
+# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp89s0.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
 vlanID: 30
-parent: enp2s0f0np0
+parent: lan0
 up: true
 ---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
+# on X710 port 1. Used by Rook for OSD<->OSD replication via
+# CephCluster.spec.network.addressRanges.cluster. No default route.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: enp2s0f1np1
+name: ceph0
 up: true
 mtu: 9000
 addresses:

kubernetes/talos/patches/k8s-2.yaml

@@ -1,99 +1,31 @@
-# k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
-# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
-# VM-era layout until neptune is converted.
 machine:
   network:
     hostname: k8s-2
+    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
+    interfaces:
+      - deviceSelector:
+          busPath: "0000:06:12.0"
+        dhcp: true
+        vip:
+          ip: 10.2.40.60
+      # v30iot: link only, no host addressing — see k8s-1.yaml.
+      - deviceSelector:
+          busPath: "0000:06:13.0"
+        dhcp: false
+      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
+      - deviceSelector:
+          driver: i40e
+        addresses:
+          - 10.99.0.2/29
+        mtu: 9000
   sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It is the
-    # macvlan parent for the v30iot pod attachments; host addresses on that L2
-    # make node replies bypass the UDM and break every UDM-DNATed flow.
+    # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
-    diskSelector:
-      serial: 50026B7382DA5F6E
+    disk: /dev/vda
   certSANs:
     - k8s-2.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: saturn
 cluster:
   allowSchedulingOnControlPlanes: true
----
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 81% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
-#
-# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
-# solely when the volume does not yet exist; applied later it is silently
-# ignored and relocating needs a disk wipe and a boot from install media.
-#
-# Matched by serial deliberately: a broad selector such as
-# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '2332E86B74BC'
-  minSize: 50GiB
-  grow: true
----
-# X710 port 0 carries the switch trunk and takes no addressing itself — the
-# port's untagged VLAN is not one this node belongs on. Management and v30iot
-# both ride it as tagged VLANs.
-#
-# Addressed by predictable kernel name, not by LinkAliasConfig. Aliases are
-# valid VLAN parents, but a selector matching zero links is skipped silently
-# and Talos disables default DHCP machine-wide as soon as any new-style network
-# document exists — so a silent miss leaves the node with no address on any
-# interface. See RUNBOOK-baremetal-conversion.md.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f0np0
-up: true
----
-# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: mgmt0
-vlanID: 40
-parent: enp2s0f0np0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: mgmt0
----
-# Shared control-plane VIP, etcd-elected. Exposed via UniFi DNS as
-# k8s.renner.uno.
-apiVersion: v1alpha1
-kind: Layer2VIPConfig
-name: 10.2.40.60
-link: mgmt0
----
-# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
-# name. VLANConfig.name is free-form, so one set of NADs serves both.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ens19
-vlanID: 30
-parent: enp2s0f0np0
-up: true
----
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. OSD<->OSD replication only. Not routed off its own
-# segment and deliberately has no default route.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f1np1
-up: true
-mtu: 9000
-addresses:
-  - address: 10.99.0.2/29

opentofu/vm_151_k8s_2.tf

@@ -0,0 +1,50 @@
+module "k8s_2" {
+  source = "./modules/proxmox_vm"
+
+  name        = "k8s-2.srv.renner.uno"
+  target_node = "saturn"
+  vmid        = 151
+  memory      = 49152
+  cores       = 8
+
+  template_name = null
+  cloud_init    = false
+  full_clone    = false
+  bios          = "ovmf"
+  machine       = "q35"
+  boot          = "order=virtio0"
+  agent_enabled = true
+
+  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
+  # VLAN from this node (previously only k8s-edge had v30iot, so any
+  # IoT-pool LB lease landing here was unreachable from devices).
+  extra_network_bridges = ["v30iot"]
+
+  disks = [
+    {
+      storage = "local-zfs"
+      size    = "64G"
+      format  = "raw"
+      slot    = "virtio0"
+      file_id = local.talos_image
+    }
+  ]
+
+  hostpci = [
+    {
+      mapping = "osd-ssd"
+    },
+    {
+      mapping = "x710-port1"
+    }
+  ]
+
+  tags            = "kubernetes;talos;controlplane"
+  ssh_public_keys = local.common_ssh_keys
+
+  vm_state = "running"
+
+  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
+  # non-responsive Talos guest can't hang the destroy for 30 minutes.
+  stop_on_destroy = true
+}