|
|
|
@@ -1,340 +0,0 @@
|
|
|
|
|
# Runbook — converting a control-plane node from Proxmox VM to bare metal
|
|
|
|
|
|
|
|
|
|
Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
|
|
|
|
|
metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
|
|
|
|
|
after doing jupiter on 2026-08-25; saturn and neptune remain.
|
|
|
|
|
|
|
|
|
|
Everything here that reads like an over-specific warning is something that
|
|
|
|
|
actually happened. The conversion itself is straightforward — the failures were
|
|
|
|
|
all in state that outlived the VM.
|
|
|
|
|
|
|
|
|
|
## What the node ends up as
|
|
|
|
|
|
|
|
|
|
| | |
|
|
|
|
|
|---|---|
|
|
|
|
|
| System disk | Kingston OM8PGP41024Q-A0, selected by serial |
|
|
|
|
|
| EPHEMERAL (`/var`) | the retired Crucial P3, selected by serial |
|
|
|
|
|
| Ceph OSD | Samsung PM983, selected by `/dev/disk/by-id` |
|
|
|
|
|
| Management | VLAN 40 tagged on X710 port 0 |
|
|
|
|
|
| v30iot | VLAN 30 on the same port, named `ens19` |
|
|
|
|
|
| Ceph cluster net | X710 port 1, `10.99.0.x/29`, MTU 9000 |
|
|
|
|
|
|
|
|
|
|
## Before you start
|
|
|
|
|
|
|
|
|
|
- [ ] All three OSDs up, all mons in quorum, PGs `active+clean`. Do not start
|
|
|
|
|
from a degraded cluster; the window itself removes one of three.
|
|
|
|
|
- [ ] Cluster fits on two nodes. Check `kubectl describe node` allocation on the
|
|
|
|
|
two survivors. Take immich down (`kubernetes/apps/immich`, both
|
|
|
|
|
Deployments to `replicas: 0`) — it frees ~1000m against a shortfall of
|
|
|
|
|
roughly 200m and takes its image pulls off the remaining nodes.
|
|
|
|
|
- [ ] `cephfs-metadata` to size 2 / min_size 1 for the duration. At size 3 on
|
|
|
|
|
two hosts every PG is undersized with min_size 2, so one further OSD loss
|
|
|
|
|
blocks IO on every CephFS mount cluster-wide.
|
|
|
|
|
- [ ] Talos metal ISO on a USB stick. Schematic
|
|
|
|
|
`a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
|
|
|
|
|
(`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
|
|
|
|
|
Verify by comparing partition offsets and sizes against the ISO with
|
|
|
|
|
`fdisk -l` — blkid labels lie after a rewrite.
|
|
|
|
|
- [ ] Physical access. Secure Boot must be turned off (see below), and that is
|
|
|
|
|
the one step nothing can do remotely.
|
|
|
|
|
|
|
|
|
|
## 1. Evacuate the OSD
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
ceph osd out <id>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Wait until no PGs are `backfilling` or `backfill_wait`. Do **not** wait for
|
|
|
|
|
`ceph osd safe-to-destroy` — pools at size 3 keep the OSD in their acting set
|
|
|
|
|
because three copies cannot be placed on two hosts, so it never returns clean.
|
|
|
|
|
|
|
|
|
|
Then remove it. The node must already be out of `cephcluster.yaml`'s
|
|
|
|
|
`storage.nodes` list, or the operator rebuilds the OSD within one reconcile and
|
|
|
|
|
starts backfilling onto the drive you are about to pull:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0
|
|
|
|
|
ceph osd purge <id> --yes-i-really-mean-it
|
|
|
|
|
kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## 2. Drain, and leave the cluster
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Single-replica workloads with `minAvailable: 1` PDBs will refuse eviction — that
|
|
|
|
|
is deliberate, they exist to surface the pod rather than let it be yanked. On
|
|
|
|
|
this cluster that is `influxdb-influxdb2-0`, `woodpecker-server` and the
|
|
|
|
|
CNPG primary. Move each one deliberately with `kubectl delete pod`; they
|
|
|
|
|
reschedule onto the survivors and their RBD volumes follow.
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
talosctl -n <node> reset --graceful --wipe-mode system-disk
|
|
|
|
|
kubectl delete node <node>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`--wipe-mode system-disk` and not the default `all`: the OSD drive is being
|
|
|
|
|
pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
|
|
|
|
|
etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
|
|
|
|
|
the drain, which the rebuilt node would otherwise inherit.
|
|
|
|
|
|
|
|
|
|
## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
|
|
|
|
|
|
|
|
|
|
Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
|
|
|
|
|
host leaves the Proxmox cluster: afterwards the provider cannot reach the node,
|
|
|
|
|
the plan errors, and the only escape is `tofu state rm`, which is off the table.
|
|
|
|
|
|
|
|
|
|
Point `proxmox_api_url` at a host that will *survive* the conversion, not the one
|
|
|
|
|
being converted.
|
|
|
|
|
|
|
|
|
|
Manual triggers are currently broken by the Woodpecker v3.18.0 regression
|
|
|
|
|
(issue #582). Until that is resolved, run it locally:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_REGION=fsn1 \
|
|
|
|
|
tofu -chdir=opentofu init -backend-config=<file>
|
|
|
|
|
… plan -target=module.k8s_N -var-file=<file> -out=<plan>
|
|
|
|
|
… apply <plan>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The `AWS_*` environment variables are required *in addition to*
|
|
|
|
|
`-backend-config`, or the S3 backend fails `HeadObject` with 403 even with
|
|
|
|
|
credentials that can read the object fine via the aws CLI.
|
|
|
|
|
|
|
|
|
|
## 2b. Proxmox quorum — matters from the second conversion onward
|
|
|
|
|
|
|
|
|
|
Two nodes remain after jupiter, with `expected_votes: 2` and quorum 2. Removing
|
|
|
|
|
one leaves the last node inquorate and `/etc/pve` read-only, which blocks the
|
|
|
|
|
final VM destroy. On the surviving node:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
pvecm expected 1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This is the official recovery. Proxmox documents it in the storage replication
|
|
|
|
|
chapter (`/usr/share/pve-docs/chapter-pvesr.html`), verbatim:
|
|
|
|
|
|
|
|
|
|
> If you have no quorum, we strongly advise to fix this first and make the node
|
|
|
|
|
> operable again. Only if this is not possible at the moment, you may use the
|
|
|
|
|
> following command to enforce quorum on the current node: `# pvecm expected 1`
|
|
|
|
|
|
|
|
|
|
The caveat there is about restoring the missing node instead. Here the missing
|
|
|
|
|
node is deliberately gone for good, so enforcing quorum is the correct branch
|
|
|
|
|
rather than a workaround.
|
|
|
|
|
|
|
|
|
|
Run it only once the node is actually alone — attempting it while both nodes are
|
|
|
|
|
up fails with `CS_ERR_INVALID_PARAM`, since corosync will not set expected votes
|
|
|
|
|
below the members it can currently see. Which is exactly why it becomes valid
|
|
|
|
|
when one remains.
|
|
|
|
|
|
|
|
|
|
Runtime only; it does not survive a reboot, so re-run it if the survivor
|
|
|
|
|
restarts before its own conversion.
|
|
|
|
|
|
|
|
|
|
## 3. Hardware
|
|
|
|
|
|
|
|
|
|
Power the host off, then:
|
|
|
|
|
|
|
|
|
|
1. Kingston boot drive out of slot 1, into slot 2.
|
|
|
|
|
2. Crucial P3 out of slot 2 — it becomes EPHEMERAL, so put it in slot 3 rather
|
|
|
|
|
than setting it aside.
|
|
|
|
|
3. Slot 1 switch to **U.2**. Set it before the drive goes in; there is nothing
|
|
|
|
|
to record afterwards, the installed drive type implies the position.
|
|
|
|
|
4. PM983 onto the included 7mm M.2-to-U.2 adapter, into slot 1.
|
|
|
|
|
|
|
|
|
|
Boot the host's existing PVE once before touching Talos and check the drive:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
nvme smart-log /dev/nvmeXn1 | grep -iE 'percentage_used|media_errors|critical'
|
|
|
|
|
nvme id-ns /dev/nvmeXn1 -H | grep -i 'in use' # want 512b data, 0 metadata
|
|
|
|
|
wipefs /dev/nvmeYn1 # Crucial: clear the old bluestore sig
|
|
|
|
|
ip link set <ceph-port> up && ethtool <ceph-port> | grep -i 'link detected'
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This is the last revert point. A bad drive here means the Crucial goes back and
|
|
|
|
|
the VM is recreated from tofu with nothing lost. Once Talos writes to the
|
|
|
|
|
Kingston, PVE on that host is gone.
|
|
|
|
|
|
|
|
|
|
Relocating the boot drive is safe: `rpool` is referenced by NVMe EUI and the
|
|
|
|
|
UEFI entries by GPT PARTUUID, neither of which encodes a slot.
|
|
|
|
|
|
|
|
|
|
## 4. Secure Boot
|
|
|
|
|
|
|
|
|
|
MS-01 ships with Secure Boot on, and Proxmox satisfies it with a
|
|
|
|
|
Microsoft-signed shim. The Image Factory `metal-amd64.iso` is an unsigned UKI
|
|
|
|
|
and the firmware rejects it outright. **Turn Secure Boot off in the BIOS.**
|
|
|
|
|
|
|
|
|
|
A signed variant exists per schematic, but it is signed with SideroLabs' key
|
|
|
|
|
which firmware does not trust by default, and the installed system would have to
|
|
|
|
|
use the secureboot installer image too. That is a separate project.
|
|
|
|
|
|
|
|
|
|
## 5. Install
|
|
|
|
|
|
|
|
|
|
Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
|
|
|
|
|
that lands on `192.168.8.x`, which is routable from the workstation.
|
|
|
|
|
|
|
|
|
|
Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
|
|
|
|
|
and `version` returns "API is not implemented in maintenance mode". You cannot
|
|
|
|
|
inspect disks or links before applying. Verify serials and MACs from the host's
|
|
|
|
|
PVE beforehand, in step 3.
|
|
|
|
|
|
|
|
|
|
Build the config from the **live cluster's secrets**, not a fresh
|
|
|
|
|
`talosctl gen config`, or the node forms a new cluster instead of joining:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
talosctl -n <surviving-node> get mc v1alpha1 -o jsonpath='{.spec}' > cp.yaml
|
|
|
|
|
talosctl gen secrets --from-controlplane-config cp.yaml -o secrets.yaml
|
|
|
|
|
talosctl gen config homelab https://k8s.renner.uno:6443 \
|
|
|
|
|
--with-secrets secrets.yaml --output-types controlplane \
|
|
|
|
|
--config-patch @talos/patches/cluster.yaml \
|
|
|
|
|
--config-patch @<cilium additive half> \
|
|
|
|
|
--config-patch @talos/patches/registry-mirror.yaml \
|
|
|
|
|
--config-patch @talos/patches/k8s-N.yaml -o install.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Confirm `cluster.id` matches the running cluster before applying.
|
|
|
|
|
|
|
|
|
|
Strip the generated `HostnameConfig` document — it collides with
|
|
|
|
|
`machine.network.hostname` from the per-node patch. `bootstrap.sh` does the same.
|
|
|
|
|
|
|
|
|
|
`cilium-cni.yaml` cannot be passed whole: its `$patch: delete` document fails
|
|
|
|
|
against a base with no `inlineManifests` (issue #581). Use only the additive
|
|
|
|
|
second document.
|
|
|
|
|
|
|
|
|
|
Validate, then apply:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
talosctl validate --config install.yaml --mode metal
|
|
|
|
|
talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## 6. The network config will strand the node if it is wrong
|
|
|
|
|
|
|
|
|
|
Talos stops running default DHCP operators **for the entire machine** as soon as
|
|
|
|
|
a single new-style network document exists — the gate is presence of a document,
|
|
|
|
|
not success of resolution. If the configured path fails to materialise there is
|
|
|
|
|
no fallback on any interface, and recovery needs a console and install media.
|
|
|
|
|
|
|
|
|
|
Concretely: do not use `LinkAliasConfig` as a `VLANConfig` parent without
|
|
|
|
|
verifying it. Aliases *are* supported (confirmed in v1.12.6 source and by a
|
|
|
|
|
maintainer), but a selector matching zero links is skipped **silently**, and
|
|
|
|
|
`permanent_addr` comes from an ethtool ioctl with no fallback — if the driver
|
|
|
|
|
does not answer, the match is against an empty string and everything downstream
|
|
|
|
|
that references the alias does nothing.
|
|
|
|
|
|
|
|
|
|
Address links by predictable kernel name. To introduce aliases later, do it on a
|
|
|
|
|
node that is already up, with `talosctl apply-config --mode=try --timeout=5m`,
|
|
|
|
|
which rolls back on its own, and check `talosctl get linkaliasspec`.
|
|
|
|
|
|
|
|
|
|
Interface names derive from PCI topology and are stable — but not immune.
|
|
|
|
|
Adding a third NVMe to jupiter renumbered the buses and renamed the unused
|
|
|
|
|
second I226 from `enp89s0` to `enp90s0`.
|
|
|
|
|
|
|
|
|
|
## 7. EPHEMERAL placement is one-shot
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: v1alpha1
|
|
|
|
|
kind: VolumeConfig
|
|
|
|
|
name: EPHEMERAL
|
|
|
|
|
provisioning:
|
|
|
|
|
diskSelector:
|
|
|
|
|
match: disk.serial == '<crucial serial>'
|
|
|
|
|
minSize: 50GiB
|
|
|
|
|
grow: true
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Talos applies volume configuration **only when the volume does not yet exist**.
|
|
|
|
|
Applied afterwards it is silently ignored, and relocating EPHEMERAL then needs a
|
|
|
|
|
disk wipe and a fresh boot from install media. It must be in the first
|
|
|
|
|
`apply-config`.
|
|
|
|
|
|
|
|
|
|
Match by serial. `disk.transport == 'nvme' && !system_disk` matches the PM983 as
|
|
|
|
|
well, and losing that coin flip formats the OSD drive.
|
|
|
|
|
|
|
|
|
|
Verify afterwards:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
talosctl -n <node> get volumestatus EPHEMERAL -o yaml | grep -E 'location|phase'
|
|
|
|
|
talosctl -n <node> get disks
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## 8. The mon will not come back on its own
|
|
|
|
|
|
|
|
|
|
**This is the step that cost the most time on jupiter.** The rebuilt node has a
|
|
|
|
|
new MAC and therefore a new DHCP lease, but Ceph has its mon pinned to the old
|
|
|
|
|
address in three places. The mon cannot bind an address the host no longer has,
|
|
|
|
|
so it crashloops; the operator then tries to restart a *healthy* mon as its next
|
|
|
|
|
step, Ceph refuses because that would drop quorum, and it retries every 60
|
|
|
|
|
seconds forever without ever reaching the storage reconcile. Nothing else
|
|
|
|
|
progresses behind it.
|
|
|
|
|
|
|
|
|
|
Fix in this order — the order matters, because doing it the other way round has
|
|
|
|
|
the operator faithfully recreate the mon from the stale ConfigMap:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
# 1. ConfigMap first
|
|
|
|
|
kubectl -n rook-ceph edit cm rook-ceph-mon-endpoints
|
|
|
|
|
# remove the dead mon from `data` and from `mapping`
|
|
|
|
|
|
|
|
|
|
# 2. then the monmap
|
|
|
|
|
ceph mon remove <id>
|
|
|
|
|
|
|
|
|
|
# 3. then the deployment — check it really carries the old address
|
|
|
|
|
kubectl -n rook-ceph get deploy rook-ceph-mon-<id> -o yaml | grep public-addr
|
|
|
|
|
kubectl -n rook-ceph delete deploy rook-ceph-mon-<id>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Rook then creates a **new** mon with the next free ID (jupiter's `c` came back
|
|
|
|
|
as `d`), on the correct address. It does not repair the old one.
|
|
|
|
|
|
|
|
|
|
Note mons use host ports under `network.provider: host`, so a replacement mon
|
|
|
|
|
cannot schedule while the broken one's deployment still holds 3300/6789 — the
|
|
|
|
|
canary sits `Pending` with "didn't have free ports" until the old deployment is
|
|
|
|
|
deleted.
|
|
|
|
|
|
|
|
|
|
## 9. Re-add the OSD
|
|
|
|
|
|
|
|
|
|
Add the node back to `cephcluster.yaml` under `storage.nodes`, naming the PM983
|
|
|
|
|
by `/dev/disk/by-id`:
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
- name: k8s-N
|
|
|
|
|
devices:
|
|
|
|
|
- fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_<serial>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Not** a bare device name. Talos enumerates these drives in a different order
|
|
|
|
|
than Proxmox did — on jupiter `nvme0n1` is the Kingston *system* disk, and the
|
|
|
|
|
PM983 is `nvme2n1`. Rook would probably decline a partitioned system disk, but
|
|
|
|
|
that is luck rather than design.
|
|
|
|
|
|
|
|
|
|
Get the path from `talosctl -n <node> get disk <dev> -o yaml`, under `symlinks`.
|
|
|
|
|
|
|
|
|
|
## 10. Verify and restore
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
kubectl get nodes # Ready
|
|
|
|
|
talosctl -n <node> etcd members # 3 members
|
|
|
|
|
ceph status # 3 mons in quorum, 3 osds
|
|
|
|
|
ceph osd tree # OSD under the right zone/host
|
|
|
|
|
ceph device ls # PM983 backing the new OSD
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
|
|
|
|
|
1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
|
|
|
|
|
raised to `high_recovery_ops` to speed the evacuation.
|
|
|
|
|
|
|
|
|
|
The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
|
|
|
|
|
`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
|
|
|
|
|
|
|
|
|
|
## Known-stale things that will bite
|
|
|
|
|
|
|
|
|
|
- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
|
|
|
|
|
cold rebuild is broken independently of this work.
|
|
|
|
|
- Manual Woodpecker pipelines are broken (issue #582), so anything gated on
|
|
|
|
|
`PIPELINE == "..."` must be run another way.
|
|
|
|
|
- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
|
|
|
|
|
`bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
|
|
|
|
|
converted node. Not observed on jupiter, but it is the first place to look if
|
|
|
|
|
the v30iot macvlan workloads misbehave there.
|
issue-fix-bot