feat(rook-ceph): re-add saturn's OSD, fix the bare-metal install sequence #595
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#595
Reference in New Issue
Block a user
saturn is converted and back in the cluster, so its PM983 returns to the storage fleet — and the runbook gets the correction that conversion earned.
Ceph
k8s-2re-added, naming the device by/dev/disk/by-idask8s-1does. Verified on the machine:Merging this takes the cluster from
OSD count 2 < osd_pool_default_size 3back to three OSDs and clears the 14 undersized PGs. Expect backfill onto the new drive.Runbook
The runbook said to apply the full per-node config in maintenance mode. That hangs the install — the node sits at
STAGE: InstallingwithKUBELET n/awhile the console repeatswhich looks like a Kubernetes problem and is not: the install never finished. The node holds a DHCP address and is reachable the whole time. On saturn the cause was the
ens19VLAN document and the sysctl naming it,net.ipv6.conf.ens19.disable_ipv6— both referencing an interface that does not exist yet at install time.Three cold installs hung on this. The identical config applied cleanly to the node once it was running, which is the evidence that the config is fine and only the timing is wrong. Wiping disks makes no difference; that was a red herring.
Install is now two stages:
EPHEMERAL(one-shot, so it cannot be deferred), applied in maintenance mode--mode=try --timeout=3m, which reverts itself if the network config is wrong, then--mode=autoto persistBoth stage-2 applies landed without a reboot on saturn.
Also corrected: the runbook claimed maintenance mode cannot be inspected. It can —
--insecureis a flag onget, so it has to come after the subcommand. Put it before and cobra resolves the legacytalosctl diskscommand and rejects the flag, which is what made it look unsupported.State after this
Ready, etcd 3 membersEPHEMERALon the Crucial (/dev/nvme1n1p1)a,d,e—mon-bwas pinned to the retired10.2.40.136and was replacedactive+cleanStill open, not in this PR:
pvecm delnode saturn, the neptune conversion, and restoringcephfs-metadatato size 3 /.mgrto 3/2 /osd_mclock_profiletobalancedonce all three nodes are converted.saturn is back on bare metal with the PM983 in place, so k8s-2 returns to the storage fleet. Like k8s-1 it names the device by /dev/disk/by-id: the machine has three NVMe drives and Talos enumerates them differently than Proxmox did, so nvme0n1 there is the Kingston system disk. The runbook prescribed applying the full per-node config in maintenance mode, which hangs the install. A document referencing an interface that does not exist yet leaves the node at STAGE: Installing with KUBELET n/a, the console repeating k8s.NodeApplyController: error getting node: nodes "k8s-N" not found which reads as a Kubernetes fault but is an install that never completed. On saturn it was the ens19 VLAN document and the sysctl naming it. Three cold installs hung; the identical config applied cleanly to the running node. Wiping disks changes nothing. Install now runs in two stages: a minimal network config plus EPHEMERAL, which is one-shot and cannot be deferred, then the full config applied to the running node behind --mode=try so a bad network config reverts itself. Also corrects the claim that maintenance mode cannot be inspected. talosctl get works there; --insecure is a subcommand flag and has to follow `get`, otherwise cobra resolves the legacy `talosctl disks` command and rejects the flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>