diff --git a/kubernetes/RUNBOOK-baremetal-conversion.md b/kubernetes/RUNBOOK-baremetal-conversion.md index 55506642..46d9adef 100644 --- a/kubernetes/RUNBOOK-baremetal-conversion.md +++ b/kubernetes/RUNBOOK-baremetal-conversion.md @@ -172,12 +172,22 @@ use the secureboot installer image too. That is a separate project. ## 5. Install Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts -that lands on `192.168.8.x`, which is routable from the workstation. +that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on +*every* link, so one machine answers on several addresses at once — they are the +same host, not several hosts. `get disks` on each confirms it. -Maintenance mode is effectively write-only — `talosctl get` has no `--insecure` -and `version` returns "API is not implemented in maintenance mode". You cannot -inspect disks or links before applying. Verify serials and MACs from the host's -PVE beforehand, in step 3. +Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the +subcommand rather than a global, so it must come *after* `get` — placed before +it, cobra resolves to the legacy `talosctl disks` command and reports +`unknown flag: --insecure`: + +```sh +talosctl -n get disks --insecure +talosctl -n get links --insecure +``` + +Verify serials and MACs here, on the machine itself, against the config about to +be applied. Build the config from the **live cluster's secrets**, not a fresh `talosctl gen config`, or the node forms a new cluster instead of joining: @@ -202,13 +212,58 @@ Strip the generated `HostnameConfig` document — it collides with against a base with no `inlineManifests` (issue #581). Use only the additive second document. -Validate, then apply: +Validate: ```sh talosctl validate --config install.yaml --mode metal -talosctl apply-config --insecure -n --file install.yaml ``` +### Install minimal, complete on the running node + +Do **not** apply the full per-node config in maintenance mode. A document that +references an interface which does not exist yet hangs the install: the node +sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`, +and the console scrolling + +``` +k8s.NodeApplyController: error getting node: nodes "k8s-N" not found +``` + +That message is a symptom of an install that never finished, not a Kubernetes or +networking fault — the node holds a DHCP address and is reachable throughout. On +saturn the offenders were the `ens19` VLAN document and the sysctl naming it, +`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical +config applied cleanly once the node was up. Wiping disks makes no difference — +it is not leftover state. + +**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`, +which is one-shot (section 7) and so cannot be deferred to stage 2: + +``` +machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels +LinkConfig +VLANConfig mgmt0 +DHCPv4Config mgmt0 +VolumeConfig EPHEMERAL +``` + +```sh +talosctl apply-config --insecure -n --file install-stage1.yaml +``` + +**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it, +apply the full config to the running node. `--mode=try` reverts on its own if the +network config is wrong, so it cannot strand the node; re-apply with +`--mode=auto` to persist. + +```sh +talosctl -e -n apply-config --mode=try --timeout=3m --file install.yaml +talosctl -e -n apply-config --mode=auto --file install.yaml +``` + +Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph +link are up afterwards with `talosctl get links` and `get addresses`. + ## 6. The network config will strand the node if it is wrong Talos stops running default DHCP operators **for the entire machine** as soon as diff --git a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml index 1d30bd04..46099df8 100644 --- a/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml +++ b/kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml @@ -167,34 +167,40 @@ spec: # impossible: the OSD is recreated within a reconcile and starts backfilling # onto the drive being replaced. # - # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and - # neptune, and is the only block device those guests can see, so a bare device - # name is unambiguous there. + # nvme0n1 on k8s-3 is the NVMe drive PCI-passed-through from neptune, and is + # the only block device that guest can see, so a bare device name is + # unambiguous there. # - # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by - # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos - # enumerates this machine's drives in a different order than Proxmox did, and - # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for - # carrying partitions, but that is luck rather than intent. The by-id path is - # derived from model and serial and does not depend on enumeration at all. + # k8s-1 and k8s-2 are bare metal and see three NVMe drives each, so they name + # their OSDs by /dev/disk/by-id path instead. A bare name would be actively + # wrong: Talos enumerates these machines' drives in a different order than + # Proxmox did, and nvme0n1 there is the Kingston SYSTEM disk. Rook would + # probably decline it for carrying partitions, but that is luck rather than + # intent. The by-id path is derived from model and serial and does not depend + # on enumeration at all. # + # k8s-1 (jupiter) # nvme0n1 50026B7382B48B86 Kingston — Talos system disk # nvme1n1 2332E86B749A Crucial — Talos EPHEMERAL (/var) # nvme2n1 S437NX0M400426 PM983 — this OSD + # + # k8s-2 (saturn) + # nvme0n1 50026B7382DA5F6E Kingston — Talos system disk + # nvme1n1 2332E86B74BC Crucial — Talos EPHEMERAL (/var) + # nvme2n1 S437NX0M403161 PM983 — this OSD storage: useAllNodes: false useAllDevices: false nodes: - # k8s-2 is absent while saturn's OSD drive is swapped and the node is - # converted to bare metal. RE-ADD IT once the PM983 is in and the node has - # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare - # metal sees three NVMe drives and the bare name is not the OSD. - name: k8s-3 devices: - name: nvme0n1 - name: k8s-1 devices: - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426 + - name: k8s-2 + devices: + - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161 config: osdsPerDevice: "1" # Storage layer must never be evicted by kubelet under node pressure