docs(runbook): drop the claim that a cold rebuild needs console access

The rewrite earlier today stated, at the top and again in known gaps, that the rebuild cannot be driven remotely: management is a tagged VLAN, a maintenance mode node cannot configure VLANs, so it comes up unreachable and every machine needs a keyboard. That is wrong. A maintenance-mode node DHCPs untagged and is reachable at that address — it is not on management, but it answers, and that is precisely the address `apply-config --insecure` targets in Step 2. All three bare-metal conversions were driven that way. The claim came from #576 and was carried into this document without being checked against what had actually happened days earlier in this repo. It is the worst place to put an unverified constraint: a disaster-recovery runbook is read when there is no capacity to discover that a stated prerequisite is fiction, and "you need physical access to three machines" is exactly the kind of statement that changes whether someone attempts a recovery at all. What survives is narrower and true: MS-01 has no IPMI and nothing here does PXE, so booting from USB is physical. Everything after the boot is remote. Also removes the follow-on error from the first correction, which claimed the switch port change on 2026-08-27 had fixed a reachability problem. There was no reachability problem to fix. That change is real and useful — a maintenance mode node now lands in management rather than the fallback network — but it did not turn an impossible rebuild into a possible one. #576 still carries the original claim and is flagged in known gaps as wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 11:25:50 +02:00
1 changed files with 21 additions and 17 deletions
@@ -11,11 +11,14 @@ are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
 Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
 resources, and physically booting three machines from install media.

-**Read this first: the rebuild cannot be done remotely today.** Management is a
-tagged VLAN, and a node in maintenance mode has no VLAN configuration — it can
-only DHCP untagged, so it comes up unreachable. MS-01 has no IPMI. Every node
-therefore needs a keyboard and a monitor until #576 puts VLAN 40 native on the
-I226-V port. Plan for physical access before starting.
+**Someone has to be at the machines to boot them from USB** — MS-01 has no IPMI
+and nothing here does PXE. Everything after that is remote.
+
+A node in maintenance mode configures no VLANs and DHCPs untagged, so it comes
+up on whatever untagged network its ports carry — not on management, which is
+a tagged VLAN on the X710 trunk. It is reachable there: that is the address
+`apply-config --insecure` talks to in Step 2, and it is how all three
+bare-metal conversions were driven. Find the leases on the UDM.

 ## The circular-bootstrap problem this runbook exists to break

@@ -77,9 +80,10 @@ These must exist outside the cluster or the rebuild cannot complete:
      Proxmox credentials — there is no Proxmox any more.
 - [ ] **Install media** written and to hand: the Image Factory ISO named in
      Step 1, on a USB stick, for all three machines.
- [ ] **Physical access to all three nodes** — keyboard and monitor. Not
-      optional today: see the note at the top. Budget for standing in front of
-      the machines, not for driving this from a desk.
+- [ ] **Someone at the machines to boot them from USB.** No IPMI, no PXE. A
+      monitor is useful for reading boot errors but is not required to find the
+      node afterwards — it DHCPs into management on its own (see the note at
+      the top). Everything past the boot is remote.

 ## Step 1a — tofu: DNS, object storage, UniFi

@@ -105,9 +109,9 @@ https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced3
 ```

 Write it to a USB stick and boot each MS-01 from it. Each comes up in
-maintenance mode with a DHCP address on the **untagged** network — not on
-management, because that is a tagged VLAN a maintenance-mode node does not
-configure. Read the address off the console; that is what Step 2 talks to.
+maintenance mode with a DHCP address on an untagged network — not on
+management, which is tagged. Read the leases off the UDM; those addresses are
+what Step 2 applies the first config to.

 The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
 interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
@@ -326,12 +330,12 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
  in-cluster commits made **after** Flux/Gitea broke are not on the mirror
  (Gitea couldn't push them if Gitea itself was down). This is the intended
  recovery point, but be aware of it when reconciling drift.
- **The rebuild cannot be driven remotely.** Management is a tagged VLAN and a
-  maintenance-mode node cannot configure VLANs, so a freshly booted machine is
-  unreachable until its config is applied — and MS-01 has no IPMI. Every node
-  needs a keyboard and a monitor. #576 fixes this by putting VLAN 40 native on
-  the I226-V port; until then, physical access is a hard prerequisite rather
-  than a convenience.
+- **Booting is physical; the rest is not.** MS-01 has no IPMI and nothing here
+  does PXE, so someone has to put a USB stick in three machines. From there it
+  is remote — a maintenance-mode node DHCPs untagged and is reachable at that
+  address, which is how all three bare-metal conversions were driven.
+  #576 claims a freshly booted node "comes up unreachable and must be installed
+  from a console". That is not what happens and has never been observed here.
 - **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.