konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Cold-rebuild runbook is unusable: Step 1 provisions Proxmox VMs that no longer exist #660

New Issue
Closed
opened 2026-08-27 08:03:11 +00:00 by claude-bot · 0 comments
claude-bot commented 2026-08-27 08:03:11 +00:00
Member
Copy Link
Copy Source

The disaster-recovery procedure in kubernetes/RUNBOOK-cold-rebuild.md cannot be followed. Its Step 1 is "tofu: provision the VMs + supporting infra", recreating the Talos VMs on Proxmox — and there are none left to recreate.

The OpenTofu stack has no VM definitions any more. What remains is modules/proxmox_vm/ (unused), plus DNS, object storage and UniFi. tofu apply provisions no nodes, and Step 2 then has no cluster to form.

The document is half-updated. It already knows one node is bare metal:

k8s-1 is not a VM and OpenTofu does not create it. jupiter runs Talos on bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).

That was true when jupiter was converted. All three have been bare metal since the neptune conversion, and Proxmox is decommissioned — the two TargetDown (pve) / TargetDown (pve-node) alerts that were closed today had been firing against monitoring targets that will never return.

Why this matters more than an ordinary stale doc: it is the document someone reaches for when everything is gone, which is exactly when there is no capacity to discover that step one is fiction. It also references bootstrap.sh, which is itself dead (#581).

Scope of the rewrite:

  • Step 1 becomes: boot all three MS-01s from Image Factory media into maintenance mode. No tofu apply for nodes — the OpenTofu step shrinks to DNS, object storage and UniFi, which are still real.
  • The bootstrap.sh path needs replacing with whatever the actual procedure is now; RUNBOOK-baremetal-conversion.md describes the working bare-metal flow and is the natural source.
  • Schematic and version references should be checked while in there — they were bumped to 0f6fb8bd… / v1.13.9 today, but the surrounding prose may still assume the Proxmox schematic.
  • The maintenance-mode reachability constraint from #576 belongs here too: with VLAN 40 only tagged, a freshly booted node cannot DHCP and needs a console. On MS-01 there is no IPMI. That is a real obstacle to a remote cold rebuild and the runbook should say so until #576 is done.

Not urgent in the sense that nothing is broken today. Urgent in the sense that it is only discovered when it is needed.

The disaster-recovery procedure in `kubernetes/RUNBOOK-cold-rebuild.md` cannot be followed. Its Step 1 is *"tofu: provision the VMs + supporting infra"*, recreating the Talos VMs on Proxmox — and there are none left to recreate. The OpenTofu stack has no VM definitions any more. What remains is `modules/proxmox_vm/` (unused), plus DNS, object storage and UniFi. `tofu apply` provisions no nodes, and Step 2 then has no cluster to form. **The document is half-updated.** It already knows one node is bare metal: > **k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune). That was true when jupiter was converted. All three have been bare metal since the neptune conversion, and Proxmox is decommissioned — the two `TargetDown (pve)` / `TargetDown (pve-node)` alerts that were closed today had been firing against monitoring targets that will never return. **Why this matters more than an ordinary stale doc:** it is the document someone reaches for when everything is gone, which is exactly when there is no capacity to discover that step one is fiction. It also references `bootstrap.sh`, which is itself dead (#581). **Scope of the rewrite:** - Step 1 becomes: boot all three MS-01s from Image Factory media into maintenance mode. No `tofu apply` for nodes — the OpenTofu step shrinks to DNS, object storage and UniFi, which are still real. - The `bootstrap.sh` path needs replacing with whatever the actual procedure is now; `RUNBOOK-baremetal-conversion.md` describes the working bare-metal flow and is the natural source. - Schematic and version references should be checked while in there — they were bumped to `0f6fb8bd…` / v1.13.9 today, but the surrounding prose may still assume the Proxmox schematic. - The maintenance-mode reachability constraint from #576 belongs here too: with VLAN 40 only tagged, a freshly booted node cannot DHCP and needs a console. On MS-01 there is no IPMI. That is a real obstacle to a remote cold rebuild and the runbook should say so until #576 is done. Not urgent in the sense that nothing is broken today. Urgent in the sense that it is only discovered when it is needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#660