docs(runbook): rewrite the cold rebuild for bare metal #661

2026-08-27T08:10:20Z

claude-bot commented

2026-08-27 08:10:20 +00:00

Closes #660.

Step 1 was "tofu: provision the VMs" and there are no VMs. The OpenTofu stack has no node definitions left — only the uninstantiated proxmox_vm module beside DNS, object storage and UniFi — so tofu apply provisions nothing and Step 2 then has no cluster to form. The document was half-updated: it knew k8s-1 was bare metal but still treated k8s-2 and k8s-3 as Proxmox VMs, which stopped being true at the neptune conversion.

That matters more than an ordinary stale doc, because it is the document someone reaches for when everything is gone — exactly when there is no capacity to discover that step one is fiction.

Step 1 splits into two unrelated halves: a tofu apply for the platform resources that do still exist, and physically booting three machines from install media.

Step 2 stops calling bootstrap.sh talos, which cannot generate a config (#581), and carries the procedure that actually works — the one RUNBOOK-baremetal-conversion.md uses. Both traps it hides are now stated: the whole cilium-cni.yaml cannot be passed as a patch, and the generated HostnameConfig collides with the per-node hostname. So is the two-stage apply — a config referencing an interface that does not exist yet hangs the install at STAGE: Installing with nothing timing out.

I scoped that claim rather than repeating "bootstrap.sh is dead" wholesale: the flux and wait-apps phases are still used in Steps 3, 5 and 6 and work. Only config generation is broken, and saying otherwise would have left the runbook contradicting itself three sections later.

Adds the constraint that decides whether this is attemptable remotely at all, at the top and again in known gaps: management is a tagged VLAN, a maintenance-mode node cannot configure VLANs, and MS-01 has no IPMI. Every node needs a keyboard and a monitor until #576 puts VLAN 40 native on the I226-V port. That was true before today and undocumented.

Also records what changed under the runbook recently: the schematic now carries i915, Ceph's cluster network is a Multus attachment rather than host networking, and OpenBao runs three replicas that each need unsealing individually — a sealed server is neither a voter nor a standby, so doing one leaves the cluster short of quorum.

Disk verification before install is now explicit. Talos enumerates NVMe in an order that is not stable across boots, the per-node configs select by serial for that reason, and the failure mode is the installer taking the Ceph OSD drive.

Closes #660. Step 1 was *"tofu: provision the VMs"* and there are no VMs. The OpenTofu stack has no node definitions left — only the uninstantiated `proxmox_vm` module beside DNS, object storage and UniFi — so `tofu apply` provisions nothing and Step 2 then has no cluster to form. The document was half-updated: it knew k8s-1 was bare metal but still treated k8s-2 and k8s-3 as Proxmox VMs, which stopped being true at the neptune conversion. That matters more than an ordinary stale doc, because it is the document someone reaches for when everything is gone — exactly when there is no capacity to discover that step one is fiction. **Step 1** splits into two unrelated halves: a `tofu apply` for the platform resources that do still exist, and physically booting three machines from install media. **Step 2** stops calling `bootstrap.sh talos`, which cannot generate a config (#581), and carries the procedure that actually works — the one `RUNBOOK-baremetal-conversion.md` uses. Both traps it hides are now stated: the whole `cilium-cni.yaml` cannot be passed as a patch, and the generated `HostnameConfig` collides with the per-node hostname. So is the two-stage apply — a config referencing an interface that does not exist yet hangs the install at `STAGE: Installing` with nothing timing out. I scoped that claim rather than repeating "bootstrap.sh is dead" wholesale: the `flux` and `wait-apps` phases are still used in Steps 3, 5 and 6 and work. Only config generation is broken, and saying otherwise would have left the runbook contradicting itself three sections later. **Adds the constraint that decides whether this is attemptable remotely at all**, at the top and again in known gaps: management is a tagged VLAN, a maintenance-mode node cannot configure VLANs, and MS-01 has no IPMI. Every node needs a keyboard and a monitor until #576 puts VLAN 40 native on the I226-V port. That was true before today and undocumented. **Also records what changed under the runbook recently:** the schematic now carries `i915`, Ceph's cluster network is a Multus attachment rather than host networking, and OpenBao runs three replicas that each need unsealing individually — a sealed server is neither a voter nor a standby, so doing one leaves the cluster short of quorum. **Disk verification before install is now explicit.** Talos enumerates NVMe in an order that is not stable across boots, the per-node configs select by serial for that reason, and the failure mode is the installer taking the Ceph OSD drive.

claude-bot added 1 commit 2026-08-27 08:10:20 +00:00

docs(runbook): rewrite the cold rebuild for bare metal

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

4678f73e9f

Step 1 was "tofu: provision the VMs" and there are no VMs. The OpenTofu stack
has no node definitions left — only the uninstantiated proxmox_vm module beside
DNS, object storage and UniFi — so `tofu apply` provisions nothing and Step 2
then has no cluster to form. The document was half-updated: it knew k8s-1 was
bare metal but still treated k8s-2 and k8s-3 as Proxmox VMs, which stopped
being true at the neptune conversion.

That matters more than an ordinary stale doc, because it is the document
someone reaches for when everything is gone — exactly when there is no capacity
to discover that step one is fiction.

Step 1 splits into two unrelated halves: a tofu apply for platform resources
that do still exist, and physically booting three machines from install media.

Step 2 stops calling `bootstrap.sh talos`, which cannot generate a config
(#581), and carries the procedure that actually works — the one
RUNBOOK-baremetal-conversion.md uses. Both traps it hides are now stated: the
whole cilium-cni.yaml cannot be passed as a patch, and the generated
HostnameConfig collides with the per-node hostname. So is the two-stage apply:
a config referencing an interface that does not exist yet hangs the install at
STAGE: Installing with nothing timing out.

Scoped that claim rather than repeating "bootstrap.sh is dead" wholesale — the
flux and wait-apps phases are still used in Steps 3, 5 and 6 and work. Only
config generation is broken, and saying otherwise would have left the runbook
contradicting itself three sections later.

Adds the constraint that decides whether this is even attemptable remotely, at
the top and again in known gaps: management is a tagged VLAN, a maintenance
mode node cannot configure VLANs, and MS-01 has no IPMI. Every node needs a
keyboard and a monitor until #576 puts VLAN 40 native on the I226-V port. That
was true before today and undocumented.

Also records what changed under the runbook recently: the schematic now carries
i915, Ceph's cluster network is a Multus attachment rather than host
networking, and OpenBao runs three replicas that each need unsealing
individually — a sealed server is neither a voter nor a standby, so doing one
leaves the cluster short of quorum.

Disk verification before install is now explicit. Talos enumerates NVMe in an
order that is not stable across boots, the per-node configs select by serial for
that reason, and the failure mode is the installer taking the Ceph OSD drive.

Closes #660.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit d1e072146f into master

2026-08-27 08:11:24 +00:00

k.renner referenced this issue from a commit

2026-08-27 08:11:25 +00:00

Merge pull request 'docs(runbook): rewrite the cold rebuild for bare metal' (#661) from runbook-cold-rebuild-baremetal into master

review-bot requested changes

2026-08-27 08:14:01 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 4678f73e9f8b673f8c016fb016fc4464fb4861d4.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This rewrites kubernetes/RUNBOOK-cold-rebuild.md to reflect that all three nodes are now bare-metal MS-01s (no Proxmox VMs), replaces the dead bootstrap.sh talos phase with the manual config-generation/two-stage-apply procedure that RUNBOOK-baremetal-conversion.md already documents, and adds several previously-undocumented constraints (VLAN-tagged management requiring physical access, OpenBao per-replica unsealing, Ceph's Multus cluster network, NVMe enumeration instability). I cross-checked the concrete claims against kubernetes/bootstrap.sh, kubernetes/talos/patches/*.yaml, kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml, kubernetes/infrastructure/openbao/helmrelease.yaml, and the opentofu/ stack — nearly everything checks out against the current code (the cilium-cni.yaml $patch: delete issue, the HostnameConfig collision, the ens19 VLAN-document stall, OpenBao's 3-replica raft, Ceph's Multus/VLAN-49 cluster network). There is one concrete factual error in the new checklist text, and one procedural gap in the new Step 2 that could stall an operator mid-rebuild.

🔴 Blocking

"No Proxmox credentials" is false; tofu apply still requires them. New text at kubernetes/RUNBOOK-cold-rebuild.md (Tooling checklist, ~line 35): "No Proxmox credentials — there is no Proxmox any more." But opentofu/providers.tf:42-52 unconditionally configures provider "proxmox" from var.proxmox_api_url and var.proxmox_token_secret, and opentofu/variables.tf declares both with no default (proxmox_token_secret is also sensitive, so tofu can't silently proceed with an empty string the way it might for an optional var). Since the provider block isn't gated behind any conditional/count and nothing was removed from providers.tf/variables.tf in this PR, tofu init/plan/apply will still prompt for — or, run non-interactively, fail on — these two variables even though no resource in the tree instantiates proxmox_vm any more. .woodpecker/deploy-infrastructure.yml still wires TF_VAR_proxmox_api_url/TF_VAR_proxmox_token_secret from secrets for exactly this reason. An operator following the new checklist literally would not have these values on hand for the local tofu apply this step tells them to run, and could get stuck on the very first command of the rebuild. Either the provider block needs to actually be removed (bigger change, likely out of scope here) or this checklist line needs to say the credentials are still required for tofu apply to succeed, even though nothing meaningful uses them.

🟡 Suggestions

Step 2's install-k8s-$N-stage1.yaml is never produced. kubernetes/RUNBOOK-cold-rebuild.md new "Generate the configs" loop (~line 138) only ever writes install-k8s-$N.yaml. The following "Apply in two stages" section (~line 174) then applies install-k8s-$N-stage1.yaml and says "the ens19 VLAN document must be removed from the first-stage config" — but doesn't show the command to actually produce that file (e.g. splitting the multi-doc YAML and dropping the ens19 VLANConfig/LinkConfig documents). RUNBOOK-baremetal-conversion.md's equivalent section (its "Install minimal, complete on the running node", ~line 239) is more explicit, spelling out exactly which document kinds belong in stage 1 (machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels, LinkConfig <uplink>, VLANConfig mgmt0, DHCPv4Config mgmt0, VolumeConfig EPHEMERAL). Given the PR's stated goal is removing exactly this kind of ambiguity for someone under pressure with no cluster, it's worth adding the equivalent explicit list or a one-line yq/manual-edit example here too.

🟢 Nits

The new "Install media" checklist bullet (~line 37) says "the Image Factory ISO named in Step 1" — the ISO link itself is actually in the new Step 1b, not 1a. Minor, since 1a/1b are adjacent and both under "Step 1", but a reader skimming for the URL could be sent to 1a first.

Tests

Documentation-only change (kubernetes/RUNBOOK-cold-rebuild.md); no tests are expected or applicable. The value of this PR is largely in whether its procedural claims match the actual repo state, which I checked directly against bootstrap.sh, the Talos patches, the Ceph/OpenBao manifests, and the OpenTofu stack (see Summary) rather than via any test suite.

Recommendation

Request changes — 1 blocking issue.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `4678f73e9f8b673f8c016fb016fc4464fb4861d4`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This rewrites `kubernetes/RUNBOOK-cold-rebuild.md` to reflect that all three nodes are now bare-metal MS-01s (no Proxmox VMs), replaces the dead `bootstrap.sh talos` phase with the manual config-generation/two-stage-apply procedure that `RUNBOOK-baremetal-conversion.md` already documents, and adds several previously-undocumented constraints (VLAN-tagged management requiring physical access, OpenBao per-replica unsealing, Ceph's Multus cluster network, NVMe enumeration instability). I cross-checked the concrete claims against `kubernetes/bootstrap.sh`, `kubernetes/talos/patches/*.yaml`, `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml`, `kubernetes/infrastructure/openbao/helmrelease.yaml`, and the `opentofu/` stack — nearly everything checks out against the current code (the `cilium-cni.yaml` `$patch: delete` issue, the `HostnameConfig` collision, the `ens19` VLAN-document stall, OpenBao's 3-replica raft, Ceph's Multus/VLAN-49 cluster network). There is one concrete factual error in the new checklist text, and one procedural gap in the new Step 2 that could stall an operator mid-rebuild. ## 🔴 Blocking - **"No Proxmox credentials" is false; `tofu apply` still requires them.** New text at `kubernetes/RUNBOOK-cold-rebuild.md` (Tooling checklist, ~line 35): *"No Proxmox credentials — there is no Proxmox any more."* But `opentofu/providers.tf:42-52` unconditionally configures `provider "proxmox"` from `var.proxmox_api_url` and `var.proxmox_token_secret`, and `opentofu/variables.tf` declares both with **no default** (`proxmox_token_secret` is also `sensitive`, so tofu can't silently proceed with an empty string the way it might for an optional var). Since the provider block isn't gated behind any conditional/count and nothing was removed from `providers.tf`/`variables.tf` in this PR, `tofu init`/`plan`/`apply` will still prompt for — or, run non-interactively, fail on — these two variables even though no resource in the tree instantiates `proxmox_vm` any more. `.woodpecker/deploy-infrastructure.yml` still wires `TF_VAR_proxmox_api_url`/`TF_VAR_proxmox_token_secret` from secrets for exactly this reason. An operator following the new checklist literally would not have these values on hand for the local `tofu apply` this step tells them to run, and could get stuck on the very first command of the rebuild. Either the provider block needs to actually be removed (bigger change, likely out of scope here) or this checklist line needs to say the credentials are still required for `tofu apply` to succeed, even though nothing meaningful uses them. ## 🟡 Suggestions - **Step 2's `install-k8s-$N-stage1.yaml` is never produced.** `kubernetes/RUNBOOK-cold-rebuild.md` new "Generate the configs" loop (~line 138) only ever writes `install-k8s-$N.yaml`. The following "Apply in two stages" section (~line 174) then applies `install-k8s-$N-stage1.yaml` and says "the `ens19` VLAN document must be removed from the first-stage config" — but doesn't show the command to actually produce that file (e.g. splitting the multi-doc YAML and dropping the `ens19` `VLANConfig`/`LinkConfig` documents). `RUNBOOK-baremetal-conversion.md`'s equivalent section (its "Install minimal, complete on the running node", ~line 239) is more explicit, spelling out exactly which document kinds belong in stage 1 (`machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels, LinkConfig <uplink>, VLANConfig mgmt0, DHCPv4Config mgmt0, VolumeConfig EPHEMERAL`). Given the PR's stated goal is removing exactly this kind of ambiguity for someone under pressure with no cluster, it's worth adding the equivalent explicit list or a one-line `yq`/manual-edit example here too. ## 🟢 Nits - The new "Install media" checklist bullet (~line 37) says "the Image Factory ISO named in Step 1" — the ISO link itself is actually in the new Step 1b, not 1a. Minor, since 1a/1b are adjacent and both under "Step 1", but a reader skimming for the URL could be sent to 1a first. ## Tests Documentation-only change (`kubernetes/RUNBOOK-cold-rebuild.md`); no tests are expected or applicable. The value of this PR is largely in whether its procedural claims match the actual repo state, which I checked directly against `bootstrap.sh`, the Talos patches, the Ceph/OpenBao manifests, and the OpenTofu stack (see Summary) rather than via any test suite. ## Recommendation Request changes — 1 blocking issue.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#661