konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

bootstrap.sh cannot generate a config: cilium-cni.yaml's $patch: delete is not a no-op #581

New Issue
Closed
opened 2026-08-25 19:14:46 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-08-25 19:14:46 +00:00
Member
Copy Link
Copy Source

kubernetes/bootstrap.sh would fail today on a cold rebuild. Found while building an install config for the jupiter bare-metal conversion.

The claim in the file

kubernetes/talos/patches/cilium-cni.yaml opens with:

Multi-document Talos config patch: delete + add. […] At bootstrap via talosctl gen config --config-patch @<file> the delete is a no-op against an empty initial config; the add inserts as usual. Same file therefore serves both paths.

What actually happens

It is not a no-op. With talosctl v1.12.6:

$ talosctl gen config homelab https://k8s.renner.uno:6443 \
    --config-patch @kubernetes/talos/patches/cilium-cni.yaml ...
failed to generate config bundle: error patching configs: patch delete:
path 'cluster.inlineManifests.[0]' in document '/v1alpha1':
failed to delete path 'cluster.inlineManifests.[0]': lookup failed

Same error from talosctl machineconfig patch. The delete document targets cluster.inlineManifests.[0], and against a base config that has no inlineManifests key at all the lookup fails rather than matching nothing.

Impact

Step 2 of RUNBOOK-cold-rebuild.md is kubernetes/bootstrap.sh talos, which passes this patch to every node via apply-config --config-patch. A cold rebuild is exactly the situation where the base config is empty, so this is broken on the path it was specifically written to support. It is not exercised in normal operation, which is why it has gone unnoticed.

Workaround used

Splitting the file and passing only the additive second document works, and is correct for a fresh install where there is nothing to delete.

Possible fixes

  • Guard the delete so it tolerates a missing path, if the patch format allows it
  • Split into two files and have bootstrap.sh (fresh install) use only the additive one, while live updates via talosctl patch mc use both
  • Verify against the current talosctl whether the "no-op at bootstrap" behaviour ever held, or whether the comment was aspirational

Worth verifying whichever fix is chosen by actually running talosctl gen config with it, since that is the step that fails.

🤖 Generated with Claude Code

`kubernetes/bootstrap.sh` would fail today on a cold rebuild. Found while building an install config for the jupiter bare-metal conversion. ## The claim in the file `kubernetes/talos/patches/cilium-cni.yaml` opens with: > Multi-document Talos config patch: delete + add. […] At bootstrap via `talosctl gen config --config-patch @<file>` the delete is a no-op against an empty initial config; the add inserts as usual. Same file therefore serves both paths. ## What actually happens It is not a no-op. With talosctl v1.12.6: ``` $ talosctl gen config homelab https://k8s.renner.uno:6443 \ --config-patch @kubernetes/talos/patches/cilium-cni.yaml ... failed to generate config bundle: error patching configs: patch delete: path 'cluster.inlineManifests.[0]' in document '/v1alpha1': failed to delete path 'cluster.inlineManifests.[0]': lookup failed ``` Same error from `talosctl machineconfig patch`. The delete document targets `cluster.inlineManifests.[0]`, and against a base config that has no `inlineManifests` key at all the lookup fails rather than matching nothing. ## Impact Step 2 of `RUNBOOK-cold-rebuild.md` is `kubernetes/bootstrap.sh talos`, which passes this patch to every node via `apply-config --config-patch`. A cold rebuild is exactly the situation where the base config is empty, so this is broken on the path it was specifically written to support. It is not exercised in normal operation, which is why it has gone unnoticed. ## Workaround used Splitting the file and passing only the additive second document works, and is correct for a fresh install where there is nothing to delete. ## Possible fixes - Guard the delete so it tolerates a missing path, if the patch format allows it - Split into two files and have `bootstrap.sh` (fresh install) use only the additive one, while live updates via `talosctl patch mc` use both - Verify against the current talosctl whether the "no-op at bootstrap" behaviour ever held, or whether the comment was aspirational Worth verifying whichever fix is chosen by actually running `talosctl gen config` with it, since that is the step that fails. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot commented 2026-08-27 07:54:48 +00:00
Author
Member
Copy Link
Copy Source

Still reproducible — the $patch: delete is still in cilium-cni.yaml.

A data point from today that narrows the scope rather than changing it: $patch: delete works correctly against a live multi-document config. Used it on all three nodes to remove the SAN addresses from a LinkConfig:

apiVersion: v1alpha1
kind: LinkConfig
name: enp2s0f1np1
addresses:
  - address: 10.99.0.1/29
    $patch: delete

talosctl patch machineconfig --dry-run showed exactly the intended removal and the apply behaved. So the mechanism is sound; the failure is specific to the bootstrap path, where the target document does not exist yet in a config being generated from empty.

Worth noting because it rules out "the $patch: delete syntax is broken" as the cause and points squarely at the asymmetry the issue already describes: a delete against something absent is not a no-op the way the file header claims.

Also relevant, since it is the same class of trap: JSON6902 patches are rejected outright on multi-document configs ("JSON6902 patches are not supported for multi-document machine configuration"), so strategic merge with $patch: delete is the only route for removals here — which makes fixing the bootstrap case more than a stylistic preference.

Still reproducible — the `$patch: delete` is still in cilium-cni.yaml. A data point from today that narrows the scope rather than changing it: `$patch: delete` works correctly against a *live* multi-document config. Used it on all three nodes to remove the SAN addresses from a LinkConfig: ```yaml apiVersion: v1alpha1 kind: LinkConfig name: enp2s0f1np1 addresses: - address: 10.99.0.1/29 $patch: delete ``` `talosctl patch machineconfig --dry-run` showed exactly the intended removal and the apply behaved. So the mechanism is sound; the failure is specific to the bootstrap path, where the target document does not exist yet in a config being generated from empty. Worth noting because it rules out "the `$patch: delete` syntax is broken" as the cause and points squarely at the asymmetry the issue already describes: a delete against something absent is not a no-op the way the file header claims. Also relevant, since it is the same class of trap: JSON6902 patches are rejected outright on multi-document configs ("JSON6902 patches are not supported for multi-document machine configuration"), so strategic merge with `$patch: delete` is the only route for removals here — which makes fixing the bootstrap case more than a stylistic preference.
claude-bot commented 2026-08-27 08:02:41 +00:00
Author
Member
Copy Link
Copy Source

Closing — bootstrap.sh is dead, so a bug in it is not worth fixing.

But looking for who still calls it turned up something worse. kubernetes/RUNBOOK-cold-rebuild.md still does, and its Step 1 is "tofu: provision the VMs + supporting infra" — recreating the Talos VMs on Proxmox.

There are no VM definitions left in the OpenTofu stack. Only the unused modules/proxmox_vm/ module remains alongside DNS, object storage and UniFi, so tofu apply would provision no nodes at all.

The runbook is half-updated: it knows k8s-1 is bare metal ("k8s-1 is not a VM and OpenTofu does not create it") but still treats k8s-2 and k8s-3 as Proxmox VMs. All three have been bare metal since the neptune conversion, and Proxmox is gone.

That makes the disaster-recovery procedure unusable as written — Step 1 produces nothing, and Step 2 then has no cluster to talk to. Filed separately; this issue is about a script that no longer matters.

Closing — bootstrap.sh is dead, so a bug in it is not worth fixing. But looking for who still calls it turned up something worse. `kubernetes/RUNBOOK-cold-rebuild.md` still does, and its Step 1 is *"tofu: provision the VMs + supporting infra"* — recreating the Talos VMs on Proxmox. There are no VM definitions left in the OpenTofu stack. Only the unused `modules/proxmox_vm/` module remains alongside DNS, object storage and UniFi, so `tofu apply` would provision no nodes at all. The runbook is half-updated: it knows k8s-1 is bare metal ("k8s-1 is not a VM and OpenTofu does not create it") but still treats k8s-2 and k8s-3 as Proxmox VMs. All three have been bare metal since the neptune conversion, and Proxmox is gone. That makes the disaster-recovery procedure unusable as written — Step 1 produces nothing, and Step 2 then has no cluster to talk to. Filed separately; this issue is about a script that no longer matters.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#581