The play generated secrets with community.general.random_string. The CI image
(ci/images/ansible/Containerfile) installs only community.proxmox and
community.docker, so the play failed at its first task when run through the
project's own tooling. It resolved on a workstation that happened to have the
collection, which is exactly why verifying it there proved nothing.
Replaced with ansible.builtin.password, which needs no collection and produces
the same thing: 48 alphanumerics, generated without persisting (path /dev/null).
Also hardens the shadow-key guard. It piped `ansible-vault view` into `grep -c`
with failed_when: false, so a failed decrypt — wrong password, moved file —
sent empty input to grep, which reported zero matches. Indistinguishable from a
clean vault, meaning the guard would have waved the rotation through precisely
when it knew least. A preceding task now proves the vault is readable, and the
assertion requires both that proof and the zero count.
Corrects the openbao rotation path in the header comment; it is under
kubernetes/.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.woodpecker/update-all.yml ran `ansible-playbook -i inventory/ plays/update.yml`
on a cron. This branch deletes that playbook, because every playbook it
imported targeted Proxmox hosts or guests that no longer exist, so the next
scheduled run would have failed outright.
Removing the pipeline rather than repointing it: there is nothing left for it
to update. Its imports were update-vm, update-lxc, update-authentik,
install-n8n, install-grist, install-music-assistant, update-docker and
update-pve-rolling — all removed here, all targeting hosts that no longer
resolve.
Swept every pipeline for references to deleted plays. The five still named
(install-claude-workstation, install-garage-synology, install-netbird-client,
install-netbird-proxy, install-netbird-server) all survive.
Also from the review:
common.yml `all:!pbs` was a no-op I introduced when simplifying the
old exclusion list — no pbs group exists. Now plain
`all`.
qemu-guest-agent dropped the `pve_cluster`/`proxmox_nodes`/
`proxmox_all_lxc` group guards; those groups are gone,
so the conditions were always true.
Caught by the review-bot on #625.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/<group>/ files merge alphabetically and the last one wins.
netbird-rotatable.yml sorts before vault.yml, which still defines
vault_netbird_turn_secret, vault_netbird_relay_secret and
vault_netbird_proxy_token — verified, all three are present.
The rotation itself would have worked, since the plays pass the new values
explicitly and play vars outrank group vars. The damage came later: the next
ordinary run of install-netbird-server.yml or install-netbird-proxy.yml would
read the stale value from vault.yml and push it back out, breaking
server<->proxy auth. Those deploys run unattended on changes to the roles, so
it would have surfaced on something unrelated to secrets.
Adds a pre-flight check that counts — never prints — matching keys in vault.yml
and refuses to rotate while any remain, plus the one-time migration step in the
header.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotates the three shared secrets where Ansible controls both ends, so the two
sides cannot disagree:
vault_netbird_turn_secret turnserver.conf (coturn) + management.json, one
role, one host
vault_netbird_relay_secret management.json on the server + relay.env on the
proxies
vault_netbird_proxy_token proxy.env on both
Opt-in, because rotation is not idempotent and briefly interrupts the mesh:
ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
New values are written to a dedicated encrypted file rather than edited into
the main vault. The plaintext is rendered at 0600 with a .new suffix, encrypted,
and only then moved into place, so a failure at any step cannot leave readable
secrets where the inventory would load them. The play asserts the installed
file carries the $ANSIBLE_VAULT header before proceeding.
Two are deliberately excluded and the play says why:
vault_netbird_datastore_key encrypts the datastore at rest; changing it
without dump/restore makes existing data
unreadable
vault_netbird_auth_client_secret its other half is in Authentik, so rotating
only the consumer breaks login
These are Ansible's own credentials, not copies of the cluster's. OpenBao holds
separate credentials for some of the same systems, deliberately different values
— one consumer, one credential, independently revocable. This play does not
touch OpenBao and is not meant to.
Proxies roll one at a time so a relay stays reachable. The server is
reconfigured first, so there is a short window where a not-yet-updated proxy
fails to authenticate; clients reconnect and peer-to-peer tunnels are not torn
down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing here had a target left. Every removal is backed by a probe rather than
an assumption:
pve_cluster (jupiter/neptune/saturn) run Talos; no SSH at all
authentik-db-1/2, authentik-app-1/2 NXDOMAIN
grafana.srv, vault.srv, paperless.srv gone (grafana/paperless resolve but
answer neither ping nor SSH)
n8n.srv, grist.srv, wiki.srv NXDOMAIN
owncloud.srv, ical-proxy.srv, loki.srv NXDOMAIN
proxmox_* dynamic groups api.proxmox.yaml queried
jupiter.renner.uno:8006
24 plays, 14 roles and the Proxmox dynamic inventory go. update.yml goes with
them: every playbook it imported is in that list, so the aggregate updater had
no fleet left to update.
Kept, because their hosts answer: netbird server and proxies, nas-1, edge-1 and
its capability groups, claude-ws. common.yml and install-node-exporter.yml lose
exclusion clauses naming groups that no longer exist.
NTP IS NO LONGER HIGHLY AVAILABLE, and this change records that rather than
causing it. ntp_cluster was jupiter (MASTER, prio 100), saturn (90), neptune
(80) and edge-1 (BACKUP, 70); the conversion took the first three. A single
VRRP member holds the address regardless of configured state, so fd40::123 is
still served — verified answering — but by one thin client with nothing behind
it. install-ntp-ha.yml still carries the four-host priority map, now describing
hosts that do not exist. Restoring redundancy needs a second chrony host that
is not a Talos node.
The gatus role is removed entirely, which supersedes the narrower edit in the
DNS cleanup that dropped its Proxmox DNS check.
Left alone: plays/archived/ still references pve_cluster, which is what an
archive is for. paperless-migration stays because an archived play uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The removal has been applied:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed, 1 forgotten.
`tofu state list` now returns no Proxmox resources at all, and a full plan with
refresh enabled produces no Proxmox errors — the "no route to host" against
10.2.40.201:8006 that failed every infrastructure plan since the conversion is
gone.
The block was always one-shot. Left in place it refers to a state entry that no
longer exists, which is harmless but misleading: it reads as pending work.
Worth recording for the next time this shape comes up, because it cost a failed
merge to learn. A `removed` block alone does not fix an unreachable provider:
OpenTofu still refreshes the resource before it can forget it, and that refresh
is exactly what cannot succeed. It takes one run with `-refresh=false` to
process. `-target` does not help — targeting does not reach `removed` blocks, so
a targeted plan reports "No changes" while the state entry sits there untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Proxmox DNS HA VIP (10.2.40.53) was decommissioned after it dead-ended
non-Ingress resolution and broke ~1374 Home Assistant entities on a pod restart
(2026-07-14). With Proxmox itself now gone, the implementation and everything
pointing at it are dead weight — and one piece was actively dangerous.
Removed:
dns-server/ the Go implementation, its systemd unit,
and a 7.4 MB checked-in binary
opentofu/scripts/remove_unifi_clients.sh
the "Proxmox DNS (VIP)" Gatus check in the ansible role
remove_unifi_clients.sh is the one worth calling out. It instructs the reader
to deploy module.proxmox_dns and point UniFi DNS at 10.2.40.53 — precisely the
configuration that caused the outage. It also works by `tofu state rm`, which
orphans resources rather than removing them. It was already marked deprecated
in its own header; anyone who ignored that and followed it would have
reintroduced a known failure.
The Gatus check queried k8s-1.srv.renner.uno against 10.2.40.53 every 30s. It
belongs to the ansible gatus role, which targets a Proxmox LXC that no longer
exists, so nothing was running it — but the check itself asserts the decommis-
sioned server should answer, which is the opposite of the intended end state.
Deliberately kept: the three comments in bootstrap.sh, coredns-config and
k8s-gateway that record 10.2.40.53 as DECOMMISSIONED and say not to reintroduce
it. They carry the outage date and blast radius, and are the reason this keeps
not happening again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every infrastructure plan has failed since neptune was wiped. module.k8s_3 still
declared a Proxmox VM, so the bpg/proxmox provider tried to reach a hypervisor
that no longer exists:
Error: failed to get resources list of type ("vm") for cluster:
dial tcp 10.2.40.201:8006: connect: no route to host
with module.k8s_3.proxmox_virtual_environment_vm.this
Deleting vm_152_k8s_3.tf alone does not fix that. Removing a resource from
configuration makes OpenTofu plan a destroy, and planning a destroy refreshes
the resource through the same unreachable provider.
A `removed` block with `lifecycle { destroy = false }` forgets the state entry
without contacting Proxmox. Nothing is orphaned: the guest, its disks and the
hypervisor underneath were all destroyed during the conversion. This is the
declarative equivalent of a state removal, reviewable in git rather than run by
hand against production state.
talos_image.tf goes with it. Its locals named the nocloud image for VM import
and had no consumer other than the VM module.
Deliberately kept for a follow-up: the proxmox provider block, the
proxmox_vm/proxmox_lxc/bpg_proxmox_container modules and sync_state_from_pve.py.
A provider with no resources is never contacted, so leaving it costs nothing,
and removing it should wait until a clean plan proves state holds no other
Proxmox resources. That check cannot be made from here — the S3 credentials
used for local runs have been revoked.
Verified with `tofu validate` and `tofu fmt -check` offline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
machine.install.image was never set, so every node installed from the stock
ghcr.io/siderolabs/installer that talosctl gen config defaults to. Extensions
live in the installer image, which makes the schematic a property of the boot
media alone — nothing lands on disk. talosctl get extensions returns empty on
all three nodes.
Two extensions are missing as a result, and the cold-rebuild runbook explains
why each was chosen for bare metal:
siderolabs/intel-ucode CPU microcode. On a VM the hypervisor supplied it;
on bare metal nothing does except the BIOS.
siderolabs/thunderbolt Inert until machine.kernel.modules asks for it.
Present so enabling the MS-01 mesh later needs
neither a new image nor a reinstall — precisely the
property installing without it destroys.
Set in the shared cluster patch rather than per node: all three are bare metal
now and want the same image. Schematic a21cec8f is the bare-metal one from
RUNBOOK-cold-rebuild.md — same net.ifnames=1 as the Proxmox schematic, minus
qemu-guest-agent.
Config-only. install.image is read at install and upgrade time, so no running
node changes until it is upgraded; the three nodes stay without extensions
until a deliberate rolling upgrade is scheduled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cilium reads datapath options once, at agent start, and the chart puts no
config checksum on the pod template. A changed ConfigMap therefore leaves every
agent running the previous datapath until something restarts it, and nothing
does — the DaemonSet is not rolled by a ConfigMap-only Helm change.
This already cost an outage. vlan-bpf-bypass landed in cilium-config and every
v30iot macvlan workload stayed broken until the agents were deleted by hand,
one node at a time (issue #592). The merge looked successful: HelmRelease
Ready, release upgraded, key present in the ConfigMap, and nothing working.
policyMapMax carries the same exposure. Its comment says "takes effect on agent
restart", which is documentation standing in for automation — as does the
equivalent note in the two bootstrap copies.
Reloader is already deployed and used across the repo for the same purpose.
Named per-ConfigMap rather than blanket, matching the existing convention.
The trade, stated plainly: any cilium-config change now rolls the CNI
cluster-wide, node by node. That is the same disruption a datapath change
already required; the difference is that it happens on its own rather than
being forgotten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step pushed a nocloud Talos image onto each Proxmox node so the k8s VMs
could boot from it. No Proxmox node remains — neptune was the last one, and all
three control-plane nodes now boot the metal image from their own installed
disk.
Left in place the step would call the Proxmox storage API for a host that no
longer answers it, then sit through its 60x10s download-wait before giving up,
on every infrastructure apply.
The block's own comment already scheduled its removal: "the whole step goes away
once neptune is converted too."
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every v30iot macvlan workload on a bare-metal node has been stuck in sandbox
creation since its node was converted. Cilium's datapath drops VLAN-tagged
frames unless bpf.vlanBypass lists the tag, and cilium-config carried no
vlan-bpf-bypass key at all.
Inside the Proxmox guests ens19 was an untagged virtio NIC on the v30iot
bridge, so nothing was tagged by the time Cilium saw it. On bare metal ens19 is
a VLAN 30 sub-interface of enp2s0f0np0, and the macvlan children's DHCP
exchange never reaches the wire.
Measured rather than inferred: cilium_drop_count_total on the affected node
reports ~123k INGRESS and ~1.1k EGRESS drops with reason "VLAN traffic
disallowed by VLAN filter", against none on the node that was still a VM, and
node_network_receive_packets_total for ens19 reads 0 there against 10 pkt/s on
the VM. Full analysis in issue #592.
The symptom hides its own cause. A tc-layer drop surfaces to an AF_PACKET
sender as ENOBUFS, so the CNI dhcp plugin reports "no buffer space available"
and "network is down" instead of a DHCP timeout.
VLAN 40 is deliberately not listed. mgmt0 rides the same parent and carries the
node IP, etcd and the API, and it passes today without a bypass entry, so
adding it would change a working control-plane datapath to fix an IoT one.
All three copies of the key move together — the HelmRelease, kubernetes/talos/
cilium.yaml, and the inline bootstrap manifest in patches/cilium-cni.yaml — so
a cold rebuild does not come up without it.
The runbook predicted this exact failure and is updated to record that it
happened, along with how to recognise it from the drop counters rather than the
misleading CNI error. music-assistant's comment claiming ens19 is "the v30iot
NIC" on every node is corrected; it is a NIC on the edge worker and a VLAN
sub-interface on the control-plane nodes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.
The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.
The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.
cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.
cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three nodes are converted and the cluster is back to three OSDs across
three zones with 217 PGs active+clean, so the conversion window that justified
2/1 on this pool is over.
The min_size override goes with it. It existed only because Ceph does not
lower min_size when size is reduced, so the pool would have been 2/2 while a
host was out. At size 3 the default min_size of 2 is what the pool wants.
The comment loses its TEMPORARY banner and now states the standing rationale
and the condition under which the pool degrades to two copies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is converted and back in the cluster, so k8s-3 returns to the storage
fleet on its new PM983. All three nodes now name their OSD by /dev/disk/by-id,
and the storage fleet is whole again.
The comment generalises to all three nodes and records neptune's serials.
k8s-3 is the clearest evidence for by-id addressing: it enumerated the Kingston
as nvme0n1 on one boot and the Crucial as nvme0n1 on the very next one, so a
kernel-name reference had no stable meaning on that machine at all.
The old osd.1 lived on the Crucial that is now EPHEMERAL. That drive was wiped
during the conversion, so the OSD is unrecoverable and is purged from the CRUSH
map separately; this change only creates its replacement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune now runs Talos directly, so k8s-3 moves from the VM-era layout to the
multi-document form k8s-1 and k8s-2 already use. The old config selected
interfaces by busPath against virtio NICs and installed to /dev/vda; neither
exists on bare metal.
Disks are selected by serial rather than kernel name. That is not defensive
styling: Talos enumerated this machine's drives differently than Proxmox did,
and differently again between two consecutive boots — nvme0n1 was the Kingston
on one boot and the Crucial on the next. A device-name reference would have
installed onto the wrong drive.
nvme 50026B7382B48B90 Kingston system disk
nvme 2326E6E8E67D Crucial EPHEMERAL, 83% of rated endurance
nvme S437NF0M501176 PM983 OSD, 3% used
Serials, interface names and drive health were read from the machine itself —
the Kingston and Crucial off the running Proxmox host before the wipe, and
confirmed again from Talos maintenance mode.
k8s-2.yaml loses the sentence claiming k8s-3 is still a Proxmox VM. No Proxmox
remains in the homelab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is the last node to convert to bare metal, and its OSD drive is swapped
in the same window. Removing k8s-3 from the explicit node list first means Rook
stops managing that OSD before the drive it lives on goes away, rather than
recreating it mid-reconcile and backfilling onto a disk that is about to be
pulled.
The device comment for k8s-3 goes with it: it described the PCI-passed-through
drive that node saw as a Proxmox guest, which stops being true here.
Ceph runs on two OSDs and size-3 pools stay degraded until the node rejoins.
cephfs-metadata is already at 2/1 for exactly this window.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is back on bare metal with the PM983 in place, so k8s-2 returns to the
storage fleet. Like k8s-1 it names the device by /dev/disk/by-id: the machine
has three NVMe drives and Talos enumerates them differently than Proxmox did,
so nvme0n1 there is the Kingston system disk.
The runbook prescribed applying the full per-node config in maintenance mode,
which hangs the install. A document referencing an interface that does not
exist yet leaves the node at STAGE: Installing with KUBELET n/a, the console
repeating
k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
which reads as a Kubernetes fault but is an install that never completed. On
saturn it was the ens19 VLAN document and the sysctl naming it. Three cold
installs hung; the identical config applied cleanly to the running node.
Wiping disks changes nothing.
Install now runs in two stages: a minimal network config plus EPHEMERAL, which
is one-shot and cannot be deferred, then the full config applied to the running
node behind --mode=try so a bad network config reverts itself.
Also corrects the claim that maintenance mode cannot be inspected. talosctl get
works there; --insecure is a subcommand flag and has to follow `get`, otherwise
cobra resolves the legacy `talosctl disks` command and rejects the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VM 605 is pinned to saturn, so it cannot survive the conversion, and it has to
go before the node leaves the Proxmox cluster — afterwards the provider cannot
reach it and every plan errors.
Nothing uses it. Its purpose was cloning new Proxmox VMs, Proxmox is being
retired, and the one remaining VM definition sets template_name = null
explicitly rather than cloning from it.
The module default is changed to null for the same reason: it named a template
that no longer exists, which would only surface as an apply-time failure for
whoever created the next VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is being converted to bare metal, so its VM definition goes and it
leaves the list of Proxmox nodes the pipeline seeds the nocloud image onto.
The apply must run while saturn is still a Proxmox cluster member. Once it
leaves, the provider cannot reach the node, the plan errors, and the only way
out is tofu state rm.
module.k8s_2 is referenced nowhere else in opentofu/, so this destroys exactly
one resource.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Structurally identical to k8s-1.yaml; only serials, MACs and addresses differ.
Written from saturn's live hardware rather than assumed: the X710 port 1 MAC
came from the guest that currently owns it, since the host cannot see a
passed-through NIC.
Kingston 50026B7382DA5F6E system disk
Crucial 2332E86B74BC EPHEMERAL, retiring from osd.0 at 81% wear
X710 p0 58:47:ca:76:6f:73 trunk, VLAN 40 + VLAN 30
X710 p1 58:47:ca:76:6f:74 Ceph cluster network, 10.99.0.2/29
Links are addressed by kernel name rather than LinkAliasConfig, and both disks
by serial. Talos and Proxmox do not agree on NVMe enumeration order, and a
network document that fails to resolve takes the whole machine's DHCP with it.
Not applied by anything on merge — talos patches are applied by hand. Landing
it before the conversion so the file under review is the file that gets
installed.
Verified with talosctl validate --mode metal against v1.12.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the node from storage.nodes before purging its OSD is what makes the
purge stick. Under useAllNodes the operator rebuilt a purged OSD within one
reconcile and began backfilling onto the drive being replaced; listing nodes
explicitly exists so a node can be taken out first.
Re-add after the conversion naming the device by /dev/disk/by-id, as k8s-1
does. A bare device name is not safe on bare metal: Talos enumerates the three
NVMe drives in a different order than Proxmox, and nvme0n1 is the system disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pvecm expected 1 is the official recovery, documented verbatim in the storage
replication chapter of the Proxmox admin guide. Its caveat is about restoring
the missing node instead; here the missing node is deliberately gone for good,
so enforcing quorum is the correct branch rather than a workaround.
Also records that it must be run only once the node is alone -- corosync
rejects a value below the members it can see, which is why the attempt on the
current two-node cluster returned CS_ERR_INVALID_PARAM -- and that it is
runtime only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.
cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.
The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written from doing jupiter, with saturn and neptune still to go. The conversion
itself was straightforward; every failure came from state that outlived the VM,
so the runbook is weighted towards those rather than towards the happy path.
The three that cost real time:
A rebuilt node gets a new MAC and a new lease, and Ceph keeps its mon pinned to
the old address in the monmap, the mon-endpoints ConfigMap and the deployment's
--public-addr. The mon then cannot bind, the operator's next step wants to
restart a healthy mon, Ceph refuses because that would drop quorum, and it
retries every minute forever without reaching the storage reconcile. Fixing the
ConfigMap before the monmap and the deployment matters, or the operator
recreates the mon from the stale entry.
Talos stops running default DHCP for the whole machine as soon as one new-style
network document exists, and a LinkAliasConfig selector that matches nothing is
skipped silently. Together those turn a small config mistake into a node with no
address on any interface.
EPHEMERAL placement is honoured only on first provision, so it has to be right
in the first apply-config; afterwards it is ignored and relocating needs a wipe.
Also records the Proxmox quorum step that starts mattering at the second
conversion, the Secure Boot blocker, and that Talos and Proxmox enumerate the
same three NVMe drives in different orders — which is why the OSD is named by
by-id path rather than device name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is back as a bare-metal Talos node, so its OSD returns to the storage
fleet on the new drive.
The device is named by /dev/disk/by-id rather than by bare device name, because
the bare name would be wrong here. Talos enumerates this machine's three NVMe
drives in a different order than Proxmox did, and nvme0n1 on k8s-1 is the
Kingston system disk:
nvme0n1 50026B7382B48B86 Kingston Talos system disk
nvme1n1 2332E86B749A Crucial Talos EPHEMERAL (/var)
nvme2n1 S437NX0M400426 PM983 this OSD
Rook would most likely have declined the system disk for carrying partitions,
but relying on that is luck rather than intent. The by-id path derives from
model and serial and does not depend on enumeration order.
k8s-2 and k8s-3 keep bare names: each guest sees exactly one block device,
the drive passed through from its host, so there is nothing to disambiguate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
k.renner
claude-botandClaude Opus 5