chore(ci): drop the Talos image seeding step

The step pushed a nocloud Talos image onto each Proxmox node so the k8s VMs
could boot from it. No Proxmox node remains — neptune was the last one, and all
three control-plane nodes now boot the metal image from their own installed
disk.

Left in place the step would call the Proxmox storage API for a host that no
longer answers it, then sit through its 60x10s download-wait before giving up,
on every infrastructure apply.

The block's own comment already scheduled its removal: "the whole step goes away
once neptune is converted too."

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-infrastructure.yml

@@ -128,42 +128,6 @@ steps:
       - printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
       - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
       - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
-      - |
-        # Ensure Talos image exists on all Proxmox nodes via API.
-        #
-        # Only nodes still running Proxmox belong in this list. jupiter and
-        # saturn run Talos on bare metal now, so the storage API returns
-        # nothing for them and the download-wait loop below would burn its full
-        # 60x10s per node before giving up. Bare-metal nodes boot the
-        # metal-amd64 image from their own installed disk instead, and the
-        # whole step goes away once neptune is converted too.
-        TALOS_VERSION="v1.12.6"
-        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
-        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
-        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
-        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in neptune; do
-          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
-            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
-          if [ "$EXISTS" -gt 0 ]; then
-            echo "Talos image already exists on $node"
-            continue
-          fi
-          echo "Downloading Talos image to $node..."
-          curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/download-url" \
-            -H "Authorization: PVEAPIToken=$API_TOKEN" \
-            --data-urlencode "url=$TALOS_URL" \
-            --data-urlencode "content=import" \
-            --data-urlencode "filename=$TALOS_FILENAME" \
-            --max-time 600
-          echo "Waiting for download on $node..."
-          for i in $(seq 1 60); do
-            sleep 10
-            EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
-              -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
-            if [ "$EXISTS" -gt 0 ]; then echo "Download complete on $node"; break; fi
-          done
-        done
       - tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
       - |
         if [ -n "$$TOFU_STATE_RM" ]; then

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -389,7 +389,23 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
   cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
   `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
+- Cilium's eBPF datapath drops VLAN-tagged traffic unless `bpf.vlanBypass` is
+  set. **This happened** — every v30iot macvlan workload on a converted node
+  stayed in sandbox creation until `vlanBypass: [30]` was added (issue #592).
+
+  Only `ens19` is affected, and the reason is worth knowing: Cilium attaches to
+  devices that carry an address. `cilium-dbg status --verbose` lists `mgmt0`
+  among its devices but not `ens19`, because `ens19` is deliberately
+  address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
+  untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
+  on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
+  VLAN carrying pod traffic will need the same treatment.
+
+  The symptom does not look like a VLAN problem. A tc-layer drop reaches an
+  AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space
+  available` and `network is down` rather than a DHCP timeout, and the pod sits
+  in `ContainerCreating` behind `plugin type="macvlan" failed (add): error
+  calling DHCP.Allocate: no more tries`. Confirm it with
+  `cilium_drop_count_total{reason=~"VLAN.*"}` per node rather than by reading
+  the CNI logs, and check `node_network_receive_packets_total{device="ens19"}`
+  — it reads 0 on an affected node.

kubernetes/apps/music-assistant/deployment.yaml

@@ -25,8 +25,11 @@ spec:
         # replaces the former hostNetwork attachment (see
         # network-attachment.yaml) so the NODE no longer holds an address
         # on the compromised IoT VLAN; MA gets its own leased IoT IP on
-        # net1 instead. master: ens19 is the v30iot NIC on every k8s-{1,2,3}/edge
-        # worker.
+        # net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
+        # worker, though it is no longer the same kind of device everywhere:
+        # on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
+        # of enp2s0f0np0 rather than a NIC, which is why the datapath needs
+        # bpf.vlanBypass to carry it (issue #592).
         k8s.v1.cni.cncf.io/networks: music-assistant-iot
     spec:
       enableServiceLinks: false

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -73,6 +73,36 @@ spec:
       # kubernetes/talos/cilium.yaml carries a third copy of the same key.
       # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
       policyMapMax: 65536
+      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
+      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
+      # filter" in cilium_drop_count_total.
+      #
+      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
+      # otbr and music-assistant attachments. On bare metal it is a VLAN
+      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
+      # an untagged virtio NIC, which is why this was not needed before.
+      #
+      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
+      # interesting part. Cilium attaches to devices that carry an address —
+      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
+      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
+      # a device and sees frames the 8021q layer has already untagged; nothing
+      # for a VLAN filter to match. ens19 is deliberately address-less (see
+      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
+      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
+      # meet the filter on the shared parent, still tagged.
+      #
+      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
+      # would touch the datapath carrying etcd and the API for no reason.
+      #
+      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
+      # symptom is the CNI dhcp plugin reporting "no buffer space available"
+      # rather than a DHCP timeout. See issue #592.
+      #
+      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
+      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
+      # must move with this list. Takes effect on agent restart.
+      vlanBypass: [30]
       # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
       # redirects the socket directly to the backend pod instead of sending
       # traffic out to the VIP and back (which fails with L2 announcements).

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,40 +167,44 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # nvme0n1 on k8s-3 is the NVMe drive PCI-passed-through from neptune, and is
-  # the only block device that guest can see, so a bare device name is
-  # unambiguous there.
+  # All three nodes are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
+  # Talos enumerates these machines' drives in a different order than Proxmox
+  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
+  # depending on the boot. Rook would probably decline the system disk for
+  # carrying partitions, but that is luck rather than intent. The by-id path is
+  # derived from model and serial and does not depend on enumeration at all.
   #
-  # k8s-1 and k8s-2 are bare metal and see three NVMe drives each, so they name
-  # their OSDs by /dev/disk/by-id path instead. A bare name would be actively
-  # wrong: Talos enumerates these machines' drives in a different order than
-  # Proxmox did, and nvme0n1 there is the Kingston SYSTEM disk. Rook would
-  # probably decline it for carrying partitions, but that is luck rather than
-  # intent. The by-id path is derived from model and serial and does not depend
-  # on enumeration at all.
+  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
+  # and the Crucial as nvme0n1 on the very next one.
   #
   #   k8s-1 (jupiter)
-  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  #   50026B7382B48B86  Kingston  — Talos system disk
+  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M400426    PM983     — this OSD
   #
   #   k8s-2 (saturn)
-  #   nvme0n1  50026B7382DA5F6E  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M403161    PM983     — this OSD
+  #   50026B7382DA5F6E  Kingston  — Talos system disk
+  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M403161    PM983     — this OSD
+  #
+  #   k8s-3 (neptune)
+  #   50026B7382B48B90  Kingston  — Talos system disk
+  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NF0M501176    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
       - name: k8s-2
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
+      - name: k8s-3
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
-  # every drive in the cluster: per-drive write volume equalled total cluster
-  # write volume, and all three drives wore at an identical rate and would have
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
-  # wear-out here is correlated by construction rather than independent.
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
+  # volume equalled total cluster write volume and all three wore at an
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that, because the wear was
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
+  # override existed only because Ceph does not lower min_size when size is
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
-  parameters:
-    min_size: "1"
+    size: 3
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -87,39 +80,26 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
-  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
-  # three conversions rather than being toggled per node: each change
-  # re-replicates the pool, and the window reopens with the next node anyway.
+  # size 3 with no min_size override. This pool holds ~253 MiB against the data
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
+  # than just the objects on the lost OSD, so it is the last pool that should
+  # ever run thin.
   #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out. size 3 on two remaining
-  # hosts leaves every PG undersized at two copies with min_size 2, so a single
-  # further OSD loss drops the pool below min_size and blocks IO on every
-  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
-  # instead, and a further loss degrades rather than blocks — availability
-  # bought at the cost of accepting writes onto a single copy while degraded.
-  #
-  # See RUNBOOK-baremetal-conversion.md.
+  # It drops to 2/1 whenever one of the three hosts is out for longer than a
+  # reboot. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss falls below min_size
+  # and blocks IO on every CephFS mount cluster-wide. At 2/1 the same two
+  # copies are active+clean instead, and a further loss degrades rather than
+  # blocks. See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
-      # the comment there for the full rationale and the accepted trade.
+      # size 3 for the same reasons as replicapool above — see the comment
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
-      parameters:
-        min_size: "1"
+        size: 3
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/talos/cilium.yaml

@@ -310,6 +310,13 @@ data:
   # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
   # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.
   bpf-policy-map-max: "65536"
+  # VLAN ids the eBPF datapath may pass; without this it drops VLAN-tagged
+  # frames. 30 is v30iot, the macvlan parent for the IoT workloads, and a VLAN
+  # sub-interface of enp2s0f0np0 on bare metal. VLAN 40 (mgmt0) rides the same
+  # parent but passes without an entry, so it is deliberately not listed.
+  # Keep in lockstep with patches/cilium-cni.yaml and values.bpf.vlanBypass in
+  # kubernetes/infrastructure/cilium/helmrelease.yaml. See issue #592.
+  vlan-bpf-bypass: "30"
   # bpf-policy-stats-map-max specifies the maximum number of entries in global
   # policy stats map
   bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/cilium-cni.yaml

@@ -331,6 +331,16 @@ cluster:
           # inline manifest, so a lower value here means a cold rebuild comes up with
           # a map that overflows until Flux reconciles.
           bpf-policy-map-max: "65536"
+          # VLAN ids the eBPF datapath may pass; without this it drops
+          # VLAN-tagged frames. 30 is v30iot, the macvlan parent for the IoT
+          # workloads, and a VLAN sub-interface of enp2s0f0np0 on bare metal.
+          # VLAN 40 (mgmt0) rides the same parent but passes without an entry,
+          # so it is deliberately not listed. Keep in lockstep with
+          # values.bpf.vlanBypass in kubernetes/infrastructure/cilium/helmrelease.yaml
+          # and the third copy in kubernetes/talos/cilium.yaml — bootstrap installs
+          # from this inline manifest, so omitting it here breaks the IoT macvlan
+          # workloads until Flux reconciles. See issue #592.
+          vlan-bpf-bypass: "30"
           # bpf-policy-stats-map-max specifies the maximum number of entries in global
           # policy stats map
           bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/k8s-2.yaml

@@ -1,7 +1,6 @@
 # k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
 # to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
-# VM-era layout until neptune is converted.
+# serials, MACs and addresses differ.
 machine:
   network:
     hostname: k8s-2

kubernetes/talos/patches/k8s-3.yaml

@@ -1,31 +1,82 @@
+# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials and addresses differ. neptune was the last node converted, so all
+# three control-plane nodes now share this layout and no Proxmox remains.
 machine:
   network:
     hostname: k8s-3
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.3/29
-        mtu: 9000
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382B48B90
   certSANs:
     - k8s-3.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: neptune
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 83% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. Available spare is still 100% and it reports no media errors.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2326E6E8E67D'
+  minSize: 50GiB
+  grow: true
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, held by whichever node wins the etcd campaign.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# v30iot: link only, no host addressing — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.3/29