feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-infrastructure.yml

@@ -128,6 +128,42 @@ steps:
       - printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
       - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
       - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
+      - |
+        # Ensure Talos image exists on all Proxmox nodes via API.
+        #
+        # Only nodes still running Proxmox belong in this list. jupiter and
+        # saturn run Talos on bare metal now, so the storage API returns
+        # nothing for them and the download-wait loop below would burn its full
+        # 60x10s per node before giving up. Bare-metal nodes boot the
+        # metal-amd64 image from their own installed disk instead, and the
+        # whole step goes away once neptune is converted too.
+        TALOS_VERSION="v1.12.6"
+        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
+        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
+        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
+        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
+        for node in neptune; do
+          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+          if [ "$EXISTS" -gt 0 ]; then
+            echo "Talos image already exists on $node"
+            continue
+          fi
+          echo "Downloading Talos image to $node..."
+          curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/download-url" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" \
+            --data-urlencode "url=$TALOS_URL" \
+            --data-urlencode "content=import" \
+            --data-urlencode "filename=$TALOS_FILENAME" \
+            --max-time 600
+          echo "Waiting for download on $node..."
+          for i in $(seq 1 60); do
+            sleep 10
+            EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+              -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+            if [ "$EXISTS" -gt 0 ]; then echo "Download complete on $node"; break; fi
+          done
+        done
       - tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
       - |
         if [ -n "$$TOFU_STATE_RM" ]; then

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -389,23 +389,7 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
   cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
   `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath drops VLAN-tagged traffic unless `bpf.vlanBypass` is
-  set. **This happened** — every v30iot macvlan workload on a converted node
-  stayed in sandbox creation until `vlanBypass: [30]` was added (issue #592).
-
-  Only `ens19` is affected, and the reason is worth knowing: Cilium attaches to
-  devices that carry an address. `cilium-dbg status --verbose` lists `mgmt0`
-  among its devices but not `ens19`, because `ens19` is deliberately
-  address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
-  untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
-  on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
-  VLAN carrying pod traffic will need the same treatment.
-
-  The symptom does not look like a VLAN problem. A tc-layer drop reaches an
-  AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space
-  available` and `network is down` rather than a DHCP timeout, and the pod sits
-  in `ContainerCreating` behind `plugin type="macvlan" failed (add): error
-  calling DHCP.Allocate: no more tries`. Confirm it with
-  `cilium_drop_count_total{reason=~"VLAN.*"}` per node rather than by reading
-  the CNI logs, and check `node_network_receive_packets_total{device="ens19"}`
-  — it reads 0 on an affected node.
+- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
+  converted node. Not observed on jupiter, but it is the first place to look if
+  the v30iot macvlan workloads misbehave there.

kubernetes/apps/music-assistant/deployment.yaml

@@ -25,11 +25,8 @@ spec:
         # replaces the former hostNetwork attachment (see
         # network-attachment.yaml) so the NODE no longer holds an address
         # on the compromised IoT VLAN; MA gets its own leased IoT IP on
-        # net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
-        # worker, though it is no longer the same kind of device everywhere:
-        # on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
-        # of enp2s0f0np0 rather than a NIC, which is why the datapath needs
-        # bpf.vlanBypass to carry it (issue #592).
+        # net1 instead. master: ens19 is the v30iot NIC on every k8s-{1,2,3}/edge
+        # worker.
         k8s.v1.cni.cncf.io/networks: music-assistant-iot
     spec:
       enableServiceLinks: false

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -73,36 +73,6 @@ spec:
       # kubernetes/talos/cilium.yaml carries a third copy of the same key.
       # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
       policyMapMax: 65536
-      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
-      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
-      # filter" in cilium_drop_count_total.
-      #
-      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
-      # otbr and music-assistant attachments. On bare metal it is a VLAN
-      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
-      # an untagged virtio NIC, which is why this was not needed before.
-      #
-      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
-      # interesting part. Cilium attaches to devices that carry an address —
-      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
-      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
-      # a device and sees frames the 8021q layer has already untagged; nothing
-      # for a VLAN filter to match. ens19 is deliberately address-less (see
-      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
-      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
-      # meet the filter on the shared parent, still tagged.
-      #
-      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
-      # would touch the datapath carrying etcd and the API for no reason.
-      #
-      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
-      # symptom is the CNI dhcp plugin reporting "no buffer space available"
-      # rather than a DHCP timeout. See issue #592.
-      #
-      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
-      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
-      # must move with this list. Takes effect on agent restart.
-      vlanBypass: [30]
       # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
       # redirects the socket directly to the backend pod instead of sending
       # traffic out to the VIP and back (which fails with L2 announcements).

kubernetes/talos/cilium.yaml

@@ -310,13 +310,6 @@ data:
   # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
   # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.
   bpf-policy-map-max: "65536"
-  # VLAN ids the eBPF datapath may pass; without this it drops VLAN-tagged
-  # frames. 30 is v30iot, the macvlan parent for the IoT workloads, and a VLAN
-  # sub-interface of enp2s0f0np0 on bare metal. VLAN 40 (mgmt0) rides the same
-  # parent but passes without an entry, so it is deliberately not listed.
-  # Keep in lockstep with patches/cilium-cni.yaml and values.bpf.vlanBypass in
-  # kubernetes/infrastructure/cilium/helmrelease.yaml. See issue #592.
-  vlan-bpf-bypass: "30"
   # bpf-policy-stats-map-max specifies the maximum number of entries in global
   # policy stats map
   bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/cilium-cni.yaml

@@ -331,16 +331,6 @@ cluster:
           # inline manifest, so a lower value here means a cold rebuild comes up with
           # a map that overflows until Flux reconciles.
           bpf-policy-map-max: "65536"
-          # VLAN ids the eBPF datapath may pass; without this it drops
-          # VLAN-tagged frames. 30 is v30iot, the macvlan parent for the IoT
-          # workloads, and a VLAN sub-interface of enp2s0f0np0 on bare metal.
-          # VLAN 40 (mgmt0) rides the same parent but passes without an entry,
-          # so it is deliberately not listed. Keep in lockstep with
-          # values.bpf.vlanBypass in kubernetes/infrastructure/cilium/helmrelease.yaml
-          # and the third copy in kubernetes/talos/cilium.yaml — bootstrap installs
-          # from this inline manifest, so omitting it here breaks the IoT macvlan
-          # workloads until Flux reconciles. See issue #592.
-          vlan-bpf-bypass: "30"
           # bpf-policy-stats-map-max specifies the maximum number of entries in global
           # policy stats map
           bpf-policy-stats-map-max: "65536"