fix(cilium): roll the agent DaemonSet when cilium-config changes

Cilium reads datapath options once, at agent start, and the chart puts no config checksum on the pod template. A changed ConfigMap therefore leaves every agent running the previous datapath until something restarts it, and nothing does — the DaemonSet is not rolled by a ConfigMap-only Helm change. This already cost an outage. vlan-bpf-bypass landed in cilium-config and every v30iot macvlan workload stayed broken until the agents were deleted by hand, one node at a time (issue #592). The merge looked successful: HelmRelease Ready, release upgraded, key present in the ConfigMap, and nothing working. policyMapMax carries the same exposure. Its comment says "takes effect on agent restart", which is documentation standing in for automation — as does the equivalent note in the two bootstrap copies. Reloader is already deployed and used across the repo for the same purpose. Named per-ConfigMap rather than blanket, matching the existing convention. The trade, stated plainly: any cilium-config change now rolls the CNI cluster-wide, node by node. That is the same disruption a datapath change already required; the difference is that it happens on its own rather than being forgotten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 02:37:31 +02:00
2 changed files with 54 additions and 0 deletions
@@ -128,6 +128,42 @@ steps:
      - printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
      - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
      - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
+      - |
+        # Ensure Talos image exists on all Proxmox nodes via API.
+        #
+        # Only nodes still running Proxmox belong in this list. jupiter and
+        # saturn run Talos on bare metal now, so the storage API returns
+        # nothing for them and the download-wait loop below would burn its full
+        # 60x10s per node before giving up. Bare-metal nodes boot the
+        # metal-amd64 image from their own installed disk instead, and the
+        # whole step goes away once neptune is converted too.
+        TALOS_VERSION="v1.12.6"
+        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
+        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
+        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
+        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
+        for node in neptune; do
+          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+          if [ "$EXISTS" -gt 0 ]; then
+            echo "Talos image already exists on $node"
+            continue
+          fi
+          echo "Downloading Talos image to $node..."
+          curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/download-url" \
+            -H "Authorization: PVEAPIToken=$API_TOKEN" \
+            --data-urlencode "url=$TALOS_URL" \
+            --data-urlencode "content=import" \
+            --data-urlencode "filename=$TALOS_FILENAME" \
+            --max-time 600
+          echo "Waiting for download on $node..."
+          for i in $(seq 1 60); do
+            sleep 10
+            EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
+              -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
+            if [ "$EXISTS" -gt 0 ]; then echo "Download complete on $node"; break; fi
+          done
+        done
      - tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
      - |
        if [ -n "$$TOFU_STATE_RM" ]; then
@@ -33,6 +33,24 @@ spec:
    remediation:
      retries: 3
  values:
+    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once, at agent start, and the chart puts no config
+    # checksum on the pod template — so a changed ConfigMap leaves every agent
+    # running the previous datapath until something restarts it. Nothing did.
+    #
+    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
+    # v30iot macvlan workload stayed broken until the agents were deleted by
+    # hand, one node at a time (issue #592). policyMapMax has the same
+    # exposure, and its comment below says "takes effect on agent restart" —
+    # documentation standing in for automation.
+    #
+    # The trade: any cilium-config change now rolls the CNI cluster-wide,
+    # node by node. That is disruptive-ish, but it is what a datapath change
+    # already required — the difference is that it now happens on its own
+    # instead of being forgotten. Reloader is already deployed for exactly
+    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    annotations:
+      configmap.reloader.stakater.com/reload: cilium-config
    # Which pod labels feed security-identity computation. Patterns are
    # start-anchored regexes; a leading "!" excludes while keeping the default
    # set. Woodpecker stamps every step pod with woodpecker-ci.org/task-uuid