fix(cilium): roll the agent DaemonSet when cilium-config changes

Cilium reads datapath options once, at agent start, and the chart puts no
config checksum on the pod template. A changed ConfigMap therefore leaves every
agent running the previous datapath until something restarts it, and nothing
does — the DaemonSet is not rolled by a ConfigMap-only Helm change.

This already cost an outage. vlan-bpf-bypass landed in cilium-config and every
v30iot macvlan workload stayed broken until the agents were deleted by hand,
one node at a time (issue #592). The merge looked successful: HelmRelease
Ready, release upgraded, key present in the ConfigMap, and nothing working.

policyMapMax carries the same exposure. Its comment says "takes effect on agent
restart", which is documentation standing in for automation — as does the
equivalent note in the two bootstrap copies.

Reloader is already deployed and used across the repo for the same purpose.
Named per-ConfigMap rather than blanket, matching the existing convention.

The trade, stated plainly: any cilium-config change now rolls the CNI
cluster-wide, node by node. That is the same disruption a datapath change
already required; the difference is that it happens on its own rather than
being forgotten.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -389,7 +389,23 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
   cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
   `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
+- Cilium's eBPF datapath drops VLAN-tagged traffic unless `bpf.vlanBypass` is
+  set. **This happened** — every v30iot macvlan workload on a converted node
+  stayed in sandbox creation until `vlanBypass: [30]` was added (issue #592).
+
+  Only `ens19` is affected, and the reason is worth knowing: Cilium attaches to
+  devices that carry an address. `cilium-dbg status --verbose` lists `mgmt0`
+  among its devices but not `ens19`, because `ens19` is deliberately
+  address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
+  untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
+  on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
+  VLAN carrying pod traffic will need the same treatment.
+
+  The symptom does not look like a VLAN problem. A tc-layer drop reaches an
+  AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space
+  available` and `network is down` rather than a DHCP timeout, and the pod sits
+  in `ContainerCreating` behind `plugin type="macvlan" failed (add): error
+  calling DHCP.Allocate: no more tries`. Confirm it with
+  `cilium_drop_count_total{reason=~"VLAN.*"}` per node rather than by reading
+  the CNI logs, and check `node_network_receive_packets_total{device="ens19"}`
+  — it reads 0 on an affected node.

kubernetes/apps/music-assistant/deployment.yaml

@@ -25,8 +25,11 @@ spec:
         # replaces the former hostNetwork attachment (see
         # network-attachment.yaml) so the NODE no longer holds an address
         # on the compromised IoT VLAN; MA gets its own leased IoT IP on
-        # net1 instead. master: ens19 is the v30iot NIC on every k8s-{1,2,3}/edge
-        # worker.
+        # net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
+        # worker, though it is no longer the same kind of device everywhere:
+        # on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
+        # of enp2s0f0np0 rather than a NIC, which is why the datapath needs
+        # bpf.vlanBypass to carry it (issue #592).
         k8s.v1.cni.cncf.io/networks: music-assistant-iot
     spec:
       enableServiceLinks: false

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -33,6 +33,24 @@ spec:
     remediation:
       retries: 3
   values:
+    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once, at agent start, and the chart puts no config
+    # checksum on the pod template — so a changed ConfigMap leaves every agent
+    # running the previous datapath until something restarts it. Nothing did.
+    #
+    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
+    # v30iot macvlan workload stayed broken until the agents were deleted by
+    # hand, one node at a time (issue #592). policyMapMax has the same
+    # exposure, and its comment below says "takes effect on agent restart" —
+    # documentation standing in for automation.
+    #
+    # The trade: any cilium-config change now rolls the CNI cluster-wide,
+    # node by node. That is disruptive-ish, but it is what a datapath change
+    # already required — the difference is that it now happens on its own
+    # instead of being forgotten. Reloader is already deployed for exactly
+    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    annotations:
+      configmap.reloader.stakater.com/reload: cilium-config
     # Which pod labels feed security-identity computation. Patterns are
     # start-anchored regexes; a leading "!" excludes while keeping the default
     # set. Woodpecker stamps every step pod with woodpecker-ci.org/task-uuid
@@ -73,6 +91,36 @@ spec:
       # kubernetes/talos/cilium.yaml carries a third copy of the same key.
       # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
       policyMapMax: 65536
+      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
+      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
+      # filter" in cilium_drop_count_total.
+      #
+      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
+      # otbr and music-assistant attachments. On bare metal it is a VLAN
+      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
+      # an untagged virtio NIC, which is why this was not needed before.
+      #
+      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
+      # interesting part. Cilium attaches to devices that carry an address —
+      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
+      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
+      # a device and sees frames the 8021q layer has already untagged; nothing
+      # for a VLAN filter to match. ens19 is deliberately address-less (see
+      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
+      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
+      # meet the filter on the shared parent, still tagged.
+      #
+      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
+      # would touch the datapath carrying etcd and the API for no reason.
+      #
+      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
+      # symptom is the CNI dhcp plugin reporting "no buffer space available"
+      # rather than a DHCP timeout. See issue #592.
+      #
+      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
+      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
+      # must move with this list. Takes effect on agent restart.
+      vlanBypass: [30]
       # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
       # redirects the socket directly to the backend pod instead of sending
       # traffic out to the VIP and back (which fails with L2 announcements).

kubernetes/talos/cilium.yaml

@@ -310,6 +310,13 @@ data:
   # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
   # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.
   bpf-policy-map-max: "65536"
+  # VLAN ids the eBPF datapath may pass; without this it drops VLAN-tagged
+  # frames. 30 is v30iot, the macvlan parent for the IoT workloads, and a VLAN
+  # sub-interface of enp2s0f0np0 on bare metal. VLAN 40 (mgmt0) rides the same
+  # parent but passes without an entry, so it is deliberately not listed.
+  # Keep in lockstep with patches/cilium-cni.yaml and values.bpf.vlanBypass in
+  # kubernetes/infrastructure/cilium/helmrelease.yaml. See issue #592.
+  vlan-bpf-bypass: "30"
   # bpf-policy-stats-map-max specifies the maximum number of entries in global
   # policy stats map
   bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/cilium-cni.yaml

@@ -331,6 +331,16 @@ cluster:
           # inline manifest, so a lower value here means a cold rebuild comes up with
           # a map that overflows until Flux reconciles.
           bpf-policy-map-max: "65536"
+          # VLAN ids the eBPF datapath may pass; without this it drops
+          # VLAN-tagged frames. 30 is v30iot, the macvlan parent for the IoT
+          # workloads, and a VLAN sub-interface of enp2s0f0np0 on bare metal.
+          # VLAN 40 (mgmt0) rides the same parent but passes without an entry,
+          # so it is deliberately not listed. Keep in lockstep with
+          # values.bpf.vlanBypass in kubernetes/infrastructure/cilium/helmrelease.yaml
+          # and the third copy in kubernetes/talos/cilium.yaml — bootstrap installs
+          # from this inline manifest, so omitting it here breaks the IoT macvlan
+          # workloads until Flux reconciles. See issue #592.
+          vlan-bpf-bypass: "30"
           # bpf-policy-stats-map-max specifies the maximum number of entries in global
           # policy stats map
           bpf-policy-stats-map-max: "65536"