fix(cni): make Multus the only writer of /etc/cni/net.d (#832 )

Phase 2 of 2 for #832. A pod sandbox created between Multus deleting 00-multus.conf (it does so on every SIGTERM, so on every node reboot) and rewriting it was built by containerd on 05-cilium.conflist alone: no multus-shim, no secondary interface, and no way to recover short of replacing the pod. Rook OSDs crashloop on it; the IoT workloads come up looking healthy with no VLAN interface. This applies the layout Multus documents for exactly that ordering problem. Cilium's cni.confPath moves to /etc/cni/multus/net.d, a directory containerd never reads, and Multus's multusAutoconfigDir points at the same directory. /etc/cni/net.d then holds only what Multus writes. While 00-multus.conf is absent the runtime has no CNI config at all and refuses to build sandboxes, which kubelet retries; the silent misattachment becomes a short, visible delay. Details that shape the values: - multusAutoconfigDir is the container-side path (/etc/cni/multus/net.d, the existing multus-conf-dir mount), not /host/...: the daemon reads the delegate file with plain os.ReadFile through its own filesystem and writes that path as clusterNetwork into 00-multus.conf. chrootDir is used only to exec plugin binaries. - cniConfigDir stays /host/etc/cni/net.d, where 00-multus.conf is written. - Cilium's chart mounts confPath with type DirectoryOrCreate, and the directory already exists (empty) on all three nodes. - The multus DaemonSet gains the reloader annotation: the daemon reads daemon-config.json once at start, and a ConfigMap change alone would not restart it. Cutover on the live cluster, all declarative: reloader rolls both DaemonSets. Each old Cilium agent's preStop hook (cni.uninstall from the previous commit) deletes /etc/cni/net.d/05-cilium.conflist; the new agent writes /etc/cni/multus/net.d/05-cilium.conflist. The new Multus daemon finds it there (polling up to 120 s if Cilium has not rolled on that node yet, then exiting and restarting) and rewrites 00-multus.conf. The order of the two rollouts on a node does not matter; the transient in either order is failed CNI ADDs that kubelet retries, never a sandbox without its attachment. Verified with helm template against cilium 1.20.x (hostPath /etc/cni/multus/net.d DirectoryOrCreate, in-container mount and write-cni-conf-when-ready unchanged) and kubectl kustomize on the multus directory (daemon-config.json parses, annotation present). kubernetes/bootstrap.sh takes these values from the HelmRelease, so a cold rebuild lands directly in the new layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
fix(cilium): arm the conflist cleanup and roll the agent node by node (#832 )
2026-09-19 21:34:26 +02:00 · 2026-09-19 21:32:31 +02:00
2 changed files with 31 additions and 8 deletions
@@ -118,14 +118,28 @@ spec:
      # traffic out to the VIP and back (which fails with L2 announcements).
      lbExternalClusterIP: true
    cni:
-      # Multus (kubernetes/infrastructure/multus) runs alongside Cilium and
-      # writes 00-multus.conf to /etc/cni/net.d so kubelet picks multus
-      # first; multus then delegates to Cilium's 05-cilium.conflist via
-      # clusterNetwork. The default `exclusive: true` makes cilium-cni's
-      # install step rename anything alphabetically before its own config
-      # to *.cilium_bak on every reconcile, which kept nuking multus.
-      # Setting it false lets multus's config coexist.
+      # Multus (kubernetes/infrastructure/multus) is the only writer of
+      # /etc/cni/net.d, the directory containerd reads. Cilium writes
+      # 05-cilium.conflist here instead; multus watches this directory and
+      # delegates to it via clusterNetwork. The runtime therefore has no CNI
+      # config at all until 00-multus.conf exists, so a sandbox can never be
+      # built on Cilium alone (#832). The multus DaemonSet mounts this path
+      # as multus-conf-dir.
+      confPath: /etc/cni/multus/net.d
+      # The chart default renames every other config in confPath to
+      # *.cilium_bak on each reconcile. Nothing else lives here now, but
+      # there is no reason to have Cilium police the directory.
      exclusive: false
+      # The agent's preStop hook deletes its own 05-cilium.conflist from the
+      # mounted conf dir. Arms the confPath move in #832: without it the file
+      # stays behind in the old directory forever.
+      uninstall: true
+    # Roll the agent one node at a time. The chart default takes two of three
+    # nodes down together, and a CNI change with wrong paths stops pod creation
+    # on every node it reaches.
+    updateStrategy:
+      rollingUpdate:
+        maxUnavailable: 1
    # Talos-specific settings
    securityContext:
      capabilities:
@@ -119,6 +119,11 @@ metadata:
  labels:
    tier: node
    app: multus
+# multusAutoconfigDir is where the daemon looks for the delegate (Cilium's
+# 05-cilium.conflist, cni.confPath in cilium/helmrelease.yaml) and is read
+# through the daemon's own filesystem, so it is the container mount path.
+# cniConfigDir is where it writes 00-multus.conf: the directory containerd
+# reads, which nothing else writes to (#832).
 data:
  daemon-config.json: |
    {
@@ -127,7 +132,7 @@ data:
        "logLevel": "verbose",
        "logToStderr": true,
        "cniConfigDir": "/host/etc/cni/net.d",
-        "multusAutoconfigDir": "/host/etc/cni/net.d",
+        "multusAutoconfigDir": "/etc/cni/multus/net.d",
        "multusConfigFile": "auto",
        "socketDir": "/host/run/multus/"
    }
@@ -141,6 +146,10 @@ metadata:
    tier: node
    app: multus
    name: multus
+  # The daemon reads daemon-config.json once at start; roll the pods when it
+  # changes, the same way cilium/helmrelease.yaml does for cilium-config.
+  annotations:
+    configmap.reloader.stakater.com/reload: multus-daemon-config
 spec:
  selector:
    matchLabels: