fix(multus): make the secondary-network DaemonSets node-critical

Three of the four DaemonSets that make up the secondary-network path ran at
priority 0. cni-dhcp-daemon.yaml, sitting in the same directory, has been
system-node-critical all along, and so is cilium's own DaemonSet — multus,
whereabouts and cni-plugins-installer were the odd ones out.

At priority 0 they cannot preempt anything. Any event that forces a fresh
scheduling decision — a rollout, a node restart, an eviction — has to fit the
pod onto a node that may since have filled up, and if it does not fit the pod
just stays Pending. The node then has no multus, or no IPAM, or no macvlan
binary, and nothing reports it: pods still start, they simply never receive
their secondary attachment.

Found on a cold rebuild. Two of four nodes sat at 99% memory requests when the
DaemonSet rolled; both new multus pods went Pending with "1 Insufficient
memory, 3 node(s) didn't satisfy plugin(s) [NodeAffinity]" and stayed there.
Rook's network canary on one of those nodes hung in Init:0/1 in
wait-for-network-status-annotation — waiting for an annotation multus was not
running to write. Its CephCluster reconcile stopped at "discovering ceph
cluster network CIDR(s)" and never returned to status.phase Ready, which is
exactly what infrastructure-ceph-cluster's healthCheckExpr gates on, so the
entire apps layer stalled behind a scheduling decision three layers away.

cni-plugins-installer.yaml already carries a comment describing this same
canary-hang, reached from a different cause (the plugin image was pulled from
the in-cluster registry, which does not exist during a rebuild). Same symptom,
same four-layer distance between cause and evidence.

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -30,6 +30,11 @@ spec:
       labels:
         app: cni-plugins-installer
     spec:
+      # Same reasoning as multus-daemonset.yaml. This one installs the macvlan
+      # binary the ceph-san attachment is made of, so a node that cannot
+      # schedule it has no plugin to delegate to — the failure the init
+      # container comment below already describes, reached by a different road.
+      priorityClassName: system-node-critical
       hostNetwork: true
       tolerations:
         - operator: Exists

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -154,6 +154,26 @@ spec:
         app: multus
         name: multus
     spec:
+      # Upstream ships this DaemonSet with no priority class, so it runs at 0
+      # and can preempt nothing. Every pod that asks for a secondary network
+      # depends on it, which puts it in the same class as cilium — and cilium's
+      # own DaemonSet is system-node-critical, as is cni-dhcp-daemon next to
+      # this file. This was the odd one out.
+      #
+      # What that costs: any event that makes this pod need a fresh scheduling
+      # decision — a rollout, a node restart, an eviction — has to fit it onto
+      # a node that may since have filled up. At priority 0 it simply stays
+      # Pending, and the node then has no multus at all. Nothing reports that
+      # as an error: pods still start, they just never get their secondary
+      # attachment, and the annotation everything waits on never appears.
+      #
+      # Seen on a cold rebuild: two of four nodes sat at 99% memory requests
+      # when the DaemonSet rolled, both multus pods went Pending, and Rook's
+      # network canary hung in Init:0/1 waiting for a network-status annotation
+      # that could never come. The CephCluster stayed in Progressing, which is
+      # what infrastructure-ceph-cluster gates on, so the whole apps layer
+      # stopped behind a scheduling decision three layers away.
+      priorityClassName: system-node-critical
       hostNetwork: true
       hostPID: true
       tolerations:

kubernetes/infrastructure/multus/whereabouts-daemonset.yaml

@@ -146,6 +146,11 @@ spec:
         app: whereabouts
         name: whereabouts
     spec:
+      # Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
+      # secondary-network path, so a node without it cannot hand out an address
+      # on ceph-san and the attachment fails just as completely as if multus
+      # itself were missing. Upstream leaves the priority class unset.
+      priorityClassName: system-node-critical
       hostNetwork: true
       serviceAccountName: whereabouts
       # Every node here is a control plane node, so the NoSchedule toleration