fix(multus): the CNI must not outrank the API server

0b76c130 gave the three secondary-network DaemonSets system-node-critical,
copying cilium and cni-dhcp-daemon. That is 2000001000, and Talos runs
kube-apiserver as a static pod at system-cluster-critical, 2000000000 — so
the change made the CNI outrank the control plane. On a full node the kubelet
did exactly what that ordering tells it to:

    message: Pod was preempted by Kubelet to accommodate a critical pod.
    reason:  Preempting
    phase:   Failed

Both runs after the change died that way, and 097a0fac's larger control plane
did not help: it reproduced unchanged at 8GiB. During a rebuild the whole
infrastructure layer lands on a single-node control plane at once, and in that
window the node is full whatever its size. Preemption is an ordering, not a
capacity question, and no amount of memory makes a lower-priority pod safe
from a higher-priority one.

system-cluster-critical instead. Equal priority cannot preempt, so the API
server is out of reach of these pods. What 0b76c130 was for still holds:
2000000000 is far above any ordinary workload, so a rollout onto a busy node
displaces application pods rather than stalling at Pending forever — which was
the original defect, where multus sat unscheduled and every secondary network
on that node silently stopped working.

The residual is symmetric: the API server cannot displace multus either. That
is the correct trade when the alternative is losing the control plane.

The 8GiB control plane in bootstrap.sh stays, on its own merits — at 6GiB it
sat at 99% of allocatable memory with nothing left for late arrivals — but its
comment no longer claims to fix the preemption, because it never did.

Note: cni-dhcp-daemon.yaml still carries system-node-critical, from long
before this. It is the same hazard and is left alone here rather than widened
into unrequested scope.

kubernetes/bootstrap.sh

@@ -412,19 +412,15 @@ if [[ "$PHASE" == "vmtest" ]]; then
     # 8GiB usable, less than OpenBao's two PVCs alone ask for. Both are raw
     # sparse files, so the larger geometry costs only what is actually written.
     #
-    # The control plane gets 8GiB, not 6. At 6 it ran at 99% of allocatable
-    # memory once the infrastructure layer was up, and the kubelet resolved the
-    # next admission by preempting kube-apiserver — Talos runs it as a
-    # system-cluster-critical static pod (2000000000), which sits below the
-    # system-node-critical (2000001000) the CNI DaemonSets carry. The API server
-    # was evicted and never returned, because the room it needed never freed.
-    # The run died three steps later at "openbao-0 never answered bao status",
-    # the real message buried in the kubelet log as "Pod evicted successfully".
+    # The control plane gets 8GiB, not 6: at 6 it ran at 99% of allocatable
+    # memory once the infrastructure layer was up, which leaves nothing for the
+    # pods that arrive late in a rebuild.
     #
-    # That priority ordering is deliberate and stays — see
-    # infrastructure/multus/multus-daemonset.yaml. What it demands in return is
-    # that a control plane keep enough headroom that admitting a node-critical
-    # pod never has to displace the API server. 6GiB did not.
+    # This is NOT what fixed the preempted API server — that was a priority
+    # ordering, not a capacity shortfall, and it reproduced unchanged at 8GiB
+    # before the CNI DaemonSets were moved off system-node-critical. See
+    # infrastructure/multus/multus-daemonset.yaml. The extra 2GiB is kept on
+    # its own merits.
     $SUDO "$TALOSCTL_BIN" cluster create qemu \
         --name "$VM_CLUSTER" \
         --state "$VM_STATE" \

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -30,11 +30,12 @@ spec:
       labels:
         app: cni-plugins-installer
     spec:
-      # Same reasoning as multus-daemonset.yaml. This one installs the macvlan
+      # Same reasoning as multus-daemonset.yaml, including why this is
+      # cluster-critical and not node-critical. This one installs the macvlan
       # binary the ceph-san attachment is made of, so a node that cannot
       # schedule it has no plugin to delegate to — the failure the init
       # container comment below already describes, reached by a different road.
-      priorityClassName: system-node-critical
+      priorityClassName: system-cluster-critical
       hostNetwork: true
       tolerations:
         - operator: Exists

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -155,17 +155,13 @@ spec:
         name: multus
     spec:
       # Upstream ships this DaemonSet with no priority class, so it runs at 0
-      # and can preempt nothing. Every pod that asks for a secondary network
-      # depends on it, which puts it in the same class as cilium — and cilium's
-      # own DaemonSet is system-node-critical, as is cni-dhcp-daemon next to
-      # this file. This was the odd one out.
-      #
-      # What that costs: any event that makes this pod need a fresh scheduling
-      # decision — a rollout, a node restart, an eviction — has to fit it onto
-      # a node that may since have filled up. At priority 0 it simply stays
-      # Pending, and the node then has no multus at all. Nothing reports that
-      # as an error: pods still start, they just never get their secondary
-      # attachment, and the annotation everything waits on never appears.
+      # and can preempt nothing. Any event that makes the pod need a fresh
+      # scheduling decision — a rollout, a node restart, an eviction — then has
+      # to fit it onto a node that may since have filled up, and if it does not
+      # fit it simply stays Pending. The node has no multus after that, and
+      # nothing reports it: pods still start, they just never get their
+      # secondary attachment, and the annotation everything waits on never
+      # appears.
       #
       # Seen on a cold rebuild: two of four nodes sat at 99% memory requests
       # when the DaemonSet rolled, both multus pods went Pending, and Rook's
@@ -174,21 +170,23 @@ spec:
       # what infrastructure-ceph-cluster gates on, so the whole apps layer
       # stopped behind a scheduling decision three layers away.
       #
-      # What this buys has a price, and it is not theoretical. system-node-
-      # critical is 2000001000; Talos runs kube-apiserver as a static pod at
-      # system-cluster-critical, 2000000000. These pods therefore outrank the
-      # API server by a thousand points, and on a node with no headroom the
-      # kubelet will evict the API server to admit one of them. That happened
-      # on the first run after this change, on a 6GiB control plane: the API
-      # server was preempted and never came back.
+      # system-cluster-critical (2000000000), NOT system-node-critical
+      # (2000001000), even though cilium and cni-dhcp-daemon carry the latter.
+      # Talos runs kube-apiserver as a static pod at system-cluster-critical,
+      # so node-critical here outranks the API server by a thousand points —
+      # and on a full node the kubelet duly evicted the API server to admit
+      # this pod. Twice, on a 6GiB control plane and again on 8GiB: during
+      # bootstrap the whole infrastructure layer lands on a single-node control
+      # plane at once, and in that window the node is full whatever its size.
+      # Preemption is not a capacity question.
       #
-      # The ordering is still the right one — a node without its CNI is a node
-      # that cannot run anything, which is why cilium and cni-dhcp-daemon carry
-      # the same class. The obligation it creates is on capacity: a control
-      # plane has to keep enough room that admitting a node-critical pod never
-      # has to displace the control plane itself. bootstrap.sh sizes the test
-      # VMs accordingly.
-      priorityClassName: system-node-critical
+      # Equal priority cannot preempt, so the API server is now safe from this
+      # pod. The original problem is still solved: 2000000000 is far above any
+      # ordinary workload, so a rollout onto a busy node displaces application
+      # pods instead of stalling. The residual is symmetric — the API server
+      # cannot displace multus either — which is the correct trade when the
+      # alternative is losing the control plane.
+      priorityClassName: system-cluster-critical
       hostNetwork: true
       hostPID: true
       tolerations:

kubernetes/infrastructure/multus/whereabouts-daemonset.yaml

@@ -146,11 +146,12 @@ spec:
         app: whereabouts
         name: whereabouts
     spec:
-      # Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
+      # Same reasoning as multus-daemonset.yaml, including why this is
+      # cluster-critical and not node-critical: this is the IPAM half of the
       # secondary-network path, so a node without it cannot hand out an address
       # on ceph-san and the attachment fails just as completely as if multus
       # itself were missing. Upstream leaves the priority class unset.
-      priorityClassName: system-node-critical
+      priorityClassName: system-cluster-critical
       hostNetwork: true
       serviceAccountName: whereabouts
       # Every node here is a control plane node, so the NoSchedule toleration