docs(multus): correct what the priority class actually does

The comment added in ce31dace claimed "preemption is not a capacity
question". That inverts the cause. The kubelet preempts because the node is
out of memory; without a shortfall nothing is evicted, whatever the priorities
say. Priority only selects the victim once the shortfall exists.

What the two runs isolate is exactly that and no more: at 8GiB with
node-critical the API server was chosen, at 8GiB with cluster-critical it was
not. The pressure did not change — the candidate set did. Under pressure the
kubelet now displaces an ordinary workload instead, or this pod stays Pending,
which is the original defect and the acceptable one of the two outcomes.

Also records why equal priority is enough: Preemptable() compares with a
strict >, and the API server is a static pod, so IsCriticalPod holds for it
and the critical-beats-noncritical branch never applies.

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -172,20 +172,27 @@ spec:
       #
       # system-cluster-critical (2000000000), NOT system-node-critical
       # (2000001000), even though cilium and cni-dhcp-daemon carry the latter.
-      # Talos runs kube-apiserver as a static pod at system-cluster-critical,
-      # so node-critical here outranks the API server by a thousand points —
-      # and on a full node the kubelet duly evicted the API server to admit
-      # this pod. Twice, on a 6GiB control plane and again on 8GiB: during
-      # bootstrap the whole infrastructure layer lands on a single-node control
-      # plane at once, and in that window the node is full whatever its size.
-      # Preemption is not a capacity question.
       #
-      # Equal priority cannot preempt, so the API server is now safe from this
-      # pod. The original problem is still solved: 2000000000 is far above any
-      # ordinary workload, so a rollout onto a busy node displaces application
-      # pods instead of stalling. The residual is symmetric — the API server
-      # cannot displace multus either — which is the correct trade when the
-      # alternative is losing the control plane.
+      # Two separate things decide whether the API server survives here, and
+      # only one of them is this line. The kubelet preempts because the node is
+      # out of memory — no shortfall, no eviction, whatever the priorities say.
+      # Priority decides only who gets chosen once that has happened. Talos runs
+      # kube-apiserver as a static pod at system-cluster-critical, so a
+      # node-critical pod outranks it by a thousand points and the kubelet duly
+      # picked the API server as the victim. Twice: on a 6GiB control plane, and
+      # again after it was grown to 8GiB.
+      #
+      #   Preemptable() in pkg/kubelet/types/pod_update.go compares with a
+      #   strict >. The API server is a static pod, so IsCriticalPod is true for
+      #   it and the critical-beats-noncritical branch does not apply; equal
+      #   priority therefore cannot preempt it.
+      #
+      # So this does not remove the memory pressure — it only takes the control
+      # plane out of the set of candidates. Under pressure the kubelet now
+      # displaces an ordinary workload instead, or this pod stays Pending, which
+      # is the failure described above and the acceptable one of the two.
+      # 2000000000 is still far above any application pod, so the rollout case
+      # this class was raised for keeps working.
       priorityClassName: system-cluster-critical
       hostNetwork: true
       hostPID: true