network-policies: revert broad ICMP-errors CCNP; add per-namespace ingress allow

The CCNP added in 7f95486 used `endpointSelector: matchExpressions Exists`
on `io.kubernetes.pod.namespace`, which matched every pod in the cluster
including kube-system. That transitioned pods like metrics-server (which
previously had NO policies and was default-allow) into policy-enforced
mode where only ICMP errors were allowed, breaking apiserver:6443 and
kubelet:10250 egress and likely impacting other kube-system workloads.

Reverts the CCNP and adds the ICMP-error ingress allow only inside
home-assistant's existing default-deny CNP — i.e. only in namespaces
that already opted into policy enforcement. Will extend to other
namespaces individually if Hubble shows ICMP-error drops there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/network-policies/allow-icmp-errors.yaml

@@ -1,60 +0,0 @@
-# When a UDP/TCP packet hits a closed port on a pod (e.g. coredns
-# pod that's draining behind a Service, or any SO_REUSEPORT race),
-# the receiver replies with an ICMP error message back to the
-# sender — ICMPv4 Destination-Unreachable (type 3) /
-# Time-Exceeded (type 11), ICMPv6 Destination-Unreachable (type 1) /
-# Time-Exceeded (type 3). Without this, the sender never learns the
-# destination was unreachable and the kernel keeps retransmitting.
-#
-# Hubble shows the symptom as e.g.
-#   `home-assistant/home-assistant <> kube-system/coredns
-#    policy-verdict:none ... DENIED (ICMPv4 DestinationUnreachable(Port))`
-# even though the actual UDP/53 query reaches coredns fine.
-#
-# This CCNP ingress-allows ICMP error-class FROM any pod TO any pod,
-# scoped via matchExpressions Exists on the namespace label so it
-# does NOT match reserved-identity endpoints (host, remote-node,
-# world, health, kube-apiserver). Egress is symmetric — pods need
-# to be able to SEND ICMP errors too (e.g. their own closed ports).
-#
-# Note: this only widens permissions for the ICMP error class. UDP
-# Echo (Type 0/8 / 128/129) is NOT allowed by this policy.
-apiVersion: cilium.io/v2
-kind: CiliumClusterwideNetworkPolicy
-metadata:
-  name: allow-icmp-errors-between-pods
-spec:
-  endpointSelector:
-    matchExpressions:
-      - key: io.kubernetes.pod.namespace
-        operator: Exists
-  ingress:
-    - fromEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - type: 3      # IPv4 Destination Unreachable
-              family: IPv4
-            - type: 11     # IPv4 Time Exceeded
-              family: IPv4
-            - type: 1      # IPv6 Destination Unreachable
-              family: IPv6
-            - type: 3      # IPv6 Time Exceeded
-              family: IPv6
-  egress:
-    - toEndpoints:
-        - matchExpressions:
-            - key: io.kubernetes.pod.namespace
-              operator: Exists
-      icmps:
-        - fields:
-            - type: 3
-              family: IPv4
-            - type: 11
-              family: IPv4
-            - type: 1
-              family: IPv6
-            - type: 3
-              family: IPv6

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -36,6 +36,26 @@ spec:
     - fromEntities:
         - host
         - remote-node
+    # ICMP error class from any cluster pod — covers e.g. coredns sending
+    # `Destination Unreachable (Port)` back when an HA DNS query lands on
+    # a draining coredns instance via SO_REUSEPORT. Without this, HA's
+    # kernel never learns the destination is unreachable and just keeps
+    # retransmitting; Hubble shows the DU drops as
+    # `policy-verdict:none DENIED (ICMPv4 DestinationUnreachable(Port))`.
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - type: 3      # IPv4 Destination Unreachable
+              family: IPv4
+            - type: 11     # IPv4 Time Exceeded
+              family: IPv4
+            - type: 1      # IPv6 Destination Unreachable
+              family: IPv6
+            - type: 3      # IPv6 Time Exceeded
+              family: IPv6
     # IoT devices reach HA on 10.2.30.80 and mosquitto on 10.2.30.81
     # via Cilium-L2 LoadBalancers (externalTrafficPolicy=Local preserves
     # source IP, so they appear as world identity from the pod's

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -18,4 +18,3 @@ resources:
   - rook-ceph-ns.yaml
   - velero-ns.yaml
   - allow-cilium-health.yaml
-  - allow-icmp-errors.yaml