The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:37:59 +02:00
15 changed files with 100 additions and 19 deletions
# Pod egress leaves over ingress0. Matching on the pod source prefix is what
# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
# source onto VLAN 47.
#
# The routing decision runs before Cilium's masquerade hook, so the rule sees
# the pod address and the masquerade then rewrites it to ingress0's.
#
# Table 47 holds nothing but the two default routes, so the destination rules
# below have to run first: without them a pod's packet to a service or to a pod
# on another node also matches the source rule, finds only the default, and
# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
#
# They send everything inside the cluster and the local networks back to main.
# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
# is not foreign traffic.
#
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
# The two families are separate rule tables, so the priorities need not match;
# the document name is the priority and must be unique.
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"990"
dst:fd00::/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"991"
dst:10.0.0.0/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1000"
src:fd10:244::/48
table:"47"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1001"
src:10.244.0.0/16
table:"47"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
k.renner
claude-botandClaude Opus 5
security-bot