The last step of the traffic-class split. The bird mesh already fills table 48 with one route per peer pod CIDR, next hop the peer's pod0 address; these two rules are what make the kernel look there.
Source AND destination are matched. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode does not list, so Cilium would classify it as world rather than host and policy would drop it. Destination alone would catch traffic from anywhere.
Priority 980/981 sits ahead of the 990/991 pair that sends the cluster and local networks to main, so pod-to-pod wins for that traffic and everything else is unchanged. Traffic to a pod on the same node falls through: table 48 holds only the peers' CIDRs, the lookup misses, and 990/991 send it to main and cilium_host.
Verified on k8s-1 with --mode=try, 200 requests per family from a pod on k8s-1 to a pod on k8s-2: pod0 gained 6957 transmit packets, matching 400 requests, while pod0 receive moved by 45 — the peer is unpatched and still answers over mgmt0, the same asymmetry the earlier measurement showed. Both rules were installed as expected, one per family.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.
Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.
That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.
Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DaemonSet came up with Init:Error, kubectl: not found. The build had pushed sha256:3bc85006 for bird:2026-09-16, but every pod resolved that tag to sha256:75e19214 — the first build of the same day, from before kubectl was added.
imagePullPolicy: Always does not help: it re-pulls, but the tag itself resolves to the stale digest. Deleting the pods changed nothing; three fresh pods on three nodes all resolved to the old digest. A one-off pod pulling the digest directly ran kubectl 1.33.1 and BIRD 2.17.2, so the registry holds the right image and only tag resolution is stale.
The dated tag cannot carry a same-day rebuild, and this image was rebuilt three times today. Referencing the digest makes the deployment deterministic. The cost is that the reference has to be updated whenever the image is rebuilt; the alternative, tagging by commit, does not work for a manifest living in the same commit.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:37:59 +02:00
22 changed files with 380 additions and 16 deletions
# Pod egress leaves over ingress0. Matching on the pod source prefix is what
# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
# source onto VLAN 47.
#
# The routing decision runs before Cilium's masquerade hook, so the rule sees
# the pod address and the masquerade then rewrites it to ingress0's.
#
# Table 47 holds nothing but the two default routes, so the destination rules
# below have to run first: without them a pod's packet to a service or to a pod
# on another node also matches the source rule, finds only the default, and
# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
#
# They send everything inside the cluster and the local networks back to main.
# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
# is not foreign traffic.
#
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
# The two families are separate rule tables, so the priorities need not match;
# the document name is the priority and must be unique.
# Pod to pod between nodes goes over pod0, via table 48, which the bird mesh
# fills with one route per peer pod CIDR and the peer's pod0 address as the
# next hop. Ahead of 990/991 so it wins for that traffic, and matching source
# AND destination so nothing else lands there: the node's own traffic to a pod
# would otherwise leave with a pod0 source, which CiliumNode does not list, so
# Cilium would classify it as world rather than host.
#
# Traffic to a pod on this node falls through — table 48 holds only the peers'
# CIDRs, so the lookup misses and 990/991 send it to main and cilium_host.
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"980"
src:fd10:244::/48
dst:fd10:244::/48
table:"48"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"981"
src:10.244.0.0/16
dst:10.244.0.0/16
table:"48"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"990"
dst:fd00::/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"991"
dst:10.0.0.0/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1000"
src:fd10:244::/48
table:"47"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1001"
src:10.244.0.0/16
table:"47"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot