The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:37:59 +02:00
16 changed files with 134 additions and 16 deletions
# Pod egress leaves over ingress0. Matching on the pod source prefix is what
# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
# source onto VLAN 47.
#
# The routing decision runs before Cilium's masquerade hook, so the rule sees
# the pod address and the masquerade then rewrites it to ingress0's.
#
# Table 47 holds nothing but the two default routes, so the destination rules
# below have to run first: without them a pod's packet to a service or to a pod
# on another node also matches the source rule, finds only the default, and
# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
#
# They send everything inside the cluster and the local networks back to main.
# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
# is not foreign traffic.
#
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
# The two families are separate rule tables, so the priorities need not match;
# the document name is the priority and must be unique.
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"990"
dst:fd00::/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"991"
dst:10.0.0.0/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1000"
src:fd10:244::/48
table:"47"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1001"
src:10.244.0.0/16
table:"47"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot