The last step of the traffic-class split. The bird mesh already fills table 48 with one route per peer pod CIDR, next hop the peer's pod0 address; these two rules are what make the kernel look there.
Source AND destination are matched. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode does not list, so Cilium would classify it as world rather than host and policy would drop it. Destination alone would catch traffic from anywhere.
Priority 980/981 sits ahead of the 990/991 pair that sends the cluster and local networks to main, so pod-to-pod wins for that traffic and everything else is unchanged. Traffic to a pod on the same node falls through: table 48 holds only the peers' CIDRs, the lookup misses, and 990/991 send it to main and cilium_host.
Verified on k8s-1 with --mode=try, 200 requests per family from a pod on k8s-1 to a pod on k8s-2: pod0 gained 6957 transmit packets, matching 400 requests, while pod0 receive moved by 45 — the peer is unpatched and still answers over mgmt0, the same asymmetry the earlier measurement showed. Both rules were installed as expected, one per family.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.
Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.
That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.
Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DaemonSet came up with Init:Error, kubectl: not found. The build had pushed sha256:3bc85006 for bird:2026-09-16, but every pod resolved that tag to sha256:75e19214 — the first build of the same day, from before kubectl was added.
imagePullPolicy: Always does not help: it re-pulls, but the tag itself resolves to the stale digest. Deleting the pods changed nothing; three fresh pods on three nodes all resolved to the old digest. A one-off pod pulling the digest directly ran kubectl 1.33.1 and BIRD 2.17.2, so the registry holds the right image and only tag resolution is stale.
The dated tag cannot carry a same-day rebuild, and this image was rebuilt three times today. Referencing the digest makes the deployment deterministic. The cost is that the reference has to be updated whenever the image is rebuilt; the alternative, tagging by commit, does not work for a manifest living in the same commit.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase B step 2 of the traffic-class split. The eBGP session source
address is what the UDM installs as the next-hop for every PodCIDR and
LoadBalancer prefix the cluster advertises. Peering over the management
LAN therefore pinned all world->pod return traffic to the 1G mgmt link,
which is the thing #576 exists to fix. Moving the peers onto VLAN 47
puts that traffic on the 10G ingress link instead.
No localAddress or sourceInterface override is needed: the kernel
already picks the ingress0 address for both families.
ip -6 route get fd02:47::1
fd02:47::1 dev ingress0 proto kernel src fd02:47::5a47:caff:fe74:7027
Both peer addresses were verified reachable from the cluster before
this change -- 10.2.47.1 and fd02:47::1 each answer HTTP with a 301,
and the UDM's neighbour entry on ingress0 is REACHABLE.
The UDM already accepts sessions from this segment: PR #835 added the
10.2.47.0/24 and fd02:47::/64 listen ranges and is uploaded. Its VLAN 40
ranges stay in place for now and get dropped in step 3, once every
session is confirmed established on 47.
gracefulRestart (120s) covers the peer replacement, so the UDM holds the
learned routes across the flap rather than withdrawing the LB VIPs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First step of moving the world-to-pod class onto ingress0 (#576). This
changes nothing about where the cluster peers from — it only widens
what the UDM will accept.
The order matters. Cilium's sessions are outbound to the UDM, and the
UDM accepts them through `bgp listen range`. Moving the cluster to
ingress0 before the UDM listens on 10.2.47.0/24 would drop every
session, and with them every advertised LoadBalancer VIP, so external
ingress would go down until the UDM caught up. Listing both segments
first removes that window entirely: a listen range only accepts a
session, it never opens one, so the 47 ranges sit idle until the
cluster actually moves.
The prefix-lists and route-maps are untouched. They filter which
prefixes the UDM accepts — pod CIDRs and the LB pool — and the cluster
advertises the same prefixes either way. Only the next hop changes.
Uploaded by hand at UniFi UI -> Settings -> Routing -> BGP, as before.
Verify with `vtysh -c "show ip bgp summary"`: the six existing sessions
on 10.2.40.x and fd40:: must still be established afterwards, and no
new ones appear until the cluster side moves.
The 40 ranges come out in a later change, once every session is
established on 47.
Refs #576
Adding a second DHCP client per node in #831 broke node DNS. Both
clients announce the same hostname, so the UDM registered each node
against VLAN 47 instead of VLAN 40: `k8s-1.srv.renner.uno` and
`k8s-3.srv.renner.uno` stopped resolving, and the reverse for
10.2.47.11 came back as bare `k8s-1`. The lease's resolver also
displaced the management one, leaving 10.2.47.1 ahead of 10.2.40.1 in
the node's DNS servers.
That matters beyond `talosctl -n <fqdn>`: bootstrap.sh resolves node
FQDNs in the talos phase, so a rebuild depends on those records.
The UDM cannot register a network's clients selectively, so the client
goes rather than the record. DHCP is now off on VLAN 47 entirely and
ingress0 declares .11/.12/.13 by node ordinal, matching pod0's
numbering. Its default route is declared with the same losing metric
2048 the lease supplied, so nothing about routing changes: mgmt0 keeps
both default routes until the flip.
IPv4 only. The problem was DHCPv4 announcing a hostname; IPv6 never had
one. The prefix is announced correctly, so ingress0 keeps taking its
IPv6 address by SLAAC exactly as mgmt0 takes its fd40:: address —
derived from the MAC, stable across reboots, nothing to allocate.
accept_ra therefore stays at its default of 2. An IPv6 default router
is a link-local address that can only be learned from the RA, and with
net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
below 2 entirely. Only accept_ra_defrtr is set to 0, because mgmt0 owns
the default until the flip; dropping that one line is what later hands
the IPv6 default to ingress0.
pod0 keeps accept_ra 0: it is a closed segment with no router on it, so
there is genuinely nothing to learn.
No reboot needed. Address, route and sysctl changes apply live, and the
whole change is provable per node with `--mode=try`, which reverts
itself.
Refs #576
! advertise extended-nexthop for the IPv6 Unicast SAFI when v6 routes
! are carried over a v4 transport, so we run a native v6 session that
! carries v6 routes with v6 next-hops and skip the capability dance
! entirely. Dynamic neighbors — k8s nodes are SLAAC-addressed on mgmt
! LAN; the listen ranges cover the SLAAC suffix space.
! entirely. Dynamic neighbors — nodes are SLAAC-addressed, so the listen
! ranges cover the SLAAC suffix space rather than naming each peer.
!
! Both segments are listed while the ingress split lands (#576). The
! cluster peers from mgmt (40) today and moves to ingress (47), and a
! listen range only accepts a session, it does not open one — so having
! both costs nothing and means the move never has a window where the UDM
! is not listening where the nodes are calling from. Drop the 40 ranges
! once every session is established on 47.
bgp listen range 10.2.40.0/24 peer-group K8S
bgp listen range fd40::/64 peer-group K8S
bgp listen range 10.2.47.0/24 peer-group K8S
bgp listen range fd02:47::/64 peer-group K8S
!
address-family ipv4 unicast
neighbor K8S activate
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot