The LB address was declared twice: on the Service as io.cilium/lb-ipam-ips,
and again as a literal PASV_ADDRESS in the ConfigMap generator.
The copy was not doing anything. vsftpd listens on a single IPv6 socket, so
an IPv4 client arrives as ::ffff:a.b.c.d and vsftpd answers PASV with
0.0.0.0 regardless of the configured address — the only such session in 30
days of logs got `227 Entering Passive Mode (0,0,0,0,82,109)` while
PASV_ADDRESS=10.2.99.82 was set. IPv6 clients negotiate EPSV, which carries
a port and no address at all; that is what the scanner uses.
Removing the literal falls back to the image default of 0.0.0.0, which is
the reply clients already receive, so behaviour is unchanged and the address
now exists in exactly one place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vsftpd 3.0.5 segfaults its standalone listener on roughly 0.3% of the
connections it accepts: the parent closes the accepted socket before it
registers the forked child's pid, so a child that exits inside that window
makes the SIGCHLD path look up an unregistered pid, get NULL, and
dereference it in hash_ip(). PID 1 is vsftpd, so the segfault takes the
container with it.
The tcpSocket probes added in 469cdbc2 opened ~8600 connections a day
against that, which turned a latent crash into a restart every ~25 minutes
and left the scanner hitting CrashLoopBackOff windows. Measured on the live
pod: 52 restarts in 21h, one node-side `vsftpd[...]: segfault at 0` per
restart, starting the hour the probes rolled out.
Assert the listening socket from inside the container instead. The probe
still catches the failure it was added for — vsftpd not serving port 21 —
without being a client of the process it watches.
vsftpd 3.0.5 is upstream's last release (2021) and Alpine ships it
unpatched in 3.21 (3.0.5-r2) and edge (3.0.5-r3), so there is no version
bump that fixes this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CronJob ran `rmcrackan/libation:latest` with imagePullPolicy Always
but was stuck on 13.7.11 (2026-08-24). Spegel answers containerd's tag
resolution from its peer cache -- a 200 from the mirror is a hit, so
upstream is never consulted and the tag never advances. Verified: the
pod's imageID was sha256:946177d3 (= tag 13.7.11) while Docker Hub's
:latest pointed at sha256:6f9d8ba1 (= 14.2.1), and querying a Spegel
pod's registry port directly returned the stale digest for that tag.
Audible changed device registration in early September; upstream shipped
an experimental fix in 14.1 and the real fix in 14.2, so the stale image
predates both. Since 2026-09-15 three newly purchased titles fail with
"Content license denied" and the library has not grown.
A digest reference is immune to the mirror's tag cache. Note that the
upstream maintainer's remedy also requires removing and re-adding the
Audible account with an interactive login, which this change does not
and cannot perform.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring VLAN 40 for the traffic-class split turned up a share of the
management segment that is not management: the node's Ceph client
talking to OSDs on the other two nodes, sourced from the node address
on mgmt0.
Ceph sends every client read to the acting primary of the PG, which
with three OSDs is remote two times out of three. Read affinity has the
client read from the replica in its own CRUSH location instead.
The preconditions hold here. replicapool and cephfs/data0 are size 3
over failureDomain host across three nodes, so every object has a copy
on every node. The OSDs are created under zone buckets
(root=default host=k8s-N zone=<name>) and the nodes carry
topology.kubernetes.io/zone, which is in Rook's default
crushLocationLabels. Ceph is v20.2.4 and the kernel 6.18, against a
documented minimum of 5.8.
Writes are unaffected: they always go to the primary, and the
replication behind them already rides the VLAN 49 cluster network.
cephfs-ci is size 2, so a local replica is not guaranteed there and the
client falls back to the primary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rolling out #857 took cross-node pod traffic down, and cluster DNS with it, until it was reverted. The table was incomplete: the node's own pod CIDR was originated as a blackhole so it could be advertised without being installed, and then filtered out of the kernel export.
The rule matches the SOURCE, and a reply from a pod on another node is forwarded traffic with a pod source — so it matches too. Its destination is a pod on this node, the table had no route for that, and it took the default and left for the UDM. The comment claiming such traffic would fall through to main was wrong: nothing falls through once a default matches.
Originating it via cilium_host instead makes it both advertisable and correct in the table, and the export filter that existed only to hide the blackhole goes with it.
Diagnosed by narrowing the rule to a single pod address, where the same path worked because the return packet no longer matched: 404 with src <pod>/32, 000 with src 10.244.0.0/16, 404 again once 10.244.0.0/24 dev cilium_host was added to table 47 by hand. bird -p parses the corrected config inside the image, exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule stack grew a carve-out layer because table 47 held nothing but a default route: it matched everything, so every cluster-internal destination had to be excluded by a rule naming its prefix. Those exclusions were wrong twice over — too broad, since fd00::/8 and 10.0.0.0/8 also caught every LAN client and sent its reply back over mgmt0, and unnecessary, since the networks in question are directly attached and a connected route already beats a default.
So the table gets the connected routes and decides on its own. bird's direct protocol generates them from the interfaces rather than from a list, alongside the peers' pod CIDRs it already learns and the egress default it now originates. One rule per family remains, on the pod source prefix; 980/981, 990/991 and table 48 are gone.
bird owns the table outright, including the default. That is what keeps the failure benign: if the daemon stops the table empties, the rule finds nothing and falls through to main, so pod traffic returns to VLAN 40 unseparated rather than misrouted. A default declared in the Talos config would have outlived the specific routes and sent pod traffic to the UDM.
The ingress gateway moves from a route in each node patch to one node annotation in the site patch, which is now the only place it is stated.
Verified: bird -p parses the rendered config inside the image, exit 0; the templates render GW6 fd02:47::1 and GW4 10.2.47.1 from the live k8s-1 object; kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the proposal, which was wrong before it was written down. It presented A, B and C as the option space and concluded pod-to-pod was blocked; policy routing was available the whole time and is what moved pod egress and then pod-to-pod. The file recorded a blind spot and gave it the authority of a checked-in document.
Three of four classes are separated now, so this describes the built design rather than proposing one, and the PROPOSAL- name goes with it: which link each class uses, the rule order and why 980/981 match source and destination, where each table comes from, how the bird mesh renders itself from the API without repeating an address, and why the node's own CIDR is originated rather than read back from the kernel.
Management is the remaining class. Its levers are listed, along with the two approaches the owner rejected — with the caveat that the same reasoning which produced that list also called pod-to-pod impossible, so it is a record of decisions and not an analysis to trust.
Failure behaviour is stated because it is what makes the design safe to roll one node at a time: an empty table 48 falls through to main and pod traffic returns to VLAN 40, unseparated rather than broken.
bootstrap.sh's reference follows the rename; its point was always that the QEMU/Hetzner sites use two segments, not the homelab's split.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last step of the traffic-class split. The bird mesh already fills table 48 with one route per peer pod CIDR, next hop the peer's pod0 address; these two rules are what make the kernel look there.
Source AND destination are matched. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode does not list, so Cilium would classify it as world rather than host and policy would drop it. Destination alone would catch traffic from anywhere.
Priority 980/981 sits ahead of the 990/991 pair that sends the cluster and local networks to main, so pod-to-pod wins for that traffic and everything else is unchanged. Traffic to a pod on the same node falls through: table 48 holds only the peers' CIDRs, the lookup misses, and 990/991 send it to main and cilium_host.
Verified on k8s-1 with --mode=try, 200 requests per family from a pod on k8s-1 to a pod on k8s-2: pod0 gained 6957 transmit packets, matching 400 requests, while pod0 receive moved by 45 — the peer is unpatched and still answers over mgmt0, the same asymmetry the earlier measurement showed. Both rules were installed as expected, one per family.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.
Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.
That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.
Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DaemonSet came up with Init:Error, kubectl: not found. The build had pushed sha256:3bc85006 for bird:2026-09-16, but every pod resolved that tag to sha256:75e19214 — the first build of the same day, from before kubectl was added.
imagePullPolicy: Always does not help: it re-pulls, but the tag itself resolves to the stale digest. Deleting the pods changed nothing; three fresh pods on three nodes all resolved to the old digest. A one-off pod pulling the digest directly ran kubectl 1.33.1 and BIRD 2.17.2, so the registry holds the right image and only tag resolution is stale.
The dated tag cannot carry a same-day rebuild, and this image was rebuilt three times today. Referencing the digest makes the deployment deterministic. The cost is that the reference has to be updated whenever the image is rebuilt; the alternative, tagging by commit, does not work for a manifest living in the same commit.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:37:59 +02:00
30 changed files with 523 additions and 242 deletions
# The UDM on the ingress segment. Published here rather than as a route
# because bird owns table 47 and originates the default from it; this is the
# one place the address is stated.
nodeAnnotations:
net.renner.uno/egress-gw-v6:fd02:47::1
net.renner.uno/egress-gw-v4:10.2.47.1
---
# All pod traffic goes to table 47, and the table decides — longest prefix
# match, no exceptions to carve out. bird fills it with the node's attached
# networks, the peers' pod CIDRs and the egress default, so a packet to a node
# on the mgmt segment takes the connected route, a packet to a pod on another
# node takes pod0, and anything else takes the default over ingress0.
#
# Matching the source is what keeps the node's own traffic out: it answers on
# mgmt0, and a reply leaving over ingress0 would carry an mgmt0 source onto
# VLAN 47. The routing decision runs before Cilium's masquerade hook, so the
# rule sees the pod address and the masquerade then rewrites it.
#
# If bird stops, table 47 empties, the lookup misses and the rule falls through
# to main — pod traffic returns to VLAN 40, unseparated rather than misrouted.
#
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
# The two families are separate rule tables, so the priorities need not match;
# the document name is the priority and must be unique.
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1000"
src:fd10:244::/48
table:"47"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1001"
src:10.244.0.0/16
table:"47"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot