fix(bird): route LoadBalancer VIP replies into table 47 (#1547) #1548

2026-11-02T06:36:14Z

hermes-bot commented

2026-11-02 06:36:14 +00:00

Disruptive (cluster-wide routing): merge only 02:00–05:00 Europe/Berlin; needs the owner's HA-impact review.

Problem

Replies from LoadBalancer VIPs leave the nodes over mgmt0 while their requests arrive over ingress0. Cilium reverses the service NAT before the reply reaches the kernel's routing decision, so the reply's source is the VIP, not the pod. The pod-source rules 1000/1001 don't match it, and main's default (10.2.40.1 via mgmt0) applies. Live on k8s-3: rules 9, 100, 1001 (10.244.0.0/16 → 47), 32766 main; no rule matches 10.2.99.0/24 or fd99::/120.

Change

Option 2 from the issue: the pool stays the only place the prefix is written, and the rule is derived from it.

infrastructure-inhouse/bird: a new vip-rules container in the bird DaemonSet (same image; busybox sh/ip, kubectl). Every 60 s it reads all enabled CiliumLoadBalancerIPPools and keeps exactly one rule per block: priority 1002, from <block> lookup 47, both families. It adds missing rules and removes stale ones.
- An API error skips that round and leaves the rules in place.
- A start/stop block exits the container, so the pod goes NotReady instead of failing silently.
- Rules stay in place when the container stops. If bird is gone, table 47 is empty and the lookup falls through to main, the same as for the pod rules.
- RBAC: get/list on cilium.io/ciliumloadbalancerippools.
post-crds/cilium-bgp.yaml: bgp-pool blocks change from 10.2.99.0–10.2.99.255 / fd99::–fd99::ff to cidr: 10.2.99.0/24 / cidr: fd99::/120. The address set is the same.
site-homelab.yaml: a comment that priority 1002 is taken. The comment is the only change: no talosctl apply needed.
NETWORK-traffic-classes.md: documents VIP replies.

Option 1 (literal RoutingRuleConfig 1002/1003) would write the pool prefix a second time, which the repo's rule ("No address is written into this repo twice") forbids.

Expected impact

bird rolls one node at a time (new container, new ConfigMap hash). While a node's bird restarts, table 47 empties and that node's pod egress falls back to main for a few seconds. This is the same as any bird change.
Pool change: no reallocation expected.
- Cilium v1.20.4 handlePoolModified keeps any range whose first and last address match a new block (LBRange.EqualCIDR). RangeFromPrefix gives 10.2.99.0–10.2.99.255 and fd99::–fd99::ff, the same as today.
- allowFirstLastIPs is unset, which means allocatable, so nothing new is reserved.
- All 10 LB Services also pin their IPs by annotation.
After rollout: VIP replies to destinations off the node's own subnets leave over ingress0 via 10.2.47.1 / fd02:47::1, with the VIP as source. That is the link the UDM learned the VIP over by BGP.
- Replies to clients on VLAN 40/46/47/48 take table 47's connected routes, the same links as today.
- Table 47 on k8s-3 has every destination main has (attached nets, peer pod CIDRs, default).
HA-relevant: Home Assistant (10.2.99.87), mosquitto-iot (10.2.99.81), shelly-manager-firmware (10.2.99.90) and k8s-gateway DNS (10.2.99.83) are all VIP services. Their replies switch path. Existing TCP sessions (MQTT) may see a retransmit at the switch. No interruption is expected beyond that.

Not established

Where on the mgmt0 → UDM → VLAN 30 path the Shelly segments were lost is still unknown. This PR restores the designed path. Whether it fixes the slow OTA is the read-out after merge.

Verification

Done:

kubectl kustomize builds infrastructure-inhouse and post-crds.
shellcheck 0.11.0 is clean.
scripts/ci/yaml-citations.sh passes.
The template was rendered with kubectl against cidr, disabled and range pools.
vip-rules.sh was run under Alpine 3.24 busybox ash with stub ip/kubectl: initial add, no-op, pool change (add/remove), API error (skip), range block (exit), no pools (remove all).
busybox ip -4/-6 rule add|del priority … from … table 47 parses; outside a privileged netns it returns EPERM.

After merge, on each node:

talosctl -n k8s-N get routingrules shows 1002 for 10.2.99.0/24 and fd99::/120.
talosctl pcap -i ingress0 shows 10.2.99.90 → device data.
mgmt0 carries none of it.
The Power Strip OTA from shelly-manager completes.

Hermes-Card: t_9c084f2c (atlas)

Closes #1547 **Disruptive (cluster-wide routing): merge only 02:00–05:00 Europe/Berlin; needs the owner's HA-impact review.** ## Problem Replies from LoadBalancer VIPs leave the nodes over `mgmt0` while their requests arrive over `ingress0`. Cilium reverses the service NAT before the reply reaches the kernel's routing decision, so the reply's source is the VIP, not the pod. The pod-source rules 1000/1001 don't match it, and main's default (`10.2.40.1` via `mgmt0`) applies. Live on k8s-3: rules 9, 100, 1001 (`10.244.0.0/16` → 47), 32766 main; no rule matches `10.2.99.0/24` or `fd99::/120`. ## Change Option 2 from the issue: the pool stays the only place the prefix is written, and the rule is derived from it. - `infrastructure-inhouse/bird`: a new `vip-rules` container in the bird DaemonSet (same image; busybox `sh`/`ip`, `kubectl`). Every 60 s it reads all enabled `CiliumLoadBalancerIPPool`s and keeps exactly one rule per block: priority 1002, `from <block> lookup 47`, both families. It adds missing rules and removes stale ones. - An API error skips that round and leaves the rules in place. - A start/stop block exits the container, so the pod goes NotReady instead of failing silently. - Rules stay in place when the container stops. If bird is gone, table 47 is empty and the lookup falls through to main, the same as for the pod rules. - RBAC: `get`/`list` on `cilium.io/ciliumloadbalancerippools`. - `post-crds/cilium-bgp.yaml`: `bgp-pool` blocks change from `10.2.99.0–10.2.99.255` / `fd99::–fd99::ff` to `cidr: 10.2.99.0/24` / `cidr: fd99::/120`. The address set is the same. - `site-homelab.yaml`: a comment that priority 1002 is taken. The comment is the only change: **no `talosctl` apply needed.** - `NETWORK-traffic-classes.md`: documents VIP replies. Option 1 (literal RoutingRuleConfig 1002/1003) would write the pool prefix a second time, which the repo's rule ("No address is written into this repo twice") forbids. ## Expected impact - **bird rolls one node at a time** (new container, new ConfigMap hash). While a node's bird restarts, table 47 empties and that node's pod egress falls back to main for a few seconds. This is the same as any bird change. - **Pool change: no reallocation expected.** - Cilium v1.20.4 `handlePoolModified` keeps any range whose first and last address match a new block (`LBRange.EqualCIDR`). `RangeFromPrefix` gives `10.2.99.0`–`10.2.99.255` and `fd99::`–`fd99::ff`, the same as today. - `allowFirstLastIPs` is unset, which means allocatable, so nothing new is reserved. - All 10 LB Services also pin their IPs by annotation. - **After rollout:** VIP replies to destinations off the node's own subnets leave over `ingress0` via `10.2.47.1` / `fd02:47::1`, with the VIP as source. That is the link the UDM learned the VIP over by BGP. - Replies to clients on VLAN 40/46/47/48 take table 47's connected routes, the same links as today. - Table 47 on k8s-3 has every destination main has (attached nets, peer pod CIDRs, default). - **HA-relevant:** Home Assistant (`10.2.99.87`), mosquitto-iot (`10.2.99.81`), shelly-manager-firmware (`10.2.99.90`) and k8s-gateway DNS (`10.2.99.83`) are all VIP services. Their replies switch path. Existing TCP sessions (MQTT) may see a retransmit at the switch. No interruption is expected beyond that. ## Not established Where on the `mgmt0` → UDM → VLAN 30 path the Shelly segments were lost is still unknown. This PR restores the designed path. Whether it fixes the slow OTA is the read-out after merge. ## Verification Done: - `kubectl kustomize` builds `infrastructure-inhouse` and `post-crds`. - `shellcheck` 0.11.0 is clean. - `scripts/ci/yaml-citations.sh` passes. - The template was rendered with kubectl against cidr, disabled and range pools. - `vip-rules.sh` was run under Alpine 3.24 busybox `ash` with stub `ip`/`kubectl`: initial add, no-op, pool change (add/remove), API error (skip), range block (exit), no pools (remove all). - busybox `ip -4/-6 rule add|del priority … from … table 47` parses; outside a privileged netns it returns EPERM. After merge, on each node: - `talosctl -n k8s-N get routingrules` shows `1002` for `10.2.99.0/24` and `fd99::/120`. - `talosctl pcap -i ingress0` shows `10.2.99.90` → device data. - `mgmt0` carries none of it. - The Power Strip OTA from shelly-manager completes. Hermes-Card: t_9c084f2c (atlas)

hermes-bot added 1 commit 2026-11-02 06:36:14 +00:00

fix(bird): route LoadBalancer VIP replies into table 47 (#1547 )

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

d0da46b054

A reply from a LoadBalancer VIP reaches the kernel's routing decision
with the VIP as its source: Cilium reverses the service NAT in
from-container before the packet enters the stack. The pod-source rules
1000/1001 do not match it, so it took main's default over mgmt0 while
the request had arrived on ingress0 as the BGP next hop.

bird's DaemonSet gets a vip-rules container that reads every enabled
CiliumLoadBalancerIPPool and keeps one rule per block, priority 1002,
`from <block> lookup 47`, reconciled every minute. The pool stays the
only place the prefixes are written. Like the pod rules, a rule into an
empty table 47 (bird gone) falls through to main.

bgp-pool's blocks move from start/stop to the equivalent CIDRs so each
block maps to one rule. Cilium v1.20.4 LB-IPAM compares a modified
pool's blocks by first and last address (handlePoolModified,
LBRange.EqualCIDR), and allowFirstLastIPs unset keeps .0/.255 and
fd99::/fd99::ff allocatable, so the ranges are kept and no service is
reallocated.

Closes #1547

k.renner closed this pull request

2026-11-02 07:58:00 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1548