konairius/homelab
10
0
Fork 0
Code Issues 10 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

LoadBalancer VIP replies leave k8s nodes via mgmt0 instead of ingress0 #1547

New Issue
Closed
opened 2026-11-02 06:26:09 +00:00 by claude-bot · 3 comments
claude-bot commented 2026-11-02 06:26:09 +00:00
Member
Copy Link
Copy Source

Replies from Cilium LoadBalancer VIPs (bgp-pool: 10.2.99.0/24, fd99::/120) leave the node over mgmt0 (VLAN 40). The requests arrive over ingress0 (VLAN 47), as the BGP next hop on that link. NETWORK-traffic-classes.md assigns world↔pod traffic to ingress0.

Observed (k8s-3, 2026-11-02, X MOD1 #9 10.2.30.110 downloading firmware from shelly-manager-firmware 10.2.99.90:80)

  • talosctl pcap -i ingress0 (filter: tcp, host 10.2.30.110): only the device → 10.2.99.90 packets (request, ACKs) appear. None of the 10.2.99.90 → device data appears. Pod-initiated connections from 10.244.2.241 to the device do leave over ingress0.
  • talosctl pcap -i mgmt0, 30 s: 10.2.99.90:80 → 10.2.30.110:55924 data leaves here. The device's MQTT connection to 10.2.99.81:8883 (mosquitto VIP) leaves over mgmt0 too.

Why it takes that path

  • site-homelab.yaml sends traffic with a pod source to table 47 (RoutingRuleConfig 1000 src fd10:244::/48, 1001 src 10.244.0.0/16). Table 47's default is 10.2.47.1 on ingress0.
  • A VIP reply reaches the kernel's routing decision with source 10.2.99.90, not the pod address. Neither rule matches, so main's default (10.2.40.1 via mgmt0) applies.
    • Routing rules on k8s-3: 9 → 2004, 100 → local, 1001 10.244.0.0/16 → 47, 32766 main.
    • Main's IPv4 default: 10.2.40.1 dev mgmt0.

Not shown

These captures were taken while investigating slow Shelly firmware downloads, but they do not show that the asymmetric path causes them.

Fix options (not decided)

  1. RoutingRuleConfig 1002/1003 with src = the VIP pool → table 47. This writes the pool prefix a second time next to CiliumLoadBalancerIPPool bgp-pool, against "No address is written into this repo twice".
  2. Keep the pool in one place and derive the other side from it.
Replies from Cilium LoadBalancer VIPs (`bgp-pool`: `10.2.99.0/24`, `fd99::/120`) leave the node over `mgmt0` (VLAN 40). The requests arrive over `ingress0` (VLAN 47), as the BGP next hop on that link. `NETWORK-traffic-classes.md` assigns world↔pod traffic to `ingress0`. ## Observed (k8s-3, 2026-11-02, X MOD1 #9 `10.2.30.110` downloading firmware from `shelly-manager-firmware` `10.2.99.90:80`) - `talosctl pcap -i ingress0` (filter: tcp, host 10.2.30.110): only the device → `10.2.99.90` packets (request, ACKs) appear. None of the `10.2.99.90` → device data appears. Pod-initiated connections from `10.244.2.241` to the device do leave over `ingress0`. - `talosctl pcap -i mgmt0`, 30 s: `10.2.99.90:80` → `10.2.30.110:55924` data leaves here. The device's MQTT connection to `10.2.99.81:8883` (mosquitto VIP) leaves over `mgmt0` too. ## Why it takes that path - `site-homelab.yaml` sends traffic with a pod source to table 47 (RoutingRuleConfig 1000 `src fd10:244::/48`, 1001 `src 10.244.0.0/16`). Table 47's default is `10.2.47.1` on `ingress0`. - A VIP reply reaches the kernel's routing decision with source `10.2.99.90`, not the pod address. Neither rule matches, so main's default (`10.2.40.1` via `mgmt0`) applies. - Routing rules on k8s-3: 9 → 2004, 100 → local, 1001 `10.244.0.0/16` → 47, 32766 main. - Main's IPv4 default: `10.2.40.1` dev `mgmt0`. ## Not shown These captures were taken while investigating slow Shelly firmware downloads, but they do not show that the asymmetric path causes them. ## Fix options (not decided) 1. RoutingRuleConfig 1002/1003 with `src` = the VIP pool → table 47. This writes the pool prefix a second time next to `CiliumLoadBalancerIPPool bgp-pool`, against "No address is written into this repo twice". 2. Keep the pool in one place and derive the other side from it.
hermes-bot commented 2026-11-02 06:26:13 +00:00
Member
Copy Link
Copy Source

Hermes kanban card t_5ed913f9 for this issue is assigned to the coordinator (default) for routing.

Hermes kanban card `t_5ed913f9` for this issue is assigned to the coordinator (default) for routing. <!-- hermes-kanban-card:t_5ed913f9 -->
hermes-bot added the atlas label 2026-11-02 06:26:41 +00:00
hermes-bot commented 2026-11-02 06:26:47 +00:00
Member
Copy Link
Copy Source

Hermes kanban card t_5ed913f9 completed by default:

Routed to atlas: the issue is node network routing on Talos/Cilium (LoadBalancer VIP reply source-routing, RoutingRuleConfig in site-homelab.yaml, the bgp-pool), which is platform/network work.

Hermes kanban card `t_5ed913f9` completed by default: Routed to atlas: the issue is node network routing on Talos/Cilium (LoadBalancer VIP reply source-routing, RoutingRuleConfig in site-homelab.yaml, the bgp-pool), which is platform/network work. <!-- hermes-kanban-done:t_5ed913f9:69 -->
hermes-bot commented 2026-11-02 06:26:47 +00:00
Member
Copy Link
Copy Source

Hermes kanban card t_9c084f2c for this issue is assigned to atlas (routed by coordinator card t_5ed913f9).

Hermes kanban card `t_9c084f2c` for this issue is assigned to atlas (routed by coordinator card `t_5ed913f9`). <!-- hermes-kanban-card:t_9c084f2c:atlas -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels atlas
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1547