docs(proposal): Traefik VIP survives an ECMP node switch (Maglev + DSR)

Long-lived connections through the Traefik VIP get RST from the node that does not hold the flow. The proposal records the Hubble evidence and moves the Traefik Service to externalTrafficPolicy: Cluster on Cilium Maglev + DSR, staged in two PRs with a live test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 15:41:29 +02:00
1 changed files with 105 additions and 0 deletions
@@ -0,0 +1,105 @@
+# Traefik VIP: survive an ECMP node switch (Maglev + DSR)
+
+## Problem
+
+Long-lived connections through the Traefik VIP (`10.2.99.86` / `fd99::86`)
+die with a TCP RST from the VIP side. Most visible on the Hermes dashboard's
+WebSockets (`/api/pty`, `/api/events`, `/api/ws`), which drop every few
+seconds to minutes; every other service behind the VIP is exposed the same way.
+
+## Evidence (2026-10-19)
+
+- Traefik Service: `externalTrafficPolicy: Local`, so the VIP is advertised by
+  BGP only from k8s-2 and k8s-3 (the nodes running a Traefik pod). The UDM runs
+  `maximum-paths 4` (`unifi/bgp.conf`), i.e. ECMP across both.
+- Hubble, live, client `fd02:50::28e1:54c6:cc63:aa61`:
+  - port 38100: connection established on k8s-2 (Traefik `9k4v8`, ACKs at
+    13:18:49Z); at 13:28:48Z Traefik `9n2bp` on **k8s-3** sends RST to the
+    client for that port.
+  - port 42698 (`2003:d6:974c:7e04:…`): Traefik on k8s-2 sends data
+    (13:28:43Z `ACK, PSH`) while the client's packets of the same 4-tuple
+    arrive at k8s-3 (RSTs 13:28:29Z…13:28:57Z, exponential back-off).
+- Test WebSocket (`logcli --tail`) 13:15:16Z: client receives
+  `connection reset by peer` from `[fd99::86]:443` at 13:17:00Z while the
+  owning Traefik pod (`9k4v8`) keeps the connection until 13:17:28Z.
+- BGP sessions stable (91 h uptime on both nodes), no pod restarts.
+
+So packets of one established flow reach a node that does not hold it, and
+that node's Traefik resets it. What makes the UDM move a flow is not
+established; the design below does not depend on it.
+
+## Why the current setup cannot survive it
+
+With `Local`, each node only knows the connections of its own Traefik pod. A
+packet that lands on the other node meets a socket that does not exist → RST.
+
+With `Cluster` in the default SNAT mode it still breaks: the second node picks
+a backend and SNATs, but the NAT/conntrack state of the flow lives on the
+first node, so the backend sees a different source and the client a foreign
+connection.
+
+## Proposal
+
+1. Cilium (`infrastructure/cilium/helmrelease.yaml`):
+   ```yaml
+   loadBalancer:
+     algorithm: maglev   # same 5-tuple → same backend on every node
+     mode: dsr           # backend replies straight to the client; no reply-path state on the ingress node
+     dsrDispatch: opt    # native routing; IPv4 option / IPv6 destination-option header
+   maglev:
+     hashSeed: <12 random bytes, base64>   # identical on all agents
+   ```
+   `hashSeed` is not secret but must be stable; it goes in git.
+2. Traefik Service: `externalTrafficPolicy: Cluster`. The VIP is then
+   advertised from all three nodes; whichever node a packet hits, Maglev picks
+   the same Traefik pod and DSR forwards it there, keeping the client source IP
+   (DSR is the documented source-IP-preserving option for `Cluster`).
+
+`mode: dsr` (not `hybrid`) so UDP/443 (HTTP/3) also keeps the client IP;
+`hybrid` would SNAT UDP.
+
+## Documented limits (Cilium v1.20 docs)
+
+- Maglev "increases the probability that all nodes select the same endpoint for
+  the same flow" — consistent hashing, not a guarantee, and only for N-S traffic
+  ([BGP operation](https://docs.cilium.io/en/v1.20/network/bgp-control-plane/bgp-control-plane-operation/)).
+  A backend change (Traefik rollout) can still move a minority of flows.
+- DSR encodes the service in the TCP SYN only; whether a mid-flow packet that
+  lands on a node that never saw the SYN reaches the right backend is not
+  stated in the docs. It is the property this change exists for, so it is the
+  first thing the rollout tests.
+- `dsrDispatch: opt` reduces the effective MTU for node-to-node forwarded
+  packets (option header). `geneve` is the alternative if IP options cause
+  trouble.
+- Changing the global `loadBalancer.mode` on a running cluster is not
+  documented as hitless; treat the agent roll as disruptive for connections
+  through LoadBalancer services.
+- The mode is global: every LoadBalancer Service gets DSR. Services that stay on
+  `Local` keep their current behaviour (no cross-node forwarding).
+- [kube-proxy-free: Maglev, DSR, Limitations](https://docs.cilium.io/en/v1.20/network/kubernetes/kubeproxy-free/)
+
+## Rollout
+
+1. PR 1, Cilium values only. Agents roll one node at a time
+   (`maxUnavailable: 1`); wait for Cilium health and Ceph `HEALTH_OK` between
+   nodes. Traefik stays on `Local`, so nothing forwards cross-node yet. Verify
+   `cilium-config`: `bpf-lb-algorithm=maglev`, `bpf-lb-mode=dsr`.
+2. PR 2, Traefik `externalTrafficPolicy: Cluster`. Verify the VIP is advertised
+   from all three nodes (`cilium-dbg shell bgp/routes`) and that client IPs in
+   Traefik's access log are still real client addresses (v4 and v6), not node
+   IPs.
+3. Test: hold a WebSocket through the VIP and watch Hubble on all three nodes
+   for RSTs to the client from a non-owning node, as in the evidence above.
+   Success = no RST from a node that does not hold the flow; Hermes dashboard
+   logs no `keepalive ping timeout`.
+
+Rollback: revert PR 2 (back to `Local`), then PR 1.
+
+## Not chosen
+
+- UDM `net.ipv6.fib_multipath_hash_policy=1` / resilient hashing: would address
+  the move itself but lives outside git on the UDM and does not cover a node
+  failure. Complementary, not a substitute.
+- Client `net.core.txrehash=0`: per client, not a fix.
+- A single advertising node: removes ECMP and with it the redundancy the BGP
+  pool exists for.