traefik: real client-IP visibility via X-Forwarded-For (CrowdSec prereq #1) #81

Merged
k.renner merged 1 commits from traefik-client-ip-visibility into master 2026-06-27 15:48:02 +00:00
Owner

Prerequisite #1 of re-enabling CrowdSec enforcement — fix the root cause of the 2026-06-27 IPv6 outage.

Why all v6 clients shared one source: fd10:244::2de1 is the netbird-ingress-qxxc5 pod's IPv6. The rootless routing peer re-originates every mesh + WAN-via-proxy connection from its own pod IP (userspace WireGuard + first-party sockets — the Cilium socketLB workaround), so Traefik's TCP peer for all that traffic is one shared pod IP. A single CrowdSec ban on it 403'd everyone.

The real client IP was there all along — the NetBird reverse-proxy (HTTP mode, on the VPS) sets X-Forwarded-For with the real WAN client IP. Traefik just wasn't configured to read it.

This PR:

  • ports.websecure.forwardedHeaders.trustedIPs: [10.244.0.0/16, fd10:244::/56] (the pod CIDRs = the re-originators) → Traefik resolves the real client IP from XFF. Verified-valid chart-40.x key (native, not additionalArguments).
  • Mirrors it in the inert crowdsec-bouncer Middleware (forwardedHeadersTrustedIPs) so the bouncer evaluates real IPs when re-enabled.

Safe — a no-op where no XFF exists (direct LAN/mesh keep their RemoteAddr). Enforcement stays OFF.

Residual (handled by prereq #2): direct mesh clients have no XFF, so they still collapse to the netbird-ingress pod IP — that's covered by an internal allowlist before enforcement goes back on.

After merge I'll verify Traefik's access logs show real external client IPs (not fd10:244::2de1) for WAN-via-proxy traffic.

🤖 Generated with Claude Code

**Prerequisite #1 of re-enabling CrowdSec enforcement** — fix the root cause of the 2026-06-27 IPv6 outage. **Why all v6 clients shared one source:** `fd10:244::2de1` is the **netbird-ingress-qxxc5 pod's IPv6**. The rootless routing peer re-originates every mesh + WAN-via-proxy connection from its own pod IP (userspace WireGuard + first-party sockets — the Cilium socketLB workaround), so Traefik's TCP peer for all that traffic is one shared pod IP. A single CrowdSec ban on it 403'd everyone. **The real client IP was there all along** — the NetBird reverse-proxy (HTTP mode, on the VPS) sets `X-Forwarded-For` with the real WAN client IP. Traefik just wasn't configured to read it. **This PR:** - `ports.websecure.forwardedHeaders.trustedIPs: [10.244.0.0/16, fd10:244::/56]` (the pod CIDRs = the re-originators) → Traefik resolves the real client IP from XFF. Verified-valid chart-40.x key (native, not `additionalArguments`). - Mirrors it in the inert `crowdsec-bouncer` Middleware (`forwardedHeadersTrustedIPs`) so the bouncer evaluates real IPs when re-enabled. **Safe** — a no-op where no XFF exists (direct LAN/mesh keep their RemoteAddr). **Enforcement stays OFF.** **Residual (handled by prereq #2):** direct mesh clients have no XFF, so they still collapse to the netbird-ingress pod IP — that's covered by an internal allowlist before enforcement goes back on. After merge I'll verify Traefik's access logs show real external client IPs (not `fd10:244::2de1`) for WAN-via-proxy traffic. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
k.renner added 1 commit 2026-06-27 14:42:47 +00:00
The netbird-ingress rootless routing peer re-originates every mesh + WAN-via-proxy connection from its own pod IP (e.g. fd10:244::2de1 = netbird-ingress-qxxc5), so Traefik's TCP peer for all that traffic is one shared pod IP — which is what let a single CrowdSec ban 403 every IPv6 client (2026-06-27 outage). The NetBird reverse-proxy (HTTP mode) already sets X-Forwarded-For with the real WAN client IP; Traefik just wasn't reading it.

Set ports.websecure.forwardedHeaders.trustedIPs to the pod CIDRs (10.244.0.0/16, fd10:244::/56 — the re-originators), so Traefik resolves the real client IP from XFF. Mirror it in the (still-inert) crowdsec-bouncer Middleware via forwardedHeadersTrustedIPs so it's ready when enforcement is re-enabled. Safe no-op where no XFF exists (direct LAN/mesh keep their RemoteAddr). Enforcement stays OFF — re-enabling still needs the internal allowlist (prereq #2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.renner merged commit 0a3e4f90e3 into master 2026-06-27 15:48:02 +00:00
Sign in to join this conversation.