traefik: disable global CrowdSec enforcement (caused v6 ingress outage) #80

2026-06-27T14:11:44Z

k.renner commented

2026-06-27 14:11:44 +00:00

Incident fix. The global CrowdSec bouncer middleware (#77/#78) caused an IPv6 ingress outage within hours of going live.

Root cause: Traefik records a single shared/SNAT'd source IP (fd10:244::2de1) for IPv6 ingress — it does not see real per-client IPs on the v6 path. So when the aggregate traffic through that source tripped crowdsecurity/http-admin-interface-probing (driven by the e2e suite's scan-like requests + my verification probing), CrowdSec banned that one shared IP and the bouncer 403'd every IPv6 client (tea/woodpecker/HTTPS-git, and any v6-first user of the renner.uno services). IPv4 + SSH were unaffected.

This PR removes the global entrypoint-middleware attachment, disabling enforcement. The plugin (experimental.plugins) and crowdsec-middleware.yaml stay in place but inert, so re-enabling is a one-line change. Detection is untouched — the agents keep parsing Traefik logs (#76).

Already done out-of-band: cleared the active ban (cscli decisions delete --ip fd10:244::2de1), which restored v6 access (verified 200).

Before re-enabling enforcement, two things must be fixed:

Client-IP visibility — Traefik must see real client IPs on the v6 path (forwardedHeaders/trustedIPs, or fix the IPv6 masquerade), so bans are per-client, not collateral.
Internal/LAN whitelist — so the e2e suite and admin activity don't ban the cluster's own ingress (same class as the existing pve-ui-polling-whitelist).

🤖 Generated with Claude Code

**Incident fix.** The global CrowdSec bouncer middleware (#77/#78) caused an **IPv6 ingress outage** within hours of going live. **Root cause:** Traefik records a single shared/SNAT'd source IP (`fd10:244::2de1`) for IPv6 ingress — it does not see real per-client IPs on the v6 path. So when the aggregate traffic through that source tripped `crowdsecurity/http-admin-interface-probing` (driven by the e2e suite's scan-like requests + my verification probing), CrowdSec banned that one shared IP and the bouncer **403'd every IPv6 client** (tea/woodpecker/HTTPS-git, and any v6-first user of the renner.uno services). IPv4 + SSH were unaffected. **This PR** removes the global entrypoint-middleware attachment, disabling enforcement. The plugin (`experimental.plugins`) and `crowdsec-middleware.yaml` stay in place but inert, so re-enabling is a one-line change. **Detection is untouched** — the agents keep parsing Traefik logs (#76). Already done out-of-band: cleared the active ban (`cscli decisions delete --ip fd10:244::2de1`), which restored v6 access (verified 200). **Before re-enabling enforcement, two things must be fixed:** 1. **Client-IP visibility** — Traefik must see real client IPs on the v6 path (forwardedHeaders/trustedIPs, or fix the IPv6 masquerade), so bans are per-client, not collateral. 2. **Internal/LAN whitelist** — so the e2e suite and admin activity don't ban the cluster's own ingress (same class as the existing `pve-ui-polling-whitelist`). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-27 14:11:45 +00:00

traefik: disable global CrowdSec enforcement (caused v6 ingress outage) 5073eed8e5

The global bouncer middleware banned all IPv6 ingress: Traefik sees one shared/SNAT'd v6 source, so a single http-admin-interface-probing hit (the e2e scan) 403'd every v6 client. Remove the global entrypoint-middleware attachment; keep the plugin + Middleware CRD inert so re-enabling is one line after fixing client-IP visibility + adding an internal whitelist. Detection (agents) stays on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 9b837ecb9f into master

2026-06-27 14:27:19 +00:00

k.renner referenced this issue from a commit

2026-06-27 14:27:20 +00:00

Merge pull request 'traefik: disable global CrowdSec enforcement (caused v6 ingress outage)' (#80) from traefik-disable-crowdsec-enforcement into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#80