The reverse-proxy wedges when an ephemeral NetBird routing peer (a k8s ingress
pod) is DELETED: its route manager pins the now-gone peer and never re-selects
a healthy HA-group peer, serving errors for every domain behind that route
until restarted. Upstream bug, unfixed as of netbird v0.74.4 (netbirdio/netbird
#3976; dnsinterceptor disables HA for an already-routed IP + recalculateRoutes
aborts on the dead peer's teardown error). A restart is the only reliable clear.
The existing URL-probe watchdog does eventually catch it, but far too slowly:
its 127.0.0.1 probe stays warm for ~3 min after the wedge starts, then needs 3
consecutive 60s rounds -- so it took 5-9 min on 2026-07-11, and both proxy
members wedged so the public outage lasted until the slower one recovered.
The wedge floods "couldn't fetch peer state" in the proxy's own log the instant
it happens, minutes before the probe notices. That flood is an unambiguous
fingerprint: 14 days of logs show it ONLY in bursts of tens per minute, never a
benign trickle. So add a persistent follower that restarts the proxy on the
first matching line.
Trade, chosen deliberately: ~1/6 of these bursts self-recover (the re-select
race sometimes wins) and get an unnecessary restart -- but a proxy restart is
near-instant, so a rare spurious restart beats a multi-minute outage.
- new netbird-proxy-flood-watchdog.sh (docker logs -f | grep, restart on match)
- new netbird-proxy-flood-watchdog.service (Type=simple, Restart=always)
- shares LAST_RESTART_FILE + cooldown with the probe watchdog (no loops, no
double-restart); URL-probe watchdog kept unchanged as the backstop.
Deploying only adds a systemd service; it does not touch the proxy or ingress,
so it can't itself cause an outage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8