netbird-proxy-watchdog: flood watchdog for instant route-manager-wedge recovery #153

2026-07-11T15:05:02Z

k.renner commented

2026-07-11 15:05:02 +00:00

Root cause (this session's investigation): the reverse-proxy wedges when an ephemeral NetBird routing peer (a k8s ingress pod) is deleted on a roll — its route manager pins the gone peer and never re-selects a healthy HA peer, serving errors for every domain behind that route until restarted. Upstream bug, unfixed as of netbird v0.74.4 (#3976; dnsinterceptor disables HA for an already-routed IP + recalculateRoutes aborts on the dead peer's teardown error). A restart is the only reliable clear.

Why the existing watchdog was too slow: its 127.0.0.1 probe stays warm ~3 min into the wedge, then needs 3×60s rounds — 5–9 min on 2026-07-11, and with both proxy members wedged the public outage lasted until the slower recovered (~6 min on the monitors).

Fix: the wedge floods couldn't fetch peer state in the proxy log the instant it starts — an unambiguous fingerprint (14 days of logs: only ever in bursts of tens/min, never a benign trickle). Add a persistent follower that restarts the proxy on the first matching line.

Deliberate trade: ~1/6 of bursts self-recover and get an unnecessary restart — but a proxy restart is near-instant, so a rare spurious restart beats a multi-minute outage (confirmed acceptable with the owner).

new netbird-proxy-flood-watchdog script (docker logs -f | grep → restart) + Type=simple, Restart=always unit
shares LAST_RESTART_FILE/cooldown with the probe watchdog → no loops, no double-restart; URL-probe watchdog unchanged as backstop for other failure modes (e.g. the 07-04 outage that had no peer-state burst)

Deploying only adds a systemd service to the two proxy hosts — it doesn't touch the proxy or ingress, so it can't itself cause an outage. Apply via the install-netbird-server / install-netbird-proxy plays.

**Root cause (this session's investigation):** the reverse-proxy wedges when an ephemeral NetBird routing peer (a k8s ingress pod) is *deleted* on a roll — its route manager pins the gone peer and never re-selects a healthy HA peer, serving errors for every domain behind that route until restarted. Upstream bug, unfixed as of netbird v0.74.4 ([#3976](https://github.com/netbirdio/netbird/issues/3976); dnsinterceptor disables HA for an already-routed IP + `recalculateRoutes` aborts on the dead peer's teardown error). A restart is the only reliable clear. **Why the existing watchdog was too slow:** its `127.0.0.1` probe stays warm ~3 min into the wedge, then needs 3×60s rounds — 5–9 min on 2026-07-11, and with both proxy members wedged the public outage lasted until the slower recovered (~6 min on the monitors). **Fix:** the wedge floods `couldn't fetch peer state` in the proxy log the instant it starts — an unambiguous fingerprint (14 days of logs: only ever in bursts of tens/min, never a benign trickle). Add a persistent follower that restarts the proxy on the first matching line. **Deliberate trade:** ~1/6 of bursts self-recover and get an unnecessary restart — but a proxy restart is near-instant, so a rare spurious restart beats a multi-minute outage (confirmed acceptable with the owner). - new `netbird-proxy-flood-watchdog` script (`docker logs -f | grep` → restart) + `Type=simple, Restart=always` unit - shares `LAST_RESTART_FILE`/cooldown with the probe watchdog → no loops, no double-restart; **URL-probe watchdog unchanged as backstop** for other failure modes (e.g. the 07-04 outage that had no peer-state burst) Deploying only adds a systemd service to the two proxy hosts — it doesn't touch the proxy or ingress, so it can't itself cause an outage. Apply via the `install-netbird-server` / `install-netbird-proxy` plays.

k.renner added 1 commit 2026-07-11 15:05:02 +00:00

netbird-proxy-watchdog: add flood watchdog for instant route-manager-wedge recovery 6c66ca3c42

The reverse-proxy wedges when an ephemeral NetBird routing peer (a k8s ingress
pod) is DELETED: its route manager pins the now-gone peer and never re-selects
a healthy HA-group peer, serving errors for every domain behind that route
until restarted. Upstream bug, unfixed as of netbird v0.74.4 (netbirdio/netbird
#3976; dnsinterceptor disables HA for an already-routed IP + recalculateRoutes
aborts on the dead peer's teardown error). A restart is the only reliable clear.

The existing URL-probe watchdog does eventually catch it, but far too slowly:
its 127.0.0.1 probe stays warm for ~3 min after the wedge starts, then needs 3
consecutive 60s rounds -- so it took 5-9 min on 2026-07-11, and both proxy
members wedged so the public outage lasted until the slower one recovered.

The wedge floods "couldn't fetch peer state" in the proxy's own log the instant
it happens, minutes before the probe notices. That flood is an unambiguous
fingerprint: 14 days of logs show it ONLY in bursts of tens per minute, never a
benign trickle. So add a persistent follower that restarts the proxy on the
first matching line.

Trade, chosen deliberately: ~1/6 of these bursts self-recover (the re-select
race sometimes wins) and get an unnecessary restart -- but a proxy restart is
near-instant, so a rare spurious restart beats a multi-minute outage.

- new netbird-proxy-flood-watchdog.sh (docker logs -f | grep, restart on match)
- new netbird-proxy-flood-watchdog.service (Type=simple, Restart=always)
- shares LAST_RESTART_FILE + cooldown with the probe watchdog (no loops, no
  double-restart); URL-probe watchdog kept unchanged as the backstop.

Deploying only adds a systemd service; it does not touch the proxy or ingress,
so it can't itself cause an outage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 11ec6449db into master

2026-07-11 19:16:27 +00:00

k.renner referenced this issue from a commit

2026-07-11 19:16:28 +00:00

Merge pull request 'netbird-proxy-watchdog: flood watchdog for instant route-manager-wedge recovery' (#153) from fix/netbird-proxy-flood-watchdog into master

k.renner referenced this pull request

2026-07-11 19:42:48 +00:00

woodpecker: auto-deploy netbird-proxy-watchdog role changes #154

k.renner referenced this issue from a commit

2026-07-11 19:42:48 +00:00

woodpecker: auto-deploy netbird-proxy-watchdog role changes

Sign in to join this conversation.