netbird: reverse-proxy watchdog (self-heal dead-route 502s) #93

2026-07-02T05:09:34Z

k.renner commented

2026-07-02 05:09:34 +00:00

Context

2026-07-02 public-edge outage: the netbird-ingress DaemonSet rollout (image auto-bump 5dffafe) minted fresh routing-peer identities — which is the intended rollout semantic. The reverse proxy's embedded NetBird client, however, keeps each proxied domain's route pinned to the old peer key and never rebinds (already routed by peer [...]. HA routing disabled, dnsinterceptor/handler.go — unchanged through upstream main). Both cluster members (VPS-1, proxy-1) served only 502/timeouts until their containers were manually restarted.

Change

New shared role netbird-proxy-watchdog, applied by both install-netbird-server.yml and install-netbird-proxy.yml:

systemd timer (1 min) probes status.renner.uno end-to-end through the local proxy (--resolve to 127.0.0.1; VPS-1 via Traefik SNI passthrough, proxy-1 binds :443 directly)
only 502 and 000 count as failures (the dead-route signatures); backend app errors don't trigger
3 consecutive failures → docker restart netbird-proxy, with a 10-min cooldown against restart loops
every probe outcome, restart, and suppression is journaled

This is the interim mitigation; the actual fix is upstream (dns-interceptor rebinding when the owning peer is deleted). Deploy via the existing deploy-netbird-server / deploy-netbird-proxy pipelines after merge.

🤖 Generated with Claude Code

## Context 2026-07-02 public-edge outage: the `netbird-ingress` DaemonSet rollout (image auto-bump `5dffafe`) minted fresh routing-peer identities — which is the intended rollout semantic. The reverse proxy's embedded NetBird client, however, keeps each proxied domain's route pinned to the old peer key and never rebinds (`already routed by peer [...]. HA routing disabled`, `dnsinterceptor/handler.go` — unchanged through upstream `main`). Both cluster members (VPS-1, proxy-1) served only 502/timeouts until their containers were manually restarted. ## Change New shared role `netbird-proxy-watchdog`, applied by both `install-netbird-server.yml` and `install-netbird-proxy.yml`: - systemd timer (1 min) probes `status.renner.uno` end-to-end through the local proxy (`--resolve` to 127.0.0.1; VPS-1 via Traefik SNI passthrough, proxy-1 binds :443 directly) - only `502` and `000` count as failures (the dead-route signatures); backend app errors don't trigger - 3 consecutive failures → `docker restart netbird-proxy`, with a 10-min cooldown against restart loops - every probe outcome, restart, and suppression is journaled This is the interim mitigation; the actual fix is upstream (dns-interceptor rebinding when the owning peer is deleted). Deploy via the existing `deploy-netbird-server` / `deploy-netbird-proxy` pipelines after merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-07-02 05:09:34 +00:00

feat(netbird): reverse-proxy watchdog (self-heal dead-route 502s)

ci/woodpecker/push/deploy-netbird-server Pipeline was successful

Details

edf5362015

The proxy's embedded client pins each proxied domain's route to a
routing-peer identity. netbird-ingress rollouts mint fresh identities by
design, and upstream's dns-interceptor never rebinds a route whose owning
peer is gone ('already routed by peer [...]. HA routing disabled') — the
proxy then serves only 502/timeouts until restarted (2026-07-02 outage).

Add a shared role deploying a systemd timer (1min) on both cluster
members (VPS-1, proxy-1): probe status.renner.uno through the local
proxy, restart the container after 3 consecutive 502/000 results, with a
10min cooldown against restart loops. Every probe outcome is journaled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

k.renner added 1 commit 2026-07-02 07:26:22 +00:00

fix(netbird-watchdog): probe home+immich, require all domains down 8eca54577d

status.renner.uno will move off-cluster, so it's a bad canary. Probe two
permanently cluster-hosted domains instead and count a round as failed
only when EVERY domain fails: one dead backend 502s on its own domain,
while the dead-route state kills all domains at once — this keeps single
app outages from restarting the proxy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

k.renner merged commit 6f519c07ac into master

2026-07-02 13:27:33 +00:00

k.renner referenced this issue from a commit

2026-07-02 13:27:33 +00:00

Merge pull request 'netbird: reverse-proxy watchdog (self-heal dead-route 502s)' (#93) from worktree-netbird-proxy-watchdog into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#93