Every netbird image bump (image-automation, ~weekly) rolls this DaemonSet and
takes public ingress (home.renner.uno, mesh-only via the HA route group) down
for ~1 min — observed 2026-07-11 08:58 on the uptime monitor, correlated exactly
with the 3 peers rolling at 06:56:18/39/58 UTC. It is NOT the IPS/firewall
(the UDM was idle; the "intrusions" were the admin laptop hitting its own
ingress) — it's the roll itself failing over slowly.
Root cause: the redundant roll (maxUnavailable:1, 2/3 peers always up) should be
seamless, but two gaps break the failover:
- No preStop drain: a rolled peer is only SIGTERM'd, so the mesh keeps the
dead peer in the route group until a keepalive timeout (~25-30s); traffic
riding through it black-holes until then. Compounded by ephemeral
enrollment (emptyDir) — the restarted pod enrolls a NEW peer and the old one
lingers until reaped.
- No minReadySeconds: the roll advances to the next node as soon as
`netbird status --check ready` passes (seconds), faster than the mesh
re-converges onto the freshly-enrolled peer — so the HA route is degraded
for the whole ~1-min roll.
Fix:
- preStop `netbird down` — deregister the peer from management before the
process dies, so the HA route fails over to the other per-node peers
immediately instead of after a timeout.
- minReadySeconds: 45 — hold each new peer Ready until the mesh has converged
onto it before rolling the next, so >=2 peers are always actually serving.
Keeps auto-updates on; turns the surprise ~1-min outage into a seamless roll.
Validation: watch the uptime monitor across the next roll (natural image bump
or a deliberate `kubectl rollout restart ds/netbird-ingress`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8