netbird-ingress: graceful drain + paced roll so rolls don't drop public ingress #152

2026-07-11T08:38:07Z

k.renner commented

2026-07-11 08:38:07 +00:00

Problem: every netbird image bump (image-automation, ~weekly) rolls the netbird-ingress DaemonSet and takes public ingress (home.renner.uno) down for ~1 min. Confirmed 2026-07-11: the uptime monitor's outage (08:58 local) lines up exactly with the 3 peers rolling at 06:56:18/39/58 UTC. Not the IPS/firewall — the UDM was idle at that time and the 'intrusion' alerts were the admin laptop hitting its own ingress (all false positives).

Root cause: the redundant roll (maxUnavailable:1, 2/3 peers always up) should be seamless, but the failover isn't:

No preStop drain — a rolled peer is only SIGTERM'd, so the mesh keeps the dead peer in the HA route group until a keepalive timeout (~25-30s); traffic through it black-holes meanwhile. Worsened by ephemeral enrollment (emptyDir): the restarted pod enrolls a new peer while the old one lingers until reaped.
No minReadySeconds — the roll advances as soon as netbird status --check ready passes (seconds), faster than the mesh re-converges onto the freshly-enrolled peer, so the route is degraded for the whole roll.

Fix:

preStop: netbird down — deregister the peer before it dies → immediate failover to the other per-node peers.
minReadySeconds: 45 — hold each new peer Ready until the mesh converges before rolling the next → always ≥2 peers actually serving.

Keeps auto-updates on; turns the surprise ~1-min outage into a seamless roll.

Validation: watch the uptime monitor across the next roll — either the next natural image bump, or a deliberate kubectl rollout restart ds/netbird-ingress -n ingress. Expect no (or sub-second) outage.

**Problem:** every netbird image bump (image-automation, ~weekly) rolls the netbird-ingress DaemonSet and takes public ingress (home.renner.uno) down for ~1 min. Confirmed 2026-07-11: the uptime monitor's outage (08:58 local) lines up exactly with the 3 peers rolling at 06:56:18/39/58 UTC. **Not** the IPS/firewall — the UDM was idle at that time and the 'intrusion' alerts were the admin laptop hitting its own ingress (all false positives). **Root cause:** the redundant roll (maxUnavailable:1, 2/3 peers always up) *should* be seamless, but the failover isn't: - **No preStop drain** — a rolled peer is only SIGTERM'd, so the mesh keeps the dead peer in the HA route group until a keepalive timeout (~25-30s); traffic through it black-holes meanwhile. Worsened by ephemeral enrollment (emptyDir): the restarted pod enrolls a *new* peer while the old one lingers until reaped. - **No minReadySeconds** — the roll advances as soon as `netbird status --check ready` passes (seconds), faster than the mesh re-converges onto the freshly-enrolled peer, so the route is degraded for the whole roll. **Fix:** - `preStop: netbird down` — deregister the peer before it dies → immediate failover to the other per-node peers. - `minReadySeconds: 45` — hold each new peer Ready until the mesh converges before rolling the next → always ≥2 peers actually serving. Keeps auto-updates on; turns the surprise ~1-min outage into a seamless roll. **Validation:** watch the uptime monitor across the next roll — either the next natural image bump, or a deliberate `kubectl rollout restart ds/netbird-ingress -n ingress`. Expect no (or sub-second) outage.

k.renner added 1 commit 2026-07-11 08:38:07 +00:00

netbird-ingress: graceful drain + paced roll so rolls don't drop public ingress 67353b8e29

Every netbird image bump (image-automation, ~weekly) rolls this DaemonSet and
takes public ingress (home.renner.uno, mesh-only via the HA route group) down
for ~1 min — observed 2026-07-11 08:58 on the uptime monitor, correlated exactly
with the 3 peers rolling at 06:56:18/39/58 UTC. It is NOT the IPS/firewall
(the UDM was idle; the "intrusions" were the admin laptop hitting its own
ingress) — it's the roll itself failing over slowly.

Root cause: the redundant roll (maxUnavailable:1, 2/3 peers always up) should be
seamless, but two gaps break the failover:
  - No preStop drain: a rolled peer is only SIGTERM'd, so the mesh keeps the
    dead peer in the route group until a keepalive timeout (~25-30s); traffic
    riding through it black-holes until then. Compounded by ephemeral
    enrollment (emptyDir) — the restarted pod enrolls a NEW peer and the old one
    lingers until reaped.
  - No minReadySeconds: the roll advances to the next node as soon as
    `netbird status --check ready` passes (seconds), faster than the mesh
    re-converges onto the freshly-enrolled peer — so the HA route is degraded
    for the whole ~1-min roll.

Fix:
  - preStop `netbird down` — deregister the peer from management before the
    process dies, so the HA route fails over to the other per-node peers
    immediately instead of after a timeout.
  - minReadySeconds: 45 — hold each new peer Ready until the mesh has converged
    onto it before rolling the next, so >=2 peers are always actually serving.

Keeps auto-updates on; turns the surprise ~1-min outage into a seamless roll.
Validation: watch the uptime monitor across the next roll (natural image bump
or a deliberate `kubectl rollout restart ds/netbird-ingress`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit b04546e14e into master

2026-07-11 08:48:21 +00:00

k.renner referenced this issue from a commit

2026-07-11 08:48:22 +00:00

Merge pull request 'netbird-ingress: graceful drain + paced roll so rolls don't drop public ingress' (#152) from fix/netbird-ingress-graceful-drain into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#152