fix(netbird-ingress): restart peers stuck in NeedsLogin, alert on NotReady #403

2026-07-25T07:07:59Z

claude-bot commented

2026-07-25 07:07:59 +00:00

Incident

On 2026-07-24 a ~24-minute WAN outage to netbird.renner.uno exceeded NetBird's 10-minute ephemeral-peer lifetime, so the management server deregistered all three ingress routing peers. On reconnect the client got PermissionDenied: peer is not registered, which it treats as unrecoverable: it exits its retry loop and parks in NeedsLogin. The daemon cannot re-enroll on its own (NB_SETUP_KEY is only consumed by netbird up in the entrypoint at container start). NeedsLogin fails the readiness probe but not the --check live liveness probe, so all three pods sat Running-but-NotReady for ~20 h — the WAN edge ran degraded with no alert — until they were deleted by hand.

Changes

Targeted liveness probe (kubernetes/infrastructure/netbird-ingress/daemonset.yaml): fail liveness only on the dead auth states (NeedsLogin/LoginFailed/SessionExpired). In this deployment those are always fatal and always fixed by a restart (re-enrolls via NB_SETUP_KEY — exactly what the manual pod delete did). Deliberately not a plain --check ready liveness: that would also fire during transient management outages, where the established WireGuard data plane still works and a restart would kill it. Transient Disconnected/Connecting states don't match the pattern and keep retrying silently as before. The ephemeral setup key stays — it is doing its intended job of reaping stale peers from the emptyDir identity churn; with this probe the reaping becomes harmless.
Alerting (kubernetes/apps/monitoring-rules/prometheusrule-netbird-ingress.yaml): warning when the DaemonSet runs with fewer Ready pods than desired for >10 m, critical when 0 are Ready for >5 m.

Verification

kubectl kustomize on monitoring-rules and a client dry-run of the DaemonSet both pass.
The exact probe command was exec'd against a live healthy pod (exit 0); the rootless image ships /bin/sh and /bin/grep.
The NeedsLogin failure string was taken verbatim from the 2026-07-24 probe events ("readiness check: daemon status is NeedsLogin").
Startup ordering is safe: liveness only starts after the startup probe (--check startup) has succeeded, i.e. after enrollment completed, so the probe cannot false-positive on a slow first netbird up.

🤖 Generated with Claude Code

## Incident On 2026-07-24 a ~24-minute WAN outage to netbird.renner.uno exceeded NetBird's 10-minute ephemeral-peer lifetime, so the management server deregistered all three ingress routing peers. On reconnect the client got `PermissionDenied: peer is not registered`, which it treats as unrecoverable: it exits its retry loop and parks in **NeedsLogin**. The daemon cannot re-enroll on its own (`NB_SETUP_KEY` is only consumed by `netbird up` in the entrypoint at container start). NeedsLogin fails the readiness probe but not the `--check live` liveness probe, so all three pods sat Running-but-NotReady for ~20 h — the WAN edge ran degraded with no alert — until they were deleted by hand. ## Changes 1. **Targeted liveness probe** (`kubernetes/infrastructure/netbird-ingress/daemonset.yaml`): fail liveness only on the dead auth states (`NeedsLogin`/`LoginFailed`/`SessionExpired`). In this deployment those are always fatal and always fixed by a restart (re-enrolls via `NB_SETUP_KEY` — exactly what the manual pod delete did). Deliberately **not** a plain `--check ready` liveness: that would also fire during transient management outages, where the established WireGuard data plane still works and a restart would kill it. Transient Disconnected/Connecting states don't match the pattern and keep retrying silently as before. The ephemeral setup key stays — it is doing its intended job of reaping stale peers from the emptyDir identity churn; with this probe the reaping becomes harmless. 2. **Alerting** (`kubernetes/apps/monitoring-rules/prometheusrule-netbird-ingress.yaml`): warning when the DaemonSet runs with fewer Ready pods than desired for >10 m, critical when 0 are Ready for >5 m. ## Verification - `kubectl kustomize` on monitoring-rules and a client dry-run of the DaemonSet both pass. - The exact probe command was exec'd against a live healthy pod (exit 0); the rootless image ships `/bin/sh` and `/bin/grep`. - The NeedsLogin failure string was taken verbatim from the 2026-07-24 probe events ("readiness check: daemon status is NeedsLogin"). - Startup ordering is safe: liveness only starts after the startup probe (`--check startup`) has succeeded, i.e. after enrollment completed, so the probe cannot false-positive on a slow first `netbird up`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-25 07:08:00 +00:00

fix(netbird-ingress): restart peers stuck in NeedsLogin, alert on NotReady

ci/woodpecker/pr/pr-review Pipeline was successful

Details

c59ad31db9

A 24-min WAN outage to netbird.renner.uno on 2026-07-24 exceeded the
10-min ephemeral-peer lifetime, so management deregistered all three
routing peers. The client treats 'peer is not registered' as
unrecoverable and parks in NeedsLogin; that fails readiness but not the
'--check live' liveness probe, so the pods sat Running-but-NotReady for
~20 h until they were deleted by hand.

- liveness probe now fails on the dead auth states (NeedsLogin /
  LoginFailed / SessionExpired) and only on those: a restart re-runs the
  entrypoint, which re-enrolls with NB_SETUP_KEY. Transient
  Disconnected/Connecting states still retry silently, so a management
  outage does not kill the working WireGuard data plane.
- new PrometheusRule: warning when the DaemonSet runs degraded >10m,
  critical when 0 peers are Ready >5m.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

review-bot reviewed

2026-07-25 07:12:18 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit c59ad31db9315b6d9a84010c546138a0fd06e60d.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Two changes: (1) narrows the netbird-ingress liveness probe to restart only on the dead auth states (NeedsLogin/LoginFailed/SessionExpired), fixing the incident where pods sat Running-but-NotReady for ~20h with no restart and no alert; (2) adds a PrometheusRule alerting on DaemonSet Ready-pod deficits. Both changes are well-reasoned, match this repo's existing conventions (compare prometheusrule-velero.yaml, prometheusrule-woodpecker-rotation.yaml), and the shell logic in the new liveness probe is correct as written. No blocking issues found.

🟡 Suggestions

kubernetes/infrastructure/netbird-ingress/daemonset.yaml:128-137 — the new probe drops the old --check live signal entirely and only restarts on a whitelisted match (NeedsLogin|LoginFailed|SessionExpired); anything else — including a daemon that's alive as a process but returns a fast, non-timeout error from --check ready that isn't one of those three strings (e.g. IPC/socket errors, corrupted state) — now falls through to exit 0 ("healthy") where the old --check live-based probe would likely have failed it via its own exit code. timeoutSeconds: 5 still catches a hung/unresponsive daemon, but not a daemon that replies quickly with an unrecognized error. This may be a deliberate, acceptable trade-off given the narrow goal of this fix, but it isn't discussed in the PR body (which only addresses the transient Disconnected/Connecting case). I could not verify from the repo alone what netbird status --check ready actually prints on a non-auth daemon fault since the netbird source isn't vendored here — worth a quick manual check (e.g. exec into a pod and kill/corrupt local daemon state, or read the netbird CLI source for the ready-check's error paths) to confirm this gap is either impossible or acceptable.
kubernetes/apps/monitoring-rules/prometheusrule-netbird-ingress.yaml:32-57 — NetbirdIngressDegraded (< desired, 10m) and NetbirdIngressDown (== 0, 5m) overlap: once all peers are down for 10m+, both a warning and a critical fire simultaneously for the same condition. Sibling rule files in this repo (e.g. prometheusrule-velero.yaml) don't appear to hit this overlap since their conditions are distinct signals rather than a strict subset. Not wrong, just double-paging for the same root cause — consider > 0 and < desired on the Degraded expr if that's undesirable.

🟢 Nits

kubernetes/infrastructure/netbird-ingress/daemonset.yaml:133 — the liveness command reruns the exact same netbird status --check ready call the readinessProbe (line 140) already issues every 10s, just parsing the text instead of trusting the exit code. Functionally fine (trivial exec cost, and the two probes need different semantics), just noting the duplication for anyone comparing the two blocks later.

Tests

No automated tests cover probe behavior or alert rules in this repo (manifests-only, validated via kubectl kustomize/client dry-run per the PR description, plus a manual exec against a live pod). That matches the repo's existing practice for this kind of infra config — kubernetes/e2e-test.sh doesn't cover netbird-ingress either. Given this is a fix for a real, already-observed incident, the verification steps described (exec'd against a live healthy pod, failure string taken verbatim from actual probe events) are a reasonable substitute for unit tests here.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `c59ad31db9315b6d9a84010c546138a0fd06e60d`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Two changes: (1) narrows the netbird-ingress liveness probe to restart only on the dead auth states (`NeedsLogin`/`LoginFailed`/`SessionExpired`), fixing the incident where pods sat Running-but-NotReady for ~20h with no restart and no alert; (2) adds a `PrometheusRule` alerting on DaemonSet Ready-pod deficits. Both changes are well-reasoned, match this repo's existing conventions (compare `prometheusrule-velero.yaml`, `prometheusrule-woodpecker-rotation.yaml`), and the shell logic in the new liveness probe is correct as written. No blocking issues found. ## 🟡 Suggestions - `kubernetes/infrastructure/netbird-ingress/daemonset.yaml:128-137` — the new probe drops the old `--check live` signal entirely and only restarts on a whitelisted match (`NeedsLogin|LoginFailed|SessionExpired`); anything else — including a daemon that's alive as a process but returns a fast, non-timeout error from `--check ready` that isn't one of those three strings (e.g. IPC/socket errors, corrupted state) — now falls through to `exit 0` ("healthy") where the old `--check live`-based probe would likely have failed it via its own exit code. `timeoutSeconds: 5` still catches a hung/unresponsive daemon, but not a daemon that replies quickly with an unrecognized error. This may be a deliberate, acceptable trade-off given the narrow goal of this fix, but it isn't discussed in the PR body (which only addresses the *transient* Disconnected/Connecting case). I could not verify from the repo alone what `netbird status --check ready` actually prints on a non-auth daemon fault since the netbird source isn't vendored here — worth a quick manual check (e.g. exec into a pod and kill/corrupt local daemon state, or read the netbird CLI source for the ready-check's error paths) to confirm this gap is either impossible or acceptable. - `kubernetes/apps/monitoring-rules/prometheusrule-netbird-ingress.yaml:32-57` — `NetbirdIngressDegraded` (`< desired`, 10m) and `NetbirdIngressDown` (`== 0`, 5m) overlap: once all peers are down for 10m+, both a warning and a critical fire simultaneously for the same condition. Sibling rule files in this repo (e.g. `prometheusrule-velero.yaml`) don't appear to hit this overlap since their conditions are distinct signals rather than a strict subset. Not wrong, just double-paging for the same root cause — consider `> 0 and < desired` on the Degraded expr if that's undesirable. ## 🟢 Nits - `kubernetes/infrastructure/netbird-ingress/daemonset.yaml:133` — the liveness command reruns the exact same `netbird status --check ready` call the readinessProbe (line 140) already issues every 10s, just parsing the text instead of trusting the exit code. Functionally fine (trivial exec cost, and the two probes need different semantics), just noting the duplication for anyone comparing the two blocks later. ## Tests No automated tests cover probe behavior or alert rules in this repo (manifests-only, validated via `kubectl kustomize`/client dry-run per the PR description, plus a manual exec against a live pod). That matches the repo's existing practice for this kind of infra config — `kubernetes/e2e-test.sh` doesn't cover netbird-ingress either. Given this is a fix for a real, already-observed incident, the verification steps described (exec'd against a live healthy pod, failure string taken verbatim from actual probe events) are a reasonable substitute for unit tests here. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 983ade2d1a into master

2026-07-25 08:40:05 +00:00

k.renner referenced this issue from a commit

2026-07-25 08:40:06 +00:00

Merge pull request 'fix(netbird-ingress): restart peers stuck in NeedsLogin, alert on NotReady' (#403) from netbird-needslogin-probe into master

claude-bot referenced this pull request

2026-07-27 19:29:28 +00:00

[alert] KubeDaemonSetRolloutStuck (ingress) #392

claude-bot referenced this pull request

2026-07-27 19:29:29 +00:00

[alert] NetbirdPublicEdgeMemberDown (monitoring) #420

Sign in to join this conversation.