konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubeDaemonSetRolloutStuck (ingress) #392

New Issue
Closed
opened 2026-07-24 13:24:31 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-07-24 13:24:31 +00:00
Member
Copy Link
Copy Source

What is firing: KubeDaemonSetRolloutStuck (severity warning) for daemonset ingress/netbird-ingress since 2026-07-24T13:13:44Z. This is the root cause for a batch of 3 accompanying KubePodNotReady alerts for the same daemonset's pods (netbird-ingress-wzmqh, -nvr9c, -dtnrq) — filed as noise pointing here.

Evidence:

  • Live daemonset status (kubectl -n ingress get daemonset netbird-ingress -o jsonpath='{.status}'): {"currentNumberScheduled":3,"desiredNumberScheduled":3,"numberReady":0,"numberUnavailable":3} — all 3 peers down, matches the alert.
  • All 3 pods are Running but 0/1 Ready with kubelet event Readiness probe failed: ... daemon status is NeedsLogin (confirmed via kubectl -n ingress describe pod).
  • Pod logs (kubectl -n ingress logs netbird-ingress-<wzmqh|nvr9c|dtnrq> --tail) all show the same failure within about 2 minutes of each other (~12:57–12:59 UTC), independent of node: 
    ERRO shared/management/client/grpc.go:601: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login
    This is an application-level PermissionDenied response from netbird.renner.uno, not a network timeout — egress connectivity (TCP/443 per the netbird-ingress-egress CiliumNetworkPolicy) is working; the management server is rejecting the credential.

Root cause: The NB_SETUP_KEY env var is correctly wired to secretKeyRef: netbird-ingress-setup-key/setup-key (verified via kubectl -n ingress get pod ... -o jsonpath='{.spec.containers[0].env}'), and the ExternalSecret is healthy (status.conditions: Ready=True, last refreshed 2026-07-24T13:19:09Z). The quarterly rotator CronJob (openbao/netbird-secret-rotate) last ran successfully 2026-07-21 — three days before this incident — so this is not an automatic key rotation that just landed. That leaves two unverified hypotheses (could not distinguish further: the read-only diagnostics ServiceAccount has no Secret-read access in ingress, so the actual key value can't be inspected):

  1. The reusable setup key stored in OpenBao (homelab/infrastructure/netbird-ingress) has been revoked, expired, or deleted on the NetBird management server (netbird.renner.uno) side, independent of the automated rotation.
  2. The value currently stored in OpenBao is stale/corrupted (e.g. from a manual edit), so ESO is faithfully syncing a bad key.
    A human with NetBird dashboard/API access should check the setup key's status at netbird.renner.uno to tell these apart — the diagnostics SA cannot reach that admin surface.

Where to fix it: This is not fixable by a plain manifest change in this repo — it needs either (a) live intervention on the external NetBird management server to check/re-issue the setup key, or (b) manually triggering the existing rotator early instead of waiting for its quarterly schedule (kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml, cron 0 4 1 1,4,7,10 *) via kubectl create job --from=cronjob/netbird-secret-rotate -n openbao <name> (mutating — a human must run this, the read-only diagnostics SA can't). If the rotator's minted key itself is bad, the rotator logic (secret-rotator image, netbird-key mode) and its OpenBao path (homelab/infrastructure/netbird-ingress) are the place to audit. Related manifests: kubernetes/infrastructure/netbird-ingress/daemonset.yaml, kubernetes/infrastructure/netbird-ingress/externalsecret.yaml, kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml.

Impact: with 0/3 routing peers up, the NetBird mesh ingress path (k8s-ingress-router route group, used for public ingress like home.renner.uno) has no HA peers serving traffic at all — this is a full outage of that ingress path, not just a degraded rollout.

Filed automatically by the alert-monitor pipeline (2026-07-24). The issue-fix pipeline may pick this up and open a PR.

**What is firing**: `KubeDaemonSetRolloutStuck` (severity warning) for daemonset `ingress/netbird-ingress` since 2026-07-24T13:13:44Z. This is the root cause for a batch of 3 accompanying `KubePodNotReady` alerts for the same daemonset's pods (`netbird-ingress-wzmqh`, `-nvr9c`, `-dtnrq`) — filed as noise pointing here. **Evidence**: - Live daemonset status (`kubectl -n ingress get daemonset netbird-ingress -o jsonpath='{.status}'`): `{"currentNumberScheduled":3,"desiredNumberScheduled":3,"numberReady":0,"numberUnavailable":3}` — all 3 peers down, matches the alert. - All 3 pods are `Running` but `0/1 Ready` with kubelet event `Readiness probe failed: ... daemon status is NeedsLogin` (confirmed via `kubectl -n ingress describe pod`). - Pod logs (`kubectl -n ingress logs netbird-ingress-<wzmqh|nvr9c|dtnrq> --tail`) all show the **same** failure within about 2 minutes of each other (~12:57–12:59 UTC), independent of node: ``` ERRO shared/management/client/grpc.go:601: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login ``` This is an application-level `PermissionDenied` response from `netbird.renner.uno`, not a network timeout — egress connectivity (TCP/443 per the `netbird-ingress-egress` CiliumNetworkPolicy) is working; the management server is rejecting the credential. **Root cause**: The `NB_SETUP_KEY` env var is correctly wired to `secretKeyRef: netbird-ingress-setup-key/setup-key` (verified via `kubectl -n ingress get pod ... -o jsonpath='{.spec.containers[0].env}'`), and the `ExternalSecret` is healthy (`status.conditions: Ready=True`, last refreshed 2026-07-24T13:19:09Z). The quarterly rotator CronJob (`openbao/netbird-secret-rotate`) last ran successfully 2026-07-21 — three days before this incident — so this is **not** an automatic key rotation that just landed. That leaves two unverified hypotheses (could not distinguish further: the read-only diagnostics ServiceAccount has no Secret-read access in `ingress`, so the actual key value can't be inspected): 1. The reusable setup key stored in OpenBao (`homelab/infrastructure/netbird-ingress`) has been revoked, expired, or deleted on the NetBird management server (`netbird.renner.uno`) side, independent of the automated rotation. 2. The value currently stored in OpenBao is stale/corrupted (e.g. from a manual edit), so ESO is faithfully syncing a bad key. A human with NetBird dashboard/API access should check the setup key's status at `netbird.renner.uno` to tell these apart — the diagnostics SA cannot reach that admin surface. **Where to fix it**: This is **not fixable by a plain manifest change** in this repo — it needs either (a) live intervention on the external NetBird management server to check/re-issue the setup key, or (b) manually triggering the existing rotator early instead of waiting for its quarterly schedule (`kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml`, cron `0 4 1 1,4,7,10 *`) via `kubectl create job --from=cronjob/netbird-secret-rotate -n openbao <name>` (mutating — a human must run this, the read-only diagnostics SA can't). If the rotator's minted key itself is bad, the rotator logic (`secret-rotator` image, `netbird-key` mode) and its OpenBao path (`homelab/infrastructure/netbird-ingress`) are the place to audit. Related manifests: `kubernetes/infrastructure/netbird-ingress/daemonset.yaml`, `kubernetes/infrastructure/netbird-ingress/externalsecret.yaml`, `kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml`. **Impact**: with 0/3 routing peers up, the NetBird mesh ingress path (`k8s-ingress-router` route group, used for public ingress like `home.renner.uno`) has no HA peers serving traffic at all — this is a full outage of that ingress path, not just a degraded rollout. --- <!-- alert-monitor:fingerprint=b38c1de9e31a07b3 --> _Filed automatically by the alert-monitor pipeline (2026-07-24). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-24 13:43:50 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change is appropriate

Alert: KubeDaemonSetRolloutStuck for ingress/netbird-ingress (3 accompanying KubePodNotReady alerts are downstream noise from the same root cause).

I re-verified every claim in the issue live this run rather than trusting it:

  • ./.ci-diag/kubectl -n ingress get daemonset netbird-ingress -o jsonpath='{.status}' →
    {"currentNumberScheduled":3,"desiredNumberScheduled":3,"numberReady":0,"numberUnavailable":3,...} —
    still 0/3 ready right now.
  • ./.ci-diag/kubectl -n ingress get pods -l app=netbird-ingress -o jsonpath=... → all 3 pods
    Running / ready=false.
  • ./.ci-diag/kubectl -n ingress logs netbird-ingress-wzmqh --tail=5 → confirmed the exact same
    error as reported: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login. This is
    an application-level rejection from netbird.renner.uno, not a network/timeout issue.
  • ./.ci-diag/kubectl -n ingress get externalsecret netbird-ingress-setup-key -o jsonpath='{.status.conditions}' → Ready=True, reason=SecretSynced — ESO is syncing fine, so
    this isn't a sync-pipeline failure; whatever key value is stored is being delivered correctly,
    it's just being rejected by the management server.
  • ./.ci-diag/kubectl -n openbao get cronjob netbird-secret-rotate -o jsonpath=... →
    lastSuccessfulTime=2026-07-21T15:27:20Z, matching the issue's claim that the quarterly rotator
    ran cleanly 3 days before the incident — rules out an in-flight automated rotation as the direct
    cause.
  • Read kubernetes/infrastructure/netbird-ingress/externalsecret.yaml,
    kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml: both are configured
    correctly (10m refresh, correct OpenBao path homelab/infrastructure/netbird-ingress, correct
    rotator mode netbird-key). No manifest bug found.

Root cause: the setup key currently stored in OpenBao at homelab/infrastructure/netbird-ingress
is being rejected by the NetBird management server (netbird.renner.uno) with PermissionDenied.
The diagnostics ServiceAccount used in this pipeline has no Secret-read access in ingress and no
reach into the NetBird admin API, so it's not possible to determine from here whether the key was
revoked/expired server-side or is simply stale/corrupt in OpenBao — both present identically to the
DaemonSet.

Why no edit: every manifest and env var involved is correctly wired and matches live cluster
state; the actual defect is an invalid credential value on an external system, which per the fix
rules is not something to paper over with a repo change (and per rule 3/6, tags/thresholds aren't
the lever here either — there's no rule or threshold to touch).

Recommended manual action (human required — the read-only CI ServiceAccount cannot do this)

  1. Check the setup key's status on the NetBird dashboard/API at netbird.renner.uno to determine
    whether it was revoked/expired/deleted, or is still valid (which would instead point at a
    corrupted/stale value in OpenBao).
  2. If the key is bad, trigger the existing rotator early rather than waiting for its quarterly
    schedule (0 4 1 1,4,7,10 *):
    kubectl create job --from=cronjob/netbird-secret-rotate -n openbao <name>
    This mints a fresh key via the secret-rotator (netbird-key mode), patches OpenBao at
    homelab/infrastructure/netbird-ingress, and ESO/Reloader will pick it up within ~10m and
    restart the DaemonSet automatically — no other manual step needed after that.
  3. After the rotation lands, confirm recovery: DaemonSet numberReady should go to 3/3, and pod
    logs should stop showing the PermissionDenied error.

Notes

The issue's own diagnosis and evidence were accurate on every point I could independently verify;
I found nothing it got wrong. This is a genuine "not fixable in the repo" case (decision rule 6),
so no edits were made.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment: no repo change is appropriate **Alert**: `KubeDaemonSetRolloutStuck` for `ingress/netbird-ingress` (3 accompanying `KubePodNotReady` alerts are downstream noise from the same root cause). I re-verified every claim in the issue live this run rather than trusting it: - `./.ci-diag/kubectl -n ingress get daemonset netbird-ingress -o jsonpath='{.status}'` → `{"currentNumberScheduled":3,"desiredNumberScheduled":3,"numberReady":0,"numberUnavailable":3,...}` — still 0/3 ready right now. - `./.ci-diag/kubectl -n ingress get pods -l app=netbird-ingress -o jsonpath=...` → all 3 pods `Running` / `ready=false`. - `./.ci-diag/kubectl -n ingress logs netbird-ingress-wzmqh --tail=5` → confirmed the exact same error as reported: `failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login`. This is an application-level rejection from `netbird.renner.uno`, not a network/timeout issue. - `./.ci-diag/kubectl -n ingress get externalsecret netbird-ingress-setup-key -o jsonpath='{.status.conditions}'` → `Ready=True, reason=SecretSynced` — ESO is syncing fine, so this isn't a sync-pipeline failure; whatever key value is stored is being delivered correctly, it's just being rejected by the management server. - `./.ci-diag/kubectl -n openbao get cronjob netbird-secret-rotate -o jsonpath=...` → `lastSuccessfulTime=2026-07-21T15:27:20Z`, matching the issue's claim that the quarterly rotator ran cleanly 3 days before the incident — rules out an in-flight automated rotation as the direct cause. - Read `kubernetes/infrastructure/netbird-ingress/externalsecret.yaml`, `kubernetes/infrastructure/openbao/rotation/netbird-ingress.yaml`: both are configured correctly (10m refresh, correct OpenBao path `homelab/infrastructure/netbird-ingress`, correct rotator mode `netbird-key`). No manifest bug found. **Root cause**: the setup key currently stored in OpenBao at `homelab/infrastructure/netbird-ingress` is being rejected by the NetBird management server (`netbird.renner.uno`) with `PermissionDenied`. The diagnostics ServiceAccount used in this pipeline has no Secret-read access in `ingress` and no reach into the NetBird admin API, so it's not possible to determine from here whether the key was revoked/expired server-side or is simply stale/corrupt in OpenBao — both present identically to the DaemonSet. **Why no edit**: every manifest and env var involved is correctly wired and matches live cluster state; the actual defect is an invalid credential value on an external system, which per the fix rules is not something to paper over with a repo change (and per rule 3/6, tags/thresholds aren't the lever here either — there's no rule or threshold to touch). ## Recommended manual action (human required — the read-only CI ServiceAccount cannot do this) 1. Check the setup key's status on the NetBird dashboard/API at `netbird.renner.uno` to determine whether it was revoked/expired/deleted, or is still valid (which would instead point at a corrupted/stale value in OpenBao). 2. If the key is bad, trigger the existing rotator early rather than waiting for its quarterly schedule (`0 4 1 1,4,7,10 *`): `kubectl create job --from=cronjob/netbird-secret-rotate -n openbao <name>` This mints a fresh key via the `secret-rotator` (`netbird-key` mode), patches OpenBao at `homelab/infrastructure/netbird-ingress`, and ESO/Reloader will pick it up within ~10m and restart the DaemonSet automatically — no other manual step needed after that. 3. After the rotation lands, confirm recovery: DaemonSet `numberReady` should go to `3/3`, and pod logs should stop showing the `PermissionDenied` error. ## Notes The issue's own diagnosis and evidence were accurate on every point I could independently verify; I found nothing it got wrong. This is a genuine "not fixable in the repo" case (decision rule 6), so no edits were made. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-07-27 19:29:28 +00:00
Member
Copy Link
Copy Source

Verified resolved during the 2026-07-27 issue round. Closing — fixed by c59ad31d (PR #403, merged 2026-07-25).

⚠️ Do NOT run the "Recommended manual action" in the earlier pipeline comment (kubectl create job --from=cronjob/netbird-secret-rotate -n openbao …). It rests on a root cause that is refuted below. Running it would mint a redundant setup key that the rotator never cleans up, churn the Secret, and trigger an unnecessary Reloader restart of all three routing peers. It would fix nothing.

Live state: healthy

NAME              DESIRED CURRENT READY UP-TO-DATE AVAILABLE AGE  IMAGES
netbird-ingress   3       3       3     3          3         31d  netbirdio/netbird:v0.75.0-rootless

netbird-ingress-cg5gs   1/1 Running 0 28h
netbird-ingress-xc58d   1/1 Running 0 2d2h
netbird-ingress-zp284   1/1 Running 0 2d2h

Zero restarts, Events: <none>, logs show peers Status: Connected with successful WireGuard handshakes. No NeedsLogin / LoginFailed / PermissionDenied anywhere.

The setup key was never the problem — confirmed, not hypothesis

This issue's stated root cause (the key in OpenBao being revoked/expired/stale) is wrong, and the evidence is decisive:

  • Secret ingress/netbird-ingress-setup-key: ESO managedFields last value change = 2026-07-21T15:27:40Z; data-hash annotation unchanged since.
  • The pods now enrolled and connected started 2026-07-25 and 2026-07-26 — they enrolled successfully using the exact same key value that was in the cluster during the 2026-07-24 incident.

A key that enrolls three peers today cannot have been "rejected as invalid" three days ago. (ExternalSecret is SecretSynced=True, refreshed 2026-07-27T10:49:59Z; ClusterSecretStore openbao Valid/Ready=True.)

Actual root cause

Per the fix commit's own analysis: a 24-minute WAN outage to netbird.renner.uno on 2026-07-24 exceeded the 10-minute ephemeral-peer lifetime, so management deregistered all three routing peers. The client treats "peer is not registered" as unrecoverable and parks in NeedsLogin — which fails readiness but not the --check live liveness probe, so the pods sat Running-but-NotReady for ~20h until deleted by hand.

That also explains the misleading log line quoted in this issue: NB_SETUP_KEY is consumed only by netbird up at container start, so on re-login the daemon has no credential to present — hence "no peer auth method provided". A symptom of deregistration, not a bad key.

The fix changed the liveness probe to fail on the parked states:

sh -c "netbird status --check ready 2>&1 | grep -qE 'NeedsLogin|LoginFailed|SessionExpired' && exit 1 || exit 0"

plus new alerting (NetbirdIngressDegraded warn/10m, NetbirdIngressDown crit/5m). The live pods show exactly this probe, confirming it is deployed.

Rotator: no time bomb

openbao/netbird-secret-rotate has never been scheduled — Last Schedule Time: <unset>, no Job objects, CronJob only 5d19h old; next fire 2026-10-01. The 2026-07-21 key write was a manual one-off whose Job was GC'd (ttlSecondsAfterFinished: 3d). The current key is reusable, unlimited-use (usage_limit: 0) and does not expire until ~2027-07-21 — comfortably inside the quarterly cadence.

Residual risk, stated plainly

The trigger is unchanged. ephemeral: true means any >10-minute outage reaching netbird.renner.uno will again deregister all three peers. The fix converts that from a silent ~20h outage into a self-healing restart within ~30s (period 10s × failureThreshold 3).

Eliminating the trigger rather than recovering from it would mean setting NB_EPHEMERAL=false in the rotator CronJob env — a deliberate trade (unbounded stale-peer accumulation vs. reap resilience), not something to change reflexively. Not doing that here.

Verified resolved during the 2026-07-27 issue round. Closing — **fixed by `c59ad31d` (PR #403, merged 2026-07-25)**. > ⚠️ **Do NOT run the "Recommended manual action" in the earlier pipeline comment** (`kubectl create job --from=cronjob/netbird-secret-rotate -n openbao …`). It rests on a root cause that is refuted below. Running it would mint a redundant setup key that the rotator never cleans up, churn the Secret, and trigger an unnecessary Reloader restart of all three routing peers. It would fix nothing. ### Live state: healthy ``` NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE AGE IMAGES netbird-ingress 3 3 3 3 3 31d netbirdio/netbird:v0.75.0-rootless netbird-ingress-cg5gs 1/1 Running 0 28h netbird-ingress-xc58d 1/1 Running 0 2d2h netbird-ingress-zp284 1/1 Running 0 2d2h ``` Zero restarts, `Events: <none>`, logs show peers `Status: Connected` with successful WireGuard handshakes. No `NeedsLogin` / `LoginFailed` / `PermissionDenied` anywhere. ### The setup key was never the problem — confirmed, not hypothesis This issue's stated root cause (the key in OpenBao being revoked/expired/stale) is **wrong**, and the evidence is decisive: - Secret `ingress/netbird-ingress-setup-key`: ESO managedFields last **value change** = `2026-07-21T15:27:40Z`; data-hash annotation unchanged since. - The pods now enrolled and connected started **2026-07-25** and **2026-07-26** — they enrolled successfully using **the exact same key value** that was in the cluster during the 2026-07-24 incident. A key that enrolls three peers today cannot have been "rejected as invalid" three days ago. (ExternalSecret is `SecretSynced=True`, refreshed `2026-07-27T10:49:59Z`; ClusterSecretStore `openbao` `Valid`/`Ready=True`.) ### Actual root cause Per the fix commit's own analysis: a 24-minute WAN outage to `netbird.renner.uno` on 2026-07-24 exceeded the **10-minute ephemeral-peer lifetime**, so management **deregistered** all three routing peers. The client treats "peer is not registered" as unrecoverable and parks in `NeedsLogin` — which fails readiness but **not** the `--check live` liveness probe, so the pods sat Running-but-NotReady for ~20h until deleted by hand. That also explains the misleading log line quoted in this issue: `NB_SETUP_KEY` is consumed only by `netbird up` at container start, so on re-login the daemon has no credential to present — hence *"no peer auth method provided"*. A **symptom of deregistration, not a bad key.** The fix changed the liveness probe to fail on the parked states: ``` sh -c "netbird status --check ready 2>&1 | grep -qE 'NeedsLogin|LoginFailed|SessionExpired' && exit 1 || exit 0" ``` plus new alerting (`NetbirdIngressDegraded` warn/10m, `NetbirdIngressDown` crit/5m). The live pods show exactly this probe, confirming it is deployed. ### Rotator: no time bomb `openbao/netbird-secret-rotate` has **never been scheduled** — `Last Schedule Time: <unset>`, no Job objects, CronJob only 5d19h old; next fire **2026-10-01**. The 2026-07-21 key write was a manual one-off whose Job was GC'd (`ttlSecondsAfterFinished: 3d`). The current key is reusable, unlimited-use (`usage_limit: 0`) and does not expire until ~2027-07-21 — comfortably inside the quarterly cadence. ### Residual risk, stated plainly The **trigger is unchanged**. `ephemeral: true` means any >10-minute outage reaching `netbird.renner.uno` will again deregister all three peers. The fix converts that from a silent ~20h outage into a self-healing restart within ~30s (period 10s × failureThreshold 3). Eliminating the trigger rather than recovering from it would mean setting `NB_EPHEMERAL=false` in the rotator CronJob env — a deliberate trade (unbounded stale-peer accumulation vs. reap resilience), not something to change reflexively. Not doing that here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#392