konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubeDeploymentRolloutStuck (paperless) #452

New Issue
Closed
opened 2026-07-30 13:33:33 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-07-30 13:33:33 +00:00
Member
Copy Link
Copy Source

What is firing

  • KubeDeploymentRolloutStuck (severity: warning) — paperless/paperless deployment rollout not progressing (reason: ProgressDeadlineExceeded) since 2026-07-30 13:20 UTC.
  • A second alert, KubeDeploymentReplicasMismatch on the same deployment, is a symptom of the exact same condition and is filed as noise pointing here.

Evidence

  • kubectl get pod -n paperless: paperless-78c659c6fd-7n4zq 0/2 Init:0/1 — pod has been stuck for 35+ minutes, still true at time of triage (live check).
  • context.txt / live kubectl describe pod: repeated FailedCreatePodSandBox events (11+ occurrences over 35 min), all with the same error: 
    plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
  • Flux confirms the deployment itself is unhealthy, not just the metric: flux-system events show repeated HealthCheckFailed for kustomization/paperless: failed early due to stalled resources: [Deployment/paperless/paperless status: 'Failed'].
  • Root cause found live in the Cilium agent on the node running the pod (k8s-2, pod cilium-bj7bf), via kubectl logs -n kube-system cilium-bj7bf --tail=200: 
    level=error msg="Failed to add PolicyMap key" ciliumEndpointName=paperless/paperless-78c659c6fd-7n4zq ... error="update map cilium_policy_v3_02441: update: no space left on device"
level=warn msg="Creation of endpoint failed" ... error="timeout while waiting for initial endpoint generation to complete: context canceled"
    This is the literal cause of the "Cilium API client timeout" the CNI shim reports — the agent can't finish programming the endpoint's BPF policy map because it has no space, so endpoint creation times out and the sandbox create fails, over and over.

Root cause

Verified: Cilium endpoint creation for this pod fails because writing to its per-endpoint BPF policy map (cilium_policy_v3_02441) returns ENOSPC ("no space left on device") on node k8s-2. Node-level conditions rule out ordinary resource pressure — kubectl describe node k8s-2 shows MemoryPressure=False, DiskPressure=False, PIDPressure=False — so this is specifically the BPF map size ceiling (Cilium sizes policy maps dynamically via bpf-map-dynamic-size-ratio, not tuned in this repo — see kubernetes/infrastructure/cilium/helmrelease.yaml), not general node exhaustion.

Hypothesis (unconfirmed, needs node-level BPF inspection which this agent's RBAC can't do — no pod exec): the node's total BPF map memory pool is close to its ceiling from the combined maps of all endpoints already running on k8s-2 (7 other pods: ftp-scanner, gotenberg, pprl-pg-2/3, protonmail-bridge, redis, syncthing, tika), so a brand-new endpoint can no longer get space allocated. Only endpoint 2441 (the current retry) shows the error in the log tail — there is no sign of a map leak from the ~15 prior failed retries piling up distinct map IDs, which argues against a leak and for a static ceiling being hit.

To confirm: someone with node access could run cilium-dbg statedb bpf-maps or cilium-dbg map list (or bpftool map list) directly on k8s-2 to see current BPF map count/utilization, or check Cilium's own cilium_bpf_map_pressure Prometheus metric via the Prometheus proxy for node k8s-2.

Where to fix it

This is not fixable purely via a repo manifest edit — it needs live node-level intervention/inspection first:

  • A human should check BPF map pressure on k8s-2 (see commands above) and, if confirmed, either restart the Cilium agent on that node (cilium-bj7bf) to force map reallocation, or drain/reschedule some pods off k8s-2 to free BPF map headroom.
  • If this turns out to be a recurring node-wide BPF map sizing issue (not specific to this one incident), the fix belongs in kubernetes/infrastructure/cilium/helmrelease.yaml under values.bpf — e.g. explicitly setting bpf.mapDynamicSizeRatio or bpf.policyMapMax higher than the chart default.
  • No change is needed in kubernetes/apps/paperless/ itself — the deployment and its CiliumNetworkPolicy in kubernetes/apps/paperless/network-policy.yaml are not implicated; the ingress/egress rules there are narrow (single outpost + in-namespace + DNS + world:443) and don't explain a per-endpoint policy map filling up.

Filed automatically by the alert-monitor pipeline (2026-07-30). The issue-fix pipeline may pick this up and open a PR.

## What is firing - `KubeDeploymentRolloutStuck` (severity: warning) — `paperless/paperless` deployment rollout not progressing (`reason: ProgressDeadlineExceeded`) since 2026-07-30 13:20 UTC. - A second alert, `KubeDeploymentReplicasMismatch` on the same deployment, is a symptom of the exact same condition and is filed as noise pointing here. ## Evidence - `kubectl get pod -n paperless`: `paperless-78c659c6fd-7n4zq 0/2 Init:0/1` — pod has been stuck for 35+ minutes, still true at time of triage (live check). - `context.txt` / live `kubectl describe pod`: repeated `FailedCreatePodSandBox` events (11+ occurrences over 35 min), all with the same error: ``` plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded ``` - Flux confirms the deployment itself is unhealthy, not just the metric: `flux-system` events show repeated `HealthCheckFailed` for `kustomization/paperless`: `failed early due to stalled resources: [Deployment/paperless/paperless status: 'Failed']`. - Root cause found live in the Cilium agent on the node running the pod (`k8s-2`, pod `cilium-bj7bf`), via `kubectl logs -n kube-system cilium-bj7bf --tail=200`: ``` level=error msg="Failed to add PolicyMap key" ciliumEndpointName=paperless/paperless-78c659c6fd-7n4zq ... error="update map cilium_policy_v3_02441: update: no space left on device" level=warn msg="Creation of endpoint failed" ... error="timeout while waiting for initial endpoint generation to complete: context canceled" ``` This is the literal cause of the "Cilium API client timeout" the CNI shim reports — the agent can't finish programming the endpoint's BPF policy map because it has no space, so endpoint creation times out and the sandbox create fails, over and over. ## Root cause **Verified**: Cilium endpoint creation for this pod fails because writing to its per-endpoint BPF policy map (`cilium_policy_v3_02441`) returns `ENOSPC` ("no space left on device") on node `k8s-2`. Node-level conditions rule out ordinary resource pressure — `kubectl describe node k8s-2` shows `MemoryPressure=False`, `DiskPressure=False`, `PIDPressure=False` — so this is specifically the BPF map size ceiling (Cilium sizes policy maps dynamically via `bpf-map-dynamic-size-ratio`, not tuned in this repo — see `kubernetes/infrastructure/cilium/helmrelease.yaml`), not general node exhaustion. **Hypothesis (unconfirmed, needs node-level BPF inspection which this agent's RBAC can't do — no pod exec)**: the node's total BPF map memory pool is close to its ceiling from the combined maps of all endpoints already running on `k8s-2` (7 other pods: ftp-scanner, gotenberg, pprl-pg-2/3, protonmail-bridge, redis, syncthing, tika), so a brand-new endpoint can no longer get space allocated. Only endpoint `2441` (the current retry) shows the error in the log tail — there is no sign of a map leak from the ~15 prior failed retries piling up distinct map IDs, which argues against a leak and for a static ceiling being hit. To confirm: someone with node access could run `cilium-dbg statedb bpf-maps` or `cilium-dbg map list` (or `bpftool map list`) directly on `k8s-2` to see current BPF map count/utilization, or check Cilium's own `cilium_bpf_map_pressure` Prometheus metric via the Prometheus proxy for node `k8s-2`. ## Where to fix it This is **not fixable purely via a repo manifest edit** — it needs live node-level intervention/inspection first: - A human should check BPF map pressure on `k8s-2` (see commands above) and, if confirmed, either restart the Cilium agent on that node (`cilium-bj7bf`) to force map reallocation, or drain/reschedule some pods off `k8s-2` to free BPF map headroom. - If this turns out to be a recurring node-wide BPF map sizing issue (not specific to this one incident), the fix belongs in `kubernetes/infrastructure/cilium/helmrelease.yaml` under `values.bpf` — e.g. explicitly setting `bpf.mapDynamicSizeRatio` or `bpf.policyMapMax` higher than the chart default. - No change is needed in `kubernetes/apps/paperless/` itself — the deployment and its `CiliumNetworkPolicy` in `kubernetes/apps/paperless/network-policy.yaml` are not implicated; the ingress/egress rules there are narrow (single outpost + in-namespace + DNS + world:443) and don't explain a per-endpoint policy map filling up. --- <!-- alert-monitor:fingerprint=edb9327d17a14d7f --> _Filed automatically by the alert-monitor pipeline (2026-07-30). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-30 14:46:11 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#456 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/456 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#452