konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubePodNotReady (paperless) #896

New Issue
Closed
opened 2026-09-21 08:48:55 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-09-21 08:48:55 +00:00
Member
Copy Link
Copy Source

What is firing

KubePodNotReady for paperless/paperless-756f45cd79-tv88m — reconstructed as a "blip" from Prometheus ALERTS history, but live verification shows the underlying condition is still active, not resolved.

Evidence

  • context.txt snapshot: pod listed under "pods not Running/Succeeded" as 1/2 CrashLoopBackOff, plus Warning events Unhealthy (readiness/startup probe: connect: connection refused on :8000/accounts/login/) and BackOff restarting failed container paperless.
  • Live (./.ci-diag/kubectl -n paperless get pods): pod is 1/2 Running with 9 restarts, container started 08:47:44 and is still failing its startup probe as of the last check (Warning Unhealthy 18s ago: Startup probe failed).
  • ./.ci-diag/kubectl -n paperless get pod ... -o jsonpath='{.status.containerStatuses[?(@.name=="paperless")].lastState}' → {"terminated":{"exitCode":137,"reason":"OOMKilled",...}}.
  • Previous container log (--previous) shows the crash happens mid-OCR: celery picks up documents.tasks.consume_file, ocrmypdf logs Starting processing with 20 workers concurrently while OCR'ing 20260921_101949_ADS-1700W_002274.pdf, then the container is OOMKilled shortly after.
  • Container resources (kubernetes/apps/paperless/paperless.yaml): requests.memory: 512Mi, limits.memory: 2Gi — no requests.cpu/limits.cpu issue, purely a memory ceiling.

Root cause

Verified: the paperless container is being OOMKilled (exit 137) while OCR-processing an incoming scanned document. ocrmypdf defaults to spawning workers equal to CPU count ("20 workers concurrently" here), which multiplies memory use for image-heavy scans and can blow past the 2Gi container limit. This has recurred at least 9 times over the pod's 10h lifetime, most recently ~08:47 today, and the pod is still unready at time of writing — this is an active crash loop, not a one-off blip.

Where to fix it

kubernetes/apps/paperless/paperless.yaml:

  • Raise the paperless container's resources.limits.memory (currently 2Gi), and/or
  • Cap OCR concurrency via PAPERLESS_OCR_PAGES/PAPERLESS_TASK_WORKERS/PAPERLESS_THREADS_PER_WORKER (or ocrmypdf's -j/jobs equivalent through PAPERLESS_CONSUMER_... / PAPERLESS_OCR_... settings) in the paperless-config ConfigMap so OCR doesn't spawn 20 concurrent workers on a memory-constrained pod.

Not fixable purely by waiting: the deployment is strategy: Recreate with replicas: 1, so every OOM leaves the archive fully down until the container restarts and re-attempts the same document, which can OOM again (visible recurrence). Once the config is adjusted, confirm with ./.ci-diag/kubectl -n paperless get pod -w and logs --previous that the same document no longer triggers exit 137.

Filed automatically by the alert-monitor pipeline (2026-09-21). The issue-fix pipeline may pick this up and open a PR.

**What is firing** `KubePodNotReady` for `paperless/paperless-756f45cd79-tv88m` — reconstructed as a "blip" from Prometheus ALERTS history, but live verification shows the underlying condition is still active, not resolved. **Evidence** - `context.txt` snapshot: pod listed under "pods not Running/Succeeded" as `1/2 CrashLoopBackOff`, plus Warning events `Unhealthy` (readiness/startup probe: `connect: connection refused` on `:8000/accounts/login/`) and `BackOff restarting failed container paperless`. - Live (`./.ci-diag/kubectl -n paperless get pods`): pod is `1/2 Running` with **9 restarts**, container started 08:47:44 and is still failing its startup probe as of the last check (`Warning Unhealthy 18s ago: Startup probe failed`). - `./.ci-diag/kubectl -n paperless get pod ... -o jsonpath='{.status.containerStatuses[?(@.name=="paperless")].lastState}'` → `{"terminated":{"exitCode":137,"reason":"OOMKilled",...}}`. - Previous container log (`--previous`) shows the crash happens mid-OCR: celery picks up `documents.tasks.consume_file`, ocrmypdf logs `Starting processing with 20 workers concurrently` while OCR'ing `20260921_101949_ADS-1700W_002274.pdf`, then the container is OOMKilled shortly after. - Container resources (`kubernetes/apps/paperless/paperless.yaml`): `requests.memory: 512Mi`, `limits.memory: 2Gi` — no `requests.cpu`/`limits.cpu` issue, purely a memory ceiling. **Root cause** Verified: the `paperless` container is being OOMKilled (exit 137) while OCR-processing an incoming scanned document. `ocrmypdf` defaults to spawning workers equal to CPU count ("20 workers concurrently" here), which multiplies memory use for image-heavy scans and can blow past the 2Gi container limit. This has recurred at least 9 times over the pod's 10h lifetime, most recently ~08:47 today, and the pod is *still* unready at time of writing — this is an active crash loop, not a one-off blip. **Where to fix it** `kubernetes/apps/paperless/paperless.yaml`: - Raise the `paperless` container's `resources.limits.memory` (currently `2Gi`), and/or - Cap OCR concurrency via `PAPERLESS_OCR_PAGES`/`PAPERLESS_TASK_WORKERS`/`PAPERLESS_THREADS_PER_WORKER` (or ocrmypdf's `-j`/jobs equivalent through `PAPERLESS_CONSUMER_...` / `PAPERLESS_OCR_...` settings) in the `paperless-config` ConfigMap so OCR doesn't spawn 20 concurrent workers on a memory-constrained pod. Not fixable purely by waiting: the deployment is `strategy: Recreate` with `replicas: 1`, so every OOM leaves the archive fully down until the container restarts and re-attempts the same document, which can OOM again (visible recurrence). Once the config is adjusted, confirm with `./.ci-diag/kubectl -n paperless get pod -w` and `logs --previous` that the same document no longer triggers exit 137. --- <!-- alert-monitor:fingerprint=8d387fd4c8fa41a0 --> _Filed automatically by the alert-monitor pipeline (2026-09-21). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-09-21 09:45:32 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#898 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/898 with a proposed fix.
claude-bot commented 2026-09-23 04:57:28 +00:00
Member
Copy Link
Copy Source

Not firing as of 2026-09-23 (Alertmanager: only Watchdog active; paperless pod Running 2/2). Same root cause as #897. Closing.

Not firing as of 2026-09-23 (Alertmanager: only Watchdog active; paperless pod Running 2/2). Same root cause as #897. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#896