fix: [alert] KubePodNotReady (paperless) (#896) #898

2026-09-21T09:45:31Z

issue-fix-bot commented

2026-09-21 09:45:31 +00:00

What this PR does

paperless is in a genuine crash loop, not a blip: the container is OOMKilled (exit 137) roughly 37 seconds into every start, while celery re-consumes the same scanned document. The cause is OCR fan-out, not a memory leak. paperless passes PAPERLESS_THREADS_PER_WORKER to ocrmypdf as its job count, and that setting defaults to multiprocessing.cpu_count() / PAPERLESS_TASK_WORKERS. The container has no CPU limit, so cpu_count() reports the node's cores — k8s-1 has exactly 20 — and OCR fans out to the "Starting processing with 20 workers concurrently" seen in the logs. Twenty pages of a 300dpi colour scan rasterised at once overruns the 2Gi limit, the kernel kills the container, and on restart celery picks up the identical document and repeats. Because the OOM lands before the webserver finishes booting, the pod never reaches Ready and the archive stays down. This PR pins OCR concurrency to a value that does not depend on which node the pod lands on, and raises the memory limit to give that bounded burst real headroom.

Changes

kubernetes/apps/paperless/paperless.yaml — in the paperless-config ConfigMap, pin PAPERLESS_THREADS_PER_WORKER: "4" (down from an effective 20) and PAPERLESS_TASK_WORKERS: "1". The thread cap is the actual fix; pinning task workers keeps the derivation explicit so neither value silently tracks a node's core count.
kubernetes/apps/paperless/paperless.yaml — raise the paperless container's resources.limits.memory from 2Gi to 3Gi. This is headroom, not the fix, and is sized from measured behaviour rather than guessed (see below).

Verification

Everything below was observed this run via ./.ci-diag/kubectl.

The crash loop is live and worse than filed. -n paperless get pods → paperless-756f45cd79-tv88m 1/2 CrashLoopBackOff 18 (2m15s ago) 11h. The issue reported 9 restarts; it is at 18.
OOMKill confirmed. get pod ... -o jsonpath='{...containerStatuses[*].lastState}' → {"terminated":{"exitCode":137,"reason":"OOMKilled","startedAt":"2026-09-21T09:39:38Z","finishedAt":"2026-09-21T09:40:15Z"}} — dead 37s after start. The bridge-tls-proxy sidecar has 0 restarts, so this is isolated to the paperless container.
Mechanism confirmed. logs --previous -c paperless shows Consuming 20260921_101949_ADS-1700W_002274.pdf then [ocrmypdf._pipelines.ocr] Starting processing with 20 workers concurrently, followed by page-rotation lines up to 11:40:14 — one second before the OOM at 11:40:15.
The "20" is the node's core count, which is what makes this a config bug. get node k8s-1 → capacity.cpu: 20. The manifest sets requests.cpu: 200m and no limits.cpu, so cpu_count() sees all 20. 20 cores / 1 task worker = the 20 jobs observed — this arithmetic is what establishes TASK_WORKERS is effectively 1 today, so pinning it to "1" preserves current behaviour exactly.
Limit sizing is measured, not guessed. Prometheus container_memory_working_set_bytes for the last long-lived paperless container (paperless-64b8458776-sjls8) gives both a max and a median of ~1046 MB — a steady ~1.0Gi baseline, leaving only ~1Gi of the 2Gi limit for an OCR burst. Note the crashlooping containers only report ~820 MB peak: they live 37s against a ~30s cadvisor scrape, so the fatal spike falls between samples. The OOMKilled status is the reliable signal here, not the metric — which is also why this reads as a sharp concurrent burst rather than a leak, and therefore why capping workers is the correct lever.
Headroom is available. describe node k8s-1 → 98Gi capacity, memory requests at 14%, MemoryPressure False. The extra 1Gi is immaterial.
Manifest validates. kubectl apply --dry-run=client -f kubernetes/apps/paperless/paperless.yaml → all five resources configured (dry run), no errors.

After Flux reconciles: the pod-template change (memory limit) triggers a rollout on its own, and the Deployment's existing configmap.reloader.stakater.com/reload: paperless-config annotation covers the ConfigMap half. Strategy is Recreate with replicas: 1, so expect a brief gap before the new pod starts. Confirm with ./.ci-diag/kubectl -n paperless get pod -w that the new pod reaches 2/2 Running and stays there, and with logs -c paperless that the line now reads Starting processing with 4 workers concurrently and that 20260921_101949_ADS-1700W_002274.pdf consumes to completion instead of being retried. KubePodNotReady for paperless should clear once the pod is Ready for longer than the rule's for: window; restart count should stop climbing.

Notes

The issue's diagnosis was correct, and I confirmed it rather than assuming it. Its two suggested levers were "raise the memory limit and/or cap OCR concurrency". I did both but weighted them deliberately: raising the limit alone would not have been a fix, because 20 unbounded workers can overrun any ceiling on a large-core node — that would have been treating the symptom. The concurrency cap is the fix; the 3Gi is margin.
The issue's specific config suggestions were partly off-target. PAPERLESS_OCR_PAGES limits how many pages get OCR'd (it would silently stop indexing the rest of every document — a data-quality regression, not a fix), and there is no PAPERLESS_CONSUMER_... job-count setting. PAPERLESS_THREADS_PER_WORKER is the one that actually reaches ocrmypdf's jobs.
The reported startup-probe failures (connect: connection refused on :8000/accounts/login/) are a symptom, not a second problem. The container dies 37s in, while the s6 boot sequence is still running and before gunicorn listens; the probe's own budget is 30 × 20s = 10 minutes and was never the constraint. No probe change is warranted.
4 is a deliberately conservative first step, chosen to fit the measured ~1.0Gi baseline inside 3Gi. If this exact document still OOMs, the next lever is PAPERLESS_THREADS_PER_WORKER: "2" — prefer that over raising memory again, since the burst scales with worker count. OCR throughput per document drops relative to a 20-way fan-out, but that fan-out was never survivable here.
I did not add a limits.cpu. It would also bound cpu_count()-derived defaults, but throttling CPU on a running workload is exactly the kind of change that can degrade it in ways I cannot verify from read-only access; pinning the setting directly achieves the same bound with no such risk.
Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit scope. No alert rule, threshold, or inhibition was modified.

Closes #896

## What this PR does `paperless` is in a genuine crash loop, not a blip: the container is OOMKilled (exit 137) roughly 37 seconds into every start, while celery re-consumes the same scanned document. The cause is OCR fan-out, not a memory leak. paperless passes `PAPERLESS_THREADS_PER_WORKER` to ocrmypdf as its job count, and that setting defaults to `multiprocessing.cpu_count() / PAPERLESS_TASK_WORKERS`. The container has **no CPU limit**, so `cpu_count()` reports the node's cores — k8s-1 has exactly 20 — and OCR fans out to the "Starting processing with 20 workers concurrently" seen in the logs. Twenty pages of a 300dpi colour scan rasterised at once overruns the 2Gi limit, the kernel kills the container, and on restart celery picks up the identical document and repeats. Because the OOM lands before the webserver finishes booting, the pod never reaches Ready and the archive stays down. This PR pins OCR concurrency to a value that does not depend on which node the pod lands on, and raises the memory limit to give that bounded burst real headroom. ## Changes - `kubernetes/apps/paperless/paperless.yaml` — in the `paperless-config` ConfigMap, pin `PAPERLESS_THREADS_PER_WORKER: "4"` (down from an effective 20) and `PAPERLESS_TASK_WORKERS: "1"`. The thread cap is the actual fix; pinning task workers keeps the derivation explicit so neither value silently tracks a node's core count. - `kubernetes/apps/paperless/paperless.yaml` — raise the `paperless` container's `resources.limits.memory` from `2Gi` to `3Gi`. This is headroom, not the fix, and is sized from measured behaviour rather than guessed (see below). ## Verification Everything below was observed this run via `./.ci-diag/kubectl`. - **The crash loop is live and worse than filed.** `-n paperless get pods` → `paperless-756f45cd79-tv88m 1/2 CrashLoopBackOff 18 (2m15s ago) 11h`. The issue reported 9 restarts; it is at 18. - **OOMKill confirmed.** `get pod ... -o jsonpath='{...containerStatuses[*].lastState}'` → `{"terminated":{"exitCode":137,"reason":"OOMKilled","startedAt":"2026-09-21T09:39:38Z","finishedAt":"2026-09-21T09:40:15Z"}}` — dead 37s after start. The `bridge-tls-proxy` sidecar has 0 restarts, so this is isolated to the `paperless` container. - **Mechanism confirmed.** `logs --previous -c paperless` shows `Consuming 20260921_101949_ADS-1700W_002274.pdf` then `[ocrmypdf._pipelines.ocr] Starting processing with 20 workers concurrently`, followed by page-rotation lines up to `11:40:14` — one second before the OOM at `11:40:15`. - **The "20" is the node's core count, which is what makes this a config bug.** `get node k8s-1` → `capacity.cpu: 20`. The manifest sets `requests.cpu: 200m` and no `limits.cpu`, so `cpu_count()` sees all 20. 20 cores / 1 task worker = the 20 jobs observed — this arithmetic is what establishes `TASK_WORKERS` is effectively 1 today, so pinning it to `"1"` preserves current behaviour exactly. - **Limit sizing is measured, not guessed.** Prometheus `container_memory_working_set_bytes` for the last *long-lived* paperless container (`paperless-64b8458776-sjls8`) gives both a max and a median of ~1046 MB — a steady ~1.0Gi baseline, leaving only ~1Gi of the 2Gi limit for an OCR burst. Note the crashlooping containers only report ~820 MB peak: they live 37s against a ~30s cadvisor scrape, so the fatal spike falls between samples. The OOMKilled status is the reliable signal here, not the metric — which is also why this reads as a sharp concurrent burst rather than a leak, and therefore why capping workers is the correct lever. - **Headroom is available.** `describe node k8s-1` → 98Gi capacity, memory requests at 14%, `MemoryPressure False`. The extra 1Gi is immaterial. - **Manifest validates.** `kubectl apply --dry-run=client -f kubernetes/apps/paperless/paperless.yaml` → all five resources `configured (dry run)`, no errors. **After Flux reconciles:** the pod-template change (memory limit) triggers a rollout on its own, and the Deployment's existing `configmap.reloader.stakater.com/reload: paperless-config` annotation covers the ConfigMap half. Strategy is `Recreate` with `replicas: 1`, so expect a brief gap before the new pod starts. Confirm with `./.ci-diag/kubectl -n paperless get pod -w` that the new pod reaches `2/2 Running` and stays there, and with `logs -c paperless` that the line now reads `Starting processing with 4 workers concurrently` and that `20260921_101949_ADS-1700W_002274.pdf` consumes to completion instead of being retried. `KubePodNotReady` for `paperless` should clear once the pod is Ready for longer than the rule's `for:` window; restart count should stop climbing. ## Notes - **The issue's diagnosis was correct, and I confirmed it rather than assuming it.** Its two suggested levers were "raise the memory limit and/or cap OCR concurrency". I did both but weighted them deliberately: raising the limit *alone* would not have been a fix, because 20 unbounded workers can overrun any ceiling on a large-core node — that would have been treating the symptom. The concurrency cap is the fix; the 3Gi is margin. - The issue's specific config suggestions were partly off-target. `PAPERLESS_OCR_PAGES` limits how many pages get OCR'd (it would silently stop indexing the rest of every document — a data-quality regression, not a fix), and there is no `PAPERLESS_CONSUMER_...` job-count setting. `PAPERLESS_THREADS_PER_WORKER` is the one that actually reaches ocrmypdf's `jobs`. - The reported startup-probe failures (`connect: connection refused` on `:8000/accounts/login/`) are a **symptom, not a second problem**. The container dies 37s in, while the s6 boot sequence is still running and before gunicorn listens; the probe's own budget is 30 × 20s = 10 minutes and was never the constraint. No probe change is warranted. - `4` is a deliberately conservative first step, chosen to fit the measured ~1.0Gi baseline inside 3Gi. If this exact document still OOMs, the next lever is `PAPERLESS_THREADS_PER_WORKER: "2"` — prefer that over raising memory again, since the burst scales with worker count. OCR throughput per document drops relative to a 20-way fan-out, but that fan-out was never survivable here. - I did **not** add a `limits.cpu`. It would also bound `cpu_count()`-derived defaults, but throttling CPU on a running workload is exactly the kind of change that can degrade it in ways I cannot verify from read-only access; pinning the setting directly achieves the same bound with no such risk. - Nothing in the issue asked me to touch CI config, credentials, or anything outside the edit scope. No alert rule, threshold, or inhibition was modified. Closes #896

issue-fix-bot added 1 commit 2026-09-21 09:45:32 +00:00

fix: [alert] KubePodNotReady (paperless) (#896 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ebb03e7063

issue-fix-bot referenced this pull request

2026-09-21 09:45:32 +00:00

[alert] KubePodNotReady (paperless) #896

review-bot reviewed

2026-09-21 09:47:18 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit ebb03e706378a1ae6716285144a4fa3a1f545f03.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes a crash loop in the paperless Kubernetes pod (issue #896) caused by unbounded OCR concurrency. The root cause is that PAPERLESS_THREADS_PER_WORKER defaults to cpu_count() / TASK_WORKERS, and without a CPU limit the container sees the node's 20 cores, causing 20 concurrent OCR jobs that exceed the 2Gi memory limit and trigger OOMKill. The fix pins OCR concurrency to 4 workers and raises the memory limit to 3Gi. The changes are technically sound, conservatively scoped, and well-reasoned.

🟡 Suggestions

paperless.yaml:25-26 — Consider documenting the relationship to upstream defaults

The PR rightly notes that pinning PAPERLESS_TASK_WORKERS: "1" reflects the current effective value. However, future maintainers may wonder why this value is chosen if they're unfamiliar with the upstream paperless default derivation. The existing comments do explain this (lines 42-44), but a one-line inline comment above the value itself (e.g., # Reflects current default: 20 cores / 1 worker = 20 jobs, now bounded by THREADS_PER_WORKER) would make it quicker to parse without reading the lengthy preceding block. This is optional and the current comments are sufficient.

paperless.yaml:46-47 — Clarify escalation path in code

The PR body documents that if 4 workers still causes OOM, the next step is PAPERLESS_THREADS_PER_WORKER: "2". Adding this guidance as a brief comment (e.g., # Conservative first step; reduce to "2" if OOM persists) would aid future troubleshooting without requiring knowledge of the PR. Again, optional but helpful.

🟢 Nits

Lines 30-45 (ConfigMap comment) and 294-301 (memory limit comment) are lengthy but justified—they explain non-obvious configuration choices and the reasoning behind measured values. No shortening suggested.
The diff is well-formatted and easy to review.

Tests

This is Kubernetes infrastructure configuration, not application code, so unit/integration tests are not applicable. The verification documented in the PR body is appropriate:

Manual confirmation of OOMKill via kubectl (exit code 137, 37s uptime before crash)
Derivation of the 20-worker value verified from node core count and current config
Prometheus metrics analysis showing ~1.0Gi baseline + ~1Gi headroom needed
Manifest validation via kubectl apply --dry-run=client (all five resources pass)
Escalation path documented (next step: reduce THREADS_PER_WORKER to "2" if needed)

Post-merge validation should confirm:

New pod reaches 2/2 Running and stays there
Logs show "Starting processing with 4 workers concurrently" instead of 20
The previously-crashing document (20260921_101949_ADS-1700W_002274.pdf) consumes to completion
Pod restart count stabilizes (was at 18 before the fix)

🟢 Design & Maintainability

The changes follow Kubernetes best practices:

ConfigMap + annotation: Uses configmap.reloader.stakater.com/reload: paperless-config to ensure config changes trigger a rollout (critical, since K8s does not auto-restart pods on ConfigMap changes when using envFrom).
Strategy: Recreate with replicas: 1 is appropriate for a single-instance stateful service.
Conservative approach: Memory increase (3Gi) is framed as headroom, not the fix. The actual fix is concurrency pinning. This prioritizes the correct lever.
No CPU limit added: The PR correctly avoids adding limits.cpu, noting that throttling CPU can have unpredictable effects in production. Pinning the config value directly achieves the same bound more safely.
Root cause focus: The PR addresses the source of the problem (unbounded OCR concurrency) rather than just symptom-treating (raising memory indefinitely).

Recommendation

Looks reasonable — no blocking issues. The fix is technically correct, appropriately scoped, and well-documented. The measured values are grounded in Prometheus metrics rather than guessed. Post-merge monitoring should confirm the pod stabilizes and the crashing document is consumed successfully.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `ebb03e706378a1ae6716285144a4fa3a1f545f03`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes a crash loop in the `paperless` Kubernetes pod (issue #896) caused by unbounded OCR concurrency. The root cause is that `PAPERLESS_THREADS_PER_WORKER` defaults to `cpu_count() / TASK_WORKERS`, and without a CPU limit the container sees the node's 20 cores, causing 20 concurrent OCR jobs that exceed the 2Gi memory limit and trigger OOMKill. The fix pins OCR concurrency to 4 workers and raises the memory limit to 3Gi. The changes are technically sound, conservatively scoped, and well-reasoned. ## 🟡 Suggestions **`paperless.yaml:25-26` — Consider documenting the relationship to upstream defaults** The PR rightly notes that pinning `PAPERLESS_TASK_WORKERS: "1"` reflects the current effective value. However, future maintainers may wonder *why* this value is chosen if they're unfamiliar with the upstream paperless default derivation. The existing comments do explain this (lines 42-44), but a one-line inline comment above the value itself (e.g., `# Reflects current default: 20 cores / 1 worker = 20 jobs, now bounded by THREADS_PER_WORKER`) would make it quicker to parse without reading the lengthy preceding block. This is optional and the current comments are sufficient. **`paperless.yaml:46-47` — Clarify escalation path in code** The PR body documents that if 4 workers still causes OOM, the next step is `PAPERLESS_THREADS_PER_WORKER: "2"`. Adding this guidance as a brief comment (e.g., `# Conservative first step; reduce to "2" if OOM persists`) would aid future troubleshooting without requiring knowledge of the PR. Again, optional but helpful. ## 🟢 Nits - Lines 30-45 (ConfigMap comment) and 294-301 (memory limit comment) are lengthy but justified—they explain non-obvious configuration choices and the reasoning behind measured values. No shortening suggested. - The diff is well-formatted and easy to review. ## Tests This is Kubernetes infrastructure configuration, not application code, so unit/integration tests are not applicable. The verification documented in the PR body is appropriate: - Manual confirmation of OOMKill via `kubectl` (exit code 137, 37s uptime before crash) - Derivation of the 20-worker value verified from node core count and current config - Prometheus metrics analysis showing ~1.0Gi baseline + ~1Gi headroom needed - Manifest validation via `kubectl apply --dry-run=client` (all five resources pass) - Escalation path documented (next step: reduce `THREADS_PER_WORKER` to "2" if needed) Post-merge validation should confirm: - New pod reaches `2/2 Running` and stays there - Logs show "Starting processing with 4 workers concurrently" instead of 20 - The previously-crashing document (`20260921_101949_ADS-1700W_002274.pdf`) consumes to completion - Pod restart count stabilizes (was at 18 before the fix) ## 🟢 Design & Maintainability The changes follow Kubernetes best practices: - **ConfigMap + annotation**: Uses `configmap.reloader.stakater.com/reload: paperless-config` to ensure config changes trigger a rollout (critical, since K8s does not auto-restart pods on ConfigMap changes when using `envFrom`). - **Strategy**: `Recreate` with `replicas: 1` is appropriate for a single-instance stateful service. - **Conservative approach**: Memory increase (3Gi) is framed as headroom, not the fix. The actual fix is concurrency pinning. This prioritizes the correct lever. - **No CPU limit added**: The PR correctly avoids adding `limits.cpu`, noting that throttling CPU can have unpredictable effects in production. Pinning the config value directly achieves the same bound more safely. - **Root cause focus**: The PR addresses the source of the problem (unbounded OCR concurrency) rather than just symptom-treating (raising memory indefinitely). ## Recommendation **Looks reasonable — no blocking issues.** The fix is technically correct, appropriately scoped, and well-documented. The measured values are grounded in Prometheus metrics rather than guessed. Post-merge monitoring should confirm the pod stabilizes and the crashing document is consumed successfully.

k.renner closed this pull request

2026-09-23 04:55:15 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#898