fix(paperclip): raise memory limit to 32Gi (OOMKilled) #980

2026-09-25T19:49:16Z

claude-bot commented

2026-09-25 19:49:16 +00:00

The Paperclip container was OOMKilled (exit 137, 2026-09-25 19:43) under its 4Gi limit. With server-local runs, every concurrent agent's Claude Code process and MCP helpers run inside this one container, so the limit caps how many agents can work at once; exceeding it kills the server and every in-flight run (process_lost / orphaned_running_run — ~50 of the last 3h's failures).

limit 4Gi → 32Gi, request 1Gi → 2Gi (idle server ~1Gi)
nodes: ~93Gi allocatable each, 12–49% used; the request stays low, so 32Gi is not reserved — on the busiest node a full 32Gi would reach ~80%.

🤖 Generated with Claude Code

The Paperclip container was **OOMKilled** (exit 137, 2026-09-25 19:43) under its 4Gi limit. With server-local runs, every concurrent agent's Claude Code process and MCP helpers run inside this one container, so the limit caps how many agents can work at once; exceeding it kills the server and every in-flight run (`process_lost` / `orphaned_running_run` — ~50 of the last 3h's failures). - limit 4Gi → 32Gi, request 1Gi → 2Gi (idle server ~1Gi) - nodes: ~93Gi allocatable each, 12–49% used; the request stays low, so 32Gi is not reserved — on the busiest node a full 32Gi would reach ~80%. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-25 19:49:16 +00:00

fix(paperclip): raise memory limit; agents run inside the container

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was canceled

Details

9319943f64

The container was OOMKilled (exit 137) at 19:43 under the 4Gi limit.
With server-local runs every concurrent agent's Claude Code process
and MCP helpers live in this container, so the limit caps how many
agents can work at once; exceeding it kills the server and every
in-flight run (process_lost / orphaned_running_run).

Limit 16Gi, request 2Gi (idle server ~1Gi). Nodes have ~93Gi
allocatable each at 12-49% use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot added 1 commit 2026-09-25 19:49:58 +00:00

fix(paperclip): memory limit 32Gi

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

a412ae4fb7

Headroom for many concurrent agents in the one container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot changed title from ~~fix(paperclip): raise memory limit to 16Gi (OOMKilled)~~ to fix(paperclip): raise memory limit to 32Gi (OOMKilled)

2026-09-25 19:50:01 +00:00

k.renner merged commit fd7b4f2eff into master

2026-09-25 19:50:42 +00:00

k.renner referenced this issue from a commit

2026-09-25 19:50:43 +00:00

Merge pull request 'fix(paperclip): raise memory limit to 32Gi (OOMKilled)' (#980) from fix/paperclip-memory into master

review-bot reviewed

2026-09-25 19:51:56 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit a412ae4fb793cde554c05a4e537507cb82fec06b.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR increases the Paperclip container's memory limit from 4Gi to 32Gi (with a request increase from 1Gi to 2Gi) to address OOMKill failures when multiple concurrent agents run inside the container. The change is well-justified by the PR description—nodes have ~93Gi allocatable with 12–49% utilization, so the 32Gi limit won't cause overallocation, and a full limit would reach ~80% on the busiest node. The added comment clarifies the constraint. No blocking issues.

🟡 Suggestions

Memory usage monitoring — Consider adding a Prometheus alert or dashboard query to track Paperclip's actual memory usage post-merge. If the container regularly approaches 32Gi, that signals either a workload growth that needs further scaling or a genuine resource leak. Without observability, you won't know until the next OOMKill.
Architectural note — This is a reactive fix to a design constraint (all agents in one pod with replicas: 1). If agent concurrency continues to grow, even 32Gi will eventually become a bottleneck. Horizontal scaling or splitting agent workloads into separate deployments might be a future consideration, but that's outside this PR's scope.
CPU limits — The container has a CPU request (250m) but no limit, allowing unbounded CPU usage. This is reasonable given you're chasing a memory issue, but you may want to add a CPU limit (e.g., 4 cores) to prevent resource contention with other workloads if a runaway process emerges.

🟢 Nits

The comment on line 93 is helpful and explains the constraint clearly.

Tests

This is infrastructure/Kubernetes configuration. No unit tests apply. The fix should be validated by deploying to the cluster and confirming that:

The container no longer OOMKills under the expected concurrent agent load.
Memory usage stays well below 32Gi under normal operation (if it regularly approaches the limit, the problem is just postponed).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `a412ae4fb793cde554c05a4e537507cb82fec06b`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR increases the Paperclip container's memory limit from 4Gi to 32Gi (with a request increase from 1Gi to 2Gi) to address OOMKill failures when multiple concurrent agents run inside the container. The change is well-justified by the PR description—nodes have ~93Gi allocatable with 12–49% utilization, so the 32Gi limit won't cause overallocation, and a full limit would reach ~80% on the busiest node. The added comment clarifies the constraint. No blocking issues. ## 🟡 Suggestions 1. **Memory usage monitoring** — Consider adding a Prometheus alert or dashboard query to track Paperclip's actual memory usage post-merge. If the container regularly approaches 32Gi, that signals either a workload growth that needs further scaling or a genuine resource leak. Without observability, you won't know until the next OOMKill. 2. **Architectural note** — This is a reactive fix to a design constraint (all agents in one pod with `replicas: 1`). If agent concurrency continues to grow, even 32Gi will eventually become a bottleneck. Horizontal scaling or splitting agent workloads into separate deployments might be a future consideration, but that's outside this PR's scope. 3. **CPU limits** — The container has a CPU request (250m) but no limit, allowing unbounded CPU usage. This is reasonable given you're chasing a memory issue, but you may want to add a CPU limit (e.g., 4 cores) to prevent resource contention with other workloads if a runaway process emerges. ## 🟢 Nits - The comment on line 93 is helpful and explains the constraint clearly. ## Tests This is infrastructure/Kubernetes configuration. No unit tests apply. The fix should be validated by deploying to the cluster and confirming that: - The container no longer OOMKills under the expected concurrent agent load. - Memory usage stays well below 32Gi under normal operation (if it regularly approaches the limit, the problem is just postponed). ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#980