fix(trmnl): make the renderer's /tmp RAM-backed (34 GB/day of node writes) #529

2026-08-20T04:26:56Z

claude-bot commented

2026-08-20 04:26:56 +00:00

Refs #434.

The finding

The TRMNL renderer is the single largest writer in the cluster — more than every Ceph OSD and every database combined — and none of it needs to touch disk.

Measured from the container own cgroup io.stat:

253:0  (node /dev/vda5)   249,092,697,088 bytes over 7.3 days  = ~34 GB/day
252:48 (its own PVC)                340 MB total               = negligible

The writes were not going to its volume at all. They were going to the node filesystem through a disk-backed emptyDir.

Why

emptyDir: {} without medium: Memory is node storage. Easy to read as "ephemeral, therefore free" — it is ephemeral, but every byte still hits the disk.

The cause is cadence, not volume. schedules.template.json renders on "* * * * *", so Chrome launches 1,440 times a day, and each launch creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*.

At rest /tmp holds under 8 MB:

/dev/vda5  60G  29G  32G  48% /tmp
7.9M  /tmp/puppeteer_dev_chrome_profile-yFnELF

Pure churn, for data that never outlives the render.

The change

One line: medium: Memory.

tmpfs pages count against the container memory limit, so the 512Mi cap is kept deliberately — measured usage is 129Mi against a 2Gi limit, so even fully consuming the cap leaves ample headroom.

Expected effect

~34 GB/day off the OS disks, which measured 215 GB/day over a clean 4h window. That is a larger cluster-wide reduction than the entire Home Assistant recorder migration achieved, for a one-line change.

I will verify against node_disk_written_bytes_total and the container cgroup counter after it rolls out, rather than assuming.

Worth a separate thought

Rendering every 60 s for an e-ink display looks generous — those panels typically refresh on the order of minutes. Halving the cadence would halve the remaining Chrome overhead (CPU, memory, and the tmpfs churn that is now in RAM). Not changed here, since that is a product decision rather than an infrastructure one.

🤖 Generated with Claude Code

Refs #434. ## The finding The TRMNL renderer is the **single largest writer in the cluster** — more than every Ceph OSD and every database combined — and none of it needs to touch disk. Measured from the container own cgroup `io.stat`: ``` 253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day 252:48 (its own PVC) 340 MB total = negligible ``` The writes were **not going to its volume at all**. They were going to the node filesystem through a disk-backed `emptyDir`. ## Why `emptyDir: {}` without `medium: Memory` is *node storage*. Easy to read as "ephemeral, therefore free" — it is ephemeral, but every byte still hits the disk. The cause is **cadence, not volume**. `schedules.template.json` renders on `"* * * * *"`, so Chrome launches **1,440 times a day**, and each launch creates, fills and deletes a `/tmp/puppeteer_dev_chrome_profile-*`. At rest `/tmp` holds **under 8 MB**: ``` /dev/vda5 60G 29G 32G 48% /tmp 7.9M /tmp/puppeteer_dev_chrome_profile-yFnELF ``` Pure churn, for data that never outlives the render. ## The change One line: `medium: Memory`. tmpfs pages count against the container memory limit, so the `512Mi` cap is kept deliberately — measured usage is **129Mi against a 2Gi limit**, so even fully consuming the cap leaves ample headroom. ## Expected effect ~34 GB/day off the OS disks, which measured **215 GB/day** over a clean 4h window. That is a **larger cluster-wide reduction than the entire Home Assistant recorder migration achieved**, for a one-line change. I will verify against `node_disk_written_bytes_total` and the container cgroup counter after it rolls out, rather than assuming. ## Worth a separate thought Rendering every 60 s for an e-ink display looks generous — those panels typically refresh on the order of minutes. Halving the cadence would halve the remaining Chrome overhead (CPU, memory, and the tmpfs churn that is now in RAM). Not changed here, since that is a product decision rather than an infrastructure one. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-20 04:26:57 +00:00

fix(trmnl): make the renderer's /tmp RAM-backed (34 GB/day of node writes)

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

b4f44b2002

The renderer was the single largest writer in the cluster -- more than every
Ceph OSD and every database combined -- and none of it needed to touch disk.

Measured 2026-08-20 from the container's own cgroup io.stat:

  253:0 (node /dev/vda5)  249,092,697,088 bytes over 7.3 days  = ~34 GB/day
  252:48 (its own PVC)             340 MB total                = negligible

So the writes were not going to its volume at all; they were going to the
node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without
`medium: Memory` is node storage, which is easy to read as "ephemeral, so
free".

The cause is cadence rather than volume. schedules.template.json renders on
"* * * * *", so Chrome is launched 1440 times a day and each launch creates,
fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp
holds under 8 MB -- it is pure churn for data that never outlives the render.

tmpfs pages count against the container's memory limit, so the 512Mi cap is
kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample
headroom even if the cap were fully consumed.

Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a
4h window. This is a bigger cluster-wide reduction than the Home Assistant
recorder migration achieved, for a one-line change.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-20 04:28:33 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit b4f44b20029280721970c82ceb94f68ce9737eaf.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR makes the TRMNL renderer's /tmp volume RAM-backed by adding medium: Memory to the emptyDir spec. The change is a one-line optimization targeting measured disk pressure (~34 GB/day) caused by Chromium being launched 1,440 times daily for ephemeral rendering work. Memory headroom is adequate, and there are no backward-compatibility or correctness concerns.

🟢 Nits

The extensive inline comment is appropriate for the non-obvious performance impact, but consider that operational metrics (disk I/O reduction, tmpfs memory pressure) should be logged as a follow-up to validate the expected benefit.

Tests

N/A — this is a Kubernetes configuration change, not code. The PR author correctly plans to verify the impact post-deployment against node_disk_written_bytes_total and container cgroup counters.

Recommendation

Looks reasonable — no blocking issues; the optimization is well-justified and safely configured.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `b4f44b20029280721970c82ceb94f68ce9737eaf`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR makes the TRMNL renderer's `/tmp` volume RAM-backed by adding `medium: Memory` to the emptyDir spec. The change is a one-line optimization targeting measured disk pressure (~34 GB/day) caused by Chromium being launched 1,440 times daily for ephemeral rendering work. Memory headroom is adequate, and there are no backward-compatibility or correctness concerns. ## 🟢 Nits The extensive inline comment is appropriate for the non-obvious performance impact, but consider that operational metrics (disk I/O reduction, tmpfs memory pressure) should be logged as a follow-up to validate the expected benefit. ## Tests N/A — this is a Kubernetes configuration change, not code. The PR author correctly plans to verify the impact post-deployment against `node_disk_written_bytes_total` and container cgroup counters. ## Recommendation Looks reasonable — no blocking issues; the optimization is well-justified and safely configured.

claude-bot referenced this pull request

2026-08-20 04:30:42 +00:00

fix: make small scratch emptyDirs RAM-backed instead of node-disk #530

k.renner merged commit ece3c93ad5 into master

2026-08-20 04:33:38 +00:00

k.renner referenced this issue from a commit

2026-08-20 04:33:38 +00:00

Merge pull request 'fix(trmnl): make the renderer's /tmp RAM-backed (34 GB/day of node writes)' (#529) from fix-trmnl-renderer-tmpfs into master

claude-bot referenced this pull request

2026-08-20 04:39:55 +00:00

Ceph observability gaps: no toolbox / no admin path, plus osd.2 endurance clock to confirm #434

Sign in to join this conversation.