fix: make small scratch emptyDirs RAM-backed instead of node-disk #530

Merged

k.renner merged 1 commits from fix-emptydir-tmpfs into master 2026-08-20 04:38:22 +00:00

Conversation 1 Commits 1 Files Changed 10

+36 -12

claude-bot commented 2026-08-20 04:30:42 +00:00

Member

Copy Link

Copy Source

Follow-up to the trmnl renderer finding (#529). Survey of the whole repo for the same pattern.

The pattern

An emptyDir without medium: Memory is node storage. Easy to read as "ephemeral, therefore free" — it is ephemeral, but every byte still hits the node disk.

Repo survey: 23 emptyDir declarations, only 2 RAM-backed. This converts 12 of the remaining ones.

Honest scope: this buys almost no write reduction

Ranking every pod by OS-disk writes over a 6h window:

67 GB/day  trmnl/trmnl-renderer      <- fixed in #529
15 GB/day  rook-ceph-mon-c           <- hostPath, not emptyDir
13 GB/day  rook-ceph-mon-b
11 GB/day  rook-ceph-mon-a
 5 GB/day  kube-apiserver-k8s-1
 2 GB/day  immich/im-pg-1            <- container logs; data is on PVC
 ...

Nothing emptyDir-driven appears after trmnl. Every volume converted here measured 8.0K in the running pod:

/var/lib/code-server   8.0K
/mosquitto/auth        8.0K
/state (z2m guards)    8.0K

So the value is correctness, not wear — plus one real security benefit: mosquitto/auth holds the generated passwd/acl files, and keeping credential material in RAM rather than on the node filesystem is strictly better.

I would rather state that plainly than dress this up as a performance fix.

Safety

• Each converted volume gets an explicit sizeLimit (16Mi where none existed), so tmpfs cannot silently eat into a container memory limit. All are orders of magnitude above measured usage.
• Behaviour is otherwise unchanged: an emptyDir persists for the pod lifetime on either medium, so the mosquitto shim assumption that the passwd file survives a container restart still holds.
• All 9 affected kustomizations build clean.

Deliberately NOT converted

• kubernetes/talos/cilium.yaml, talos/patches/cilium-cni.yaml — vendored/generated from the chart; a hand edit would be lost on the next upgrade.
• trmnl/renderer.yaml — handled in #529, where the actual 34 GB/day lives.

Cost

This rolls the affected pods once (audiobookshelf, home-assistant, immich, paperless, pve-exporter, netbird-ingress, velero restore-test, rook-ceph toolbox). Worth weighing against the modest benefit — happy to drop the low-value ones and keep only mosquitto/auth if you would rather not churn the rest.

🤖 Generated with Claude Code

Follow-up to the trmnl renderer finding (#529). Survey of the whole repo for the same pattern. ## The pattern An `emptyDir` **without** `medium: Memory` is **node storage**. Easy to read as "ephemeral, therefore free" — it is ephemeral, but every byte still hits the node disk. Repo survey: **23 emptyDir declarations, only 2 RAM-backed.** This converts 12 of the remaining ones. ## Honest scope: this buys almost no write reduction Ranking every pod by OS-disk writes over a 6h window: ``` 67 GB/day trmnl/trmnl-renderer <- fixed in #529 15 GB/day rook-ceph-mon-c <- hostPath, not emptyDir 13 GB/day rook-ceph-mon-b 11 GB/day rook-ceph-mon-a 5 GB/day kube-apiserver-k8s-1 2 GB/day immich/im-pg-1 <- container logs; data is on PVC ... ``` **Nothing emptyDir-driven appears after trmnl.** Every volume converted here measured 8.0K in the running pod: ``` /var/lib/code-server 8.0K /mosquitto/auth 8.0K /state (z2m guards) 8.0K ``` So the value is **correctness**, not wear — plus one real security benefit: `mosquitto/auth` holds the generated `passwd`/`acl` files, and keeping credential material in RAM rather than on the node filesystem is strictly better. I would rather state that plainly than dress this up as a performance fix. ## Safety - Each converted volume gets an explicit `sizeLimit` (16Mi where none existed), so tmpfs cannot silently eat into a container memory limit. All are orders of magnitude above measured usage. - Behaviour is otherwise unchanged: an `emptyDir` persists for the **pod** lifetime on either medium, so the mosquitto shim assumption that the passwd file survives a *container* restart still holds. - All 9 affected kustomizations build clean. ## Deliberately NOT converted - `kubernetes/talos/cilium.yaml`, `talos/patches/cilium-cni.yaml` — vendored/generated from the chart; a hand edit would be lost on the next upgrade. - `trmnl/renderer.yaml` — handled in #529, where the actual 34 GB/day lives. ## Cost This rolls the affected pods once (audiobookshelf, home-assistant, immich, paperless, pve-exporter, netbird-ingress, velero restore-test, rook-ceph toolbox). Worth weighing against the modest benefit — happy to drop the low-value ones and keep only `mosquitto/auth` if you would rather not churn the rest. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-20 04:30:42 +00:00

fix: make small scratch emptyDirs RAM-backed instead of node-disk ... b34450b22e

Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.

Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.

HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:

  /var/lib/code-server   8.0K
  /mosquitto/auth        8.0K
  /state (z2m guards)    8.0K

So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.

Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.

Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.

DELIBERATELY NOT CONVERTED:
  - kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
    and generated from the chart; a hand edit would be lost on upgrade.
  - trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.

Cost: this rolls the affected pods once.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot requested changes 2026-08-20 04:36:28 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit b34450b22ec0bbb9335e028289334c9aceacc334.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR converts 12 emptyDir: {} volumes across 9 kustomizations to medium: Memory (tmpfs) with an explicit sizeLimit: 16Mi, framed honestly as a correctness/security fix rather than a meaningful write-reduction. The individual conversions are safe and match the PR's own measurements — but the PR's central scoping argument (skip trmnl/renderer.yaml because its 67 GB/day tmp volume was "fixed in #529") does not hold against the code actually in this checkout, and one of the 12 conversions (code-server's vscode-state) caps a volume the surrounding comments say is meant to hold installed VS Code extensions at 16Mi, which is unrealistically small for that purpose.

🔴 Blocking

• kubernetes/apps/trmnl/renderer.yaml:363-365 — the PR's stated reason for excluding this file is that it was "fixed in #529" and is therefore already RAM-backed, calling it the source of "the actual 34 GB/day" (elsewhere in the same PR body: "67 GB/day … <- fixed in #529"). But as checked out here, that volume is still a plain node-disk emptyDir — sizeLimit: 512Mi, no medium: Memory:

  - name: tmp
    emptyDir:
      sizeLimit: 512Mi
  

  A repo-wide grep for medium: Memory returns exactly the 10 files this PR itself touches (Grep "medium: Memory" → 10 hits, all in this diff) — there is no pre-existing RAM-backed volume anywhere in kubernetes/, trmnl included. So either #529 did not actually convert this volume to tmpfs (it only added the sizeLimit bound, which caps unbounded growth but still writes to node disk on every render — the container comment at renderer.yaml:296-303 confirms Chromium relaunches into this dir "once a minute for the life of the pod"), or the PR's premise about #529 is simply wrong. Either way, the single largest disk-write source identified in this PR's own survey (4-8x every other volume converted here, which each measured 8.0K) is left untouched while the PR's description asserts it's handled. This should be corrected before merge — either fix renderer.yaml's tmp volume in this PR, or correct the PR body so reviewers don't approve believing the actual problem is resolved.
  • Also worth reconciling: the PR body cites two different figures for the same volume, "67 GB/day" in the ranking table vs "34 GB/day" in the "Deliberately NOT converted" section.

🟡 Suggestions

• kubernetes/apps/home-assistant/deployment.yaml:288-291 (vscode-state, mounted at /var/lib/code-server, i.e. code-server's $HOME) — the adjacent comment (deployment.yaml:285-287) says this volume exists so "installed extensions don't pollute HA's /config listing" and explicitly accepts the trade-off that "extensions reset across pod restarts." That implies real extension installs are expected here. A single non-trivial VS Code/code-server extension (language servers, Python tooling, etc.) commonly runs into the tens of MB, so 16Mi will be exhausted quickly. Unlike the default-medium emptyDir this replaces, a medium: Memory volume enforces its sizeLimit as a hard tmpfs mount cap, so once full, writes fail with ENOSPC immediately rather than triggering a slower capacity-based eviction — i.e., this will manifest as code-server itself failing to install/save extensions, not a graceful degradation. The PR's own measurement (8.0K) reflects a pod that has never had extensions installed, so it doesn't validate 16Mi against the volume's actual intended use. Consider either sizing this one significantly larger (100s of Mi) or leaving it off tmpfs given it's explicitly meant to hold larger, session-lived state.

• kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml:1-9 — the file's header comment enumerates exactly three deliberate divergences from the upstream verbatim manifest and states "no other divergence; keep it that way so upstream diffs stay readable." This PR adds a fourth (the medium: Memory/sizeLimit on ceph-config) without updating that header, so the comment is now stale/inaccurate for future readers diffing against upstream.

🟢 Nits

• The "23 emptyDir declarations, only 2 RAM-backed" claim in the PR body isn't reflected in the checkout: a repo-wide search for medium: Memory finds only the volumes this PR itself adds it to (10 files, matching this diff) — no pre-existing RAM-backed emptyDir was found. Not load-bearing for the PR's mechanics, but worth double-checking the survey numbers given the trmnl discrepancy above.

Tests

No test suite applies to this style of change (declarative Kubernetes manifests reconciled by Flux). The PR claims "All 9 affected kustomizations build clean" — I did not independently re-run kustomize build in this review, but every hunk is a mechanically straightforward emptyDir: {} → emptyDir: {medium: Memory, sizeLimit: 16Mi} substitution using standard, long-stable API fields, so a build failure is unlikely. The more relevant verification — actually exercising code-server with an extension install against the new vscode-state tmpfs cap — is a runtime check, not something a build validates, and is exactly the gap the Suggestion above is about.

Recommendation

Request changes — 1 blocking issue (the trmnl exclusion rationale doesn't match the current code, leaving the actual largest disk-write source unaddressed while the PR implies it's handled).

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `b34450b22ec0bbb9335e028289334c9aceacc334`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR converts 12 `emptyDir: {}` volumes across 9 kustomizations to `medium: Memory` (tmpfs) with an explicit `sizeLimit: 16Mi`, framed honestly as a correctness/security fix rather than a meaningful write-reduction. The individual conversions are safe and match the PR's own measurements — but the PR's central scoping argument (skip `trmnl/renderer.yaml` because its 67 GB/day `tmp` volume was "fixed in #529") does not hold against the code actually in this checkout, and one of the 12 conversions (`code-server`'s `vscode-state`) caps a volume the surrounding comments say is meant to hold installed VS Code extensions at 16Mi, which is unrealistically small for that purpose. ## 🔴 Blocking - **`kubernetes/apps/trmnl/renderer.yaml:363-365`** — the PR's stated reason for excluding this file is that it was "fixed in #529" and is therefore already RAM-backed, calling it the source of "the actual 34 GB/day" (elsewhere in the same PR body: "67 GB/day … `<- fixed in #529`"). But as checked out here, that volume is still a plain node-disk `emptyDir` — `sizeLimit: 512Mi`, no `medium: Memory`: ``` - name: tmp emptyDir: sizeLimit: 512Mi ``` A repo-wide grep for `medium: Memory` returns exactly the 10 files this PR itself touches (`Grep "medium: Memory"` → 10 hits, all in this diff) — there is no pre-existing RAM-backed volume anywhere in `kubernetes/`, trmnl included. So either #529 did not actually convert this volume to tmpfs (it only added the `sizeLimit` bound, which caps unbounded growth but still writes to node disk on every render — the container comment at `renderer.yaml:296-303` confirms Chromium relaunches into this dir "once a minute for the life of the pod"), or the PR's premise about #529 is simply wrong. Either way, the single largest disk-write source identified in this PR's own survey (4-8x every other volume converted here, which each measured 8.0K) is left untouched while the PR's description asserts it's handled. This should be corrected before merge — either fix `renderer.yaml`'s `tmp` volume in this PR, or correct the PR body so reviewers don't approve believing the actual problem is resolved. - Also worth reconciling: the PR body cites two different figures for the same volume, "67 GB/day" in the ranking table vs "34 GB/day" in the "Deliberately NOT converted" section. ## 🟡 Suggestions - **`kubernetes/apps/home-assistant/deployment.yaml:288-291`** (`vscode-state`, mounted at `/var/lib/code-server`, i.e. code-server's `$HOME`) — the adjacent comment (`deployment.yaml:285-287`) says this volume exists so "installed extensions don't pollute HA's /config listing" and explicitly accepts the trade-off that "extensions reset across pod restarts." That implies real extension installs are expected here. A single non-trivial VS Code/code-server extension (language servers, Python tooling, etc.) commonly runs into the tens of MB, so 16Mi will be exhausted quickly. Unlike the default-medium emptyDir this replaces, a `medium: Memory` volume enforces its `sizeLimit` as a hard tmpfs mount cap, so once full, writes fail with ENOSPC immediately rather than triggering a slower capacity-based eviction — i.e., this will manifest as code-server itself failing to install/save extensions, not a graceful degradation. The PR's own measurement (8.0K) reflects a pod that has never had extensions installed, so it doesn't validate 16Mi against the volume's actual intended use. Consider either sizing this one significantly larger (100s of Mi) or leaving it off tmpfs given it's explicitly meant to hold larger, session-lived state. - **`kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml:1-9`** — the file's header comment enumerates exactly three deliberate divergences from the upstream verbatim manifest and states "no other divergence; keep it that way so upstream diffs stay readable." This PR adds a fourth (the `medium: Memory`/`sizeLimit` on `ceph-config`) without updating that header, so the comment is now stale/inaccurate for future readers diffing against upstream. ## 🟢 Nits - The "23 emptyDir declarations, only 2 RAM-backed" claim in the PR body isn't reflected in the checkout: a repo-wide search for `medium: Memory` finds only the volumes this PR itself adds it to (10 files, matching this diff) — no pre-existing RAM-backed `emptyDir` was found. Not load-bearing for the PR's mechanics, but worth double-checking the survey numbers given the trmnl discrepancy above. ## Tests No test suite applies to this style of change (declarative Kubernetes manifests reconciled by Flux). The PR claims "All 9 affected kustomizations build clean" — I did not independently re-run `kustomize build` in this review, but every hunk is a mechanically straightforward `emptyDir: {}` → `emptyDir: {medium: Memory, sizeLimit: 16Mi}` substitution using standard, long-stable API fields, so a build failure is unlikely. The more relevant verification — actually exercising code-server with an extension install against the new `vscode-state` tmpfs cap — is a runtime check, not something a build validates, and is exactly the gap the Suggestion above is about. ## Recommendation Request changes — 1 blocking issue (the trmnl exclusion rationale doesn't match the current code, leaving the actual largest disk-write source unaddressed while the PR implies it's handled). <!-- pr-review:sha=b34450b22ec0bbb9335e028289334c9aceacc334 -->

k.renner merged commit 0d65fb0982 into master 2026-08-20 04:38:22 +00:00

k.renner referenced this issue from a commit 2026-08-20 04:38:23 +00:00

Merge pull request 'fix: make small scratch emptyDirs RAM-backed instead of node-disk' (#530) from fix-emptydir-tmpfs into master

claude-bot referenced this pull request 2026-08-20 04:39:55 +00:00

Ceph observability gaps: no toolbox / no admin path, plus osd.2 endurance clock to confirm #434

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#530