fix(trmnl): give Chromium a writable HOME and stamp obtained_at #484

Merged

k.renner merged 2 commits from fix-trmnl-renderer-runtime into master 2026-08-12 17:02:11 +00:00

Conversation 2 Commits 2 Files Changed 1

+52 -1

claude-bot commented 2026-08-12 16:49:09 +00:00

Member

Copy Link

Copy Source

Two runtime faults found by deploying #483. Neither is visible statically — both needed the pod running.

Current state on the cluster: pod Running and Ready, /health passing, schedule firing every minute, and zero screens reaching Terminus. Quiet in exactly the way this stack keeps being quiet.

1. Chromium never launched

Every render died with:

chrome_crashpad_handler: --database is required
[ERROR] [browser] Recovery attempt 5/5 failed: Browser crashed

Chromium can't start without a writable HOME — crashpad derives its database path from it. Diagnosed on the pod rather than inferred:

HOME=/          uid=1000 gid=1000
/ writable?     NO      (drwxr-xr-x root root)
/tmp writable?  yes

The cause is our own securityContext, not the image. Forcing runAsUser: 1000 with no matching passwd entry makes HOME fall back to /. Sets HOME=/tmp; crash dumps are throwaway and don't belong on the PVC.

Worth being precise here, because the review on #483 flagged the dropped capabilities as the likely Chromium risk, and the comment I added in response said they were fine. The capabilities were fine — that part of the analysis held. But something adjacent to that same securityContext broke it anyway, which is a fair hit on the confidence of the comment rather than its content. The comment now tells the next person to check HOME and /tmp before loosening the capability drop, so this doesn't get "fixed" by weakening security.

2. Nothing would have been pushed even once it rendered

[WARNING] [scheduler] BYOS tokens for "Entrance e-ink panel" expired
          beyond the refresh window — re-authenticate in the schedule settings

The scheduler derives token age from obtained_at, and the ConfigMap template shipped a literal 0 — so freshly-minted tokens read as dating from 1970 and the webhook was skipped entirely. The init container now stamps it with jq's now * 1000 | floor at write time; verified the filter yields a 13-digit epoch-ms value.

Stock Terminus disables session expiry and issues effectively permanent tokens, so a write-time stamp rather than the true mint time is honest enough — and the tokens themselves were already verified good (200 against /api/models).

What already worked

Everything else from #483 came up correctly on first deploy, which narrows this to the two faults above:

• init container wrote /data/schedules.json (1980 bytes) with the tokens spliced in
• [Config] Data directory: /data — the PVC mount is doing its job, so the silent-fallback trap didn't fire
• [Config] Running in standalone mode — no Supervisor assumptions
• schedule loaded: Entrance e-ink panel [every 1 min] → webhook
• trmnl-renderer-secrets ExternalSecret SecretSynced
• the HA-reachability fix from review holds — no connection errors to HA in the logs

🤖 Generated with Claude Code

Two runtime faults found by deploying #483. Neither is visible statically — both needed the pod running. Current state on the cluster: pod **Running and Ready**, `/health` passing, schedule firing every minute, and **zero screens reaching Terminus**. Quiet in exactly the way this stack keeps being quiet. ## 1. Chromium never launched Every render died with: ``` chrome_crashpad_handler: --database is required [ERROR] [browser] Recovery attempt 5/5 failed: Browser crashed ``` Chromium can't start without a writable `HOME` — crashpad derives its database path from it. Diagnosed on the pod rather than inferred: ``` HOME=/ uid=1000 gid=1000 / writable? NO (drwxr-xr-x root root) /tmp writable? yes ``` **The cause is our own securityContext, not the image.** Forcing `runAsUser: 1000` with no matching passwd entry makes `HOME` fall back to `/`. Sets `HOME=/tmp`; crash dumps are throwaway and don't belong on the PVC. Worth being precise here, because the review on #483 flagged the dropped capabilities as the likely Chromium risk, and the comment I added in response said they were fine. **The capabilities were fine** — that part of the analysis held. But something adjacent to that same securityContext broke it anyway, which is a fair hit on the confidence of the comment rather than its content. The comment now tells the next person to check `HOME` and `/tmp` *before* loosening the capability drop, so this doesn't get "fixed" by weakening security. ## 2. Nothing would have been pushed even once it rendered ``` [WARNING] [scheduler] BYOS tokens for "Entrance e-ink panel" expired beyond the refresh window — re-authenticate in the schedule settings ``` The scheduler derives token age from `obtained_at`, and the ConfigMap template shipped a literal `0` — so freshly-minted tokens read as dating from 1970 and the webhook was skipped entirely. The init container now stamps it with jq's `now * 1000 | floor` at write time; verified the filter yields a 13-digit epoch-ms value. Stock Terminus disables session expiry and issues effectively permanent tokens, so a write-time stamp rather than the true mint time is honest enough — and the tokens themselves were already verified good (`200` against `/api/models`). ## What already worked Everything else from #483 came up correctly on first deploy, which narrows this to the two faults above: - init container wrote `/data/schedules.json` (1980 bytes) with the tokens spliced in - `[Config] Data directory: /data` — the PVC mount is doing its job, so the silent-fallback trap didn't fire - `[Config] Running in standalone mode` — no Supervisor assumptions - schedule loaded: `Entrance e-ink panel [every 1 min] → webhook` - `trmnl-renderer-secrets` ExternalSecret `SecretSynced` - the HA-reachability fix from review holds — no connection errors to HA in the logs 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-12 16:49:09 +00:00

fix(trmnl): give Chromium a writable HOME and stamp obtained_at ... d824263759

Two runtime faults from the first deploy. Neither is visible statically
— both needed the pod running to find.

1. The browser never launched. Every render died with

     chrome_crashpad_handler: --database is required

   Chromium cannot start without a writable HOME: crashpad builds its
   database path from it. Diagnosed on the pod rather than guessed —
   HOME=/, uid=1000, / is root-owned 0755 and not writable, /tmp is.
   The cause is OUR securityContext, not the image: forcing
   runAsUser 1000 with no matching passwd entry makes HOME fall back
   to "/". Sets HOME=/tmp; crash dumps are throwaway, so they do not
   belong on the PVC.

   Worth being precise about, because review had flagged the dropped
   capabilities as the likely Chromium risk and the comment I added
   said they were fine. The capabilities were fine. Something adjacent
   to that securityContext broke it anyway, and the comment now says
   to check HOME before loosening the capability block.

2. Even once the browser works, nothing would have been pushed. The
   scheduler logged

     BYOS tokens ... expired beyond the refresh window

   and skipped the webhook entirely. It derives token age from
   obtained_at, and the template shipped a literal 0, so
   freshly-minted tokens read as dating from 1970. The init container
   now stamps it with jq's `now * 1000 | floor` at write time.
   Verified the filter produces a 13-digit epoch-ms value. Stock
   Terminus disables session expiry and issues effectively permanent
   tokens, so a write-time stamp is honest enough.

Failure mode of the pair was quiet in the way this stack keeps being
quiet: pod Running and Ready, /health passing, schedule firing once a
minute, zero screens reaching Terminus.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed 2026-08-12 16:52:37 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit d8242637591b93a5c82ad48758c2cfc8f4104ce3.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Two targeted, pod-diagnosed runtime fixes to kubernetes/apps/trmnl/renderer.yaml: (1) HOME=/tmp on the renderer container so Chromium/crashpad has a writable home directory under the non-root securityContext, and (2) the init container's jq filter now stamps obtained_at with now * 1000 | floor instead of shipping a literal 0, so the scheduler no longer treats freshly-minted BYOS tokens as expired. Both changes are small, precise, and match the failure signatures quoted in the PR description. No blocking issues found.

🟡 Suggestions

• kubernetes/apps/trmnl/renderer.yaml:241-242 — HOME=/tmp uses the container's writable root-fs layer (no emptyDir backing it, unlike the /tmp mount convention this repo uses elsewhere for containers with readOnlyRootFilesystem: true, e.g. kubernetes/apps/monitoring/pve-exporter-deployment.yaml:60-67). That's consistent here since this container doesn't set readOnlyRootFilesystem, so it's not a bug — but now that Chromium will actually launch and run every minute (previously it crashed before writing anything), crashpad/profile data under /tmp will accumulate for the life of the pod. There's no ephemeral-storage limit on the container (resources.limits only sets memory: 2Gi). Worth a note-to-self to watch node disk pressure on this node if the panel runs for extended periods, though this isn't something to block on.
• PR body: the fix is verified piecemeal (writable-HOME diagnosed on the crashed pod; the jq filter's output format verified in isolation), but the description doesn't state an end-to-end redeploy confirming a screen actually reached Terminus with both fixes applied together. Given the PR's own point about a previous comment's confidence being wrong despite being locally correct, it'd strengthen this PR to confirm post-merge that obtained_at freshness plus the working browser together produce a pushed screen, not just that each symptom individually looks fixed.

🟢 Nits

• The same rationale (writable-HOME root cause, obtained_at stamping) is now documented in three places (ConfigMap header at renderer.yaml:39-44, the jq script at renderer.yaml:23-31/155-163, and the HOME env var at renderer.yaml:228-240, plus a fourth cross-reference at the capabilities block renderer.yaml:296-300). This matches the file's existing very-verbose commenting convention, so it's not out of place, but it's worth knowing the explanation is duplicated rather than centralized if it ever needs correcting later.

Tests

No automated tests apply here — this is a GitOps manifest, and the repo's ci-manifests-test.yml (kustomize-build, kubeconform-style schema checks, shellcheck) will validate structure but not runtime behavior; the embedded jq/sh snippet in the init container command isn't covered by the shellcheck step (which only scans *.sh files under kubernetes/). The PR's own methodology — diagnosing both faults on the live pod and quoting the exact log lines — is the appropriate verification for this kind of change, and the jq filter's ms-epoch output was independently checked. The one gap is the lack of a stated full end-to-end confirmation (screen pushed to Terminus) with both fixes combined, noted above as a suggestion.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `d8242637591b93a5c82ad48758c2cfc8f4104ce3`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Two targeted, pod-diagnosed runtime fixes to `kubernetes/apps/trmnl/renderer.yaml`: (1) `HOME=/tmp` on the `renderer` container so Chromium/crashpad has a writable home directory under the non-root `securityContext`, and (2) the init container's `jq` filter now stamps `obtained_at` with `now * 1000 | floor` instead of shipping a literal `0`, so the scheduler no longer treats freshly-minted BYOS tokens as expired. Both changes are small, precise, and match the failure signatures quoted in the PR description. No blocking issues found. ## 🟡 Suggestions - `kubernetes/apps/trmnl/renderer.yaml:241-242` — `HOME=/tmp` uses the container's writable root-fs layer (no `emptyDir` backing it, unlike the `/tmp` mount convention this repo uses elsewhere for containers with `readOnlyRootFilesystem: true`, e.g. `kubernetes/apps/monitoring/pve-exporter-deployment.yaml:60-67`). That's consistent here since this container doesn't set `readOnlyRootFilesystem`, so it's not a bug — but now that Chromium will actually launch and run every minute (previously it crashed before writing anything), crashpad/profile data under `/tmp` will accumulate for the life of the pod. There's no `ephemeral-storage` limit on the container (`resources.limits` only sets `memory: 2Gi`). Worth a note-to-self to watch node disk pressure on this node if the panel runs for extended periods, though this isn't something to block on. - PR body: the fix is verified piecemeal (writable-`HOME` diagnosed on the crashed pod; the `jq` filter's output format verified in isolation), but the description doesn't state an end-to-end redeploy confirming a screen actually reached Terminus with both fixes applied together. Given the PR's own point about a previous comment's confidence being wrong despite being locally correct, it'd strengthen this PR to confirm post-merge that `obtained_at` freshness plus the working browser together produce a pushed screen, not just that each symptom individually looks fixed. ## 🟢 Nits - The same rationale (writable-HOME root cause, `obtained_at` stamping) is now documented in three places (ConfigMap header at `renderer.yaml:39-44`, the `jq` script at `renderer.yaml:23-31`/`155-163`, and the `HOME` env var at `renderer.yaml:228-240`, plus a fourth cross-reference at the `capabilities` block `renderer.yaml:296-300`). This matches the file's existing very-verbose commenting convention, so it's not out of place, but it's worth knowing the explanation is duplicated rather than centralized if it ever needs correcting later. ## Tests No automated tests apply here — this is a GitOps manifest, and the repo's `ci-manifests-test.yml` (`kustomize-build`, `kubeconform`-style schema checks, `shellcheck`) will validate structure but not runtime behavior; the embedded `jq`/`sh` snippet in the init container `command` isn't covered by the `shellcheck` step (which only scans `*.sh` files under `kubernetes/`). The PR's own methodology — diagnosing both faults on the live pod and quoting the exact log lines — is the appropriate verification for this kind of change, and the jq filter's ms-epoch output was independently checked. The one gap is the lack of a stated full end-to-end confirmation (screen pushed to Terminus) with both fixes combined, noted above as a suggestion. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=d8242637591b93a5c82ad48758c2cfc8f4104ce3 -->

claude-bot added 1 commit 2026-08-12 16:54:04 +00:00

fix(trmnl): bound the renderer's /tmp with a sized emptyDir ... 2dd3e8f0b2

Follow-up to the HOME=/tmp fix in this PR, from review.

Pointing HOME at /tmp was correct but left it on the container's
writable layer, which was harmless while Chromium crashed on launch and
wrote nothing. With the browser actually starting once a minute for the
life of the pod, crashpad state and profile data now accumulate there
with nothing to bound them — resources.limits sets memory only, so
there is no ephemeral-storage ceiling to catch it either.

Backs it with an emptyDir at a 512Mi sizeLimit, matching the /tmp mount
convention already used in pve-exporter-deployment.yaml. The point of
the limit is the failure mode, not the size: unbounded it degrades into
node disk pressure, which surfaces far from this pod and hurts whatever
else lands on that node; bounded it is an eviction of the renderer,
attributable on sight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed 2026-08-12 16:57:22 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 2dd3e8f0b28580a19cabff30d23d10dc788b9b47.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Two small, targeted runtime fixes to kubernetes/apps/trmnl/renderer.yaml, both diagnosed against the actually-running pod rather than inferred: (1) HOME=/tmp plus a sized emptyDir mount at /tmp so Chromium's crashpad has a writable home under the runAsUser: 1000 securityContext, and (2) the init container's jq filter now stamps obtained_at with now * 1000 | floor instead of shipping a literal 0, so the scheduler doesn't treat freshly-minted BYOS tokens as expired. Both changes are consistent with the surrounding manifest and I found no blocking issues.

🟡 Suggestions

• kubernetes/apps/trmnl/renderer.yaml:322-324 — the new tmp emptyDir has no medium set, so it lands on node disk (correct choice here, given the PR's own reasoning about crashpad dumps). Worth double-checking after this deploys that the once-a-minute Chromium launch cycle actually cleans up its per-launch temp/profile dir on the happy path — if it doesn't, 512Mi could fill from healthy operation rather than only from the crash-loop case the comment anticipates, causing the exact bounded eviction described but on a much shorter timeline than intended. Not blocking since the PR already accepts eviction as the deliberate failure mode.
• The same rationale for HOME=/tmp and for obtained_at stamping is written out twice each — once as a manifest comment (lines 39-44, 228-240, 255-262, 305-310) and once inline in the shell script embedded in the ConfigMap (lines 23-31 in the diff). That's a documentation choice, not a defect, but it does mean any future change to this logic needs both copies updated in lockstep or the comments drift out of sync with each other.

Tests

No automated tests apply to this class of change — it's a manifest-only fix, and per the PR body the actual verification was done by redeploying and inspecting the running pod (crashpad launch succeeded, /data/schedules.json written with a plausible byte count, and the jq filter's output format confirmed as 13-digit epoch-ms). That's the right kind of verification for this repo; nothing here would be meaningfully improved by a unit test. ci-manifests-test.yml should still validate the YAML/kustomize build mechanically.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `2dd3e8f0b28580a19cabff30d23d10dc788b9b47`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Two small, targeted runtime fixes to `kubernetes/apps/trmnl/renderer.yaml`, both diagnosed against the actually-running pod rather than inferred: (1) `HOME=/tmp` plus a sized `emptyDir` mount at `/tmp` so Chromium's crashpad has a writable home under the `runAsUser: 1000` securityContext, and (2) the init container's `jq` filter now stamps `obtained_at` with `now * 1000 | floor` instead of shipping a literal `0`, so the scheduler doesn't treat freshly-minted BYOS tokens as expired. Both changes are consistent with the surrounding manifest and I found no blocking issues. ## 🟡 Suggestions - `kubernetes/apps/trmnl/renderer.yaml:322-324` — the new `tmp` `emptyDir` has no `medium` set, so it lands on node disk (correct choice here, given the PR's own reasoning about crashpad dumps). Worth double-checking after this deploys that the once-a-minute Chromium launch cycle actually cleans up its per-launch temp/profile dir on the happy path — if it doesn't, 512Mi could fill from healthy operation rather than only from the crash-loop case the comment anticipates, causing the exact bounded eviction described but on a much shorter timeline than intended. Not blocking since the PR already accepts eviction as the deliberate failure mode. - The same rationale for `HOME=/tmp` and for `obtained_at` stamping is written out twice each — once as a manifest comment (lines 39-44, 228-240, 255-262, 305-310) and once inline in the shell script embedded in the ConfigMap (lines 23-31 in the diff). That's a documentation choice, not a defect, but it does mean any future change to this logic needs both copies updated in lockstep or the comments drift out of sync with each other. ## Tests No automated tests apply to this class of change — it's a manifest-only fix, and per the PR body the actual verification was done by redeploying and inspecting the running pod (crashpad launch succeeded, `/data/schedules.json` written with a plausible byte count, and the jq filter's output format confirmed as 13-digit epoch-ms). That's the right kind of verification for this repo; nothing here would be meaningfully improved by a unit test. `ci-manifests-test.yml` should still validate the YAML/kustomize build mechanically. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=2dd3e8f0b28580a19cabff30d23d10dc788b9b47 -->

k.renner merged commit fc53334b31 into master 2026-08-12 17:02:11 +00:00

k.renner referenced this issue from a commit 2026-08-12 17:02:11 +00:00

Merge pull request 'fix(trmnl): give Chromium a writable HOME and stamp obtained_at' (#484) from fix-trmnl-renderer-runtime into master

claude-bot referenced this pull request 2026-08-12 17:16:14 +00:00

fix(trmnl): make Terminus issue long-lived API tokens #485

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#484