From e84ea0210046c0b94ba1a6763c80685bebe52f07 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 12 Aug 2026 19:15:43 +0200 Subject: [PATCH 1/4] fix(trmnl): make Terminus issue long-lived API tokens MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The renderer renders correctly and pushes nothing. Every webhook is rejected: [webhook] Webhook failed: 400 {"error":"expired JWT access token"} [webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."} Terminus issues API JWTs with API_ACCESS_TOKEN_PERIOD, default 1800s. Minting a replacement via POST /api/jwt requires a still-VALID access token in the Authorization header, so an expired token cannot refresh itself — the second line above is the refresh attempt being rejected by the very expiry it was trying to repair. There is no recovery path once the window closes. That makes "mint once, store in OpenBao" unworkable for an unattended appliance, which is what this stack is. The renderer's own code assumes "stock Terminus disables session expiration and issues effectively permanent tokens"; that is not true of this deployment. Rather than work around the assumption in the client, this makes it true: sets API_ACCESS_TOKEN_PERIOD to 10 years. The trade is deliberate. A long-lived JWT with full Terminus API rights now sits in OpenBao. The alternative was putting the Terminus account PASSWORD into the cluster so an init container could mint a short-lived token on every pod start — a strictly worse secret to hold, for a LAN-only service whose token is revocable by rotating the account. Also corrects the obtained_at comment on the renderer, which justified stamping "now" by repeating that same false permanence assumption. Stamping now is safe ONLY because of this setting; against the 30-minute default it is actively harmful, because the app would believe a stale token had a fresh window, skip its proactive refresh, and strand itself. The comment now says so, and says what to do if this setting is reverted. The period is fixed at mint time, so this does not revive the existing tokens — they must be re-minted after it deploys. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/apps/trmnl/renderer.yaml | 17 +++++++++++------ kubernetes/apps/trmnl/terminus.yaml | 28 ++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/kubernetes/apps/trmnl/renderer.yaml b/kubernetes/apps/trmnl/renderer.yaml index 0ee7b019..88f47ea4 100644 --- a/kubernetes/apps/trmnl/renderer.yaml +++ b/kubernetes/apps/trmnl/renderer.yaml @@ -154,13 +154,18 @@ spec: set -eu # obtained_at is stamped at write time, in MILLISECONDS. # It is not cosmetic: the scheduler derives token age from - # it, and a literal 0 in the template made the app treat - # freshly-minted tokens as ancient and refuse to push — + # it, and a literal 0 made the app treat freshly-minted + # tokens as ancient and skip the webhook entirely — # "BYOS tokens ... expired beyond the refresh window" - # — with no render attempt and no error, just a warning at - # startup. Stamping it here is honest enough: stock - # Terminus disables session expiry and issues effectively - # permanent tokens, so age is not load-bearing. + # + # Stamping it "now" is only safe because Terminus is + # configured with API_ACCESS_TOKEN_PERIOD = 10 years (see + # terminus.yaml). Against the 30-minute default it would be + # actively harmful: the app would believe a stale token had + # 30 fresh minutes left, skip its proactive refresh, and + # then be unable to recover, because refreshing requires a + # still-valid token. If that setting is ever reverted, this + # must become a real mint time carried alongside the tokens. jq --arg at "$BYOS_ACCESS_TOKEN" \ --arg rt "$BYOS_REFRESH_TOKEN" \ '(.[].webhook_format.byosConfig.auth.access_token) = $at diff --git a/kubernetes/apps/trmnl/terminus.yaml b/kubernetes/apps/trmnl/terminus.yaml index 339d7ee3..46846659 100644 --- a/kubernetes/apps/trmnl/terminus.yaml +++ b/kubernetes/apps/trmnl/terminus.yaml @@ -18,6 +18,34 @@ data: # pool (fd99::/120) and the LAN ULA — without it every IPv6 client, # including the device and Traefik, is rate-limited as if it were remote. RACK_ATTACK_ALLOWED_SUBNETS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1,::1,fc00::/7" + # 10 years, against a default of 1800s (30 minutes). + # + # This exists for the renderer, which pushes screens to /api/screens with + # a JWT held in OpenBao. At the default period that arrangement cannot + # work unattended, and it fails in a way that has no recovery: minting a + # replacement token via POST /api/jwt requires a still-VALID access token + # in the Authorization header, so once one expires it cannot be refreshed + # from itself. Observed exactly that on first deploy — + # [webhook] Webhook failed: 400 {"error":"expired JWT access token"} + # [webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."} + # — the refresh attempt being rejected by the very expiry it was trying + # to repair. + # + # The renderer's own code assumes "stock Terminus disables session + # expiration and issues effectively permanent tokens". That is simply not + # true of this deployment, so the assumption is made true here rather + # than worked around in the client. + # + # The trade is deliberate: a long-lived JWT with full Terminus API rights + # now sits in OpenBao. Accepted because the alternative was putting the + # Terminus account PASSWORD into the cluster so an init container could + # mint a short-lived token on every pod start, which is a strictly worse + # secret to hold. The token is revocable by rotating the account's + # credentials; it grants access only to this LAN-only instance. + # + # NOTE: the period is fixed at mint time. Changing this does not extend + # tokens that already exist — they must be re-minted afterwards. + API_ACCESS_TOKEN_PERIOD: "315360000" --- apiVersion: apps/v1 kind: Deployment -- 2.54.0 From 18c8b74baadc803b07d3384bc1f098f2665e73dd Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 12 Aug 2026 19:20:29 +0200 Subject: [PATCH 2/4] fix(trmnl): let Reloader roll terminus on ConfigMap changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review caught that this PR, as it stood, would not have fixed the outage it describes. terminus-config is consumed via envFrom, and env vars are read once at container start — Kubernetes does not hot-reload them. The ConfigMap is a plain resource rather than a configMapGenerator, so editing it does not change the pod template hash and Flux will not roll the pods either. Stakater Reloader is deployed but acts only on annotated workloads, and this Deployment carried no annotation. Net effect would have been a silent no-op: git and the ConfigMap object would show a 10-year token period while the running app kept issuing 30-minute tokens, and the re-mint step in the PR plan would have produced another short-lived token that looked correct and expired anyway. The manifest and reality would disagree with nothing to indicate it. Annotates for terminus-config specifically rather than the blanket reloader.stakater.com/auto: "true", matching the reasoning already recorded on mosquitto-deployment.yaml — enrolling anything further stays a visible, reviewable edit to that line. Verified Reloader is actually running (kube-system/reloader-reloader). This also removes a manual step rather than documenting one, which is the point: a fix that needs a human to remember `kubectl rollout restart` is not deployed, it is a runbook entry waiting to be missed. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/apps/trmnl/terminus.yaml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/kubernetes/apps/trmnl/terminus.yaml b/kubernetes/apps/trmnl/terminus.yaml index 46846659..a13e95ca 100644 --- a/kubernetes/apps/trmnl/terminus.yaml +++ b/kubernetes/apps/trmnl/terminus.yaml @@ -54,6 +54,20 @@ metadata: namespace: trmnl labels: app.kubernetes.io/name: terminus + annotations: + # terminus-config is consumed via envFrom, and env vars are read ONCE at + # container start — Kubernetes does not hot-reload them. The ConfigMap is + # a plain resource, not a configMapGenerator, so editing it does not + # change the pod template hash and Flux will not roll the pods either. + # Without this annotation a ConfigMap-only change (such as raising + # API_ACCESS_TOKEN_PERIOD) applies to git and to the ConfigMap object + # while the running app keeps the old value indefinitely — a silent + # no-op, with the manifest and reality disagreeing. + # + # Named per-ConfigMap rather than the blanket `reloader.stakater.com/ + # auto: "true"`, matching mosquitto-deployment.yaml: enrolling anything + # further stays a visible, reviewable edit to this line. + configmap.reloader.stakater.com/reload: terminus-config spec: replicas: 1 # RWO PVC shared by both containers in this pod, and APP_SETUP runs -- 2.54.0 From 8031a0ad7a3e6579de0ab18051063ecf807073a6 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 12 Aug 2026 19:25:32 +0200 Subject: [PATCH 3/4] docs(openbao): record the trmnl renderer tokens as deliberately unrotated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review's remaining point: with API_ACCESS_TOKEN_PERIOD at 10 years, nothing will ever expire the renderer's BYOS token or prompt anyone to rotate it. Left undocumented, "never rotated" is indistinguishable from "forgotten". Records it where it is discoverable — the rotation directory itself, whose purpose is that unrotated credentials are a visible choice rather than an omission. Notes why there is no rotator: minting a Terminus JWT requires the account password, which is precisely the secret this design declined to hold in-cluster, so a rotator would have to store the thing the long-lived token exists to avoid. Rotation is therefore manual, tied to the Terminus account password changing. Also flags that HA_ACCESS_TOKEN in the same OpenBao path is a different case: secret-rotator already has an `ha-token` mode (home-assistant.yaml uses it), so enrolling that one is a real follow-up rather than a limitation, and doing so would leave only the BYOS pair unrotated. Co-Authored-By: Claude Opus 5 (1M context) --- .../openbao/rotation/kustomization.yaml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/kubernetes/infrastructure/openbao/rotation/kustomization.yaml b/kubernetes/infrastructure/openbao/rotation/kustomization.yaml index dc26fe3f..5eaf696d 100644 --- a/kubernetes/infrastructure/openbao/rotation/kustomization.yaml +++ b/kubernetes/infrastructure/openbao/rotation/kustomization.yaml @@ -59,3 +59,22 @@ resources: - review-bot.yaml - claude-oauth.yaml - mosquitto-health.yaml +# NOT ROTATED, deliberately — recorded here because the whole point of this +# directory is that unrotated credentials should be a visible choice: +# +# homelab/apps/trmnl/renderer → BYOS_ACCESS_TOKEN / BYOS_REFRESH_TOKEN +# Terminus API JWTs used by the e-ink renderer to push screens. Terminus +# is configured with API_ACCESS_TOKEN_PERIOD = 10 years +# (kubernetes/apps/trmnl/terminus.yaml), so nothing will ever expire +# these or prompt a rotation. That is deliberate: the alternative was +# holding the Terminus account password in-cluster to mint short-lived +# tokens per pod start. There is no rotator because minting requires the +# account password, which is exactly what we declined to store. +# Rotate BY HAND whenever the Terminus account password changes, by +# re-running POST /login and re-seeding both keys. +# +# HA_ACCESS_TOKEN in the same path is a dedicated Home Assistant +# long-lived token. `ha-token` mode in secret-rotator already handles +# this class (see home-assistant.yaml) — enrolling it is a genuine +# follow-up, not a limitation, and would leave only the BYOS pair +# unrotated. -- 2.54.0 From 463e493ec024ccd3b5c987cb105d1f75a25fdb8c Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 12 Aug 2026 19:30:03 +0200 Subject: [PATCH 4/4] docs(trmnl): cite the source for API_ACCESS_TOKEN_PERIOD MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The setting was treated as an unverified assumption in review, and I repeated that framing. It was not one. Upstream's doc/configuration.adoc documents the variable and its 1800-second default in as many words, and this instance corroborated it empirically: a token minted here was rejected roughly thirty minutes later, which is the documented default behaving exactly as written. Documentation plus matching observed behaviour is not an assumption. The reviewer's uncertainty was reasonable — Terminus's source is not vendored in this repo, so from the checkout alone the claim is unsupported. The failing was mine for citing nothing at the point of use, which is the same gap as the /health probe earlier: the evidence existed and was not written where a reader could find it. Quotes the documentation inline so the question is answerable from the manifest. Note this does not make the post-merge check redundant. Whether Terminus honours the variable is settled here; whether this ConfigMap edit actually reached the running process via Reloader is a separate question, and that is what decoding the new token's exp claim answers. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/apps/trmnl/terminus.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kubernetes/apps/trmnl/terminus.yaml b/kubernetes/apps/trmnl/terminus.yaml index a13e95ca..5cc5d3bc 100644 --- a/kubernetes/apps/trmnl/terminus.yaml +++ b/kubernetes/apps/trmnl/terminus.yaml @@ -20,6 +20,13 @@ data: RACK_ATTACK_ALLOWED_SUBNETS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1,::1,fc00::/7" # 10 years, against a default of 1800s (30 minutes). # + # Documented upstream in doc/configuration.adoc: + # API_ACCESS_TOKEN_PERIOD — "The amount of time, in seconds, the JWT + # access token remains valid before needing to refresh your access + # token." Default: 1,800 seconds (30 minutes). + # Corroborated on this instance: a token minted here was rejected about + # thirty minutes later, i.e. the documented default behaving as written. + # # This exists for the renderer, which pushes screens to /api/screens with # a JWT held in OpenBao. At the default period that arrangement cannot # work unattended, and it fails in a way that has no recovery: minting a -- 2.54.0