Commit Graph
4354 Commits
Author SHA1 Message Date
k.renner 8573cdf220 Merge pull request 'fix(trmnl): let Reloader restart the renderer on config and secret changes' (#489) from fix-trmnl-renderer-reloader into master
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#489
2026-08-12 20:23:03 +00:00
claude-botandClaude Opus 5 507e86cb83 fix(trmnl): let Reloader restart the renderer on config and secret changes
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The renderer's init container writes /data/schedules.json ONCE, at pod
start, and nothing re-reads either source afterwards. So a ConfigMap or
Secret change lands in git, updates the object, and the running
renderer keeps the old values indefinitely.

The app re-reads schedules.json every 60 seconds, which makes it look
self-updating. It is not — it re-reads the FILE, and only the init
container writes the file.

This was hit three times in practice, each needing `kubectl delete pod`
by hand:

  * switching the theme to Graphite E-ink Light — the pod kept
    rendering with a theme name that no longer existed
  * setting the schedule's lang field to "de"
  * every re-mint of the BYOS tokens, which is the Secret half

The theme case is the instructive one: nothing errored. HA silently
served the default theme for a name that did not resolve, so the panel
just looked wrong, and I was about to debug the design.

Annotates both sources. Named per-resource rather than the blanket
reloader.stakater.com/auto: "true", matching terminus.yaml and
mosquitto-deployment.yaml.

Terminus got this annotation earlier; the renderer did not, despite
having the same init-container pattern and hitting the same failure in
the same session.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 22:22:04 +02:00
k.renner b2ed2beae4 Merge pull request 'fix(trmnl): render the panel in German via the schedule's lang field' (#488) from fix-trmnl-render-lang into master
Reviewed-on: k.renner/homelab#488
2026-08-12 20:18:58 +00:00
claude-botandClaude Opus 5 f8c968c5a7 fix(trmnl): render the panel in German via the schedule's lang field
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
lang was null, so the screenshot session rendered in English — and the
dashboard grew a German weekday array, a German month array and a
fifteen-entry weather-condition dict to translate around it by hand.

The field is documented as "UI language code (HA mode only)" and calls
Home Assistant's own _selectLanguage() on the page before capture;
upstream's tests use lang: 'de'. So localisation is HA's job and the
field was sitting in the schedule template, already written out, set to
null because I copied the shape without asking what it did.

Pairs with homeassistant#220, which deletes both hand-written mappings:
the weather badge points at weather.garten directly and the date is a
device_class: timestamp the frontend formats.

The instance stays language "en". This affects only the render session,
so the panel is German without changing anyone's UI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 22:16:43 +02:00
k.renner 595e5eb7a3 Merge pull request 'fix(trmnl): disable Terminus session expiry, the other half of the token fix' (#487) from fix-trmnl-session-expiry into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#487
2026-08-12 20:02:38 +00:00
claude-botandClaude Opus 5 7e293d3de7 fix(trmnl): disable Terminus session expiry, the other half of the token fix
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The renderer stopped pushing and the panel preview froze. Every webhook:

  HTTP 401: Unauthorized - This session has expired, please login again

Terminus has TWO independent expiries and the earlier fix only addressed
one. API_ACCESS_TOKEN_PERIOD governs the JWT's own lifetime and is now
10 years. SESSION_EXPIRATION_ENABLED / SESSION_INACTIVITY_LIMIT govern
the SESSION BEHIND that token and were left at their defaults, so a
valid, decade-long token stopped being accepted after about half an
hour of session inactivity.

Worth being precise, because the two look alike and are not: the first
failure was 400 "expired JWT access token", the token itself. This is
401 "This session has expired", the session. Different mechanism,
different error, same symptom — a panel that silently stops updating.

I noticed both expiries when diagnosing the first one, wrote that
Terminus "has two independent expiries", and then fixed one and moved
on. That is the actual mistake here, not the missing variable.

Disabling session expiry is the state the renderer's own code already
assumes ("stock Terminus disables session expiration"), which is false
by default and has to be made true here — the same shape as the token
period. And it is the deciding constraint for an unattended appliance:
a session that expires needs a human with a password to restore it, and
a hallway panel has neither.

Reloader will roll the pod; the ConfigMap annotation added earlier
covers this. Existing tokens must be re-minted afterwards — the current
session is already dead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 22:01:34 +02:00
k.renner 9423381102 Merge pull request 'fix(trmnl): point the renderer at Graphite E-ink Light' (#486) from fix-trmnl-graphite-theme into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#486
2026-08-12 19:36:24 +00:00
claude-botandClaude Opus 5 ae12ce760c fix(trmnl): point the renderer at Graphite E-ink Light
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The renderer's own docs recommend Graphite by name — "Use e-ink theme
(Graphite recommended)", with theme=Graphite in the example — and the
Home Assistant community converges on it independently. The dashboard
now ships graphite-eink-light.yaml (homeassistant repo) in place of the
theme I hand-wrote from a spec sheet.

Pairs with homeassistant#216. The theme name must match the key inside
the vendored file exactly ("Graphite E-ink Light"); a name that does not
resolve fails silently — HA serves the default theme and the panel just
looks wrong rather than erroring.

Note upstream's example also passes `dark`. Not used here: the e-ink
LIGHT variant is the one that gives black-on-white, which is what an
e-paper panel wants. `dark: false` stays.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 21:34:55 +02:00
k.renner 641e8bce84 Merge pull request 'fix(trmnl): make Terminus issue long-lived API tokens' (#485) from fix-trmnl-token-lifetime into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#485
2026-08-12 17:34:47 +00:00
claude-botandClaude Opus 5 463e493ec0 docs(trmnl): cite the source for API_ACCESS_TOKEN_PERIOD
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The setting was treated as an unverified assumption in review, and I
repeated that framing. It was not one.

Upstream's doc/configuration.adoc documents the variable and its
1800-second default in as many words, and this instance corroborated it
empirically: a token minted here was rejected roughly thirty minutes
later, which is the documented default behaving exactly as written.
Documentation plus matching observed behaviour is not an assumption.

The reviewer's uncertainty was reasonable — Terminus's source is not
vendored in this repo, so from the checkout alone the claim is
unsupported. The failing was mine for citing nothing at the point of
use, which is the same gap as the /health probe earlier: the evidence
existed and was not written where a reader could find it. Quotes the
documentation inline so the question is answerable from the manifest.

Note this does not make the post-merge check redundant. Whether Terminus
honours the variable is settled here; whether this ConfigMap edit
actually reached the running process via Reloader is a separate
question, and that is what decoding the new token's exp claim answers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 19:30:03 +02:00
claude-botandClaude Opus 5 8031a0ad7a docs(openbao): record the trmnl renderer tokens as deliberately unrotated
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review's remaining point: with API_ACCESS_TOKEN_PERIOD at 10 years,
nothing will ever expire the renderer's BYOS token or prompt anyone to
rotate it. Left undocumented, "never rotated" is indistinguishable from
"forgotten".

Records it where it is discoverable — the rotation directory itself,
whose purpose is that unrotated credentials are a visible choice rather
than an omission. Notes why there is no rotator: minting a Terminus JWT
requires the account password, which is precisely the secret this design
declined to hold in-cluster, so a rotator would have to store the thing
the long-lived token exists to avoid. Rotation is therefore manual, tied
to the Terminus account password changing.

Also flags that HA_ACCESS_TOKEN in the same OpenBao path is a different
case: secret-rotator already has an `ha-token` mode (home-assistant.yaml
uses it), so enrolling that one is a real follow-up rather than a
limitation, and doing so would leave only the BYOS pair unrotated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 19:25:32 +02:00
claude-botandClaude Opus 5 18c8b74baa fix(trmnl): let Reloader roll terminus on ConfigMap changes
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review caught that this PR, as it stood, would not have fixed the
outage it describes.

terminus-config is consumed via envFrom, and env vars are read once at
container start — Kubernetes does not hot-reload them. The ConfigMap is
a plain resource rather than a configMapGenerator, so editing it does
not change the pod template hash and Flux will not roll the pods
either. Stakater Reloader is deployed but acts only on annotated
workloads, and this Deployment carried no annotation.

Net effect would have been a silent no-op: git and the ConfigMap object
would show a 10-year token period while the running app kept issuing
30-minute tokens, and the re-mint step in the PR plan would have
produced another short-lived token that looked correct and expired
anyway. The manifest and reality would disagree with nothing to
indicate it.

Annotates for terminus-config specifically rather than the blanket
reloader.stakater.com/auto: "true", matching the reasoning already
recorded on mosquitto-deployment.yaml — enrolling anything further
stays a visible, reviewable edit to that line.

Verified Reloader is actually running (kube-system/reloader-reloader).

This also removes a manual step rather than documenting one, which is
the point: a fix that needs a human to remember `kubectl rollout
restart` is not deployed, it is a runbook entry waiting to be missed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 19:20:29 +02:00
claude-botandClaude Opus 5 e84ea02100 fix(trmnl): make Terminus issue long-lived API tokens
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The renderer renders correctly and pushes nothing. Every webhook is
rejected:

  [webhook] Webhook failed: 400 {"error":"expired JWT access token"}
  [webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."}

Terminus issues API JWTs with API_ACCESS_TOKEN_PERIOD, default 1800s.
Minting a replacement via POST /api/jwt requires a still-VALID access
token in the Authorization header, so an expired token cannot refresh
itself — the second line above is the refresh attempt being rejected by
the very expiry it was trying to repair. There is no recovery path once
the window closes.

That makes "mint once, store in OpenBao" unworkable for an unattended
appliance, which is what this stack is. The renderer's own code assumes
"stock Terminus disables session expiration and issues effectively
permanent tokens"; that is not true of this deployment. Rather than work
around the assumption in the client, this makes it true: sets
API_ACCESS_TOKEN_PERIOD to 10 years.

The trade is deliberate. A long-lived JWT with full Terminus API rights
now sits in OpenBao. The alternative was putting the Terminus account
PASSWORD into the cluster so an init container could mint a short-lived
token on every pod start — a strictly worse secret to hold, for a
LAN-only service whose token is revocable by rotating the account.

Also corrects the obtained_at comment on the renderer, which justified
stamping "now" by repeating that same false permanence assumption.
Stamping now is safe ONLY because of this setting; against the 30-minute
default it is actively harmful, because the app would believe a stale
token had a fresh window, skip its proactive refresh, and strand itself.
The comment now says so, and says what to do if this setting is reverted.

The period is fixed at mint time, so this does not revive the existing
tokens — they must be re-minted after it deploys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 19:15:43 +02:00
k.renner fc53334b31 Merge pull request 'fix(trmnl): give Chromium a writable HOME and stamp obtained_at' (#484) from fix-trmnl-renderer-runtime into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#484
2026-08-12 17:02:10 +00:00
claude-botandClaude Opus 5 2dd3e8f0b2 fix(trmnl): bound the renderer's /tmp with a sized emptyDir
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Follow-up to the HOME=/tmp fix in this PR, from review.

Pointing HOME at /tmp was correct but left it on the container's
writable layer, which was harmless while Chromium crashed on launch and
wrote nothing. With the browser actually starting once a minute for the
life of the pod, crashpad state and profile data now accumulate there
with nothing to bound them — resources.limits sets memory only, so
there is no ephemeral-storage ceiling to catch it either.

Backs it with an emptyDir at a 512Mi sizeLimit, matching the /tmp mount
convention already used in pve-exporter-deployment.yaml. The point of
the limit is the failure mode, not the size: unbounded it degrades into
node disk pressure, which surfaces far from this pod and hurts whatever
else lands on that node; bounded it is an eviction of the renderer,
attributable on sight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:53:55 +02:00
claude-botandClaude Opus 5 d824263759 fix(trmnl): give Chromium a writable HOME and stamp obtained_at
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Two runtime faults from the first deploy. Neither is visible statically
— both needed the pod running to find.

1. The browser never launched. Every render died with

     chrome_crashpad_handler: --database is required

   Chromium cannot start without a writable HOME: crashpad builds its
   database path from it. Diagnosed on the pod rather than guessed —
   HOME=/, uid=1000, / is root-owned 0755 and not writable, /tmp is.
   The cause is OUR securityContext, not the image: forcing
   runAsUser 1000 with no matching passwd entry makes HOME fall back
   to "/". Sets HOME=/tmp; crash dumps are throwaway, so they do not
   belong on the PVC.

   Worth being precise about, because review had flagged the dropped
   capabilities as the likely Chromium risk and the comment I added
   said they were fine. The capabilities were fine. Something adjacent
   to that securityContext broke it anyway, and the comment now says
   to check HOME before loosening the capability block.

2. Even once the browser works, nothing would have been pushed. The
   scheduler logged

     BYOS tokens ... expired beyond the refresh window

   and skipped the webhook entirely. It derives token age from
   obtained_at, and the template shipped a literal 0, so
   freshly-minted tokens read as dating from 1970. The init container
   now stamps it with jq's `now * 1000 | floor` at write time.
   Verified the filter produces a 13-digit epoch-ms value. Stock
   Terminus disables session expiry and issues effectively permanent
   tokens, so a write-time stamp is honest enough.

Failure mode of the pair was quiet in the way this stack keeps being
quiet: pod Running and Ready, /health passing, schedule firing once a
minute, zero screens reaching Terminus.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:48:38 +02:00
k.renner b0f0f9242b Merge pull request 'feat(trmnl): deploy the headless dashboard renderer' (#483) from feat-trmnl-renderer into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#483
2026-08-12 16:43:16 +00:00
claude-botandClaude Opus 5 a79a5417e3 docs(trmnl): record why the probe path and dropped caps are safe
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both were verified against the upstream image while writing this, but
the evidence never made it into the manifest, so review flagged them as
unverified assumptions. Fair — the reasoning was invisible.

/health is not a guess: the image's own Dockerfile declares
`HEALTHCHECK ... curl -f http://localhost:10000/health`, so the route is
part of its published contract. That is the difference from
terminus.yaml, which got TCP-only probes precisely because Terminus
publishes no health endpoint.

Dropping ALL capabilities is safe despite the container running headless
Chromium because the app launches it with --no-sandbox
--disable-setuid-sandbox --no-zygote --disable-dev-shm-usage, baked into
its own PUPPETEER_ARGS. That last flag is also why no /dev/shm emptyDir
is mounted. Both hold only for the pinned tag, so the comment says to
re-check on a bump.

No behaviour change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:40:51 +02:00
claude-botandClaude Opus 5 b6ada58b7a fix(trmnl): reach HA over the in-cluster Service, not the public name
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review caught a blocking bug, correctly. HOME_ASSISTANT_URL pointed at
http://ha.renner.uno:8123, which is unusable from inside a pod.

That hostname is served by the home-assistant-bgp LoadBalancer with
externalTrafficPolicy: Local, advertised to the LAN over BGP. It is
not backed by an Ingress, so CoreDNS's k8s_gateway delegation does not
intercept it — a pod resolving it gets the real external address
(10.2.99.87) and hairpins off the LB.

This cluster has hit precisely this before with precisely this
Service. kubernetes/infrastructure/openbao/rotation/home-assistant.yaml
reaches HA from a pod for token rotation and says so outright: "The
public name is unusable from a pod (ingress-LB hairpin)." The same
language recurs in the gitea and woodpecker network policies. I took
the hostname from the repo's own "internal http://ha.renner.uno:8123"
note, which is true for a workstation and false from a pod.

Now uses home-assistant-bgp.home-assistant.svc.cluster.local:8123,
verified to exist with ClusterIP 10.96.79.213 on 8123/TCP.

Fixing the URL alone was not enough. The home-assistant namespace's
default-deny-ingress had no rule admitting trmnl pods on 8123 — the
only 8123 rule is scoped to the openbao secret-rotator — so traffic
would still have been dropped. Adds a companion ingress rule modelled
on the rotator's, scoped to the renderer's pod label rather than the
whole trmnl namespace, since Terminus shares that namespace and has no
business reaching Home Assistant. Only the ingress half is needed:
trmnl has no default-deny egress selecting the renderer, unlike
openbao.

Worth naming the failure mode, because it is the fourth "fails
quietly" case in this stack: nothing crashes. The schedule fires every
minute, each render errors on connect, and the panel simply never
updates.

Also updates the trmnl-daily Velero comment, which enumerated PVCs and
so undersold itself once trmnl-renderer-data existed — the schedule
selects the namespace minus CNPG-labelled PVCs, so new PVCs are
already covered without editing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:34:53 +02:00
k.renner 0581bc334a Merge pull request 'chore(ansible): retire the monitoring plays for decommissioned hosts' (#479) from retire-dead-monitoring into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#479
2026-08-12 16:30:30 +00:00
claude-botandClaude Opus 5 f466f00029 fix(trmnl): model_id is the integer 1, not the name "v2"
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Resolves the one field flagged as unverified on the renderer PR, now
that a Terminus API token exists to check it with.

GET /api/screens on the running instance returns model_id as a NUMBER
(an existing welcome screen carries model_id 11), and GET /api/models
gives TRMNL X as id 1, name "v2", 1872x1404, 16 colours. Upstream's
example value "og_1bit" matches no model on this server at all — the
2-bit OG model here is id 11, name "og_plus" — so that example was
illustrative rather than a real identifier.

Would have surfaced as Terminus rejecting every pushed screen with a
422, i.e. a permanently blank panel with the renderer looking healthy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:27:57 +02:00
security-bot 3f4846bbbc chore(image-automation): bump ghcr.io/music-assistant/server:2.9.13
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-12 14:35:43 +00:00
claude-botandClaude Opus 5 7e2a439b9e feat(trmnl): deploy the headless dashboard renderer
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Phase 4. Screenshots the Home Assistant e-ink dashboard, dithers it to
16 greys and hands it to Terminus, which the panel then polls.

Upstream ships this as a Home Assistant add-on. Ours is a plain HA
container with no Supervisor, so it runs standalone — viable because
the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env vars over
/data/options.json and authenticates with an ordinary long-lived
token. SUPERVISOR_TOKEN is deliberately left unset: it only flips an
internal isAddOn flag and would switch the app to add-on behaviour.

The non-obvious part is the direction of travel. With the byos-hanami
webhook in `uri` delivery mode the renderer does NOT push image bytes
— it POSTs a URL pointing at its own /output/<file> endpoint and
TERMINUS FETCHES THE IMAGE BACK. That is why this lives in the trmnl
namespace: the fetch is intra-namespace and already permitted by
default-deny-ingress. Splitting the two across namespaces would have
needed a new policy for a dependency that is easy to miss.

Fully declarative, no clicks. Schedules are a JSON file at
/data/schedules.json which the app re-reads every 60s, so the whole
schedule — viewport 1872x1404, floyd-steinberg at bitDepth 4, the
trmnl-eink theme, the dashboard path — is reviewable in git rather
than typed into the app's web UI. An init container splices the BYOS
tokens into the ConfigMap template with jq; jq rather than sed because
the tokens are JWTs going into a JSON string and sed would corrupt on
any character it treats as special (verified with a token containing
/ + &).

No HTTP in the init container on purpose. The app exposes a
/api/byos/login proxy that trades a Terminus username+password for
these tokens, but using it would mean shipping the Terminus PASSWORD
into the cluster and making every pod start depend on a network call.
Stock Terminus disables session expiry and issues effectively
permanent tokens, so they are minted once out of band and kept in
OpenBao; the password never enters the cluster.

Three things that are load-bearing and would fail quietly otherwise:

  * The PVC at /data is REQUIRED. The app picks its data directory by
    testing whether /data exists — with nothing mounted it silently
    falls back to /app/data in the container, so schedules and rendered
    output vanish on restart and the panel goes blank.
  * The Service is ClusterIP with NO Ingress. The app's web UI on this
    port has no authentication whatsoever, per upstream, so it must not
    be routable from the LAN.
  * The memory limit is 2Gi, not the 512Mi upstream's troubleshooting
    note suggests. The app's own watchdog restarts it past 900MB but
    only measures its OWN rss — Chromium is a separate process, and it
    is rendering a 1872x1404 page.

The image is arch-suffixed (trmnl-ha-amd64); upstream publishes no
multi-arch manifest. Pinned and not enrolled in image automation,
since a bad bump here surfaces as a corrupted or blank panel rather
than a failed rollout — rationale recorded in the image-automation
README alongside Terminus's.

PREREQUISITE: homelab/apps/trmnl/renderer must be seeded in OpenBao
with HA_ACCESS_TOKEN, BYOS_ACCESS_TOKEN and BYOS_REFRESH_TOKEN before
this merges, or the ExternalSecret never syncs, the init container
blocks, and the trmnl Kustomization fails its wait.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 05:58:26 +02:00
security-bot dc1ea300a8 chore(image-automation): bump ghcr.io/music-assistant/server:2.9.12
ci/woodpecker/cron/security-scan Pipeline failed
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-11 00:35:18 +00:00
k.renner 73c2674344 Merge pull request 'fix(trmnl): request an explicit device LB address, document the sidekiq race' (#481) from fix-trmnl-lb-address into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#481
2026-08-10 21:08:53 +00:00
claude-botandClaude Opus 5 931810e28b fix(trmnl): request an explicit device LB address, document the sidekiq race
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Two things the first reconcile surfaced.

Cilium IPAM allocated 10.2.99.0 for the device LoadBalancer. Nothing
requested it — the Service carried no lbipam.cilium.io/ips annotation, so
IPAM handed out the first free address in the pool block, which is the base
address of the 10.2.99.0-10.2.99.255 range. Each LB address is advertised
to the UDM as its own BGP route, so it is routable in principle, but a .0
host address is exactly the sort of thing constrained embedded IP stacks
get wrong, and the TRMNL firmware is the one client that has to work here.
Requesting .89 explicitly also brings this in line with every other LB
Service in the repo, all of which pin their address, and matches the
fd99::89 IPAM had already chosen for v6.

Omitting the annotation was my call on the original PR, on the reasoning
that the device is provisioned with a hostname so the address could move
freely. That reasoning still holds; it just is not worth spending it on an
address nobody picked.

Second, the sidekiq container exits 1 once on a fresh deploy with
`PG::UndefinedTable: relation "account" does not exist` — it loses the race
against the web container's migration, then comes up healthy on the
restart. Observed on the first deploy, converged in about a second. There
is no upstream ordering hook, and gating it behind an initContainer that
probes for a named table would couple us to Terminus's schema, so this is
left to Kubernetes' restart deliberately. Documented in place so the next
person reading `RESTARTS 1` can tell the benign case from a real one.

Verified on the live cluster before this commit: Kustomization Ready, tr-pg
"Cluster in healthy state" on PostgreSQL 18, both ExternalSecrets
SecretSynced, terminus and valkey Running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 23:04:17 +02:00
security-bot 2d8324b3f9 chore(image-automation): bump valkey/valkey:9.1.1-alpine
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-10 21:00:08 +00:00
k.renner b529864453 Merge pull request 'feat(trmnl): deploy Terminus BYOS server for the entrance e-ink panel' (#480) from feat-trmnl-terminus into master
Reviewed-on: k.renner/homelab#480
2026-08-10 20:59:01 +00:00
claude-botandClaude Opus 5 ac2d728108 feat(trmnl): deploy Terminus BYOS server for the entrance e-ink panel
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Phase 1 of the TRMNL X dashboard plan: stand up the self-hosted Terminus
(BYOS) server the panel will poll. The Home Assistant dashboard, the e-ink
theme and the screenshot renderer are later phases and are not in here.

Phase 0 verification, all confirmed against upstream before writing any of
this:

  * TRMNL X IS a first-class model in Terminus (id v2 / label "TRMNL X",
    1872x1404, 16 colours, 4-bit) — the stop-and-report condition in the
    plan does not trigger. Note it is NOT in config/db/seeds.rb: the model
    list is populated by the MODEL_SYNCHRONIZER job pulling trmnl.com, so
    the Models list is empty until that job has run once.
  * The HA add-on can run standalone. Our HA is plain HA Core, not
    Supervised, but the add-on reads env vars in preference to
    /data/options.json and authenticates with an ordinary long-lived
    token — SUPERVISOR_TOKEN only flips an isAddOn boolean. So the
    fallback path in the plan (upstream container, not add-on) is viable.
    That deployment lands with Phase 4.

Deviations from the repo's house patterns, each deliberate:

  * tr-pg runs PostgreSQL 18, not the standard 16. Terminus ships a
    committed config/db/structure.sql containing `SET transaction_timeout
    = 0;`, a GUC that does not exist before PG 17, so schema load against
    a 16 server fails outright. This is the first non-16 CNPG cluster
    here. The existing 16 pin is inherited from 739a67e9 (Apr 2026) and
    has no documented rationale.
  * The device endpoint is cleartext HTTP on a Cilium BGP LoadBalancer
    (trmnl-device.renner.uno) rather than an Ingress. The firmware cannot
    be given the internal CA, and a TLS-less Ingress would be a new
    pattern for this repo — a dedicated LB address is not. The admin UI
    keeps a normal TLS Ingress at trmnl.renner.uno; both front the same
    Service. Same split as home.renner.uno / ha.renner.uno.
  * ghcr.io/usetrmnl/terminus is NOT enrolled in image automation despite
    publishing a usable tag stream. It serves the firmware and
    /api/display contract to a wall-mounted device, upstream is pre-1.0
    with no releases or changelog, and an unattended bump that breaks
    either leaves the panel dark. Bumps are manual and reviewed. The
    Valkey sidecar is enrolled — it touches no device-facing contract.
  * Egress is an FQDN allowlist (trmnl.com, plus the S3 host that serves
    firmware binaries), not toEntities: world. The S3 host is returned by
    the API at request time rather than being a source constant, so it can
    drift; the policy comment says where to look when firmware sync
    starts failing. This also means Terminus's Extensions/Poll feature,
    which fetches admin-configured URLs, will not work — accepted, since
    screen content is pushed in by the renderer.
  * The CNPG pods are deliberately left OUT of that egress policy so the
    barman WAL archive path is not behind a rule that fails closed.

Images verified to exist in their registries before pinning:
ghcr.io/usetrmnl/terminus:0.68.0, valkey/valkey:9.1-alpine,
ghcr.io/cloudnative-pg/postgresql:18.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 22:54:38 +02:00
claude-botandClaude Opus 5 8e5b91e6d6 chore(ansible): retire the monitoring plays for decommissioned hosts
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Prometheus and Loki run in Kubernetes -- statefulsets
prometheus-kube-prometheus-stack-prometheus and the Loki release, both up 93
days. The standalone VMs these plays targeted are gone: neither
prometheus.srv.renner.uno nor loki.srv.renner.uno resolves at all, and the
proxmox_logging group install-log-aggregation.yml targets is not in the
inventory.

So these rendered scrape configs, exporter installs and log shippers for
hosts that do not exist. Removed:

  plays/install-prometheus.yml          -> prometheus.srv.renner.uno (gone)
  plays/install-blackbox-exporter.yml   -> prometheus.srv.renner.uno (gone)
  plays/install-loki.yml                -> loki.srv.renner.uno (gone)
  plays/install-log-aggregation.yml     -> proxmox_logging (no such group)
  roles/prometheus-server, snmp-exporter, pve-exporter, unifi-poller,
  roles/blackbox-exporter, loki-server, promtail

Each role was consumed only by the plays above; promtail was referenced by
nothing at all. Verified no dangling references remain.

plays/update.yml imported install-log-aggregation.yml, so deleting it alone
would have broken the fleet update play. That import is removed and
update.yml re-syntax-checked.

This also retires #473, which changed IPv4 literals to FQDNs in
prometheus.yml.j2 and nodes.yml.j2. That was a change to a template nothing
renders. Its PR body described the instance-label consequences for
dashboards and alert rules, which was fiction -- no dashboard reads those
labels. When it would not apply I reported it as blocked on Proxmox API
credentials; the actual reason was that the target host had not existed for
months, and `--list-hosts` returning 0 was the evidence I misread.

Worth checking separately, not assumed here: whether anything scrapes the
PVE node_exporter and chrony_exporter today. install-node-exporter-pve.yml
still installs them on pve_cluster, but the config that pointed a Prometheus
at them was in the templates just deleted. If that was the only reference,
those metrics have been dark since the VM went away -- and the FQDN change
in #473 would have masked the question rather than raising it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 22:10:50 +02:00
k.renner c8bc6380a1 Merge pull request 'fix(heating-can): give supervisord a control socket' (#478) from heating-can-supervisorctl into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#478
2026-08-10 19:53:51 +00:00
claude-botandClaude Opus 5 e3dea169b0 fix(heating-can): give supervisord a control socket
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The image shipped without [unix_http_server], [rpcinterface:supervisor] or
[supervisorctl], so supervisorctl could not connect:

  Error: .ini file does not include supervisorctl section

That means no way to query or restart an individual service inside the
container -- docker logs was the only introspection. Per-service lifecycle
management was the whole argument for running a supervisor instead of one
process per container, and the first revision did not actually deliver it.

Unix socket only, inside the container. Nothing outside needs it, and this
container runs with host networking, so a TCP listener would be exposed on
the host.

The build-time self-test now asserts the section exists and supervisorctl
runs, so this cannot regress silently the way it shipped.

NOT fixed here, because my diagnosis was wrong: the repeated
"mqtt broker disconnected. reason_code = Unspecified error" is not a client
ID collision. E3onCAN has no -mcid option at all -- verified against the
pinned source -- and generates its own ID; the native broker log showed
E3onCANclient.py-e380-7f847281 alongside open3e-edge, so the two never
collided. I nearly "fixed" it by passing a flag that does not exist.

Telemetry is flowing (each reconnect succeeds), so this is churn rather than
an outage, but the cause is still unidentified. The next step is the broker's
own log on edge-1 -- mosquitto records the disconnect reason, which the
client side reports only as "Unspecified error".

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:49:00 +02:00
k.renner 6384223415 Merge pull request 'fix(heating-can): stop the native services before removing their units' (#477) from fix-native-teardown into master
Reviewed-on: k.renner/homelab#477
2026-08-10 19:48:34 +00:00
claude-botandClaude Opus 5 1b7e3dc100 fix(heating-can): stop the native services before removing their units
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The migration did not stop anything. It ran stop + disable + mask in one
call with failed_when: false, then deleted the unit files. Masking a unit
whose file still exists conflicts, so the call failed, failed_when hid the
failure, nothing was stopped -- and the next task deleted the unit files out
from under two running processes.

That left systemctl is-enabled reporting "not-found", which looks clean,
while is-active still said "active". I reported the cutover successful on
the strength of the former.

On 2026-08-10 the result was two open3e instances issuing UDS requests on
the same CAN bus from 19:28 to 19:37, taking over each other's
open3e-edge MQTT session -- the exact failure the block's own comment said
it existed to prevent.

Fixed in three parts:

  - Stop and disable FIRST, and do not swallow the result. Only units that
    actually exist are touched (stat first), so a fresh host does not need
    an error hidden to get through.

  - No masking. It writes a symlink at the same path the next task removes,
    so the two cancel out. With the unit files gone there is nothing to
    start.

  - Assert the invariant that actually matters -- no native process on the
    CAN bus -- rather than inferring it from systemd reporting success.
    pgrep, then fail loudly with the surviving command lines and why it
    matters.

Verified against the live host: ok=15 changed=1 failed=0, the assert passing
with "No native CAN processes remain; safe to start the container", and the
run now idempotent (the single change is the compose up, not the teardown).

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:45:47 +02:00
k.renner 1ee4c2b61c Merge pull request 'refactor(edge): split the plays by capability; pin open3e and register it with the audit' (#475) from edge-capability-groups into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#475
2026-08-10 19:26:11 +00:00
claude-botandClaude Opus 5 33d92e133c fix(audit): track the heating-can image pins, not the deleted Ansible vars
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#476 merged first, which removed edge_open3e_pkg, edge_e3oncan_repo and
edge_e3oncan_version in favour of the image. The audit entry added here still
told the agent to read those, so it would have gone looking for variables
that no longer exist and reported nothing -- the same silent non-auditing the
entry was written to prevent.

Repointed at where the pins actually live now: OPEN3E_REF and E3ONCAN_REF as
ARGs in ci/images/heating-can/Containerfile. The awkward specifics carry over
unchanged, since they are properties of the upstreams rather than of how we
package them: open3e publishes no tags so the comparison must be against
commits, and E3onCAN's pin is an untagged commit later than its newest tag.

Added a note the previous version could not have made: heating_can_image is
:latest, so the node follows whatever CI last built. The Containerfile ARGs
are what actually fix the versions, and bumping one is what requires a
rebuild -- otherwise a reader could reasonably conclude the tag is the pin.

The pin commit itself is now a no-op against master and survives only as the
audit registration; its conflict on heating-can/defaults resolved in favour of
the image.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:17:42 +02:00
claude-botandClaude Opus 5 150a83ee38 fix(edge): make the capability-group invariant structural, and unstale a note
Both from review on #475.

The rule "a host in a capability group must also be in [edge_host]" was
enforced by a comment, in a file that already uses :children a few lines
below for ntp_cluster. The three capability groups are now children of
edge_host, so the relation is structural: a host added only to [heating_can]
still resolves group_vars/edge_host, where the local MQTT contract lives, and
cannot silently come up without edge_mqtt_host and friends.

That matters more than it looks. The MQTT contract only moved to group_vars
in this PR because splitting the plays exposed that the roles could not
render without each other's defaults -- so an edge host outside [edge_host]
would hit exactly the failure that change was made to fix.

Verified all four groups still resolve to edge-1, and the redundant
ansible_user on the capability entries is dropped: it is inherited from the
[edge_host] declaration rather than repeated three times.

Also unstaled a Day-2 note that still read "the pip task tracks the git
default branch (pin edge_open3e_pkg to a tag if that ever bites)". Both
services are pinned to commits now, deliberately, and pinning to a tag is not
available for open3e -- it publishes none. Replaced with the actual workflow:
the weekly update-audit reports drift, bump the pin and re-run.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:16:37 +02:00
claude-botandClaude Opus 5 dd6553ab07 fix(heating-can): pin open3e, and register both CAN services with the audit
open3e was installed from an unpinned git URL, so the heating bridge
silently tracked upstream main: whatever HEAD happened to be on the day a
converge ran was what ended up driving the heat pump. Now pinned to
6553c104, which was simultaneously upstream HEAD and the copy already
installed on edge-1, so the pin froze the running state and changed nothing.

A commit rather than a tag because open3e publishes none -- its tags API
returns an empty array. The package self-reports 0.7.5 at that commit, which
is useful context but is not a tag.

Honest about what this does NOT fix: pip still re-clones and rebuilds a VCS
requirement even when the SHA is pinned, because it cannot compare an
installed VCS dist against a requested URL without fetching it first.
Verified directly on the host:

  pip install --dry-run "git+...@6553c104"
    -> Cloning ... Resolved ... to commit 6553c104
       Installing build dependencies: started

So the task still reports changed and still restarts open3e on every
converge. Pinning buys deliberate versioning, not idempotence.

Pinning without an update path only converts churn into silent rot, so both
services are now registered with the weekly update-audit -- which had been
auditing neither, despite E3onCAN having been pinned since July. The audit
prompt warns that anything not listed is silently not audited, and these two
were exactly that.

The new scope entry records the awkward specifics so the agent does not
report normal states as problems: open3e has no tags at all, so it must
compare commits against the default branch; E3onCAN does tag, but sparsely,
and its pin sits between the newest tag and HEAD.

Also corrected while there: edge_e3oncan_version was commented "# V0.5.2",
which does not exist upstream. The repo's only tag is v0.5.1 (6f43b3fb), and
461dcbdf is a later untagged commit from 2026-06-21, "Fixed: MQTT client ID
collision caused stale retained values; added publish error logging".

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:16:37 +02:00
claude-botandClaude Opus 5 0b845e7c76 refactor(edge): split the plays by capability, not just the roles
Splitting edge-host into single-concern roles left the coupling one layer
up: install-edge-host.yml was still one play applying all eleven roles to
one group. A host in [edge_host] got the Z-Wave gateway and the CAN services
whether or not it had the hardware -- the monolith again, in the play.

Inventory groups now state which hardware a host physically has, and each
playbook targets the group for its own concern:

  [edge_host]       install-edge-host.yml       base: flash-wear, log
                                                shipper, MQTT bridge, VLAN,
                                                docker
  [heating_can]     install-heating-can.yml     Innomaker USB2CAN
  [zwave_gateway]   install-zwave-gateway.yml   Nabu Casa ZWA-2
  [power_meter]     install-power-meter.yml     FTDI meter head

No flags and no `when:` on the roles. Moving a stick to another box is a
one-line inventory change, which is what it should be. A host with nothing
attached is simply absent from the capability groups -- which is what lets a
second thin client join as an NTP peer without being handed peripherals it
does not have.

The split exposed a circular dependency the single play had been hiding: the
local MQTT contract was scattered across two roles' defaults, in both
directions. mqtt-bridge's config needs edge_open3e_mqtt_user, because the
bridge authenticates to the cluster broker AS that user; heating-can and
power-meter-sml need edge_mqtt_host, because that is where they publish.
Neither role could render without the other's defaults, and it only ever
worked because one play loaded every role at once.

Those four values are the interface between the broker and its clients, not
the property of either side, so they move to group_vars/edge_host -- where
the matching secrets (edge_open3e_mqtt_password, edge_sml2mqtt_mqtt_password)
already lived. Removed from the role defaults so there is one source of
truth. Each role keeps what is genuinely its own: mqtt-bridge the cluster
endpoint and CA path, heating-can the datapoints and poll interval.

Also dropped: the legacy shared-compose teardown pre_tasks. That migration
completed on the 2026-08-08 apply (verified: /opt/edge-host/docker-compose.yml
is gone), and carrying migration cruft in a base play would be the same
mixing this commit is undoing.

Verified by --check --diff against the live host, all four plays:

  install-edge-host      ok=33 changed=0 failed=0
  install-zwave-gateway  ok=8  changed=0 failed=0
  install-power-meter    ok=4  changed=0 failed=0
  install-heating-can    ok=18 changed=4 failed=0

Three of four are fully idempotent. heating-can's four are the known open3e
pip churn plus the E3onCAN venv, which reports changed only under check mode
(it was ok in the real apply) -- and their two restart handlers.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 21:16:17 +02:00
k.renner 685905b1df Merge pull request 'build(heating-can): image for the CAN readers, replacing pip-into-venv' (#476) from heating-can-image into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#476
2026-08-10 19:04:49 +00:00
claude-botandClaude Opus 5 f9848d2bf5 fix(heating-can): correct the overlay path, secret mode and writable CWD
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #476 found a real regression plus three things worth fixing before
this reaches edge-1.

Blocking: edge_open3e_devices.vx3.dpList still pointed at the old host path
/opt/edge-host/open3e/vx3plus.py. The compose file mounts that overlay at
/config/vx3plus.py and nothing mounts anything under /opt/edge-host inside
the container, so open3e would have looked for it at a path that does not
exist -- silently falling back to the stock vx3 datapoint list and dropping
the battery-module DIDs (vx3.1577-1582 and friends) the overlay exists to
add. Now /config/vx3plus.py.

Worth noting the build-time self-test could not have caught this: it proves
the binaries execute, not that devices.json resolves against the mounted
files at runtime.

Writable working directory. WORKDIR was /opt, which the COPY layers create
root-owned, while the process runs as open3e. udsoncan writes its log file
into the CWD -- the native install gave open3e's directory `owner: open3e`
for exactly that reason, and I dropped that when containerising. Now a
dedicated /var/lib/open3e, owned by the service user.

Secret mode. MQTT_PASSWORD was inline in docker-compose.yml, which is
rendered 0644. The env file this replaced was 0600, and the sibling roles on
this host keep secrets in a 0600 file beside a world-readable compose file.
Restored: credentials move to heating-can.env at 0600, referenced via
env_file.

edge_e3oncan_mqtt_format had become dead -- both services were fed
MQTT_FORMAT from edge_open3e_mqtt_format. The two default to the same string
so behaviour was unchanged, but overriding it did nothing. Wired through as
E3ONCAN_MQTT_FORMAT, which supervisord passes to e3oncan only.

Also from review: dropped the registered-but-unread heating_can_native_stopped,
and the migration now removes the emptied {{ edge_base_dir }}/open3e and
/e3oncan directories rather than leaving them behind.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 20:58:09 +02:00
security-bot 2e62216bc7 chore(image-automation): bump prompve/prometheus-pve-exporter:3.10.0
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-10 06:35:03 +00:00
claude-botandClaude Opus 5 fac46aa7ac fix(heating-can): self-test at build time instead of a CI step
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The smoke-test step I added errored the whole pipeline before any step ran.
It used the freshly built image as its own step image, but the Gitea registry
is private: the build steps authenticate kaniko for PUSH, nothing gives the
runner credentials to PULL. So the step could never have worked.

Moved into the Containerfile as a build-time RUN, which is better anyway --
it needs no registry credentials, and a broken image is never published
rather than published and then flagged.

It asserts what the previous revision got wrong: both services actually
execute, and supervisord's config carries both programs. That is exactly the
failure that shipped green -- supervisord pointed at
/opt/e3oncan/src/E3onCANcollect.py, which does not exist (the file is at the
repo root), so the image would have run open3e while e3oncan crash-looped
and the E380 meter went silently missing.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:13:59 +02:00
claude-botandClaude Opus 5 6fb760edae fix(heating-can): correct the e3oncan path, and smoke-test the image in CI
The image shipped a path that does not exist. supervisord ran
`python /opt/e3oncan/src/E3onCANcollect.py`, but that file is at the repo
ROOT -- there is no src/ directory. The `src` came from the old Ansible task,
which cloned into `{{ edge_base_dir }}/e3oncan/src`; that was the clone
DESTINATION, not part of the upstream layout, and I carried it into the
Containerfile without checking. Confirmed against the upstream tree at the
pinned commit: E3onCANcollect.py sits at the root alongside requirements.txt,
which is why the build succeeded while the runtime path was wrong.

So the image built green and would have started open3e and failed e3oncan on
every restart -- the E380 meter silently absent, with supervisord retrying
forever.

Nothing caught that, because nothing ran the image. Now CI does: a smoke-test
step uses the freshly built image as its own step image and asserts both
services execute and supervisord's config carries both programs.

It cannot exercise the CAN bus or the broker, neither of which exists on a
runner, so it verifies exactly what is verifiable without hardware. That is
still the difference between "built" and "works" -- and it belongs on a
throwaway runner, not on the production edge host, which is where I wrongly
went looking for this the first time.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:12:03 +02:00
claude-botandClaude Opus 5 96fcb426aa build(heating-can): one container, both readers supervised inside it
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Reverts my own reinterpretation. The instruction was to put both in one
container because they belong together; I argued against it on crash-isolation
grounds, then read the correction "two services, one image" as agreement with
my version rather than as a restatement of the original. It was not.

A container runs one PID 1. With two processes that PID 1 is a supervisor,
which is how a container handles multiple services. supervisord now runs both
readers: it restarts either on its own if it dies, reaps children and forwards
shutdown -- none of which a bare process as PID 1 can do for a sibling, which
was the actual answer to the concern I raised.

Also gone: the dispatch entrypoint and the argv in the compose file. Both were
the same mistake in different places -- the invocation living outside the
artifact. open3e's and E3onCAN's command lines are an implementation detail of
this image now; supervisord builds them from environment variables it expands
itself. Nothing outside the image knows either CLI, so an upstream flag rename
cannot break the compose file or an Ansible template.

The compose file supplies configuration and nothing else: one service, one
container, environment only. The credential env file is dropped -- the
credentials are environment already.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:06:43 +02:00
claude-botandClaude Opus 5 56a9859e50 refactor(heating-can): run the image, install nothing on the node
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Nothing is installed on the host any more. The image is built in CI with
both sources pinned, published to the registry, and edge-1 pulls and runs
it. That is the whole point of building it -- installation happens at build
time, once, not on every converge.

Removed from the node: both venvs, the E3onCAN checkout, and the two systemd
units. Kept and still rendered: devices.json, the vx3 datapoint overlay and
the broker credentials, now bind-mounted into the containers rather than
read from a venv working directory.

Two services from one image, as before: open3e drives the Vitocal/VX3 over
UDS, e3oncan listens passively for the E380 meter. Separate containers so
either can crash and restart without a supervisor as PID 1.

The migration stops AND masks the old units before the containers start.
Masked rather than disabled because two open3e instances issuing UDS
requests on one CAN bus is the failure mode to avoid, and a stray
`systemctl start` or leftover dependency should not be able to bring the
native one back alongside the container.

No missing-device tolerance here, unlike the zwave and meter roles: can0 is
this host's purpose. If it is absent the heating telemetry is down and the
run should say so rather than pass quietly.

heating_can_image replaces edge_open3e_pkg / edge_e3oncan_repo /
edge_e3oncan_version. The source pins move into the Containerfile, so what
is deployed here is a tag -- and `docker compose up -d` on an unchanged tag
is a genuine no-op, which is what removes the restart-per-converge that
pinning the pip URL could not.

Verification is limited and worth stating: plays/install-heating-can.yml
lives in #475 and is not on this branch, so only the master-side play
syntax-checks here. A --check run cannot exercise this at all -- command
tasks are skipped under check mode, so neither the pull nor the compose up
would run. It cannot be meaningfully verified until the image has been built
and #475 is merged.

Depends on #475 (the capability-split playbooks, and the MQTT contract
moving to group_vars). This branch is cut from master, so heating-can's
defaults will conflict there; #475 should merge first and this be rebased
onto it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 09:59:45 +02:00
claude-botandClaude Opus 5 4df1018e05 build(heating-can): make both services first-class commands in the image
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The image had no ENTRYPOINT and no CMD, with a comment claiming that was
deliberate because the commands are rendered by Ansible. That was
rationalising a defect.

An image with nothing runnable is not a self-contained artifact. The
knowledge of how to START each service leaked into Ansible, and the two were
not even symmetric: open3e installs a console script via pip, while E3onCAN
is an unpackaged script in a checkout, so the compose file would have had to
carry `python /opt/e3oncan/src/E3onCANcollect.py` -- an implementation detail
of this image's internal layout, in a file that has no business knowing it.
Move the checkout and every caller breaks.

E3onCAN now gets a wrapper on PATH, so both services are invoked the same
way and `docker run <image> e3oncan --help` works.

CMD prints what the image provides and exits 64 (EX_USAGE) rather than
silently doing nothing, so `docker run <image>` is self-describing. Which
service to run, and its arguments, stays the caller's choice -- the compose
file renders those, because the datapoint list and broker settings live
there.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 09:54:18 +02:00
claude-botandClaude Opus 5 0e4d516722 build(heating-can): image for the CAN readers, replacing pip-into-venv
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Builds one image containing open3e (Vitocal heat pump, Vitocharge VX3) and
E3onCAN (E380 grid energy meter), both pinned in the Containerfile.

Why: both were pip-installed into venvs on edge-1, open3e straight from an
unpinned git URL. pip cannot compare an installed VCS dist against a
requested one without fetching it, so `pip install` re-cloned and rebuilt on
every Ansible converge -- reporting changed, firing the restart handler, and
putting a gap in heating telemetry each time. Pinning the SHA does not fix
that; verified directly on the host that pip still clones and rebuilds a
pinned VCS requirement. A tagged image does: `docker compose up -d` on an
unchanged tag is a genuine no-op.

It also gives the weekly update-audit a normal artifact instead of two
awkward special cases. open3e publishes no tags at all, and E3onCAN is
pinned to an untagged commit sitting between its newest tag and HEAD -- both
of which the audit currently has to be told about in prose.

One image, two services. They read the same physical installation over the
same bus and are versioned together, so they are one artifact; but they will
run as two containers from it, so either can crash and restart on its own
without a supervisor as PID 1.

Fixes a long-standing blind spot in passing: e3oncan.service set
PYTHONUNBUFFERED=1 and open3e.service did not, so open3e's stdout was
block-buffered and effectively never reached the journal or Loki. The image
sets it for both.

Build follows the existing kaniko pattern exactly -- path-scoped `when:` so
only this image rebuilds, date tag plus latest, pushed to the Gitea registry
alongside the ci-* images.

Deliberately artifact-only. The heating-can role still deploys venvs and
systemd units; moving it to a compose project is a separate change, once
this image exists and has been pulled and run once. Rewriting the role that
drives the heat pump against an image that has never been built would be the
wrong order.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 08:52:53 +02:00
k.renner 603db4ed95 Merge pull request 'fix(edge-host): make the bootstrap self-contained and the runbook followable' (#474) from edge-bootstrap-selfcontained into master
ci/woodpecker/cron/update-all Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/update-audit Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#474
2026-08-08 10:39:47 +00:00
claude-botandClaude Opus 5 6cfa63f6f3 fix(edge): keep the docker log-driver config out of the shared role
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #474 caught that the previous commit put the daemon.json task in
docker-install, which is shared by ten plays: install-netbird-server,
install-netbird-proxy, install-n8n, install-authentik-ha, install-owncloud,
install-gatus, install-ical-proxy, install-grist and
install-music-assistant.

None of those hosts has an /etc/docker/daemon.json today -- the template and
the default were both new here -- so the next routine run of any of those
plays would have created one, reported changed, fired the Restart docker
handler, and bounced every container on that host. Unannounced restarts
across authentik, n8n, owncloud, grist, netbird, gatus, music-assistant and
ical-proxy, from a change about edge-host bootstrapping.

The --check verification could not have caught it: it only ever exercised
install-edge-host.yml, and the damage was in the other nine plays.

The comment I wrote there was the tell. It said "the services that use this
host run with network_mode: host" -- true of edge-1, false of most consumers
of that role. A justification that only held for the one host I was looking
at, written into a shared file.

Moved to edge-flash-wear, which is edge-host-only and already owns this
rationale: routing container stdout to the journal is flash-wear handling,
not generic docker setup. The networking aside is dropped -- in a flash-wear
role the only thing worth saying about daemon.json is why logging goes to
the journal.

docker-install is now byte-identical to master (git diff origin/master --
roles/docker-install/ is empty).

Two details the move needed: /etc/docker is created explicitly, because
edge-flash-wear runs before docker-install on a first bootstrap and the
directory will not exist yet; and the Restart docker handler fires at
end-of-play, by which time docker is installed.

Verified: ok=67 changed=25. The changed count is unchanged from before the
move, and no docker task appears in it -- daemon.json renders byte-identical
to what the old role produced, so it reports ok and the handler never fires
on edge-1.

Also from review: noted on the play that heating-can, zwave-gateway and
power-meter-sml each declare edge_base_dir in their own defaults so they
stay usable standalone, that play vars win, and that the two must be kept in
step.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 12:27:13 +02:00