fix(trmnl): request an explicit device LB address, document the sidekiq race

Two things the first reconcile surfaced.

Cilium IPAM allocated 10.2.99.0 for the device LoadBalancer. Nothing
requested it — the Service carried no lbipam.cilium.io/ips annotation, so
IPAM handed out the first free address in the pool block, which is the base
address of the 10.2.99.0-10.2.99.255 range. Each LB address is advertised
to the UDM as its own BGP route, so it is routable in principle, but a .0
host address is exactly the sort of thing constrained embedded IP stacks
get wrong, and the TRMNL firmware is the one client that has to work here.
Requesting .89 explicitly also brings this in line with every other LB
Service in the repo, all of which pin their address, and matches the
fd99::89 IPAM had already chosen for v6.

Omitting the annotation was my call on the original PR, on the reasoning
that the device is provisioned with a hostname so the address could move
freely. That reasoning still holds; it just is not worth spending it on an
address nobody picked.

Second, the sidekiq container exits 1 once on a fresh deploy with
`PG::UndefinedTable: relation "account" does not exist` — it loses the race
against the web container's migration, then comes up healthy on the
restart. Observed on the first deploy, converged in about a second. There
is no upstream ordering hook, and gating it behind an initContainer that
probes for a named table would couple us to Terminus's schema, so this is
left to Kubernetes' restart deliberately. Documented in place so the next
person reading `RESTARTS 1` can tell the benign case from a real one.

Verified on the live cluster before this commit: Kustomization Ready, tr-pg
"Cluster in healthy state" on PostgreSQL 18, both ExternalSecrets
SecretSynced, terminus and valkey Running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci/images/heating-can/Containerfile

@@ -103,6 +103,8 @@ RUN open3e --help > /dev/null \
  && supervisord --help > /dev/null \
  && grep -q '^\[program:open3e\]' /etc/supervisord.conf \
  && grep -q '^\[program:e3oncan\]' /etc/supervisord.conf \
+ && grep -q '^\[supervisorctl\]' /etc/supervisord.conf \
+ && supervisorctl --help > /dev/null \
  && echo "self-test OK: both services execute; supervisord has both programs"
 
 # CAN_RAW needs no capabilities, so nothing here runs as root.

ci/images/heating-can/supervisord.conf

@@ -10,6 +10,24 @@
 # expands %(ENV_NAME)s at start, so neither the compose file nor a wrapper
 # script constructs open3e's or E3onCAN's argv -- those CLIs stay an
 # implementation detail of this image.
+# Control socket. Without these two sections supervisorctl cannot connect, so
+# there is no way to query or restart an individual service inside the
+# container -- which is the entire reason for running a supervisor rather than
+# one process per container. The first revision shipped without them.
+#
+# Unix socket only, inside the container: nothing outside needs to reach it,
+# and the container runs with host networking, so a TCP listener would be
+# exposed on the host.
+[unix_http_server]
+file=/tmp/supervisor.sock
+chmod=0700
+
+[rpcinterface:supervisor]
+supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
+
+[supervisorctl]
+serverurl=unix:///tmp/supervisor.sock
+
 [supervisord]
 nodaemon=true
 user=open3e

kubernetes/apps/gatus/configmap.yaml

@@ -160,6 +160,24 @@ data:
         conditions:
           - "[STATUS] == any(200, 301, 302)"
 
+      # Terminus admin UI. Checks the TLS route only — the cleartext
+      # device endpoint (trmnl-device.renner.uno) is the same pod, so a
+      # failure here covers both. Terminus exposes no health endpoint, so
+      # this is a root-page check: the login page answers 200, and an
+      # unauthenticated request may redirect.
+      - name: Web Service 10
+        group: services
+        url: https://trmnl.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
       # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
       # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
       # no longer deployed. Re-add when/if they come back.

kubernetes/apps/trmnl/cluster.yaml

@@ -0,0 +1,41 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: tr-pg
+  namespace: trmnl
+spec:
+  instances: 1
+  smartShutdownTimeout: 10
+  stopDelay: 60
+  # PG 18, NOT the repo-standard 16. Terminus ships a committed
+  # config/db/structure.sql containing `SET transaction_timeout = 0;` — a GUC
+  # that does not exist before PostgreSQL 17, so schema load against a 16
+  # server dies with `unrecognized configuration parameter`. Upstream's
+  # compose.yml develops against 18.4, so 18 is the tested target; 17 is the
+  # bare minimum. Bare major tag matches the other clusters (minor security
+  # bumps flow automatically, major upgrades stay an explicit decision).
+  imageName: ghcr.io/cloudnative-pg/postgresql:18
+  enableSuperuserAccess: true
+  storage:
+    storageClass: ceph-block
+    size: 5Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: tr-pg-store
+  bootstrap:
+    initdb:
+      database: terminus
+      owner: terminus
+  postgresql:
+    parameters:
+      shared_buffers: "128MB"
+      max_connections: "50"
+      wal_compression: "on"
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      memory: 512Mi

kubernetes/apps/trmnl/device-service.yaml

@@ -0,0 +1,48 @@
+# Device-facing endpoint. Cleartext HTTP, deliberately.
+#
+# The TRMNL firmware initiates every request and cannot be handed the
+# internal CA, so terminating TLS here would simply break polling. Scope of
+# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
+# record), it carries no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
+# UI stays on the TLS route at trmnl.renner.uno.
+#
+# Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
+# pool rather than a TLS-less Ingress, which would be a new pattern for this
+# repo. The device is provisioned with the HOSTNAME, not the address.
+#
+# The address is requested explicitly, matching every other LB service here.
+# Left to allocate freely, Cilium IPAM handed out the first free address in
+# the pool block (10.2.99.0-10.2.99.255) — which is 10.2.99.0, the base
+# address of the range. Each LB address is advertised to the UDM as its own
+# BGP route so that is routable in principle, but a .0 host address is the
+# kind of thing constrained embedded IP stacks get wrong, and the TRMNL
+# firmware is the single client that has to work here. Not worth the risk
+# for an address nobody chose on purpose.
+#
+# .89 continues the existing sequence (.81-.88 are in use) and matches the
+# fd99::89 that IPAM had already picked for v6.
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-device
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+    lb-pool: bgp
+  annotations:
+    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
+    lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/externalsecret.yaml

@@ -0,0 +1,18 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: terminus-secrets
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: terminus-secrets
+  # Holds APP_SECRET only (session/cookie/CSRF key). DATABASE_URL is not in
+  # here — it comes straight off the CNPG-generated `tr-pg-app` Secret's
+  # `uri` key, so the password rotates with the cluster and is never copied.
+  dataFrom:
+    - extract:
+        key: apps/trmnl

kubernetes/apps/trmnl/hetzner-creds-externalsecret.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/trmnl/hetzner-creds

kubernetes/apps/trmnl/ingress.yaml

@@ -0,0 +1,28 @@
+# Admin UI only — login, device registration, playlists, screen previews.
+# The device never touches this route: the firmware speaks plain HTTP and
+# would reject the internal CA, so it uses the cleartext LoadBalancer in
+# device-service.yaml instead. Both front the same Service.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: terminus
+  namespace: trmnl
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - trmnl.renner.uno
+      secretName: trmnl-tls
+  rules:
+    - host: trmnl.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: terminus
+                port:
+                  number: 80

kubernetes/apps/trmnl/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - pvcs.yaml
+  - valkey.yaml
+  - terminus.yaml
+  - ingress.yaml
+  - device-service.yaml
+  # tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
+  # PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
+  - cluster.yaml
+  - objectstore.yaml
+  - scheduled-backup.yaml
+  - hetzner-creds-externalsecret.yaml

kubernetes/apps/trmnl/objectstore.yaml

@@ -0,0 +1,30 @@
+---
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: tr-pg-store
+  namespace: trmnl
+spec:
+  configuration:
+    # Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
+    # clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
+    # (plugin #646), so at-rest encryption is unavailable; transit is TLS.
+    destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      # Region is required (REGION=fsn1) — without it boto3's HeadBucket
+      # (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  retentionPolicy: "30d"

kubernetes/apps/trmnl/pvcs.yaml

@@ -0,0 +1,36 @@
+---
+# Uploads + synchronised fonts for the Terminus pod. Mounted twice by
+# subPath (uploads/, fonts/) into both the web and sidekiq containers, which
+# share one pod — so RWO is sufficient and the Deployment uses `Recreate`.
+#
+# Contents: rendered device screens and the font set pulled by the
+# FONT_SYNCHRONIZER job. Persisting fonts here (rather than an emptyDir)
+# means a pod restart does not depend on upstream being reachable.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 5Gi
+---
+# Valkey RDB snapshots. The Sidekiq queue is reconstructable — the
+# synchroniser jobs re-enqueue on boot — so this is small and losing it is
+# an inconvenience, not data loss.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-valkey-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 1Gi

kubernetes/apps/trmnl/scheduled-backup.yaml

@@ -0,0 +1,16 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: tr-pg-daily
+  namespace: trmnl
+spec:
+  # 03:00 local, same slot as the other CNPG clusters (the Velero PVC
+  # schedule for this namespace runs later, at 08:15).
+  schedule: "0 0 3 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: tr-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/trmnl/terminus.yaml

@@ -0,0 +1,184 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: terminus-config
+  namespace: trmnl
+data:
+  TZ: "Europe/Berlin"
+  HANAMI_PORT: "2345"
+  # What the device is told to fetch from. Terminus bakes this into the
+  # absolute `image_url` it hands back on /api/display, so it MUST be the
+  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
+  # HTTP and would reject the internal CA. No trailing slash: the same
+  # string is typed into the captive portal at provisioning time.
+  API_URI: "http://trmnl-device.renner.uno"
+  KEYVALUE_URL: "redis://terminus-valkey:6379"
+  # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
+  # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB
+  # pool (fd99::/120) and the LAN ULA — without it every IPv6 client,
+  # including the device and Traefik, is rate-limited as if it were remote.
+  RACK_ATTACK_ALLOWED_SUBNETS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1,::1,fc00::/7"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+spec:
+  replicas: 1
+  # RWO PVC shared by both containers in this pod, and APP_SETUP runs
+  # migrations on boot — two concurrent web pods would race on both.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Tag pinned deliberately and NOT enrolled in Flux image automation.
+        # Terminus is pre-1.0 with no GitHub Releases and no changelog; an
+        # unattended bump that changes the firmware or /api/display contract
+        # is a device-bricking class of risk. Bumps are a reviewed, manual
+        # act — see kubernetes/infrastructure/image-automation/README.md.
+        - name: terminus
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          ports:
+            - name: http
+              containerPort: 2345
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          env:
+            # Runs `hanami assets compile` + `hanami db migrate` in the
+            # entrypoint before exec'ing puma. Set on the web container
+            # only — the sidekiq sidecar must not race the migration.
+            - name: APP_SETUP
+              value: "true"
+            # CNPG generates this Secret; `uri` already carries the rotated
+            # password and points at the tr-pg-rw service.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          # TCP probes, not HTTP: Terminus exposes no documented health
+          # endpoint, and every real route is behind either auth or
+          # Rack::Attack. Tighten to an HTTP GET once the route set is
+          # confirmed against the running app.
+          startupProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            # ~10 min: first boot compiles assets and migrates a fresh DB.
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+        - name: sidekiq
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          # EXPECTED on a fresh deploy: this container loses the race against
+          # the web container's migration and exits 1 with
+          # `PG::UndefinedTable: relation "account" does not exist`, then
+          # comes up healthy on the restart once the schema exists.
+          # Observed on first deploy 2026-08-10; converged in ~1s. There is
+          # no upstream ordering hook, and gating it behind an initContainer
+          # that probes for a specific table would couple us to Terminus's
+          # schema. So this is left to Kubernetes' restart, deliberately —
+          # a single sidekiq restart shortly after a rollout is benign. More
+          # than a couple, or restarts long after a rollout, are NOT this.
+          env:
+            # No APP_SETUP here — see the web container.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+spec:
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/valkey.yaml

@@ -0,0 +1,79 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus-valkey
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus-valkey
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus-valkey
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
+        # is the only consumer; a major bump is a separate migration.
+        - name: valkey
+          image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
+          args:
+            - valkey-server
+            - --save
+            - "60"
+            - "1"
+            - --loglevel
+            - warning
+          ports:
+            - name: valkey
+              containerPort: 6379
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          livenessProbe:
+            tcpSocket:
+              port: valkey
+            initialDelaySeconds: 10
+          readinessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-valkey-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+spec:
+  selector:
+    app.kubernetes.io/name: terminus-valkey
+  ports:
+    - port: 6379
+      targetPort: valkey

kubernetes/clusters/homelab/trmnl.yaml

@@ -0,0 +1,28 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: trmnl
+  namespace: flux-system
+spec:
+  # Terminus (TRMNL BYOS server) plus its tr-pg CNPG Cluster + ObjectStore +
+  # ScheduledBackup, the Valkey queue, and the uploads/fonts PVC. Drives the
+  # entrance-way e-ink panel; the device polls the cleartext LoadBalancer.
+  interval: 10m
+  retryInterval: 1m
+  timeout: 30m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/trmnl
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster
+    - name: infrastructure-cnpg-barman
+  healthCheckExprs:
+    - apiVersion: postgresql.cnpg.io/v1
+      kind: Cluster
+      failed: "has(status.phase) && status.phase.startsWith('Failed')"
+      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"

kubernetes/infrastructure/image-automation/README.md

@@ -96,6 +96,19 @@ The security-scan agent will continue to surface vuln findings on
 these as "Skipped / Flux image-automation owns this" even without
 a policy.
 
+- `ghcr.io/usetrmnl/terminus` — publishes a usable `0.x.0` tag stream, so
+  unlike the three above this is **not** a tag-format problem. It is left
+  unenrolled because the blast radius is hardware: Terminus is pre-1.0,
+  cuts no GitHub Releases and ships no changelog, and it serves the
+  firmware and `/api/display` contract that the TRMNL X panel polls. An
+  unattended bump that changes either can leave a wall-mounted device
+  dark, with no rollback path that doesn't involve the captive portal.
+  Bumps are a reviewed, manual act: read the commit range, then edit the
+  tag in `kubernetes/apps/trmnl/terminus.yaml` (both containers).
+
+  The Valkey sidecar alongside it *is* enrolled (`valkey.yaml`) — it holds
+  only the Sidekiq queue and touches no device-facing contract.
+
 ## Re-import hazard: cnpg-barman-plugin
 
 `kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml` is an

kubernetes/infrastructure/image-automation/kustomization.yaml

@@ -20,6 +20,7 @@ resources:
   - redis.yaml
   - shelly-manager.yaml
   - tika.yaml
+  - valkey.yaml
   - velero-plugin-for-aws.yaml
   - vsftpd.yaml
   - woodpecker.yaml

kubernetes/infrastructure/image-automation/valkey.yaml

@@ -0,0 +1,27 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: valkey-trmnl
+  namespace: flux-system
+spec:
+  image: valkey/valkey
+  interval: 1h
+---
+# Alpine tags only, semver 9.x — matches the major version pinned in
+# upstream Terminus's compose.yml. Valkey holds nothing but the Sidekiq
+# queue here, so minor bumps are safe to take unattended; a 10.x cutover
+# is a deliberate migration.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: valkey-trmnl
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: valkey-trmnl
+  filterTags:
+    pattern: '^(?P<ver>\d+\.\d+\.\d+)-alpine$'
+    extract: '$ver'
+  policy:
+    semver:
+      range: "9.x"

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -145,3 +145,9 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: external-secrets
+---
+# Terminus (TRMNL BYOS server) driving the entrance-way e-ink panel.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: trmnl

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -21,4 +21,5 @@ resources:
   - openbao-ns.yaml
   - external-secrets-ns.yaml
   - woodpecker-ci-ns.yaml
+  - trmnl-ns.yaml
   - allow-cilium-health.yaml

kubernetes/infrastructure/network-policies/trmnl-ns.yaml

@@ -0,0 +1,177 @@
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: trmnl
+spec:
+  endpointSelector: {}
+  ingress:
+    # Intra-namespace — terminus ↔ sidekiq ↔ valkey ↔ tr-pg.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": trmnl
+    # Admin UI via Traefik (trmnl.renner.uno). The device does NOT come
+    # this way — see allow-device-http below.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    # Prometheus scrape.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    # Velero backups via PVC mount.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": velero
+    # CNPG operator (in `database` ns) reconciles tr-pg — needs to reach
+    # the postgres pods for health, failover, role checks.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
+    # home-assistant-ns.yaml for the rationale. INGRESS ONLY.
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 4,   family: IPv4 }
+            - { type: 5,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 9,   family: IPv4 }
+            - { type: 10,  family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 13,  family: IPv4 }
+            - { type: 14,  family: IPv4 }
+            - { type: 15,  family: IPv4 }
+            - { type: 16,  family: IPv4 }
+            - { type: 17,  family: IPv4 }
+            - { type: 18,  family: IPv4 }
+            - { type: 30,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 130, family: IPv6 }
+            - { type: 131, family: IPv6 }
+            - { type: 132, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 137, family: IPv6 }
+            - { type: 138, family: IPv6 }
+            - { type: 141, family: IPv6 }
+            - { type: 142, family: IPv6 }
+            - { type: 143, family: IPv6 }
+            - { type: 148, family: IPv6 }
+            - { type: 149, family: IPv6 }
+            - { type: 151, family: IPv6 }
+            - { type: 152, family: IPv6 }
+            - { type: 153, family: IPv6 }
+---
+# The TRMNL panel polls /api/display over cleartext HTTP on the
+# terminus-device LoadBalancer. The firmware cannot be handed the internal
+# CA, so this bypasses Traefik entirely and the namespace default-deny has
+# to be opened for it — same shape as paperless's allow-ftp-scanner.
+#
+# Scoped to the terminus pods and port 2345 only. Requests still have to
+# carry a known device MAC in the `ID` header plus its api_key; this rule
+# opens the port, not the application.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: allow-device-http
+  namespace: trmnl
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: terminus
+  ingress:
+    - fromEntities:
+        - world
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - port: "2345"
+              protocol: TCP
+---
+# Egress allowlist for the Terminus pod only.
+#
+# Deliberately NOT applied to the tr-pg CNPG pods: selecting them here
+# would make their egress restricted too, and the barman-cloud WAL archive
+# path to Hetzner object storage is not worth putting behind an FQDN rule
+# that fails closed. They keep unrestricted egress, matching immich/authentik.
+#
+# Terminus needs exactly three outbound destinations, all HTTPS:
+#   - trmnl.com            → GET /api/models, /api/palettes (MODEL_SYNCHRONIZER,
+#                            which is what seeds the TRMNL X model definition —
+#                            1872x1404, 16 colours — into the Models list),
+#                            GET /api/firmware/latest (FIRMWARE_SYNCHRONIZER),
+#                            GET /fonts/<name> (FONT_SYNCHRONIZER)
+#   - trmnl-fw.s3.us-east-2.amazonaws.com
+#                          → the actual firmware binary. This host is NOT a
+#                            constant in the source: it is the `url` field of
+#                            the /api/firmware/latest response, so TRMNL can
+#                            move bucket/region without a Terminus release and
+#                            firmware sync would start failing closed here.
+#                            If firmware sync breaks, re-check that response
+#                            before assuming the policy is wrong.
+#
+# KNOWN LIMITATION: Terminus's "Extensions" (private plugin) feature polls
+# arbitrary admin-configured URLs rendered from a Liquid template stored in
+# the database. Those are unknowable from source and WILL be blocked by this
+# allowlist. We do not use Extensions — screen content is pushed in from the
+# Home Assistant renderer — so this is an accepted trade, not an oversight.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: terminus-egress
+  namespace: trmnl
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: terminus
+  egress:
+    # Intra-namespace — tr-pg-rw (postgres) and terminus-valkey.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": trmnl
+    # DNS. The L7 `rules.dns` block is mandatory, not decorative: Cilium's
+    # DNS proxy only learns an FQDN→IP mapping for names it has actually
+    # observed being resolved, so without this the toFQDNs rules below
+    # never match anything.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+          rules:
+            dns:
+              - matchPattern: "*"
+    - toFQDNs:
+        - matchName: "trmnl.com"
+        - matchName: "trmnl-fw.s3.us-east-2.amazonaws.com"
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+    - toEntities:
+        - host
+        - remote-node

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -28,6 +28,7 @@ spec:
         - monitoring
         - netbird-backup # netbird-backup-ssh: the VPS-pull backup key
         - paperless
+        - trmnl
         - velero
   provider:
     vault: # ESO's provider name; speaks OpenBao fine

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -283,3 +283,30 @@ spec:
     storageLocation: default
     ttl: 720h0m0s
     snapshotMoveData: true
+---
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: trmnl-daily
+  namespace: velero
+  labels:
+    app.kubernetes.io/part-of: homelab-backup
+spec:
+  # Daily at 08:15 local (next free stagger slot after netbird-backup-daily
+  # at 07:45). Covers the terminus-data PVC (rendered device screens +
+  # synchronised fonts) and terminus-valkey-data. The tr-pg CNPG cluster is
+  # covered separately by Barman, hence the cnpg.io/cluster exclusion.
+  schedule: "15 8 * * *"
+  useOwnerReferencesInBackup: false
+  template:
+    includedNamespaces:
+      - trmnl
+    includedResources:
+      - persistentvolumeclaims
+    storageLocation: default
+    ttl: 720h0m0s
+    snapshotMoveData: true
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist