feat(trmnl): serve the device endpoint over TLS

The cleartext LoadBalancer was justified with "the firmware cannot be
handed the internal CA". There is no internal CA in that path:
letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
so a LAN-only name gets a publicly-trusted certificate without ever being
publicly reachable — which is why trmnl.renner.uno has always served an
issuer CN=YR2 cert.

trmnl-device.renner.uno therefore moves onto Traefik with its own
Ingress, and API_URI follows it to https so the image_url handed back on
/api/display is fetched over the same scheme the device polls with. The
name moves rather than being duplicated: external-dns runs --policy=sync
and would flap the record between the Traefik VIP and the LB address if
both an Ingress and the Service claimed it.

The LoadBalancer stays, without a hostname. http://10.2.99.89 remains a
path to /api/setup that depends on neither Traefik nor cert renewal, and
the captive portal takes a bare address, so it is the recovery route if
TLS is ever what breaks the panel.

Note Traefik's web entrypoint carries a global http -> websecure
redirect, so http://trmnl-device.renner.uno now answers 308 rather than
serving. The device is provisioned with the https URL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/llama-server/embed.yaml

@@ -44,37 +44,12 @@ spec:
             - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
-            # A short, stable id for consumers. Without it the model is
-            # advertised as the whole --hf-repo string, which would end up
-            # verbatim in Paperless's config and break on the next model swap.
-            # One name only: llama-server keeps aliases in a sorted set and
-            # reports the first as the id, so a second name can steal it.
-            - --alias
-            - qwen3-embedding-0.6b
-            # Qwen3-Embedding is trained for last-token pooling. Left
-            # unspecified llama-server takes the GGUF's default, which is not
-            # visible from outside and would silently produce vectors the model
-            # was not trained to emit. Set before anything is indexed: changing
-            # pooling later invalidates every stored embedding.
-            - --pooling
-            - last
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks documents at 1024 tokens. The defaults are
-            # batch 2048 / ubatch 512, and with --embeddings llama-server
-            # clamps both down to 512 rather than run them mismatched — below
-            # the chunk size. Matching both to 1024 fits a whole chunk in one
-            # pass; 2048 is deliberately avoided, since that is where this
-            # Vulkan stack fails with ErrorDeviceLost.
-            - --batch-size
-            - "1024"
-            - --ubatch-size
-            - "1024"
-            # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
-            - "4096"
+            - "2048"
             # No browser ever talks to this Service, so the WebUI is dead weight
             # and one more unauthenticated surface.
             - --no-webui

kubernetes/apps/paperless/network-policy.yaml

@@ -57,20 +57,6 @@ spec:
               protocol: UDP
             - port: "53"
               protocol: TCP
-    # The cluster's own LLM, for AI suggestions and RAG embeddings. Scoped to
-    # the two pods and the one port rather than the ai namespace as a whole, so
-    # this grant cannot widen if something else is ever placed there.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-server
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-embed
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     # External HTTPS only — Tika model downloads, runtime image
     # fetches, etc. RFC1918 + ULA are excluded from `world` via the
     # CIDR-deny pattern. `world` covers public IPs / DNS-resolvable

kubernetes/apps/paperless/paperless.yaml

@@ -31,14 +31,12 @@ data:
   PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
   PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
   PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE: "true"
-  # No CONSUMER_BARCODE_SCANNER: zxing-cpp is the only backend, so the
-  # setting no longer exists. pyzbar and libzbar are gone from the image.
-  #
+  PAPERLESS_CONSUMER_BARCODE_SCANNER: "ZXING"
   # CephFS does not propagate inotify across nodes — files written
   # via the RWX consume PVC from a pod on a different node than
   # paperless never fire IN_CREATE in paperless's kernel. Polling
   # is the reliable backstop. See wiki: cephfs-inotify-no-cross-node-propagation.
-  PAPERLESS_CONSUMER_POLLING_INTERVAL: "10"
+  PAPERLESS_CONSUMER_POLLING: "10"
   PAPERLESS_REDIS: "redis://redis:6379"
   PAPERLESS_DBENGINE: "postgresql"
   PAPERLESS_DBHOST: "pprl-pg-rw"
@@ -46,44 +44,6 @@ data:
   PAPERLESS_TIKA_ENABLED: "1"
   PAPERLESS_TIKA_ENDPOINT: "http://tika:9998"
   PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://gotenberg:3000"
-  # AI suggestions against the cluster's own LLM (kubernetes/apps/llama-server).
-  # Nothing leaves the cluster: both endpoints are ClusterIP Services, so
-  # document text is never sent to a third party and there is no API key to
-  # rotate — llama-server has no authentication, and the ai namespace's network
-  # policy is what restricts who may reach it.
-  #
-  # CAVEAT, and it is a real one: paperless reads these from an
-  # ApplicationConfiguration row in the database and falls back to the
-  # environment only when that row is empty. A value set in the admin UI
-  # therefore overrides this ConfigMap silently. Configure AI in ONE place —
-  # here — and leave those fields untouched in the UI.
-  PAPERLESS_AI_ENABLED: "true"
-  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://llama-server.ai.svc.cluster.local:8080/v1"
-  # Must match --alias in kubernetes/apps/llama-server/deployment.yaml. The
-  # server ignores the field on the wire, but paperless sends it and the value
-  # is what appears in its logs.
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  # ClusterIP addresses are private, and paperless blocks non-public endpoints
-  # unless this is set. Default is already true; explicit because the whole
-  # integration silently stops working if it ever flips.
-  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  # Under llama-server's own 16384, leaving room for the response.
-  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  # The 120s default is too tight at ~11 tok/s: a classification is roughly
-  # 2-4k prompt tokens plus a second round trip for localisation.
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "300"
-  # Matched against Django's settings.LANGUAGES, so the region is required —
-  # "de" alone does not resolve. Without it the model returns German tags but
-  # English document types, which was reproducible against this endpoint.
-  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
-  # RAG embeddings from llama-embed. Its Qwen3-Embedding covers 100+ languages,
-  # where paperless's default all-MiniLM-L6-v2 is documented as suited to
-  # primarily English archives — the better fit for a German corpus.
-  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
-  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -160,7 +120,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -212,25 +172,6 @@ spec:
           # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
           # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
           # means giving these probes an explicit Host header.
-
-          # The container does not serve until its init sequence finishes:
-          # s6 makes svc-webserver depend on init-migrations and
-          # init-search-index. On the first 3.x start those are the SHA256
-          # checksum backfill, which opens and reads every original and every
-          # archive file, and a full Tantivy index build, since the Whoosh
-          # index cannot be converted. Both scale with the corpus.
-          #
-          # startupProbe holds liveness and readiness off until the webserver
-          # answers, giving that boot 30 * 20s = 10 minutes. Without it the
-          # 120s liveness delay plus three 30s periods kills the pod 210s in,
-          # part-way through a migration, and the next boot starts over.
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            timeoutSeconds: 5
-            failureThreshold: 30
           livenessProbe:
             httpGet:
               path: /accounts/login/

kubernetes/apps/trmnl/device-ingress.yaml

@@ -0,0 +1,52 @@
+# Device-facing endpoint, over TLS.
+#
+# This replaces the reason the cleartext LoadBalancer in device-service.yaml
+# was given a hostname at all. That file claimed the firmware "would reject
+# the internal CA" — but nothing here is signed by an internal CA:
+# letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
+# so trmnl.renner.uno already serves a public Let's Encrypt certificate
+# (issuer CN=YR2) that any client with a normal CA bundle trusts. A
+# LAN-only name gets the same treatment — DNS-01 never needs the host to be
+# publicly reachable — so the device can speak HTTPS like everything else.
+#
+# The hostname moves here from the Service's external-dns annotation rather
+# than being duplicated: two sources claiming trmnl-device.renner.uno would
+# have external-dns flapping the record between the Traefik VIP and the
+# device LB. The LoadBalancer keeps its address and stays reachable at
+# http://10.2.99.89 as a nameless fallback if TLS ever turns out to be the
+# thing that breaks.
+#
+# CONSEQUENCE, and the reason API_URI in terminus.yaml changes with this:
+# Traefik's `web` entrypoint carries a global http -> websecure redirect
+# (helmrelease.yaml: `to: websecure, scheme: https, permanent: true`), so
+# once this name points at Traefik, http://trmnl-device.renner.uno answers
+# 308 rather than serving. Whether the firmware follows that is not
+# documented, so the device is provisioned with the https URL and Terminus
+# hands back https image_urls to match. No mixed-scheme fetch.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: terminus-device
+  namespace: trmnl
+  # No external-dns hostname annotation: external-dns-unifi runs with
+  # --source=ingress, so it registers the hosts in spec.rules on its own.
+  # Same as the terminus Ingress next door.
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - trmnl-device.renner.uno
+      secretName: trmnl-device-tls
+  rules:
+    - host: trmnl-device.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: terminus
+                port:
+                  number: 80

kubernetes/apps/trmnl/device-service.yaml

@@ -1,15 +1,24 @@
-# Device-facing endpoint. Cleartext HTTP, deliberately.
+# Cleartext fallback for the device, reachable by address only.
 #
-# The TRMNL firmware initiates every request and cannot be handed the
-# internal CA, so terminating TLS here would simply break polling. Scope of
-# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
-# record), it carries no admin session — the device authenticates with its
-# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
-# UI stays on the TLS route at trmnl.renner.uno.
+# This used to be THE device endpoint, on the premise that the firmware
+# "cannot be handed the internal CA". The premise was wrong: nothing here
+# hands out an internal CA for these names — letsencrypt-cluster solves
+# DNS-01 against the renner.uno Cloudflare zone and issues publicly-trusted
+# certificates for LAN-only hosts, which is why trmnl.renner.uno has always
+# served an issuer CN=YR2 certificate. The device speaks HTTPS via
+# device-ingress.yaml, which now owns the trmnl-device.renner.uno name.
+#
+# Kept, without a hostname, because it is the one path to Terminus that
+# depends on neither Traefik nor certificate renewal: http://10.2.99.89
+# still serves /api/setup and /api/display. If TLS is ever what breaks the
+# panel, the recovery is to type that address into the captive portal.
+#
+# Scope of the exposure is unchanged: LAN-only (no public DNS label, no
+# Cloudflare record), no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key.
 #
 # Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
-# pool rather than a TLS-less Ingress, which would be a new pattern for this
-# repo. The device is provisioned with the HOSTNAME, not the address.
+# pool.
 #
 # The address is requested explicitly, matching every other LB service here.
 # Left to allocate freely, Cilium IPAM handed out the first free address in
@@ -30,8 +39,10 @@ metadata:
   labels:
     app.kubernetes.io/name: terminus
     lb-pool: bgp
+  # No external-dns hostname: device-ingress.yaml claims that name. Two
+  # sources claiming trmnl-device.renner.uno would flap the record between
+  # the Traefik VIP and this address (external-dns runs --policy=sync).
   annotations:
-    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
     lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
 spec:
   type: LoadBalancer

kubernetes/apps/trmnl/kustomization.yaml

@@ -6,6 +6,9 @@ resources:
   - valkey.yaml
   - terminus.yaml
   - ingress.yaml
+  # TLS device endpoint (owns trmnl-device.renner.uno) plus the nameless
+  # cleartext LoadBalancer it falls back to.
+  - device-ingress.yaml
   - device-service.yaml
   # Headless HA screenshotter. Lives in this namespace because Terminus
   # FETCHES rendered screens back from it (uri delivery mode), so the

kubernetes/apps/trmnl/terminus.yaml

@@ -7,11 +7,17 @@ data:
   TZ: "Europe/Berlin"
   HANAMI_PORT: "2345"
   # What the device is told to fetch from. Terminus bakes this into the
-  # absolute `image_url` it hands back on /api/display, so it MUST be the
-  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
-  # HTTP and would reject the internal CA. No trailing slash: the same
-  # string is typed into the captive portal at provisioning time.
-  API_URI: "http://trmnl-device.renner.uno"
+  # absolute `image_url` it hands back on /api/display, so it has to be a
+  # name the DEVICE can resolve and fetch — and it has to carry the same
+  # scheme the device polls with, or every screen fetch is a second,
+  # cleartext connection. No trailing slash: the same string is typed into
+  # the captive portal at provisioning time.
+  #
+  # https since device-ingress.yaml took over trmnl-device.renner.uno. The
+  # earlier value was http on the theory that the firmware would reject an
+  # internal CA; there is no internal CA here — letsencrypt-cluster issues
+  # publicly-trusted certs for LAN-only names via DNS-01.
+  API_URI: "https://trmnl-device.renner.uno"
   KEYVALUE_URL: "redis://terminus-valkey:6379"
   # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
   # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/infrastructure/image-automation/paperless-ngx.yaml

@@ -7,7 +7,7 @@ spec:
   image: ghcr.io/paperless-ngx/paperless-ngx
   interval: 1h
 ---
-# Allow only patch + minor bumps within the 3.x line. A 4.0 release
+# Allow only patch + minor bumps within the 2.x line. A 3.0 release
 # requires manual policy widening — major upgrades typically need a
 # migration plan and shouldn't land via auto-merge.
 apiVersion: image.toolkit.fluxcd.io/v1
@@ -20,4 +20,4 @@ spec:
     name: paperless-ngx
   policy:
     semver:
-      range: "3.x"
+      range: "2.x"

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -24,16 +24,6 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # Paperless calls both models for AI suggestions and RAG embeddings. The
-    # matching egress grant on its side is scoped to these two pods on :8080.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": paperless
-            app.kubernetes.io/name: paperless
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring