feat(paperless): AI suggestions against the cluster's own LLM

Paperless 3.x has LLM suggestions built in, so this is configuration rather
than a sidecar: no extra image, no second database, no Paperless API token to
mint. Both endpoints are ClusterIP Services, so document text never leaves the
cluster and there is no key to rotate.

The make-or-break requirement was verified against the live endpoint before any
of this was written. Paperless's openai-like client calls chat_with_tools with
tool_required=True and error_on_no_tool_call=True — there is no free-text JSON
fallback, so the model must emit real tool_calls. It does: finish_reason
"tool_calls", a populated tool_calls array, empty content. A German utility
bill came back tagged Rechnung / Strom / Versorger.

That same probe showed the tags in German but document_types in English, which
is the documented weak spot and exactly what OUTPUT_LANGUAGE's second
localisation pass addresses. The value must be "de-de": it is matched against
Django's settings.LANGUAGES, where a bare "de" does not resolve.

llama-embed needed three fixes to serve as the RAG embedder:

  - --alias, so the model is not advertised as its whole --hf-repo string.
    That string would otherwise sit verbatim in this ConfigMap and break on the
    next model swap.
  - --pooling last. Qwen3-Embedding is trained for it; unspecified,
    llama-server takes the GGUF default, which is invisible from outside. Set
    now because nothing is indexed yet — changing pooling later invalidates
    every stored vector.
  - --batch-size/--ubatch-size 1024. The logs showed llama-server clamping both
    to 512 under --embeddings, below Paperless's 1024-token chunk size. 2048 is
    avoided deliberately: that is where this Vulkan stack fails with
    ErrorDeviceLost.

Both network policies are scoped to the two pods and the one port on each side,
so neither grant widens if something else is placed in either namespace.

REQUEST_TIMEOUT goes to 300s. At ~11 tok/s a classification is 2-4k prompt
tokens plus a second round trip for localisation, and the 120s default is
marginal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/llama-server/embed.yaml

@@ -44,12 +44,37 @@ spec:
             - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
+            # A short, stable id for consumers. Without it the model is
+            # advertised as the whole --hf-repo string, which would end up
+            # verbatim in Paperless's config and break on the next model swap.
+            # One name only: llama-server keeps aliases in a sorted set and
+            # reports the first as the id, so a second name can steal it.
+            - --alias
+            - qwen3-embedding-0.6b
+            # Qwen3-Embedding is trained for last-token pooling. Left
+            # unspecified llama-server takes the GGUF's default, which is not
+            # visible from outside and would silently produce vectors the model
+            # was not trained to emit. Set before anything is indexed: changing
+            # pooling later invalidates every stored embedding.
+            - --pooling
+            - last
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
+            # Paperless chunks documents at 1024 tokens. The defaults are
+            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # clamps both down to 512 rather than run them mismatched — below
+            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # pass; 2048 is deliberately avoided, since that is where this
+            # Vulkan stack fails with ErrorDeviceLost.
+            - --batch-size
+            - "1024"
+            - --ubatch-size
+            - "1024"
+            # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
-            - "2048"
+            - "4096"
             # No browser ever talks to this Service, so the WebUI is dead weight
             # and one more unauthenticated surface.
             - --no-webui

kubernetes/apps/paperless/network-policy.yaml

@@ -57,6 +57,20 @@ spec:
               protocol: UDP
             - port: "53"
               protocol: TCP
+    # The cluster's own LLM, for AI suggestions and RAG embeddings. Scoped to
+    # the two pods and the one port rather than the ai namespace as a whole, so
+    # this grant cannot widen if something else is ever placed there.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-server
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-embed
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # External HTTPS only — Tika model downloads, runtime image
     # fetches, etc. RFC1918 + ULA are excluded from `world` via the
     # CIDR-deny pattern. `world` covers public IPs / DNS-resolvable

kubernetes/apps/paperless/paperless.yaml

@@ -31,12 +31,14 @@ data:
   PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
   PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
   PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE: "true"
-  PAPERLESS_CONSUMER_BARCODE_SCANNER: "ZXING"
+  # No CONSUMER_BARCODE_SCANNER: zxing-cpp is the only backend, so the
+  # setting no longer exists. pyzbar and libzbar are gone from the image.
+  #
   # CephFS does not propagate inotify across nodes — files written
   # via the RWX consume PVC from a pod on a different node than
   # paperless never fire IN_CREATE in paperless's kernel. Polling
   # is the reliable backstop. See wiki: cephfs-inotify-no-cross-node-propagation.
-  PAPERLESS_CONSUMER_POLLING: "10"
+  PAPERLESS_CONSUMER_POLLING_INTERVAL: "10"
   PAPERLESS_REDIS: "redis://redis:6379"
   PAPERLESS_DBENGINE: "postgresql"
   PAPERLESS_DBHOST: "pprl-pg-rw"
@@ -44,6 +46,44 @@ data:
   PAPERLESS_TIKA_ENABLED: "1"
   PAPERLESS_TIKA_ENDPOINT: "http://tika:9998"
   PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://gotenberg:3000"
+  # AI suggestions against the cluster's own LLM (kubernetes/apps/llama-server).
+  # Nothing leaves the cluster: both endpoints are ClusterIP Services, so
+  # document text is never sent to a third party and there is no API key to
+  # rotate — llama-server has no authentication, and the ai namespace's network
+  # policy is what restricts who may reach it.
+  #
+  # CAVEAT, and it is a real one: paperless reads these from an
+  # ApplicationConfiguration row in the database and falls back to the
+  # environment only when that row is empty. A value set in the admin UI
+  # therefore overrides this ConfigMap silently. Configure AI in ONE place —
+  # here — and leave those fields untouched in the UI.
+  PAPERLESS_AI_ENABLED: "true"
+  PAPERLESS_AI_LLM_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_ENDPOINT: "http://llama-server.ai.svc.cluster.local:8080/v1"
+  # Must match --alias in kubernetes/apps/llama-server/deployment.yaml. The
+  # server ignores the field on the wire, but paperless sends it and the value
+  # is what appears in its logs.
+  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
+  # ClusterIP addresses are private, and paperless blocks non-public endpoints
+  # unless this is set. Default is already true; explicit because the whole
+  # integration silently stops working if it ever flips.
+  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
+  # Under llama-server's own 16384, leaving room for the response.
+  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
+  # The 120s default is too tight at ~11 tok/s: a classification is roughly
+  # 2-4k prompt tokens plus a second round trip for localisation.
+  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "300"
+  # Matched against Django's settings.LANGUAGES, so the region is required —
+  # "de" alone does not resolve. Without it the model returns German tags but
+  # English document types, which was reproducible against this endpoint.
+  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
+  # RAG embeddings from llama-embed. Its Qwen3-Embedding covers 100+ languages,
+  # where paperless's default all-MiniLM-L6-v2 is documented as suited to
+  # primarily English archives — the better fit for a German corpus.
+  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -120,7 +160,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -172,6 +212,25 @@ spec:
           # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
           # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
           # means giving these probes an explicit Host header.
+
+          # The container does not serve until its init sequence finishes:
+          # s6 makes svc-webserver depend on init-migrations and
+          # init-search-index. On the first 3.x start those are the SHA256
+          # checksum backfill, which opens and reads every original and every
+          # archive file, and a full Tantivy index build, since the Whoosh
+          # index cannot be converted. Both scale with the corpus.
+          #
+          # startupProbe holds liveness and readiness off until the webserver
+          # answers, giving that boot 30 * 20s = 10 minutes. Without it the
+          # 120s liveness delay plus three 30s periods kills the pod 210s in,
+          # part-way through a migration, and the next boot starts over.
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 30
           livenessProbe:
             httpGet:
               path: /accounts/login/

kubernetes/apps/trmnl/device-ingress.yaml

@@ -1,52 +0,0 @@
-# Device-facing endpoint, over TLS.
-#
-# This replaces the reason the cleartext LoadBalancer in device-service.yaml
-# was given a hostname at all. That file claimed the firmware "would reject
-# the internal CA" — but nothing here is signed by an internal CA:
-# letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
-# so trmnl.renner.uno already serves a public Let's Encrypt certificate
-# (issuer CN=YR2) that any client with a normal CA bundle trusts. A
-# LAN-only name gets the same treatment — DNS-01 never needs the host to be
-# publicly reachable — so the device can speak HTTPS like everything else.
-#
-# The hostname moves here from the Service's external-dns annotation rather
-# than being duplicated: two sources claiming trmnl-device.renner.uno would
-# have external-dns flapping the record between the Traefik VIP and the
-# device LB. The LoadBalancer keeps its address and stays reachable at
-# http://10.2.99.89 as a nameless fallback if TLS ever turns out to be the
-# thing that breaks.
-#
-# CONSEQUENCE, and the reason API_URI in terminus.yaml changes with this:
-# Traefik's `web` entrypoint carries a global http -> websecure redirect
-# (helmrelease.yaml: `to: websecure, scheme: https, permanent: true`), so
-# once this name points at Traefik, http://trmnl-device.renner.uno answers
-# 308 rather than serving. Whether the firmware follows that is not
-# documented, so the device is provisioned with the https URL and Terminus
-# hands back https image_urls to match. No mixed-scheme fetch.
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: terminus-device
-  namespace: trmnl
-  # No external-dns hostname annotation: external-dns-unifi runs with
-  # --source=ingress, so it registers the hosts in spec.rules on its own.
-  # Same as the terminus Ingress next door.
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-spec:
-  ingressClassName: traefik
-  tls:
-    - hosts:
-        - trmnl-device.renner.uno
-      secretName: trmnl-device-tls
-  rules:
-    - host: trmnl-device.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: terminus
-                port:
-                  number: 80

kubernetes/apps/trmnl/device-service.yaml

@@ -1,24 +1,15 @@
-# Cleartext fallback for the device, reachable by address only.
+# Device-facing endpoint. Cleartext HTTP, deliberately.
 #
-# This used to be THE device endpoint, on the premise that the firmware
-# "cannot be handed the internal CA". The premise was wrong: nothing here
-# hands out an internal CA for these names — letsencrypt-cluster solves
-# DNS-01 against the renner.uno Cloudflare zone and issues publicly-trusted
-# certificates for LAN-only hosts, which is why trmnl.renner.uno has always
-# served an issuer CN=YR2 certificate. The device speaks HTTPS via
-# device-ingress.yaml, which now owns the trmnl-device.renner.uno name.
-#
-# Kept, without a hostname, because it is the one path to Terminus that
-# depends on neither Traefik nor certificate renewal: http://10.2.99.89
-# still serves /api/setup and /api/display. If TLS is ever what breaks the
-# panel, the recovery is to type that address into the captive portal.
-#
-# Scope of the exposure is unchanged: LAN-only (no public DNS label, no
-# Cloudflare record), no admin session — the device authenticates with its
-# MAC in the `ID` header plus a per-device api_key.
+# The TRMNL firmware initiates every request and cannot be handed the
+# internal CA, so terminating TLS here would simply break polling. Scope of
+# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
+# record), it carries no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
+# UI stays on the TLS route at trmnl.renner.uno.
 #
 # Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
-# pool.
+# pool rather than a TLS-less Ingress, which would be a new pattern for this
+# repo. The device is provisioned with the HOSTNAME, not the address.
 #
 # The address is requested explicitly, matching every other LB service here.
 # Left to allocate freely, Cilium IPAM handed out the first free address in
@@ -39,10 +30,8 @@ metadata:
   labels:
     app.kubernetes.io/name: terminus
     lb-pool: bgp
-  # No external-dns hostname: device-ingress.yaml claims that name. Two
-  # sources claiming trmnl-device.renner.uno would flap the record between
-  # the Traefik VIP and this address (external-dns runs --policy=sync).
   annotations:
+    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
     lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
 spec:
   type: LoadBalancer

kubernetes/apps/trmnl/kustomization.yaml

@@ -6,9 +6,6 @@ resources:
   - valkey.yaml
   - terminus.yaml
   - ingress.yaml
-  # TLS device endpoint (owns trmnl-device.renner.uno) plus the nameless
-  # cleartext LoadBalancer it falls back to.
-  - device-ingress.yaml
   - device-service.yaml
   # Headless HA screenshotter. Lives in this namespace because Terminus
   # FETCHES rendered screens back from it (uri delivery mode), so the

kubernetes/apps/trmnl/terminus.yaml

@@ -7,17 +7,11 @@ data:
   TZ: "Europe/Berlin"
   HANAMI_PORT: "2345"
   # What the device is told to fetch from. Terminus bakes this into the
-  # absolute `image_url` it hands back on /api/display, so it has to be a
-  # name the DEVICE can resolve and fetch — and it has to carry the same
-  # scheme the device polls with, or every screen fetch is a second,
-  # cleartext connection. No trailing slash: the same string is typed into
-  # the captive portal at provisioning time.
-  #
-  # https since device-ingress.yaml took over trmnl-device.renner.uno. The
-  # earlier value was http on the theory that the firmware would reject an
-  # internal CA; there is no internal CA here — letsencrypt-cluster issues
-  # publicly-trusted certs for LAN-only names via DNS-01.
-  API_URI: "https://trmnl-device.renner.uno"
+  # absolute `image_url` it hands back on /api/display, so it MUST be the
+  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
+  # HTTP and would reject the internal CA. No trailing slash: the same
+  # string is typed into the captive portal at provisioning time.
+  API_URI: "http://trmnl-device.renner.uno"
   KEYVALUE_URL: "redis://terminus-valkey:6379"
   # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
   # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/infrastructure/image-automation/paperless-ngx.yaml

@@ -7,7 +7,7 @@ spec:
   image: ghcr.io/paperless-ngx/paperless-ngx
   interval: 1h
 ---
-# Allow only patch + minor bumps within the 2.x line. A 3.0 release
+# Allow only patch + minor bumps within the 3.x line. A 4.0 release
 # requires manual policy widening — major upgrades typically need a
 # migration plan and shouldn't land via auto-merge.
 apiVersion: image.toolkit.fluxcd.io/v1
@@ -20,4 +20,4 @@ spec:
     name: paperless-ngx
   policy:
     semver:
-      range: "2.x"
+      range: "3.x"

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -24,6 +24,16 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
+    # Paperless calls both models for AI suggestions and RAG embeddings. The
+    # matching egress grant on its side is scoped to these two pods on :8080.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless
+            app.kubernetes.io/name: paperless
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring