fix(llama-embed): raise the memory limit to match its own context and batch

#739 raised the context from 2048 to 4096 and the micro-batch from an effective
512 to 1024, both to match how Paperless chunks documents. It left memory at
2Gi/4Gi, which was sized for the smaller values.

The pod ran fine interactively and was OOMKilled (exit 137) partway through the
first sustained run, around 300 documents into an embedding pass. Nothing in the
server log marked it — the process was killed outright, so the last line is a
half-finished task and the reason is only visible in the container's
lastState.terminated.

This would have surfaced when Paperless first indexed the archive, i.e. on the
first real use rather than under test.

The weights are only ~600 MiB. What sets the footprint is four slots at 4096
context with a 1024-token micro-batch, plus Vulkan allocating the iGPU working
set from system RAM where it counts against this cgroup — the same effect that
put llama-server at 32Gi for 19 GiB of weights.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/authentik/blueprints-configmap.yaml

@@ -907,6 +907,17 @@ data:
           name: llama-gate
         state: present
         attrs:
+          # Both flows, mirroring the Audiobookshelf provider. Setting only
+          # authorization_flow leaves authentication on Authentik's default
+          # flow, which is what made the first login clumsy — the step-up ran
+          # on top of a password prompt instead of replacing it. With the
+          # passwordless flow named explicitly, signing in is the passkey
+          # itself.
+          #
+          # Referenced by !Find rather than !KeyOf: both flows are defined in
+          # the passkey-stepup document of this ConfigMap, and !KeyOf only
+          # resolves within one document.
+          authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
           authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
           invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
           mode: proxy

kubernetes/apps/llama-server/embed.yaml

@@ -44,12 +44,37 @@ spec:
             - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
+            # A short, stable id for consumers. Without it the model is
+            # advertised as the whole --hf-repo string, which would end up
+            # verbatim in Paperless's config and break on the next model swap.
+            # One name only: llama-server keeps aliases in a sorted set and
+            # reports the first as the id, so a second name can steal it.
+            - --alias
+            - qwen3-embedding-0.6b
+            # Qwen3-Embedding is trained for last-token pooling. Left
+            # unspecified llama-server takes the GGUF's default, which is not
+            # visible from outside and would silently produce vectors the model
+            # was not trained to emit. Set before anything is indexed: changing
+            # pooling later invalidates every stored embedding.
+            - --pooling
+            - last
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
+            # Paperless chunks documents at 1024 tokens. The defaults are
+            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # clamps both down to 512 rather than run them mismatched — below
+            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # pass; 2048 is deliberately avoided, since that is where this
+            # Vulkan stack fails with ErrorDeviceLost.
+            - --batch-size
+            - "1024"
+            - --ubatch-size
+            - "1024"
+            # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
-            - "2048"
+            - "4096"
             # No browser ever talks to this Service, so the WebUI is dead weight
             # and one more unauthenticated surface.
             - --no-webui
@@ -66,14 +91,24 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # The weights are only ~600 MiB, but that is not what sets the
+          # footprint: four slots at 4096 context with a 1024-token micro-batch,
+          # and Vulkan allocating the iGPU's working set from system RAM where
+          # it counts against this cgroup.
+          #
+          # 4Gi was sized for the earlier 2048 context and 512 micro-batch. Both
+          # were raised to match Paperless's chunking without raising this, and
+          # the pod then survived interactive use and was OOMKilled (137) partway
+          # through the first sustained run — roughly 300 documents in. Indexing
+          # the archive would have hit it on day one.
           resources:
             requests:
               cpu: 250m
-              memory: 2Gi
+              memory: 4Gi
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 10Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -1,97 +0,0 @@
-# paperless-data: CephFS → RBD
-
-## Why
-
-`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
-Deployment mounts it**. Verified 2026-08-20:
-
-```
-$ pods mounting paperless-data
-paperless-68cb8b8744-7hjb8      <- one, and only one
-
-paperless-data     ReadWriteMany  ceph-filesystem
-paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-```
-
-Its contents are single-writer state — the 50 MB Whoosh index,
-`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
-`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
-
-It is on CephFS by default, not by requirement.
-
-## What it costs
-
-Every access goes through the MDS. Measured with **zero CI pipelines running**:
-
-```
-paperless-data client   request_load_avg 22   (all other sessions 0-1)
-MDS over 180s           4,052 client requests  = 22.5/s
-                        2,718 journal events   = 15/s
-                            2 dir commits      <- almost none: not file writes
-```
-
-So it is cap/inode churn from repeated access, not data being written. That is
-most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
-pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
-
-`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
-filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
-depends on it. Cutting the writes at source is correct; cutting the replication
-is not.
-
-## Sequence
-
-Staged deliberately: the PVC lands first and empty, so merging it changes
-nothing. The copy and the repoint are separate.
-
-### Phase 1 — provision (this PR)
-
-`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
-nothing. Inert.
-
-Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
-
-### Phase 2 — copy
-
-The Whoosh index must come across intact. Paperless *can* rebuild it
-(`document_index reindex`), but that reprocesses every document, so copying is
-both faster and less disruptive.
-
-1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
-   and the Whoosh index are both live-written, and copying them under an active
-   writer gives a torn read.
-2. Run a one-off pod that mounts BOTH PVCs and copies:
-   ```
-   cp -a /src/. /dst/
-   ```
-   The RWO target pins the pod to one node; the RWX source will follow.
-3. Verify file count and index size match:
-   ```
-   find /src -type f | wc -l   ==   find /dst -type f | wc -l
-   du -sh /src/index           ==   du -sh /dst/index
-   ```
-
-### Phase 3 — repoint
-
-Change the Deployment's `data` volume to `claimName: paperless-data-block`,
-merge, let it roll.
-
-Verify afterwards:
-- paperless starts, search returns results (proves the Whoosh index survived)
-- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
-  client
-- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
-
-**Do not delete `paperless-data` until search has been confirmed working.** It is
-the rollback, and it is `prune: disabled` so Flux will not remove it.
-
-## Expected effect
-
-~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
-replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
-worth doing, but it is a correctness fix (a single-writer volume does not belong
-on a shared filesystem) more than a large win.
-
-It also removes 22 req/sec of MDS load, which benefits the workloads that
-genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/network-policy.yaml

@@ -57,6 +57,20 @@ spec:
               protocol: UDP
             - port: "53"
               protocol: TCP
+    # The cluster's own LLM, for AI suggestions and RAG embeddings. Scoped to
+    # the two pods and the one port rather than the ai namespace as a whole, so
+    # this grant cannot widen if something else is ever placed there.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-server
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-embed
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # External HTTPS only — Tika model downloads, runtime image
     # fetches, etc. RFC1918 + ULA are excluded from `world` via the
     # CIDR-deny pattern. `world` covers public IPs / DNS-resolvable

kubernetes/apps/paperless/paperless.yaml

@@ -31,12 +31,14 @@ data:
   PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
   PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
   PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE: "true"
-  PAPERLESS_CONSUMER_BARCODE_SCANNER: "ZXING"
+  # No CONSUMER_BARCODE_SCANNER: zxing-cpp is the only backend, so the
+  # setting no longer exists. pyzbar and libzbar are gone from the image.
+  #
   # CephFS does not propagate inotify across nodes — files written
   # via the RWX consume PVC from a pod on a different node than
   # paperless never fire IN_CREATE in paperless's kernel. Polling
   # is the reliable backstop. See wiki: cephfs-inotify-no-cross-node-propagation.
-  PAPERLESS_CONSUMER_POLLING: "10"
+  PAPERLESS_CONSUMER_POLLING_INTERVAL: "10"
   PAPERLESS_REDIS: "redis://redis:6379"
   PAPERLESS_DBENGINE: "postgresql"
   PAPERLESS_DBHOST: "pprl-pg-rw"
@@ -44,6 +46,44 @@ data:
   PAPERLESS_TIKA_ENABLED: "1"
   PAPERLESS_TIKA_ENDPOINT: "http://tika:9998"
   PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://gotenberg:3000"
+  # AI suggestions against the cluster's own LLM (kubernetes/apps/llama-server).
+  # Nothing leaves the cluster: both endpoints are ClusterIP Services, so
+  # document text is never sent to a third party and there is no API key to
+  # rotate — llama-server has no authentication, and the ai namespace's network
+  # policy is what restricts who may reach it.
+  #
+  # CAVEAT, and it is a real one: paperless reads these from an
+  # ApplicationConfiguration row in the database and falls back to the
+  # environment only when that row is empty. A value set in the admin UI
+  # therefore overrides this ConfigMap silently. Configure AI in ONE place —
+  # here — and leave those fields untouched in the UI.
+  PAPERLESS_AI_ENABLED: "true"
+  PAPERLESS_AI_LLM_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_ENDPOINT: "http://llama-server.ai.svc.cluster.local:8080/v1"
+  # Must match --alias in kubernetes/apps/llama-server/deployment.yaml. The
+  # server ignores the field on the wire, but paperless sends it and the value
+  # is what appears in its logs.
+  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
+  # ClusterIP addresses are private, and paperless blocks non-public endpoints
+  # unless this is set. Default is already true; explicit because the whole
+  # integration silently stops working if it ever flips.
+  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
+  # Under llama-server's own 16384, leaving room for the response.
+  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
+  # The 120s default is too tight at ~11 tok/s: a classification is roughly
+  # 2-4k prompt tokens plus a second round trip for localisation.
+  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "300"
+  # Matched against Django's settings.LANGUAGES, so the region is required —
+  # "de" alone does not resolve. Without it the model returns German tags but
+  # English document types, which was reproducible against this endpoint.
+  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
+  # RAG embeddings from llama-embed. Its Qwen3-Embedding covers 100+ languages,
+  # where paperless's default all-MiniLM-L6-v2 is documented as suited to
+  # primarily English archives — the better fit for a German corpus.
+  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -56,6 +96,13 @@ metadata:
     # Reloader restarts paperless when paperless-secrets changes (e.g. after a
     # secret-key rotation). See kubernetes/infrastructure/openbao/rotation/paperless.yaml.
     secret.reloader.stakater.com/reload: paperless-secrets
+    # And when paperless-config changes. The ConfigMap is consumed with
+    # envFrom, and Kubernetes does not restart a pod when a ConfigMap it reads
+    # that way is updated — the running container keeps the environment it
+    # started with. Without this, a settings change applies cleanly in git and
+    # in the cluster while the application never sees it, which is how the AI
+    # settings landed with no effect at all.
+    configmap.reloader.stakater.com/reload: paperless-config
 spec:
   replicas: 1
   strategy:
@@ -120,7 +167,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -172,6 +219,25 @@ spec:
           # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
           # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
           # means giving these probes an explicit Host header.
+
+          # The container does not serve until its init sequence finishes:
+          # s6 makes svc-webserver depend on init-migrations and
+          # init-search-index. On the first 3.x start those are the SHA256
+          # checksum backfill, which opens and reads every original and every
+          # archive file, and a full Tantivy index build, since the Whoosh
+          # index cannot be converted. Both scale with the corpus.
+          #
+          # startupProbe holds liveness and readiness off until the webserver
+          # answers, giving that boot 30 * 20s = 10 minutes. Without it the
+          # 120s liveness delay plus three 30s periods kills the pod 210s in,
+          # part-way through a migration, and the next boot starts over.
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 30
           livenessProbe:
             httpGet:
               path: /accounts/login/
@@ -220,22 +286,16 @@ spec:
             seccompProfile:
               type: RuntimeDefault
       volumes:
-        # paperless-data-block (RWO, ceph-block), NOT paperless-data
-        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
-        # and its contents are single-writer state — the Whoosh index,
-        # classification model, celerybeat schedule, logs. Being on CephFS meant
-        # every access went through the MDS: 22 requests/sec sustained with zero
-        # CI running, ~15 journal events/sec against 2 directory commits per
-        # 180s, i.e. cap/inode churn rather than file writes. That was most of
-        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
+        # `data` is RWO on ceph-block, not RWX on ceph-filesystem. Only this
+        # Deployment mounts it and its contents are single-writer state — the
+        # search index, classification model, celerybeat schedule, logs. On
+        # CephFS every access went through the MDS: 22 requests/sec sustained
+        # with zero CI running, ~15 journal events/sec against 2 directory
+        # commits per 180s, i.e. cap/inode churn rather than file writes. That
+        # was most of the constant 7-8 GB/day on cephfs-metadata, a size 3 pool.
         #
         # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
         # those from another node, so they genuinely need RWX.
-        #
-        # paperless-data is retained (prune: disabled) as the rollback. Do not
-        # delete it until search has been confirmed working — that is what
-        # proves the Whoosh index survived the copy, not merely that paperless
-        # starts. Migration record: MIGRATION-data-to-block.md.
         - name: data
           persistentVolumeClaim:
             claimName: paperless-data-block

kubernetes/apps/paperless/pvcs.yaml

@@ -3,33 +3,18 @@
 # (e.g. when the seed initContainer or the Garage migration creds are
 # removed once we're done with the migration phase). Velero's
 # bootstrap-restore then owns their contents.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-data
-  namespace: paperless
-  annotations:
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 10Gi
----
-# Replacement for paperless-data, on RBD instead of CephFS.
+
+# paperless's single-writer state, on RBD rather than CephFS.
 #
-# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
-# pod ever mounts it — the paperless Deployment itself. Its contents are all
-# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
-# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
-# and `media` genuinely need RWX, because the FTP scanner writes to those from a
-# different node.
+# WHY RWO: only ONE pod mounts it — the paperless Deployment itself. Its
+# contents are all single-writer state: the Tantivy search index,
+# classification_model.pickle, celerybeat-schedule.db, logs, migration_lock.
+# Nothing shares it. Only `consume` and `media` genuinely need RWX, because the
+# FTP scanner writes to those from a different node.
 #
-# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
-# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
-# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# WHY NOT CEPHFS: every access goes through the MDS. With zero CI running, this
+# volume's client sustained 22 MDS requests/sec, and the MDS logged ~15 journal
+# events/sec against only 2 directory commits per
 # 180s — i.e. cap and inode churn from repeated access, not file writes. That is
 # most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
 # pool, so ~21-24 GB/day actually hits the drives.
@@ -38,10 +23,6 @@ spec:
 # the metadata pool destroys the whole filesystem, not just the objects on one
 # OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
 # is the right fix; reducing the replication is not.
-#
-# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
-# and the deployment repoint are a separate, staged step — see
-# MIGRATION-data-to-block.md. Merging this alone is inert.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/infrastructure/image-automation/paperless-ngx.yaml

@@ -7,7 +7,7 @@ spec:
   image: ghcr.io/paperless-ngx/paperless-ngx
   interval: 1h
 ---
-# Allow only patch + minor bumps within the 2.x line. A 3.0 release
+# Allow only patch + minor bumps within the 3.x line. A 4.0 release
 # requires manual policy widening — major upgrades typically need a
 # migration plan and shouldn't land via auto-merge.
 apiVersion: image.toolkit.fluxcd.io/v1
@@ -20,4 +20,4 @@ spec:
     name: paperless-ngx
   policy:
     semver:
-      range: "2.x"
+      range: "3.x"

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -24,6 +24,16 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
+    # Paperless calls both models for AI suggestions and RAG embeddings. The
+    # matching egress grant on its side is scoped to these two pods on :8080.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless
+            app.kubernetes.io/name: paperless
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring