refactor(paperless): retire paperless-data, the CephFS predecessor volume

The `data` volume moved from paperless-data (RWX, ceph-filesystem) to
paperless-data-block (RWO, ceph-block) on 2026-08-28. paperless-data was
kept as the rollback, on the condition that search be confirmed working
first — the point being to prove the Whoosh index survived the copy, not
merely that paperless started.

Both halves of that condition are now settled. index_optimize succeeded on
every night between the repoint and 2026-09-01, and that task opens and
rewrites the index rather than just reading it. Then the 3.1.2 upgrade
replaced Whoosh with Tantivy outright, so the index paperless-data was
preserving no longer exists in a form anything can read.

Removed: the PVC, the migration record, and the comments that described the
volume as in-flight. The rationale for RWO-on-RBD stays where it is useful,
on the PVC and on the Deployment's volume list, since it is the reason not
to move this back onto a shared filesystem later.

paperless-data carries kustomize.toolkit.fluxcd.io/prune: disabled, so this
commit does not delete anything by itself. The PVC, its Retain-policy PV and
the backing CephFS subvolume have to be removed by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -1,97 +0,0 @@
-# paperless-data: CephFS → RBD
-
-## Why
-
-`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
-Deployment mounts it**. Verified 2026-08-20:
-
-```
-$ pods mounting paperless-data
-paperless-68cb8b8744-7hjb8      <- one, and only one
-
-paperless-data     ReadWriteMany  ceph-filesystem
-paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-```
-
-Its contents are single-writer state — the 50 MB Whoosh index,
-`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
-`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
-
-It is on CephFS by default, not by requirement.
-
-## What it costs
-
-Every access goes through the MDS. Measured with **zero CI pipelines running**:
-
-```
-paperless-data client   request_load_avg 22   (all other sessions 0-1)
-MDS over 180s           4,052 client requests  = 22.5/s
-                        2,718 journal events   = 15/s
-                            2 dir commits      <- almost none: not file writes
-```
-
-So it is cap/inode churn from repeated access, not data being written. That is
-most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
-pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
-
-`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
-filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
-depends on it. Cutting the writes at source is correct; cutting the replication
-is not.
-
-## Sequence
-
-Staged deliberately: the PVC lands first and empty, so merging it changes
-nothing. The copy and the repoint are separate.
-
-### Phase 1 — provision (this PR)
-
-`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
-nothing. Inert.
-
-Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
-
-### Phase 2 — copy
-
-The Whoosh index must come across intact. Paperless *can* rebuild it
-(`document_index reindex`), but that reprocesses every document, so copying is
-both faster and less disruptive.
-
-1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
-   and the Whoosh index are both live-written, and copying them under an active
-   writer gives a torn read.
-2. Run a one-off pod that mounts BOTH PVCs and copies:
-   ```
-   cp -a /src/. /dst/
-   ```
-   The RWO target pins the pod to one node; the RWX source will follow.
-3. Verify file count and index size match:
-   ```
-   find /src -type f | wc -l   ==   find /dst -type f | wc -l
-   du -sh /src/index           ==   du -sh /dst/index
-   ```
-
-### Phase 3 — repoint
-
-Change the Deployment's `data` volume to `claimName: paperless-data-block`,
-merge, let it roll.
-
-Verify afterwards:
-- paperless starts, search returns results (proves the Whoosh index survived)
-- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
-  client
-- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
-
-**Do not delete `paperless-data` until search has been confirmed working.** It is
-the rollback, and it is `prune: disabled` so Flux will not remove it.
-
-## Expected effect
-
-~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
-replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
-worth doing, but it is a correctness fix (a single-writer volume does not belong
-on a shared filesystem) more than a large win.
-
-It also removes 22 req/sec of MDS load, which benefits the workloads that
-genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/paperless.yaml

@@ -31,12 +31,14 @@ data:
   PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
   PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
   PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE: "true"
-  PAPERLESS_CONSUMER_BARCODE_SCANNER: "ZXING"
+  # No CONSUMER_BARCODE_SCANNER: zxing-cpp is the only backend, so the
+  # setting no longer exists. pyzbar and libzbar are gone from the image.
+  #
   # CephFS does not propagate inotify across nodes — files written
   # via the RWX consume PVC from a pod on a different node than
   # paperless never fire IN_CREATE in paperless's kernel. Polling
   # is the reliable backstop. See wiki: cephfs-inotify-no-cross-node-propagation.
-  PAPERLESS_CONSUMER_POLLING: "10"
+  PAPERLESS_CONSUMER_POLLING_INTERVAL: "10"
   PAPERLESS_REDIS: "redis://redis:6379"
   PAPERLESS_DBENGINE: "postgresql"
   PAPERLESS_DBHOST: "pprl-pg-rw"
@@ -120,7 +122,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -172,6 +174,25 @@ spec:
           # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
           # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
           # means giving these probes an explicit Host header.
+
+          # The container does not serve until its init sequence finishes:
+          # s6 makes svc-webserver depend on init-migrations and
+          # init-search-index. On the first 3.x start those are the SHA256
+          # checksum backfill, which opens and reads every original and every
+          # archive file, and a full Tantivy index build, since the Whoosh
+          # index cannot be converted. Both scale with the corpus.
+          #
+          # startupProbe holds liveness and readiness off until the webserver
+          # answers, giving that boot 30 * 20s = 10 minutes. Without it the
+          # 120s liveness delay plus three 30s periods kills the pod 210s in,
+          # part-way through a migration, and the next boot starts over.
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 30
           livenessProbe:
             httpGet:
               path: /accounts/login/
@@ -220,22 +241,16 @@ spec:
             seccompProfile:
               type: RuntimeDefault
       volumes:
-        # paperless-data-block (RWO, ceph-block), NOT paperless-data
-        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
-        # and its contents are single-writer state — the Whoosh index,
-        # classification model, celerybeat schedule, logs. Being on CephFS meant
-        # every access went through the MDS: 22 requests/sec sustained with zero
-        # CI running, ~15 journal events/sec against 2 directory commits per
-        # 180s, i.e. cap/inode churn rather than file writes. That was most of
-        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
+        # `data` is RWO on ceph-block, not RWX on ceph-filesystem. Only this
+        # Deployment mounts it and its contents are single-writer state — the
+        # search index, classification model, celerybeat schedule, logs. On
+        # CephFS every access went through the MDS: 22 requests/sec sustained
+        # with zero CI running, ~15 journal events/sec against 2 directory
+        # commits per 180s, i.e. cap/inode churn rather than file writes. That
+        # was most of the constant 7-8 GB/day on cephfs-metadata, a size 3 pool.
         #
         # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
         # those from another node, so they genuinely need RWX.
-        #
-        # paperless-data is retained (prune: disabled) as the rollback. Do not
-        # delete it until search has been confirmed working — that is what
-        # proves the Whoosh index survived the copy, not merely that paperless
-        # starts. Migration record: MIGRATION-data-to-block.md.
         - name: data
           persistentVolumeClaim:
             claimName: paperless-data-block

kubernetes/apps/paperless/pvcs.yaml

@@ -3,33 +3,18 @@
 # (e.g. when the seed initContainer or the Garage migration creds are
 # removed once we're done with the migration phase). Velero's
 # bootstrap-restore then owns their contents.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-data
-  namespace: paperless
-  annotations:
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 10Gi
----
-# Replacement for paperless-data, on RBD instead of CephFS.
+
+# paperless's single-writer state, on RBD rather than CephFS.
 #
-# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
-# pod ever mounts it — the paperless Deployment itself. Its contents are all
-# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
-# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
-# and `media` genuinely need RWX, because the FTP scanner writes to those from a
-# different node.
+# WHY RWO: only ONE pod mounts it — the paperless Deployment itself. Its
+# contents are all single-writer state: the Tantivy search index,
+# classification_model.pickle, celerybeat-schedule.db, logs, migration_lock.
+# Nothing shares it. Only `consume` and `media` genuinely need RWX, because the
+# FTP scanner writes to those from a different node.
 #
-# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
-# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
-# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# WHY NOT CEPHFS: every access goes through the MDS. With zero CI running, this
+# volume's client sustained 22 MDS requests/sec, and the MDS logged ~15 journal
+# events/sec against only 2 directory commits per
 # 180s — i.e. cap and inode churn from repeated access, not file writes. That is
 # most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
 # pool, so ~21-24 GB/day actually hits the drives.
@@ -38,10 +23,6 @@ spec:
 # the metadata pool destroys the whole filesystem, not just the objects on one
 # OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
 # is the right fix; reducing the replication is not.
-#
-# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
-# and the deployment repoint are a separate, staged step — see
-# MIGRATION-data-to-block.md. Merging this alone is inert.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/infrastructure/image-automation/paperless-ngx.yaml

@@ -7,7 +7,7 @@ spec:
   image: ghcr.io/paperless-ngx/paperless-ngx
   interval: 1h
 ---
-# Allow only patch + minor bumps within the 2.x line. A 3.0 release
+# Allow only patch + minor bumps within the 3.x line. A 4.0 release
 # requires manual policy widening — major upgrades typically need a
 # migration plan and shouldn't land via auto-merge.
 apiVersion: image.toolkit.fluxcd.io/v1
@@ -20,4 +20,4 @@ spec:
     name: paperless-ngx
   policy:
     semver:
-      range: "2.x"
+      range: "3.x"