From d2d06d853234428f4990a074ed6aded50ca42592 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 20 Aug 2026 07:18:31 +0200 Subject: [PATCH] feat(paperless): provision RBD volume to move data off CephFS (phase 1) paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod mounts it -- the paperless Deployment itself. Its contents are all single-writer state: the 50 MB Whoosh index, classification_model.pickle, celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only `consume` and `media` genuinely need RWX, because the ftp-scanner writes to those from another node. It is on CephFS by default, not by requirement, and every access goes through the MDS. Measured 2026-08-20 with ZERO CI pipelines running: paperless-data client request_load_avg 22 (all other sessions 0-1) MDS over 180s 4,052 client requests = 22.5/s 2,718 journal events = 15/s 2 dir commits <- not file writes Cap and inode churn from repeated access, not data being written. That accounts for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is size 3, roughly 21-24 GB/day reaches the drives. cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0 rides on it. Cutting writes at source is the fix; cutting replication is not. PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data copy and the deployment repoint are separate steps, documented in MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the Whoosh index and celerybeat schedule are both live-written, so copying under an active writer gives a torn read). Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so about 5%. This is a correctness fix -- a single-writer volume does not belong on a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS load, which benefits Woodpecker CI workspaces sharing the same MDS. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) --- .../apps/paperless/MIGRATION-data-to-block.md | 97 +++++++++++++++++++ kubernetes/apps/paperless/pvcs.yaml | 39 ++++++++ 2 files changed, 136 insertions(+) create mode 100644 kubernetes/apps/paperless/MIGRATION-data-to-block.md diff --git a/kubernetes/apps/paperless/MIGRATION-data-to-block.md b/kubernetes/apps/paperless/MIGRATION-data-to-block.md new file mode 100644 index 00000000..0bfdccfc --- /dev/null +++ b/kubernetes/apps/paperless/MIGRATION-data-to-block.md @@ -0,0 +1,97 @@ +# paperless-data: CephFS → RBD + +## Why + +`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless +Deployment mounts it**. Verified 2026-08-20: + +``` +$ pods mounting paperless-data +paperless-68cb8b8744-7hjb8 <- one, and only one + +paperless-data ReadWriteMany ceph-filesystem +paperless-media ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner) +paperless-consume ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner) +``` + +Its contents are single-writer state — the 50 MB Whoosh index, +`classification_model.pickle`, `celerybeat-schedule.db`, `log/`, +`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them. + +It is on CephFS by default, not by requirement. + +## What it costs + +Every access goes through the MDS. Measured with **zero CI pipelines running**: + +``` +paperless-data client request_load_avg 22 (all other sessions 0-1) +MDS over 180s 4,052 client requests = 22.5/s + 2,718 journal events = 15/s + 2 dir commits <- almost none: not file writes +``` + +So it is cap/inode churn from repeated access, not data being written. That is +most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that +pool is `size: 3`, roughly **21-24 GB/day** reaches the drives. + +`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire +filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0` +depends on it. Cutting the writes at source is correct; cutting the replication +is not. + +## Sequence + +Staged deliberately: the PVC lands first and empty, so merging it changes +nothing. The copy and the repoint are separate. + +### Phase 1 — provision (this PR) + +`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by +nothing. Inert. + +Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound. + +### Phase 2 — copy + +The Whoosh index must come across intact. Paperless *can* rebuild it +(`document_index reindex`), but that reprocesses every document, so copying is +both faster and less disruptive. + +1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db` + and the Whoosh index are both live-written, and copying them under an active + writer gives a torn read. +2. Run a one-off pod that mounts BOTH PVCs and copies: + ``` + cp -a /src/. /dst/ + ``` + The RWO target pins the pod to one node; the RWX source will follow. +3. Verify file count and index size match: + ``` + find /src -type f | wc -l == find /dst -type f | wc -l + du -sh /src/index == du -sh /dst/index + ``` + +### Phase 3 — repoint + +Change the Deployment's `data` volume to `claimName: paperless-data-block`, +merge, let it roll. + +Verify afterwards: +- paperless starts, search returns results (proves the Whoosh index survived) +- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data` + client +- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline + +**Do not delete `paperless-data` until search has been confirmed working.** It is +the rollback, and it is `prune: disabled` so Flux will not remove it. + +## Expected effect + +~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once +replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% — +worth doing, but it is a correctness fix (a single-writer volume does not belong +on a shared filesystem) more than a large win. + +It also removes 22 req/sec of MDS load, which benefits the workloads that +genuinely need CephFS — Woodpecker CI workspaces share the same MDS. diff --git a/kubernetes/apps/paperless/pvcs.yaml b/kubernetes/apps/paperless/pvcs.yaml index ac3d1d33..7b56c326 100644 --- a/kubernetes/apps/paperless/pvcs.yaml +++ b/kubernetes/apps/paperless/pvcs.yaml @@ -18,6 +18,45 @@ spec: requests: storage: 10Gi --- +# Replacement for paperless-data, on RBD instead of CephFS. +# +# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE +# pod ever mounts it — the paperless Deployment itself. Its contents are all +# single-writer state: the 50 MB Whoosh index, classification_model.pickle, +# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume` +# and `media` genuinely need RWX, because the FTP scanner writes to those from a +# different node. +# +# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with +# zero CI running, this volume's client was doing 22 MDS requests/sec sustained, +# and the MDS logged ~15 journal events/sec against only 2 directory commits per +# 180s — i.e. cap and inode churn from repeated access, not file writes. That is +# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3 +# pool, so ~21-24 GB/day actually hits the drives. +# +# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing +# the metadata pool destroys the whole filesystem, not just the objects on one +# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source +# is the right fix; reducing the replication is not. +# +# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy +# and the deployment repoint are a separate, staged step — see +# MIGRATION-data-to-block.md. Merging this alone is inert. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: paperless-data-block + namespace: paperless + annotations: + kustomize.toolkit.fluxcd.io/prune: disabled +spec: + accessModes: + - ReadWriteOnce + storageClassName: ceph-block + resources: + requests: + storage: 10Gi +--- apiVersion: v1 kind: PersistentVolumeClaim metadata: -- 2.54.0