Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a003df0ef8 | ||
|
|
f0e48b226a | ||
|
|
669ef1c5ee | ||
|
|
11cd02ec44 |
@@ -1,97 +0,0 @@
|
||||
# paperless-data: CephFS → RBD
|
||||
|
||||
## Why
|
||||
|
||||
`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
|
||||
Deployment mounts it**. Verified 2026-08-20:
|
||||
|
||||
```
|
||||
$ pods mounting paperless-data
|
||||
paperless-68cb8b8744-7hjb8 <- one, and only one
|
||||
|
||||
paperless-data ReadWriteMany ceph-filesystem
|
||||
paperless-media ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner)
|
||||
paperless-consume ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner)
|
||||
```
|
||||
|
||||
Its contents are single-writer state — the 50 MB Whoosh index,
|
||||
`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
|
||||
`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
|
||||
|
||||
It is on CephFS by default, not by requirement.
|
||||
|
||||
## What it costs
|
||||
|
||||
Every access goes through the MDS. Measured with **zero CI pipelines running**:
|
||||
|
||||
```
|
||||
paperless-data client request_load_avg 22 (all other sessions 0-1)
|
||||
MDS over 180s 4,052 client requests = 22.5/s
|
||||
2,718 journal events = 15/s
|
||||
2 dir commits <- almost none: not file writes
|
||||
```
|
||||
|
||||
So it is cap/inode churn from repeated access, not data being written. That is
|
||||
most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
|
||||
pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
|
||||
|
||||
`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
|
||||
filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
|
||||
depends on it. Cutting the writes at source is correct; cutting the replication
|
||||
is not.
|
||||
|
||||
## Sequence
|
||||
|
||||
Staged deliberately: the PVC lands first and empty, so merging it changes
|
||||
nothing. The copy and the repoint are separate.
|
||||
|
||||
### Phase 1 — provision (this PR)
|
||||
|
||||
`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
|
||||
nothing. Inert.
|
||||
|
||||
Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
|
||||
|
||||
### Phase 2 — copy
|
||||
|
||||
The Whoosh index must come across intact. Paperless *can* rebuild it
|
||||
(`document_index reindex`), but that reprocesses every document, so copying is
|
||||
both faster and less disruptive.
|
||||
|
||||
1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
|
||||
and the Whoosh index are both live-written, and copying them under an active
|
||||
writer gives a torn read.
|
||||
2. Run a one-off pod that mounts BOTH PVCs and copies:
|
||||
```
|
||||
cp -a /src/. /dst/
|
||||
```
|
||||
The RWO target pins the pod to one node; the RWX source will follow.
|
||||
3. Verify file count and index size match:
|
||||
```
|
||||
find /src -type f | wc -l == find /dst -type f | wc -l
|
||||
du -sh /src/index == du -sh /dst/index
|
||||
```
|
||||
|
||||
### Phase 3 — repoint
|
||||
|
||||
Change the Deployment's `data` volume to `claimName: paperless-data-block`,
|
||||
merge, let it roll.
|
||||
|
||||
Verify afterwards:
|
||||
- paperless starts, search returns results (proves the Whoosh index survived)
|
||||
- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
|
||||
client
|
||||
- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
|
||||
|
||||
**Do not delete `paperless-data` until search has been confirmed working.** It is
|
||||
the rollback, and it is `prune: disabled` so Flux will not remove it.
|
||||
|
||||
## Expected effect
|
||||
|
||||
~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
|
||||
replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
|
||||
worth doing, but it is a correctness fix (a single-writer volume does not belong
|
||||
on a shared filesystem) more than a large win.
|
||||
|
||||
It also removes 22 req/sec of MDS load, which benefits the workloads that
|
||||
genuinely need CephFS — Woodpecker CI workspaces share the same MDS.
|
||||
@@ -96,6 +96,13 @@ metadata:
|
||||
# Reloader restarts paperless when paperless-secrets changes (e.g. after a
|
||||
# secret-key rotation). See kubernetes/infrastructure/openbao/rotation/paperless.yaml.
|
||||
secret.reloader.stakater.com/reload: paperless-secrets
|
||||
# And when paperless-config changes. The ConfigMap is consumed with
|
||||
# envFrom, and Kubernetes does not restart a pod when a ConfigMap it reads
|
||||
# that way is updated — the running container keeps the environment it
|
||||
# started with. Without this, a settings change applies cleanly in git and
|
||||
# in the cluster while the application never sees it, which is how the AI
|
||||
# settings landed with no effect at all.
|
||||
configmap.reloader.stakater.com/reload: paperless-config
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
@@ -279,22 +286,16 @@ spec:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumes:
|
||||
# paperless-data-block (RWO, ceph-block), NOT paperless-data
|
||||
# (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
|
||||
# and its contents are single-writer state — the Whoosh index,
|
||||
# classification model, celerybeat schedule, logs. Being on CephFS meant
|
||||
# every access went through the MDS: 22 requests/sec sustained with zero
|
||||
# CI running, ~15 journal events/sec against 2 directory commits per
|
||||
# 180s, i.e. cap/inode churn rather than file writes. That was most of
|
||||
# the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
|
||||
# `data` is RWO on ceph-block, not RWX on ceph-filesystem. Only this
|
||||
# Deployment mounts it and its contents are single-writer state — the
|
||||
# search index, classification model, celerybeat schedule, logs. On
|
||||
# CephFS every access went through the MDS: 22 requests/sec sustained
|
||||
# with zero CI running, ~15 journal events/sec against 2 directory
|
||||
# commits per 180s, i.e. cap/inode churn rather than file writes. That
|
||||
# was most of the constant 7-8 GB/day on cephfs-metadata, a size 3 pool.
|
||||
#
|
||||
# `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
|
||||
# those from another node, so they genuinely need RWX.
|
||||
#
|
||||
# paperless-data is retained (prune: disabled) as the rollback. Do not
|
||||
# delete it until search has been confirmed working — that is what
|
||||
# proves the Whoosh index survived the copy, not merely that paperless
|
||||
# starts. Migration record: MIGRATION-data-to-block.md.
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: paperless-data-block
|
||||
|
||||
@@ -3,33 +3,18 @@
|
||||
# (e.g. when the seed initContainer or the Garage migration creds are
|
||||
# removed once we're done with the migration phase). Velero's
|
||||
# bootstrap-restore then owns their contents.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: paperless-data
|
||||
namespace: paperless
|
||||
annotations:
|
||||
kustomize.toolkit.fluxcd.io/prune: disabled
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ceph-filesystem
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
# Replacement for paperless-data, on RBD instead of CephFS.
|
||||
|
||||
# paperless's single-writer state, on RBD rather than CephFS.
|
||||
#
|
||||
# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
|
||||
# pod ever mounts it — the paperless Deployment itself. Its contents are all
|
||||
# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
|
||||
# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
|
||||
# and `media` genuinely need RWX, because the FTP scanner writes to those from a
|
||||
# different node.
|
||||
# WHY RWO: only ONE pod mounts it — the paperless Deployment itself. Its
|
||||
# contents are all single-writer state: the Tantivy search index,
|
||||
# classification_model.pickle, celerybeat-schedule.db, logs, migration_lock.
|
||||
# Nothing shares it. Only `consume` and `media` genuinely need RWX, because the
|
||||
# FTP scanner writes to those from a different node.
|
||||
#
|
||||
# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
|
||||
# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
|
||||
# and the MDS logged ~15 journal events/sec against only 2 directory commits per
|
||||
# WHY NOT CEPHFS: every access goes through the MDS. With zero CI running, this
|
||||
# volume's client sustained 22 MDS requests/sec, and the MDS logged ~15 journal
|
||||
# events/sec against only 2 directory commits per
|
||||
# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
|
||||
# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
|
||||
# pool, so ~21-24 GB/day actually hits the drives.
|
||||
@@ -38,10 +23,6 @@ spec:
|
||||
# the metadata pool destroys the whole filesystem, not just the objects on one
|
||||
# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
|
||||
# is the right fix; reducing the replication is not.
|
||||
#
|
||||
# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
|
||||
# and the deployment repoint are a separate, staged step — see
|
||||
# MIGRATION-data-to-block.md. Merging this alone is inert.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
|
||||
Reference in New Issue
Block a user