feat(paperless): repoint data volume to RBD (phase 3) #534

2026-08-20T05:44:24Z

claude-bot commented

2026-08-20 05:44:24 +00:00

Refs #434. Phase 3 of #533. Paperless is currently scaled to 0 pending this merge.

Copy is done and verified

Performed with paperless stopped, so the Whoosh index and celerybeat schedule were quiescent:

src files: 75
dst files: 75
IDENTICAL: name, size and mtime match for every file
index segments: src 3, dst 3

mtime parity matters specifically for the Whoosh segments, which carry their own consistency expectations — cp -a preserved them. (The can not preserve lines in the copy log were for the mount root /dst/., not the files.)

What changes

Only the data volume moves. Verified in the render:

claimName: paperless-consume       <- stays CephFS (ftp-scanner, needs RWX)
claimName: paperless-data-block    <- CHANGED
claimName: paperless-media         <- stays CephFS (ftp-scanner, needs RWX)

Rollback

paperless-data is retained and prune: disabled. Do not delete it until search has been confirmed returning results — that is what proves the index survived, not merely that paperless starts.

Note on timing

Paperless is down right now because the copy required a quiescent source. If you would rather not merge immediately, tell me and I will scale it back up on the old volume — the cutover can then happen whenever you merge. The only cost of that path is that documents consumed in the interval would land in the old index and need a document_index reindex after cutover.

After merge I will verify

paperless starts and search returns results
the paperless-data MDS session is gone from ceph tell mds.cephfs-b session ls
rate(ceph_pool_wr_bytes{pool_id="3"}) actually drops from its 7-8 GB/day baseline

That last one measured rather than assumed — I have been wrong three times in this thread about the metadata source, and the fix is only real if the number moves.

🤖 Generated with Claude Code

Refs #434. Phase 3 of #533. **Paperless is currently scaled to 0 pending this merge.** ## Copy is done and verified Performed with paperless stopped, so the Whoosh index and celerybeat schedule were quiescent: ``` src files: 75 dst files: 75 IDENTICAL: name, size and mtime match for every file index segments: src 3, dst 3 ``` mtime parity matters specifically for the Whoosh segments, which carry their own consistency expectations — `cp -a` preserved them. (The `can not preserve` lines in the copy log were for the mount root `/dst/.`, not the files.) ## What changes Only the `data` volume moves. Verified in the render: ``` claimName: paperless-consume <- stays CephFS (ftp-scanner, needs RWX) claimName: paperless-data-block <- CHANGED claimName: paperless-media <- stays CephFS (ftp-scanner, needs RWX) ``` ## Rollback `paperless-data` is retained and `prune: disabled`. **Do not delete it until search has been confirmed returning results** — that is what proves the index survived, not merely that paperless starts. ## Note on timing Paperless is down right now because the copy required a quiescent source. If you would rather not merge immediately, tell me and I will scale it back up on the old volume — the cutover can then happen whenever you merge. The only cost of that path is that documents consumed in the interval would land in the old index and need a `document_index reindex` after cutover. ## After merge I will verify - paperless starts and **search returns results** - the `paperless-data` MDS session is gone from `ceph tell mds.cephfs-b session ls` - `rate(ceph_pool_wr_bytes{pool_id="3"})` actually drops from its 7-8 GB/day baseline That last one measured rather than assumed — I have been wrong three times in this thread about the metadata source, and the fix is only real if the number moves. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-20 05:44:24 +00:00

feat(paperless): repoint data volume to RBD (phase 3)

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

3a279d2dcd

Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.

The copy is already done and verified byte-for-byte with paperless stopped:

  src files: 75
  dst files: 75
  IDENTICAL: name, size and mtime match for every file
  index segments: src 3, dst 3

mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)

Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.

cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.

paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.

Refs #434

Co-Authored-By: Claude Bot <noreply@anthropic.com>

k.renner merged commit eafd99faca into master

2026-08-20 05:45:26 +00:00

k.renner referenced this issue from a commit

2026-08-20 05:45:27 +00:00

Merge pull request 'feat(paperless): repoint data volume to RBD (phase 3)' (#534) from feat-paperless-data-repoint into master

review-bot reviewed

2026-08-20 05:47:34 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 3a279d2dcdc655d9ab8fbc92fd1baa8d3a03336f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Phase 3 of the paperless data-volume migration: repoints the Deployment's data
volume from paperless-data (RWX, ceph-filesystem) to paperless-data-block
(RWO, ceph-block), which was already provisioned empty in an earlier phase
(pvcs.yaml). The diff itself is a one-line claimName change plus a thorough
explanatory comment; consume and media correctly stay on CephFS since the
ftp-scanner needs RWX for those. No blocking issues found — one thing worth
verifying before/after cutover, noted below.

🟡 Suggestions

Confirm Velero can still back up paperless-data-block. It's RWO
ceph-block and will be mounted RW by the single paperless pod on one node —
exactly the shape that two other PVCs in this repo were excluded from backup
for: redis (kubernetes/apps/paperless/pvcs.yaml:96-100) and
music-assistant-data (kubernetes/apps/music-assistant/pvcs.yaml:10-15),
both carrying velero.io/exclude-from-backup: "true" because of a Velero
CSI snapshot-exposer failure ("cannot mount /dev/rbdN read-only") under that
same RWO/mounted-on-same-node condition. paperless-data-block
(kubernetes/apps/paperless/pvcs.yaml:45-58) carries no such label, and the
paperless-daily Velero Schedule (kubernetes/infrastructure/post-crds/velero-schedules.yaml:29-49)
includes all PVCs in the namespace by default with snapshotMoveData: true.
It's possible this is already moot — node-agent-config.yaml:14-26 shows
ceph-block backups now use the default RW-clone path (no readOnly), which
was adopted specifically to avoid RO-mount ext4 issues — but that fix was
motivated by a different symptom (ENOTCLEAN on a dirty journal), not
explicitly the same-node RW-mount conflict described in the redis/music-assistant
comments, and unlike redis (cache) or music-assistant (reseedable from an LXC),
this PVC now holds paperless's only live copy of the Whoosh index once
paperless-data is eventually deleted. Worth explicitly checking that the
next paperless-daily backup run succeeds for paperless-data-block rather
than assuming it does, and adding velero.io/exclude-from-backup (with
something to reintroduce coverage later, e.g. a fs-backup annotation) if it
doesn't.
The post-merge verification list in the PR body (paperless starts, search
works, MDS session gone, metadata write rate drops) doesn't include a backup
check — worth adding once the point above is resolved either way.

Tests

No automated tests apply to this kind of infra/manifest change; verification is
manual and operational, and the PR body lays out a reasonable post-merge check
sequence (search results, MDS session, write-rate metric). The pre-merge copy
was verified byte-for-byte (file count, mtime, index segment count) with
paperless quiesced, which is the right way to validate a Whoosh-index-bearing
copy.

Recommendation

Looks reasonable — no blocking issues; the Velero backup-coverage question
above is worth a quick check but shouldn't hold up this merge given the
retained paperless-data rollback.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `3a279d2dcdc655d9ab8fbc92fd1baa8d3a03336f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Phase 3 of the paperless data-volume migration: repoints the Deployment's `data` volume from `paperless-data` (RWX, ceph-filesystem) to `paperless-data-block` (RWO, ceph-block), which was already provisioned empty in an earlier phase (`pvcs.yaml`). The diff itself is a one-line `claimName` change plus a thorough explanatory comment; `consume` and `media` correctly stay on CephFS since the ftp-scanner needs RWX for those. No blocking issues found — one thing worth verifying before/after cutover, noted below. ## 🟡 Suggestions - **Confirm Velero can still back up `paperless-data-block`.** It's RWO `ceph-block` and will be mounted RW by the single paperless pod on one node — exactly the shape that two other PVCs in this repo were excluded from backup for: `redis` (`kubernetes/apps/paperless/pvcs.yaml:96-100`) and `music-assistant-data` (`kubernetes/apps/music-assistant/pvcs.yaml:10-15`), both carrying `velero.io/exclude-from-backup: "true"` because of a Velero CSI snapshot-exposer failure ("cannot mount /dev/rbdN read-only") under that same RWO/mounted-on-same-node condition. `paperless-data-block` (`kubernetes/apps/paperless/pvcs.yaml:45-58`) carries no such label, and the `paperless-daily` Velero Schedule (`kubernetes/infrastructure/post-crds/velero-schedules.yaml:29-49`) includes all PVCs in the namespace by default with `snapshotMoveData: true`. It's possible this is already moot — `node-agent-config.yaml:14-26` shows ceph-block backups now use the default RW-clone path (no `readOnly`), which was adopted specifically to avoid RO-mount ext4 issues — but that fix was motivated by a different symptom (ENOTCLEAN on a dirty journal), not explicitly the same-node RW-mount conflict described in the redis/music-assistant comments, and unlike redis (cache) or music-assistant (reseedable from an LXC), this PVC now holds paperless's only live copy of the Whoosh index once `paperless-data` is eventually deleted. Worth explicitly checking that the next `paperless-daily` backup run succeeds for `paperless-data-block` rather than assuming it does, and adding `velero.io/exclude-from-backup` (with something to reintroduce coverage later, e.g. a fs-backup annotation) if it doesn't. - The post-merge verification list in the PR body (paperless starts, search works, MDS session gone, metadata write rate drops) doesn't include a backup check — worth adding once the point above is resolved either way. ## Tests No automated tests apply to this kind of infra/manifest change; verification is manual and operational, and the PR body lays out a reasonable post-merge check sequence (search results, MDS session, write-rate metric). The pre-merge copy was verified byte-for-byte (file count, mtime, index segment count) with paperless quiesced, which is the right way to validate a Whoosh-index-bearing copy. ## Recommendation Looks reasonable — no blocking issues; the Velero backup-coverage question above is worth a quick check but shouldn't hold up this merge given the retained `paperless-data` rollback.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#534