Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 a003df0ef8 fix(paperless): restart on paperless-config changes, not just secrets
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The AI settings from #739 applied cleanly to git and to the ConfigMap in the
cluster, and the application never saw them: the pod was 13 minutes old with no
PAPERLESS_AI_* in its environment at all.

paperless-config is consumed with envFrom, and Kubernetes does not restart a pod
when a ConfigMap read that way changes — the container keeps the environment it
started with. The Deployment carried a Reloader annotation for paperless-secrets
only, so secret rotations restarted it but settings changes did not.

Adding the ConfigMap to Reloader closes that, and makes the annotation pair
match how the ConfigMap is actually consumed rather than only the Secret.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 07:19:22 +02:00
k.renner f0e48b226a Merge pull request 'feat(paperless): AI suggestions against the cluster's own LLM' (#739) from paperless-llm into master
Reviewed-on: k.renner/homelab#739
2026-09-02 05:16:01 +00:00
k.renner 669ef1c5ee Merge pull request 'refactor(paperless): retire paperless-data, the CephFS predecessor volume' (#740) from close-paperless-data-migration into master
Reviewed-on: k.renner/homelab#740
2026-09-02 05:15:31 +00:00
claude-botandClaude Opus 5 11cd02ec44 refactor(paperless): retire paperless-data, the CephFS predecessor volume
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The `data` volume moved from paperless-data (RWX, ceph-filesystem) to
paperless-data-block (RWO, ceph-block) on 2026-08-28. paperless-data was
kept as the rollback, on the condition that search be confirmed working
first — the point being to prove the Whoosh index survived the copy, not
merely that paperless started.

Both halves of that condition are now settled. index_optimize succeeded on
every night between the repoint and 2026-09-01, and that task opens and
rewrites the index rather than just reading it. Then the 3.1.2 upgrade
replaced Whoosh with Tantivy outright, so the index paperless-data was
preserving no longer exists in a form anything can read.

Removed: the PVC, the migration record, and the comments that described the
volume as in-flight. The rationale for RWO-on-RBD stays where it is useful,
on the PVC and on the Deployment's volume list, since it is the reason not
to move this back onto a shared filesystem later.

paperless-data carries kustomize.toolkit.fluxcd.io/prune: disabled, so this
commit does not delete anything by itself. The PVC, its Retain-policy PV and
the backing CephFS subvolume have to be removed by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 07:13:07 +02:00
3 changed files with 24 additions and 139 deletions
@@ -1,97 +0,0 @@
# paperless-data: CephFS → RBD
## Why
`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
Deployment mounts it**. Verified 2026-08-20:
```
$ pods mounting paperless-data
paperless-68cb8b8744-7hjb8 <- one, and only one
paperless-data ReadWriteMany ceph-filesystem
paperless-media ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner)
paperless-consume ReadWriteMany ceph-filesystem <- genuinely shared (ftp-scanner)
```
Its contents are single-writer state — the 50 MB Whoosh index,
`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
It is on CephFS by default, not by requirement.
## What it costs
Every access goes through the MDS. Measured with **zero CI pipelines running**:
```
paperless-data client request_load_avg 22 (all other sessions 0-1)
MDS over 180s 4,052 client requests = 22.5/s
2,718 journal events = 15/s
2 dir commits <- almost none: not file writes
```
So it is cap/inode churn from repeated access, not data being written. That is
most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
depends on it. Cutting the writes at source is correct; cutting the replication
is not.
## Sequence
Staged deliberately: the PVC lands first and empty, so merging it changes
nothing. The copy and the repoint are separate.
### Phase 1 — provision (this PR)
`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
nothing. Inert.
Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
### Phase 2 — copy
The Whoosh index must come across intact. Paperless *can* rebuild it
(`document_index reindex`), but that reprocesses every document, so copying is
both faster and less disruptive.
1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
and the Whoosh index are both live-written, and copying them under an active
writer gives a torn read.
2. Run a one-off pod that mounts BOTH PVCs and copies:
```
cp -a /src/. /dst/
```
The RWO target pins the pod to one node; the RWX source will follow.
3. Verify file count and index size match:
```
find /src -type f | wc -l == find /dst -type f | wc -l
du -sh /src/index == du -sh /dst/index
```
### Phase 3 — repoint
Change the Deployment's `data` volume to `claimName: paperless-data-block`,
merge, let it roll.
Verify afterwards:
- paperless starts, search returns results (proves the Whoosh index survived)
- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
client
- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
**Do not delete `paperless-data` until search has been confirmed working.** It is
the rollback, and it is `prune: disabled` so Flux will not remove it.
## Expected effect
~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
worth doing, but it is a correctness fix (a single-writer volume does not belong
on a shared filesystem) more than a large win.
It also removes 22 req/sec of MDS load, which benefits the workloads that
genuinely need CephFS — Woodpecker CI workspaces share the same MDS.
+14 -13
View File
@@ -96,6 +96,13 @@ metadata:
# Reloader restarts paperless when paperless-secrets changes (e.g. after a
# secret-key rotation). See kubernetes/infrastructure/openbao/rotation/paperless.yaml.
secret.reloader.stakater.com/reload: paperless-secrets
# And when paperless-config changes. The ConfigMap is consumed with
# envFrom, and Kubernetes does not restart a pod when a ConfigMap it reads
# that way is updated — the running container keeps the environment it
# started with. Without this, a settings change applies cleanly in git and
# in the cluster while the application never sees it, which is how the AI
# settings landed with no effect at all.
configmap.reloader.stakater.com/reload: paperless-config
spec:
replicas: 1
strategy:
@@ -279,22 +286,16 @@ spec:
seccompProfile:
type: RuntimeDefault
volumes:
# paperless-data-block (RWO, ceph-block), NOT paperless-data
# (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
# and its contents are single-writer state — the Whoosh index,
# classification model, celerybeat schedule, logs. Being on CephFS meant
# every access went through the MDS: 22 requests/sec sustained with zero
# CI running, ~15 journal events/sec against 2 directory commits per
# 180s, i.e. cap/inode churn rather than file writes. That was most of
# the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
# `data` is RWO on ceph-block, not RWX on ceph-filesystem. Only this
# Deployment mounts it and its contents are single-writer state — the
# search index, classification model, celerybeat schedule, logs. On
# CephFS every access went through the MDS: 22 requests/sec sustained
# with zero CI running, ~15 journal events/sec against 2 directory
# commits per 180s, i.e. cap/inode churn rather than file writes. That
# was most of the constant 7-8 GB/day on cephfs-metadata, a size 3 pool.
#
# `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
# those from another node, so they genuinely need RWX.
#
# paperless-data is retained (prune: disabled) as the rollback. Do not
# delete it until search has been confirmed working — that is what
# proves the Whoosh index survived the copy, not merely that paperless
# starts. Migration record: MIGRATION-data-to-block.md.
- name: data
persistentVolumeClaim:
claimName: paperless-data-block
+10 -29
View File
@@ -3,33 +3,18 @@
# (e.g. when the seed initContainer or the Garage migration creds are
# removed once we're done with the migration phase). Velero's
# bootstrap-restore then owns their contents.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperless-data
namespace: paperless
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
accessModes:
- ReadWriteMany
storageClassName: ceph-filesystem
resources:
requests:
storage: 10Gi
---
# Replacement for paperless-data, on RBD instead of CephFS.
# paperless's single-writer state, on RBD rather than CephFS.
#
# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
# pod ever mounts it — the paperless Deployment itself. Its contents are all
# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
# and `media` genuinely need RWX, because the FTP scanner writes to those from a
# different node.
# WHY RWO: only ONE pod mounts it — the paperless Deployment itself. Its
# contents are all single-writer state: the Tantivy search index,
# classification_model.pickle, celerybeat-schedule.db, logs, migration_lock.
# Nothing shares it. Only `consume` and `media` genuinely need RWX, because the
# FTP scanner writes to those from a different node.
#
# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
# and the MDS logged ~15 journal events/sec against only 2 directory commits per
# WHY NOT CEPHFS: every access goes through the MDS. With zero CI running, this
# volume's client sustained 22 MDS requests/sec, and the MDS logged ~15 journal
# events/sec against only 2 directory commits per
# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
# pool, so ~21-24 GB/day actually hits the drives.
@@ -38,10 +23,6 @@ spec:
# the metadata pool destroys the whole filesystem, not just the objects on one
# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
# is the right fix; reducing the replication is not.
#
# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
# and the deployment repoint are a separate, staged step — see
# MIGRATION-data-to-block.md. Merging this alone is inert.
apiVersion: v1
kind: PersistentVolumeClaim
metadata: