feat(paperless): repoint data volume to RBD (phase 3)

Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.

The copy is already done and verified byte-for-byte with paperless stopped:

  src files: 75
  dst files: 75
  IDENTICAL: name, size and mtime match for every file
  index segments: src 3, dst 3

mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)

Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.

cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.

paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.

Refs #434

Co-Authored-By: Claude Bot <noreply@anthropic.com>

kubernetes/apps/authentik/cluster.yaml

@@ -43,6 +43,27 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "400"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "on"
     # Async replication makes CNPG switchover slow because the standby has
     # to chase missing WAL via barman-cloud and replay queued records
     # before it can promote. With one synchronous standby, the standby is

kubernetes/apps/grafana-db/postgres.yaml

@@ -29,6 +29,27 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "on"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -55,7 +55,21 @@ spec:
       owner: homeassistant
   postgresql:
     parameters:
-      shared_buffers: "256MB"
+      # 512MB, not the 256MB this shipped with. Measured after 15h on Postgres:
+      # buffers_backend was 629,704 of 939,025 total page writes — 67% written
+      # by backend processes rather than the checkpointer or bgwriter. That
+      # ratio means the working set does not fit in shared_buffers, so backends
+      # evict dirty pages themselves and the same pages are written out
+      # repeatedly instead of coalescing in memory.
+      #
+      # The recorder touches a wide set of pages continuously (~30 state
+      # changes/sec across states, state_attributes and five indexes), which is
+      # exactly the workload that punishes a small buffer pool.
+      #
+      # Memory headroom checked before raising: the instance was using 383Mi of
+      # a 1Gi limit with 256MB of that being shared_buffers, so the limit goes
+      # to 1.5Gi to keep the same margin.
+      shared_buffers: "512MB"
       max_connections: "50"
       # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
       # entire reason for the migration. Defaults would give a correct database
@@ -87,4 +101,4 @@ spec:
       cpu: 200m
       memory: 512Mi
     limits:
-      memory: 1Gi
+      memory: 1536Mi

kubernetes/apps/immich/cluster.yaml

@@ -43,6 +43,25 @@ spec:
       shared_buffers: "512MB"
       max_wal_size: "2GB"
       wal_compression: "on"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
   resources:
     requests:
       cpu: 500m

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -13,6 +13,10 @@ resources:
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
+  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
+  # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
+  # endpoint, which proves reachability, not that anything was written.
+  - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - home-assistant-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -0,0 +1,78 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cnpg-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cnpg-backup.rules
+      rules:
+        # WHY THIS FILE EXISTS
+        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
+        # stopped. The only related rule was HetznerBackupEndpointDown, which
+        # probes the S3 endpoint — it proves the bucket is reachable, not that
+        # anything was written to it. Same failure shape as the WordPress backup
+        # that failed nightly for three days against a bucket that was declared
+        # in tofu but never applied: the status surface looked fine because
+        # nothing checked the thing that mattered.
+
+        - alert: CNPGWalArchiveBacklog
+          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
+          # to be archived. Steady state is 0 across every cluster (verified
+          # 2026-08-19). A sustained backlog means the archiver is failing:
+          # point-in-time recovery is silently broken from that moment, and
+          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          #
+          # This is the load-bearing alert of the two. It depends only on the
+          # instance-level collector, so unlike the backup-age metrics below it
+          # is unaffected by the missing Backup CRs (#522).
+          #
+          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
+          # 80 MiB of un-archived WAL, well below anything that threatens the
+          # volume but far above idle noise.
+          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGBackupFailed
+          # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
+          # when no failure is recorded. Guard on > 0 so the epoch-zero default
+          # is not read as "failed in 1970".
+          expr: |
+            cnpg_collector_last_failed_backup_timestamp > 0
+            and
+            (time() - cnpg_collector_last_failed_backup_timestamp) < 86400
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
+
+        # DELIBERATELY ABSENT: a backup-age alert, which is the one you would
+        # most want here.
+        #
+        # The natural expression is
+        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
+        # and it CANNOT be used today: that metric reads exactly 0 on every
+        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
+        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
+        # successful backup anywhere, because the Backup CRs it derives that
+        # from are being deleted within a minute of completing — see #522.
+        #
+        # Backups themselves DO work; that was verified independently by
+        # listing the bucket with barman-cloud-backup-list and restoring a WAL
+        # segment back out of S3. The gap is in the record, not the data.
+        #
+        # Adding the rule now would fire permanently for all seven clusters and
+        # be silenced within a day, which is worse than not having it. Add it as
+        # part of resolving #522, once the metric reflects reality.

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -0,0 +1,97 @@
+# paperless-data: CephFS → RBD
+
+## Why
+
+`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
+Deployment mounts it**. Verified 2026-08-20:
+
+```
+$ pods mounting paperless-data
+paperless-68cb8b8744-7hjb8      <- one, and only one
+
+paperless-data     ReadWriteMany  ceph-filesystem
+paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+```
+
+Its contents are single-writer state — the 50 MB Whoosh index,
+`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
+`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
+
+It is on CephFS by default, not by requirement.
+
+## What it costs
+
+Every access goes through the MDS. Measured with **zero CI pipelines running**:
+
+```
+paperless-data client   request_load_avg 22   (all other sessions 0-1)
+MDS over 180s           4,052 client requests  = 22.5/s
+                        2,718 journal events   = 15/s
+                            2 dir commits      <- almost none: not file writes
+```
+
+So it is cap/inode churn from repeated access, not data being written. That is
+most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
+pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
+
+`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
+filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
+depends on it. Cutting the writes at source is correct; cutting the replication
+is not.
+
+## Sequence
+
+Staged deliberately: the PVC lands first and empty, so merging it changes
+nothing. The copy and the repoint are separate.
+
+### Phase 1 — provision (this PR)
+
+`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
+nothing. Inert.
+
+Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
+
+### Phase 2 — copy
+
+The Whoosh index must come across intact. Paperless *can* rebuild it
+(`document_index reindex`), but that reprocesses every document, so copying is
+both faster and less disruptive.
+
+1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
+   and the Whoosh index are both live-written, and copying them under an active
+   writer gives a torn read.
+2. Run a one-off pod that mounts BOTH PVCs and copies:
+   ```
+   cp -a /src/. /dst/
+   ```
+   The RWO target pins the pod to one node; the RWX source will follow.
+3. Verify file count and index size match:
+   ```
+   find /src -type f | wc -l   ==   find /dst -type f | wc -l
+   du -sh /src/index           ==   du -sh /dst/index
+   ```
+
+### Phase 3 — repoint
+
+Change the Deployment's `data` volume to `claimName: paperless-data-block`,
+merge, let it roll.
+
+Verify afterwards:
+- paperless starts, search returns results (proves the Whoosh index survived)
+- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
+  client
+- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
+
+**Do not delete `paperless-data` until search has been confirmed working.** It is
+the rollback, and it is `prune: disabled` so Flux will not remove it.
+
+## Expected effect
+
+~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
+replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
+worth doing, but it is a correctness fix (a single-writer volume does not belong
+on a shared filesystem) more than a large win.
+
+It also removes 22 req/sec of MDS load, which benefits the workloads that
+genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/cluster.yaml

@@ -34,6 +34,27 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "200"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "on"
     # See ak-pg cluster.yaml for the rationale on synchronous replication.
     synchronous:
       method: any

kubernetes/apps/paperless/paperless.yaml

@@ -208,9 +208,25 @@ spec:
             seccompProfile:
               type: RuntimeDefault
       volumes:
+        # paperless-data-block (RWO, ceph-block), NOT paperless-data
+        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
+        # and its contents are single-writer state — the Whoosh index,
+        # classification model, celerybeat schedule, logs. Being on CephFS meant
+        # every access went through the MDS: 22 requests/sec sustained with zero
+        # CI running, ~15 journal events/sec against 2 directory commits per
+        # 180s, i.e. cap/inode churn rather than file writes. That was most of
+        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
+        #
+        # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
+        # those from another node, so they genuinely need RWX.
+        #
+        # paperless-data is retained (prune: disabled) as the rollback. Do not
+        # delete it until search has been confirmed working — that is what
+        # proves the Whoosh index survived the copy, not merely that paperless
+        # starts. Migration record: MIGRATION-data-to-block.md.
         - name: data
           persistentVolumeClaim:
-            claimName: paperless-data
+            claimName: paperless-data-block
         - name: media
           persistentVolumeClaim:
             claimName: paperless-media

kubernetes/apps/paperless/pvcs.yaml

@@ -18,6 +18,45 @@ spec:
     requests:
       storage: 10Gi
 ---
+# Replacement for paperless-data, on RBD instead of CephFS.
+#
+# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
+# pod ever mounts it — the paperless Deployment itself. Its contents are all
+# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
+# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
+# and `media` genuinely need RWX, because the FTP scanner writes to those from a
+# different node.
+#
+# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
+# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
+# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
+# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
+# pool, so ~21-24 GB/day actually hits the drives.
+#
+# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing
+# the metadata pool destroys the whole filesystem, not just the objects on one
+# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
+# is the right fix; reducing the replication is not.
+#
+# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
+# and the deployment repoint are a separate, staged step — see
+# MIGRATION-data-to-block.md. Merging this alone is inert.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data-block
+  namespace: paperless
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/trmnl/cluster.yaml

@@ -33,6 +33,26 @@ spec:
       shared_buffers: "128MB"
       max_connections: "50"
       wal_compression: "on"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/trmnl/renderer.yaml

@@ -362,6 +362,22 @@ spec:
             name: trmnl-renderer-schedule
         - name: tmp
           emptyDir:
+            # RAM-backed, NOT disk. Without `medium: Memory` an emptyDir lands
+            # on the node's filesystem, and this one was the single largest
+            # writer in the cluster: 249 GB over 7.3 days = ~34 GB/day onto
+            # /dev/vda5, more than every Ceph OSD and every database combined
+            # (measured 2026-08-20 from the container's cgroup io.stat).
+            #
+            # The cause is cadence, not volume. schedules.template.json renders
+            # every 60 s, so Chrome is launched 1440 times a day and each launch
+            # creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*
+            # directory. At rest /tmp holds under 8 MB; it is pure churn for
+            # data that never outlives the render.
+            #
+            # tmpfs pages count against the container's memory limit, which is
+            # why the cap stays at 512Mi: measured usage is 129Mi against a 2Gi
+            # limit, so even the full cap leaves ample headroom.
+            medium: Memory
             sizeLimit: 512Mi
 ---
 apiVersion: v1

kubernetes/apps/woodpecker/cluster.yaml

@@ -56,6 +56,27 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "on"
   resources:
     requests:
       cpu: 100m