feat(paperless): provision RBD volume to move data off CephFS (phase 1)

paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.

It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:

  paperless-data client   request_load_avg 22   (all other sessions 0-1)
  MDS over 180s           4,052 client requests = 22.5/s
                          2,718 journal events  = 15/s
                              2 dir commits     <- not file writes

Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.

cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.

PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).

Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/audiobookshelf/deployment.yaml

@@ -121,7 +121,9 @@ spec:
           persistentVolumeClaim:
             claimName: audiobookshelf-books
         - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -66,7 +66,52 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
 status — the WordPress backup failed nightly for three days against a bucket
 that was declared but never created (see `opentofu/object_storage.tf`).
 
-## Phase 2 — migrate the data (NOT in this PR)
+## Phase 2b — bulk statistics copy: DONE 2026-08-19
+
+Completed with **HA running** — the SQLite file was read through a read-only
+mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
+was needed for the bulk copy.
+
+```
+statistics_meta:        sqlite=1449       pg=1449       OK
+statistics:             sqlite=10799904   pg=10799904   OK   (153.9s)
+statistics_short_term:  sqlite=1969804    pg=1969804    OK   ( 27.0s)
+
+statistics span in Postgres: 2024-03-13 -> 2026-08-19
+entities with long-term history: 1449
+rows carrying sum (energy dashboard): 2627859
+database size: 2252 MB   (PVC is 20Gi)
+```
+
+`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
+`COPY` the planner has no statistics, and HA queries these tables constantly for
+history and energy views, so without it every view seq-scans 10.8M rows.
+
+Two schema details that cost a failed first attempt, recorded so the top-up does
+not repeat them:
+
+- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
+  will not coerce; convert explicitly and preserve NULL.
+- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
+  both bulk tables (superseded by the `_ts` variants), which is what makes a
+  tab-separated `COPY` safe — nothing in those tables needs escaping.
+
+The schema itself was created with HA's own models
+(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
+left EMPTY on purpose. HA's `_initialize_database` detects a new database by
+looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
+itself. Pre-stamping that row would skip HA's own initialisation path.
+
+### High-water marks for the cutover top-up
+
+Rows written after the bulk copy are not yet in Postgres. Resume from:
+
+```
+statistics             max_id=13323904    max_start_ts=1787162400.0
+statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
+```
+
+## Phase 2c — cutover (REMAINING)
 
 Do not start until phase 1 is verified.
 
@@ -90,7 +135,18 @@ HA outage, and buys a table that is discarded within 7 days anyway.
    SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
    ```
 5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
-   there, not here:
+   there, not here. The `recorder_db_url` secret it references is already
+   rendered into `secrets.yaml` by the `render-db-url` init container in
+   `deployment.yaml` (phase 2a), so this order is mandatory: **the init
+   container must be deployed first**, or HA refuses to start on a missing
+   secret.
+
+   The init container percent-encodes the credentials and validates the
+   resulting YAML rather than counting lines. That is not incidental — the CNPG
+   password is random, and a naive `"postgresql://user:pass@host/db"` string
+   makes `secrets.yaml` unparseable the moment the password contains a quote,
+   which takes out *every* secret HA loads, not just this one. Verified against
+   a deliberately hostile password before merge.
    ```yaml
    recorder:
      db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant

kubernetes/apps/home-assistant/deployment.yaml

@@ -36,6 +36,64 @@ spec:
       # (sml2mqtt bridge owns the USB device on edge), so it no longer needs
       # the edge USB device and schedules on the general cluster pool. This
       # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
+      initContainers:
+        # Puts CNPG's connection URI into /config/secrets.yaml as
+        # `recorder_db_url`, for the recorder's move off SQLite
+        # (MIGRATION-recorder-postgres.md).
+        #
+        # HA has no way to read a db_url from the environment — `!env_var` was
+        # removed from its YAML loader — so `!secret` is the only mechanism, and
+        # secrets.yaml is PVC state that config-sync deliberately does not
+        # manage. Hence a small init container rather than a env: entry.
+        #
+        # The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
+        # percent-encodes it, so there is nothing to escape here: the value is
+        # restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
+        # An earlier version of this assembled the URI from user/password/dbname
+        # and needed percent-encoding plus YAML round-trip validation to be
+        # safe. That was solving a problem CNPG had already solved.
+        #
+        # Re-rendered on every pod start, so a CNPG password rotation takes
+        # effect rather than silently leaving a stale URI behind.
+        #
+        # INERT until the recorder is repointed: nothing reads this key until
+        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
+        # homeassistant repo. That ordering is required — HA refuses to start
+        # when a referenced secret is missing.
+        - name: render-db-url
+          image: busybox:1.36
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              S=/config/secrets.yaml
+              T=/config/.secrets.render.tmp
+              [ -f "$S" ] || : > "$S"
+              # Filter-then-append, never rewrite in place: this file holds
+              # every other HA secret (influxdb_token and friends).
+              grep -v '^recorder_db_url:' "$S" > "$T" || true
+              printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
+              # Refuse to shrink the file. Cheap guard against the one failure
+              # that would actually hurt: losing the other keys.
+              [ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
+                echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
+              chmod 600 "$T"
+              mv "$T" "$S"
+              echo "recorder_db_url rendered"
+          env:
+            - name: DB_URI
+              valueFrom:
+                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
+          volumeMounts:
+            - name: config
+              mountPath: /config
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 32Mi
       containers:
         - name: home-assistant
           image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
@@ -221,9 +279,13 @@ spec:
         # Writable emptyDir for the nginx entrypoint's envsubst output
         # (/etc/nginx/conf.d/default.conf rendered from the .template).
         - name: proxy-conf-d
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
         # code-server's settings/extensions cache. Not on the config PVC
         # so installed extensions don't pollute HA's /config listing.
         # Trade-off: extensions reset across pod restarts.
         - name: vscode-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/apps/home-assistant/mosquitto-deployment.yaml

@@ -313,7 +313,9 @@ spec:
           configMap:
             name: mosquitto-config
         - name: auth
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
         - name: tls
           secret:
             secretName: mosquitto-tls

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -344,9 +344,13 @@ spec:
           configMap:
             name: zigbee2mqtt-interview-watch-script
         - name: interview-watch-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
         - name: discovery-guard-script
           configMap:
             name: zigbee2mqtt-discovery-guard-script
         - name: discovery-guard-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/apps/immich/server.yaml

@@ -147,7 +147,9 @@ spec:
               - key: immich-config.json
                 path: immich-config.json
         - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/monitoring/pve-exporter-deployment.yaml

@@ -64,7 +64,9 @@ spec:
           secret:
             secretName: pve-exporter-config
         - name: tmp
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -0,0 +1,97 @@
+# paperless-data: CephFS → RBD
+
+## Why
+
+`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
+Deployment mounts it**. Verified 2026-08-20:
+
+```
+$ pods mounting paperless-data
+paperless-68cb8b8744-7hjb8      <- one, and only one
+
+paperless-data     ReadWriteMany  ceph-filesystem
+paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+```
+
+Its contents are single-writer state — the 50 MB Whoosh index,
+`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
+`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
+
+It is on CephFS by default, not by requirement.
+
+## What it costs
+
+Every access goes through the MDS. Measured with **zero CI pipelines running**:
+
+```
+paperless-data client   request_load_avg 22   (all other sessions 0-1)
+MDS over 180s           4,052 client requests  = 22.5/s
+                        2,718 journal events   = 15/s
+                            2 dir commits      <- almost none: not file writes
+```
+
+So it is cap/inode churn from repeated access, not data being written. That is
+most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
+pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
+
+`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
+filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
+depends on it. Cutting the writes at source is correct; cutting the replication
+is not.
+
+## Sequence
+
+Staged deliberately: the PVC lands first and empty, so merging it changes
+nothing. The copy and the repoint are separate.
+
+### Phase 1 — provision (this PR)
+
+`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
+nothing. Inert.
+
+Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
+
+### Phase 2 — copy
+
+The Whoosh index must come across intact. Paperless *can* rebuild it
+(`document_index reindex`), but that reprocesses every document, so copying is
+both faster and less disruptive.
+
+1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
+   and the Whoosh index are both live-written, and copying them under an active
+   writer gives a torn read.
+2. Run a one-off pod that mounts BOTH PVCs and copies:
+   ```
+   cp -a /src/. /dst/
+   ```
+   The RWO target pins the pod to one node; the RWX source will follow.
+3. Verify file count and index size match:
+   ```
+   find /src -type f | wc -l   ==   find /dst -type f | wc -l
+   du -sh /src/index           ==   du -sh /dst/index
+   ```
+
+### Phase 3 — repoint
+
+Change the Deployment's `data` volume to `claimName: paperless-data-block`,
+merge, let it roll.
+
+Verify afterwards:
+- paperless starts, search returns results (proves the Whoosh index survived)
+- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
+  client
+- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
+
+**Do not delete `paperless-data` until search has been confirmed working.** It is
+the rollback, and it is `prune: disabled` so Flux will not remove it.
+
+## Expected effect
+
+~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
+replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
+worth doing, but it is a correctness fix (a single-writer volume does not belong
+on a shared filesystem) more than a large win.
+
+It also removes 22 req/sec of MDS load, which benefits the workloads that
+genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/paperless.yaml

@@ -218,7 +218,9 @@ spec:
           persistentVolumeClaim:
             claimName: paperless-consume
         - name: bridge-cert
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/paperless/pvcs.yaml

@@ -18,6 +18,45 @@ spec:
     requests:
       storage: 10Gi
 ---
+# Replacement for paperless-data, on RBD instead of CephFS.
+#
+# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
+# pod ever mounts it — the paperless Deployment itself. Its contents are all
+# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
+# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
+# and `media` genuinely need RWX, because the FTP scanner writes to those from a
+# different node.
+#
+# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
+# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
+# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
+# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
+# pool, so ~21-24 GB/day actually hits the drives.
+#
+# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing
+# the metadata pool destroys the whole filesystem, not just the objects on one
+# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
+# is the right fix; reducing the replication is not.
+#
+# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
+# and the deployment repoint are a separate, staged step — see
+# MIGRATION-data-to-block.md. Merging this alone is inert.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data-block
+  namespace: paperless
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/trmnl/renderer.yaml

@@ -362,6 +362,22 @@ spec:
             name: trmnl-renderer-schedule
         - name: tmp
           emptyDir:
+            # RAM-backed, NOT disk. Without `medium: Memory` an emptyDir lands
+            # on the node's filesystem, and this one was the single largest
+            # writer in the cluster: 249 GB over 7.3 days = ~34 GB/day onto
+            # /dev/vda5, more than every Ceph OSD and every database combined
+            # (measured 2026-08-20 from the container's cgroup io.stat).
+            #
+            # The cause is cadence, not volume. schedules.template.json renders
+            # every 60 s, so Chrome is launched 1440 times a day and each launch
+            # creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*
+            # directory. At rest /tmp holds under 8 MB; it is pure churn for
+            # data that never outlives the render.
+            #
+            # tmpfs pages count against the container's memory limit, which is
+            # why the cap stays at 512Mi: measured usage is 129Mi against a 2Gi
+            # limit, so even the full cap leaves ample headroom.
+            medium: Memory
             sizeLimit: 512Mi
 ---
 apiVersion: v1

kubernetes/infrastructure/netbird-ingress/daemonset.yaml

@@ -156,4 +156,6 @@ spec:
         # each pod enrolls its own peer and a restart re-enrolls cleanly
         # (the old peer is reaped).
         - name: state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml

@@ -164,7 +164,9 @@ spec:
             name: rook-config-override
             optional: true
         - name: ceph-config
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
       tolerations:
         - key: "node.kubernetes.io/unreachable"
           operator: "Exists"

kubernetes/infrastructure/velero/restore-test-cronjob.yaml

@@ -85,4 +85,6 @@ spec:
                 name: restore-test-script
                 defaultMode: 0755
             - name: tmp
-              emptyDir: {}
+              emptyDir:
+                medium: Memory
+                sizeLimit: 16Mi