fix: make small scratch emptyDirs RAM-backed instead of node-disk

Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory` is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is ephemeral, but every byte still hits the node's disk. Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts 12 of the remaining ones. HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then nothing emptyDir-driven -- the rest of the top of that list is Ceph mons (hostPath, not emptyDir), kube-apiserver, and container logs from databases. Every volume converted here measured 8.0K in the running pod: /var/lib/code-server 8.0K /mosquitto/auth 8.0K /state (z2m guards) 8.0K So the value is correctness and, for mosquitto, keeping credential material off the node's filesystem: that volume holds the generated passwd/acl files. Each converted volume gets an explicit sizeLimit (16Mi where none existed), so tmpfs cannot silently consume a container's memory limit. All are orders of magnitude above measured usage. Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on either medium, so the mosquitto shim's assumption that the passwd file survives a container restart still holds. DELIBERATELY NOT CONVERTED: - kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored and generated from the chart; a hand edit would be lost on upgrade. - trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was. Cost: this rolls the affected pods once. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'feat(home-assistant): render recorder_db_url from the ha-pg Secret' (#524 ) from feat-ha-recorder-db-url into master
2026-08-20 06:30:23 +02:00 · 2026-08-19 19:24:38 +00:00 · 2026-08-19 21:18:27 +02:00 · 2026-08-19 21:10:26 +02:00 · 2026-08-19 15:43:41 +00:00 · 2026-08-19 15:58:50 +02:00
20 changed files with 634 additions and 16 deletions
@@ -121,7 +121,9 @@ spec:
          persistentVolumeClaim:
            claimName: audiobookshelf-books
        - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service
@@ -0,0 +1,179 @@
+# Home Assistant recorder: SQLite → Postgres
+
+## Why
+
+The recorder's SQLite database is the single largest client writer in the
+cluster. Measured 2026-08-19 (cgroup `io.stat` on the home-assistant container,
+corroborated by kernel diskstats and Ceph pool counters):
+
+| | |
+|---|---|
+| HA writes to its RBD volume | **40 GB/day** |
+| share of all client writes | **~26%** |
+| actual row data underneath | **~5 GB/day** |
+
+The 8× gap is the write *pattern*, not the volume:
+
+```
+~30 state changes/sec  (107k rows/hour, 2.57M/day)
+  ×2   SQLite WAL + checkpoint — every dirtied page written twice
+  ×4   ext4 journal + commit, because synchronous=FULL fsyncs every commit
+       (measured: 819 KB of 4 KiB writes → 3.28 MB at the block device)
+  = ~15 KB written per recorded state change
+```
+
+Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
+This attacks the multiplier rather than trimming what gets recorded — excluding
+entities would hide the problem, not fix it.
+
+**Context:** this is one of three write-reduction changes. The other two are
+merged and verified — `size 3→2` on the Ceph pools (#519) and
+`mon_compact_on_trim=false` (#520, which alone removed ~228 GB/day). See #434.
+
+## What is at risk
+
+`states` is disposable — `purge_keep_days: 7`, so it regenerates in a week.
+
+**`statistics` is not.** As of 2026-08-19:
+
+| table | rows | note |
+|---|---|---|
+| `statistics` | 10,793,712 | **spans 2024-03-13 → today, 2.4 years** |
+| `statistics_short_term` | 1,902,624 | 5-minute buckets |
+| `statistics_meta` | 1,448 | entity ↔ id mapping, required by both |
+| `states` | 18,289,190 | disposable, 7-day retention |
+| `events` | 483,989 | disposable |
+
+**Home Assistant does not migrate data between database engines.** Repointing
+`db_url` at an empty Postgres silently starts from zero and the energy dashboard
+loses its entire history. The long-term statistics are irreplaceable — they are
+not in InfluxDB, which HA does not read for the energy dashboard.
+
+## Phase 1 — deploy the cluster (this PR)
+
+`ha-pg` comes up **empty** and nothing consumes it. HA stays on SQLite, so a
+failure here is inert and there is nothing to roll back.
+
+Verify before going further:
+
+```sh
+kubectl -n home-assistant get cluster ha-pg          # Cluster in healthy state, 1/1
+kubectl -n home-assistant get backup                 # ha-pg-daily Completed (immediate: true)
+kubectl -n home-assistant logs ha-pg-1 -c postgres | grep -i archive   # WAL archiving, no errors
+```
+
+Confirm the backup actually landed in the bucket rather than trusting the CR
+status — the WordPress backup failed nightly for three days against a bucket
+that was declared but never created (see `opentofu/object_storage.tf`).
+
+## Phase 2b — bulk statistics copy: DONE 2026-08-19
+
+Completed with **HA running** — the SQLite file was read through a read-only
+mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
+was needed for the bulk copy.
+
+```
+statistics_meta:        sqlite=1449       pg=1449       OK
+statistics:             sqlite=10799904   pg=10799904   OK   (153.9s)
+statistics_short_term:  sqlite=1969804    pg=1969804    OK   ( 27.0s)
+
+statistics span in Postgres: 2024-03-13 -> 2026-08-19
+entities with long-term history: 1449
+rows carrying sum (energy dashboard): 2627859
+database size: 2252 MB   (PVC is 20Gi)
+```
+
+`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
+`COPY` the planner has no statistics, and HA queries these tables constantly for
+history and energy views, so without it every view seq-scans 10.8M rows.
+
+Two schema details that cost a failed first attempt, recorded so the top-up does
+not repeat them:
+
+- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
+  will not coerce; convert explicitly and preserve NULL.
+- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
+  both bulk tables (superseded by the `_ts` variants), which is what makes a
+  tab-separated `COPY` safe — nothing in those tables needs escaping.
+
+The schema itself was created with HA's own models
+(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
+left EMPTY on purpose. HA's `_initialize_database` detects a new database by
+looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
+itself. Pre-stamping that row would skip HA's own initialisation path.
+
+### High-water marks for the cutover top-up
+
+Rows written after the bulk copy are not yet in Postgres. Resume from:
+
+```
+statistics             max_id=13323904    max_start_ts=1787162400.0
+statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
+```
+
+## Phase 2c — cutover (REMAINING)
+
+Do not start until phase 1 is verified.
+
+**Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
+HA outage, and buys a table that is discarded within 7 days anyway.
+
+1. **Snapshot first.** Velero backup of the HA PVC, plus a `.backup` copy of
+   `home-assistant_v2.db`. This is the rollback.
+2. **Stop the recorder** — scale `deploy/home-assistant` to 0. The database must
+   be quiescent; copying a live SQLite file gives a torn read.
+3. **Create the schema.** Start HA once against the empty Postgres so it runs
+   its own migrations. Never hand-create the tables — the schema version must
+   match the HA version exactly or HA will refuse to start or silently migrate.
+4. **Copy** `statistics_meta` → `statistics` → `statistics_short_term`, in that
+   order (the other two carry `metadata_id` foreign keys into the first).
+   Sequences must be reset afterwards, or the first insert collides on a
+   duplicate primary key:
+   ```sql
+   SELECT setval('statistics_id_seq', (SELECT MAX(id) FROM statistics));
+   SELECT setval('statistics_meta_id_seq', (SELECT MAX(id) FROM statistics_meta));
+   SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
+   ```
+5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
+   there, not here. The `recorder_db_url` secret it references is already
+   rendered into `secrets.yaml` by the `render-db-url` init container in
+   `deployment.yaml` (phase 2a), so this order is mandatory: **the init
+   container must be deployed first**, or HA refuses to start on a missing
+   secret.
+
+   The init container percent-encodes the credentials and validates the
+   resulting YAML rather than counting lines. That is not incidental — the CNPG
+   password is random, and a naive `"postgresql://user:pass@host/db"` string
+   makes `secrets.yaml` unparseable the moment the password contains a quote,
+   which takes out *every* secret HA loads, not just this one. Verified against
+   a deliberately hostile password before merge.
+   ```yaml
+   recorder:
+     db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
+   ```
+   The password is in the CNPG-generated `ha-pg-app` Secret. It must not be
+   committed — render it into `secrets.yaml` the same way the Z2M MQTT password
+   is handled in `zigbee2mqtt-deployment.yaml`.
+6. **Verify** the energy dashboard renders history back to 2024, not just today.
+   That is the check that proves the statistics migration worked; "HA starts and
+   records" does not.
+
+### Rollback
+
+Revert `db_url` to the SQLite path and restore the `.backup` copy. Keep the
+SQLite file until the energy dashboard has been confirmed over a full day —
+deleting it early removes the only rollback that preserves the 2.4 years.
+
+## After the migration
+
+- The HA config PVC shrinks from 5.5 GB to a few hundred MB. Do not shrink the
+  PVC itself: `ceph-block` allows expansion, not reduction.
+- Re-measure. `kubectl -n home-assistant exec deploy/home-assistant -c
+  home-assistant -- cat /sys/fs/cgroup/io.stat` was 40 GB/day before. If it does
+  not drop substantially, the migration did not achieve its purpose and that
+  should be recorded rather than assumed.
+- `synchronous_commit: off` in `ha-pg-cluster.yaml` is the single biggest
+  contributor to the reduction. It is safe for telemetry (Postgres stays crash
+  consistent; at most a few hundred ms of commits are lost on unclean shutdown)
+  but it is a deliberate trade and should not be "corrected" later without
+  understanding why it is set.
@@ -36,6 +36,64 @@ spec:
      # (sml2mqtt bridge owns the USB device on edge), so it no longer needs
      # the edge USB device and schedules on the general cluster pool. This
      # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
+      initContainers:
+        # Puts CNPG's connection URI into /config/secrets.yaml as
+        # `recorder_db_url`, for the recorder's move off SQLite
+        # (MIGRATION-recorder-postgres.md).
+        #
+        # HA has no way to read a db_url from the environment — `!env_var` was
+        # removed from its YAML loader — so `!secret` is the only mechanism, and
+        # secrets.yaml is PVC state that config-sync deliberately does not
+        # manage. Hence a small init container rather than a env: entry.
+        #
+        # The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
+        # percent-encodes it, so there is nothing to escape here: the value is
+        # restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
+        # An earlier version of this assembled the URI from user/password/dbname
+        # and needed percent-encoding plus YAML round-trip validation to be
+        # safe. That was solving a problem CNPG had already solved.
+        #
+        # Re-rendered on every pod start, so a CNPG password rotation takes
+        # effect rather than silently leaving a stale URI behind.
+        #
+        # INERT until the recorder is repointed: nothing reads this key until
+        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
+        # homeassistant repo. That ordering is required — HA refuses to start
+        # when a referenced secret is missing.
+        - name: render-db-url
+          image: busybox:1.36
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              S=/config/secrets.yaml
+              T=/config/.secrets.render.tmp
+              [ -f "$S" ] || : > "$S"
+              # Filter-then-append, never rewrite in place: this file holds
+              # every other HA secret (influxdb_token and friends).
+              grep -v '^recorder_db_url:' "$S" > "$T" || true
+              printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
+              # Refuse to shrink the file. Cheap guard against the one failure
+              # that would actually hurt: losing the other keys.
+              [ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
+                echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
+              chmod 600 "$T"
+              mv "$T" "$S"
+              echo "recorder_db_url rendered"
+          env:
+            - name: DB_URI
+              valueFrom:
+                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
+          volumeMounts:
+            - name: config
+              mountPath: /config
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 32Mi
      containers:
        - name: home-assistant
          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
@@ -221,9 +279,13 @@ spec:
        # Writable emptyDir for the nginx entrypoint's envsubst output
        # (/etc/nginx/conf.d/default.conf rendered from the .template).
        - name: proxy-conf-d
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
        # code-server's settings/extensions cache. Not on the config PVC
        # so installed extensions don't pollute HA's /config listing.
        # Trade-off: extensions reset across pod restarts.
        - name: vscode-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
@@ -0,0 +1,90 @@
+---
+# Postgres backend for the Home Assistant recorder.
+#
+# WHY THIS EXISTS
+# The recorder's SQLite database is the single largest client writer in the
+# cluster. Measured 2026-08-19 (cgroup io.stat on the home-assistant container,
+# corroborated by kernel diskstats): 40 GB/day to its RBD volume, ~26% of all
+# client writes, against only ~5 GB/day of actual row data. The gap is the write
+# PATTERN, not the volume:
+#
+#   ~30 state changes/sec (107k rows/hour, 2.57M/day)
+#     x2  SQLite WAL + checkpoint — every dirtied page is written twice
+#     x4  ext4 journal + commit, because synchronous=FULL fsyncs every commit
+#         (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
+#     = ~15 KB written per recorded state change
+#
+# Postgres replaces scattered fsync'd page rewrites with sequential, batched WAL.
+# That is the point of the move — it attacks the multiplier rather than trimming
+# what gets recorded, which would only hide the problem.
+#
+# STAGING
+# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
+# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
+# to Hetzner, and a successful base backup before going further.
+# Phase 2: migrate the statistics tables and repoint recorder.db_url.
+# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: ha-pg
+  namespace: home-assistant
+spec:
+  instances: 1
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  storage:
+    storageClass: ceph-block
+    # Sized for the migrated statistics plus 7 days of states at the current
+    # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with
+    # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window
+    # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn.
+    # ceph-block has allowVolumeExpansion: true, so this can grow in place.
+    size: 20Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: ha-pg-store
+  bootstrap:
+    initdb:
+      # Explicit, not left to CNPG's `app` default: the generated ha-pg-app
+      # Secret and the -app Service point at whatever is named here, and the
+      # recorder db_url in phase 2 must match. Same trap that made the wp-pg
+      # namespace migration point credentials at an empty database (#433).
+      database: homeassistant
+      owner: homeassistant
+  postgresql:
+    parameters:
+      shared_buffers: "256MB"
+      max_connections: "50"
+      # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
+      # entire reason for the migration. Defaults would give a correct database
+      # that writes far more than it needs to on replicated Ceph storage.
+      #
+      # synchronous_commit off: the recorder is telemetry, not transactional
+      # record-keeping. This does NOT risk corruption — Postgres stays crash
+      # consistent; it only means the last few hundred ms of commits can be lost
+      # on an unclean shutdown, which for state history is a non-event. In
+      # exchange every commit stops forcing an fsync, which is precisely the
+      # cost that makes SQLite expensive here.
+      synchronous_commit: "off"
+      # Compress full-page images in WAL. Recorder writes are many small row
+      # inserts scattered across pages, so full-page writes after each
+      # checkpoint dominate WAL volume; compression cuts that directly.
+      wal_compression: "on"
+      # Fewer checkpoints => fewer full-page writes, since a page is only
+      # fully-logged on its first modification after a checkpoint. 15min with a
+      # 2GB ceiling keeps recovery time sane while cutting repeat full-page
+      # images for the hot rows the recorder rewrites constantly.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      min_wal_size: "512MB"
+      # full_page_writes is deliberately LEFT ON (the default). RBD can tear a
+      # page across a 4 MiB object boundary, and turning this off would trade
+      # write volume for the risk of an unrecoverable database. Not worth it.
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1Gi
@@ -0,0 +1,38 @@
+---
+# Barman S3 credentials for ha-pg.
+#
+# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
+# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
+# problem and the same note. The values are the same today, so this is correct
+# but coupled: rotating paperless's Hetzner credentials would silently change
+# the backup credentials for Home Assistant and Woodpecker too.
+#
+# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
+# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
+#
+#   bao kv put homelab/seed/home-assistant/hetzner-creds \
+#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
+#
+# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
+# 403 on `bao kv list seed`, which was a wrong-path error rather than a
+# permissions problem.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace or ESO silently never syncs (#505). home-assistant is already
+# allowlisted — it has several ExternalSecrets in this directory already.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: home-assistant
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/paperless/hetzner-creds
@@ -0,0 +1,35 @@
+---
+# Barman object store for ha-pg: base backups + continuous WAL archiving to
+# Hetzner Object Storage, same target bucket as the other CNPG clusters.
+#
+# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
+# their streams and corrupt both — see the note in
+# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: ha-pg-store
+  namespace: home-assistant
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  # Matches the 30d retention the other CNPG clusters use and the
+  # renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
+  # opentofu/object_storage.tf, so old S3 versions don't outlive the backups
+  # that superseded them.
+  retentionPolicy: "30d"
@@ -0,0 +1,23 @@
+---
+# Daily base backups for ha-pg. Continuous WAL archiving runs separately
+# (ha-pg-cluster.yaml spec.plugins[].isWALArchiver); without periodic base
+# backups the prefix would have no recovery point of its own, and
+# ha-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:50 UTC, offset from the other CNPG clusters (02:30, wp-pg at 02:40) so they
+# do not all hit the Hetzner bucket at once.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: ha-pg-daily
+  namespace: home-assistant
+spec:
+  schedule: "0 50 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: ha-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io
@@ -52,6 +52,14 @@ resources:
  # CNP ingress rule.
  - zigbee2mqtt-gate-service.yaml
  - zigbee2mqtt-ingress.yaml
+  # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
+  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
+  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
+  # tables and repoints recorder.db_url in the homeassistant repo.
+  - ha-pg-hetzner-creds-externalsecret.yaml
+  - ha-pg-objectstore.yaml
+  - ha-pg-cluster.yaml
+  - ha-pg-scheduled-backup.yaml
 configMapGenerator:
  # nginx config for the reverse-proxy sidecar. Hash suffix disabled so
  # the cm name stays stable; editing proxy.conf does not auto-roll the
@@ -313,7 +313,9 @@ spec:
          configMap:
            name: mosquitto-config
        - name: auth
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
        - name: tls
          secret:
            secretName: mosquitto-tls
@@ -344,9 +344,13 @@ spec:
          configMap:
            name: zigbee2mqtt-interview-watch-script
        - name: interview-watch-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
        - name: discovery-guard-script
          configMap:
            name: zigbee2mqtt-discovery-guard-script
        - name: discovery-guard-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
@@ -147,7 +147,9 @@ spec:
              - key: immich-config.json
                path: immich-config.json
        - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service
@@ -99,3 +99,26 @@ spec:
      interval: 30s
  podTargetLabels:
    - role
+---
+# ha-pg — Postgres backend for the Home Assistant recorder
+# (kubernetes/apps/home-assistant/ha-pg-cluster.yaml). Added with the cluster
+# rather than after it, so the migration has metrics from the first moment
+# instead of discovering the gap later, which is how wp-pg's scrape hole was
+# found (#433).
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-ha-pg
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role
@@ -64,7 +64,9 @@ spec:
          secret:
            secretName: pve-exporter-config
        - name: tmp
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service
@@ -218,7 +218,9 @@ spec:
          persistentVolumeClaim:
            claimName: paperless-consume
        - name: bridge-cert
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service
@@ -156,4 +156,6 @@ spec:
        # each pod enrolls its own peer and a restart re-enrolls cleanly
        # (the old peer is reaped).
        - name: state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
@@ -358,3 +358,67 @@ spec:
        - ports:
            - port: "8123"
              protocol: TCP
+---
+# ha-pg (CNPG) — instance-manager access for the CloudNativePG operator.
+#
+# DEDICATED policy with its own endpointSelector, NOT another rule on the
+# namespace-wide default-deny above. That policy has endpointSelector:{}, so any
+# rule added there applies to EVERY pod in the namespace — and :8000 is served
+# both by the CNPG instance manager and by other workloads, so a namespace-wide
+# :8000 grant would expose far more than intended. Same reasoning as
+# wp-pg-allow-cnpg-operator in woodpecker-ns.yaml, which was corrected during
+# review for exactly this (#433).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-cnpg-operator
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP
+    # ICMP to the ha-pg pods. REQUIRED because the L4 rules in this policy
+    # select those endpoints, and Cilium drops ICMP to an endpoint selected by
+    # any layer 4 policy unless the traffic is CT-related. Without this, PMTUD
+    # Destination Unreachable / Packet Too Big toward ha-pg are dropped
+    # silently, and the symptom is stalled transfers rather than a policy
+    # denial — the failure mode documented in woodpecker-ns.yaml.
+    - fromEntities:
+        - cluster
+      icmps:
+        - fields:
+            - { type: 3,   family: IPv4 }   # Destination Unreachable — PMTUD
+            - { type: 1,   family: IPv6 }   # Destination Unreachable
+            - { type: 2,   family: IPv6 }   # Packet Too Big — PMTUD, mandatory
+---
+# ha-pg — Postgres access for the Home Assistant recorder.
+#
+# Scoped to the HA pod on both ends: only home-assistant reaches ha-pg, and only
+# on 5432. Nothing else in this namespace has any business in the recorder
+# database.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-home-assistant
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app.kubernetes.io/name: home-assistant
+      toPorts:
+        - ports:
+            - port: "5432"
+              protocol: TCP
@@ -5,7 +5,7 @@ metadata:
  namespace: rook-ceph
 spec:
  cephVersion:
-    image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  mon:
@@ -45,6 +45,49 @@ spec:
      # would be circular, and Rook only applies mon.volumeClaimTemplate to
      # newly created mons, never migrating existing ones.
      mon_data_avail_warn: "20"
+      # Stop compacting the whole RocksDB prefix on every paxos trim.
+      #
+      # Upstream default is true — "Compact a certain prefix (including paxos)
+      # when we trim its old states" — which is fine at low paxos rates and
+      # pathological here. Measured 2026-08-19:
+      #
+      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
+      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
+      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
+      #                                        (cgroup io.stat: 2.15 TB/27 d)
+      #   => ~98x write amplification, for a ~106 MB database
+      #
+      # That was ~62% of all OS-disk writes on these nodes and roughly a
+      # quarter of everything the cluster writes — more than every client
+      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
+      # means hundreds of forced range compactions per day, each rewriting the
+      # store.
+      #
+      # The churn itself is 99% cluster log and mgr stats, not real cluster
+      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
+      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
+      # report / osd dump — 23 audit entries per minute per mon) becomes a
+      # cluster-log entry, hence a paxos transaction, replicated to all three
+      # mons. Reducing that polling is the complementary fix and is tracked
+      # separately; this setting removes the amplification regardless of rate.
+      #
+      # This does NOT disable compaction. RocksDB's own levelled compaction
+      # still runs and still reclaims trimmed keys; what stops is the extra
+      # forced compact_range on every trim. Store growth is therefore bounded
+      # by normal compaction, and there are two backstops if that assumption is
+      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
+      # CephHealthWarn in monitoring-rules already alerts on, and
+      # mon_compact_on_start below compacts fully on every mon restart.
+      #
+      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
+      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
+      # not come back down, revisit rather than leaving it.
+      mon_compact_on_trim: "false"
+      # Compensating control for the above: a full compaction on mon start.
+      # Upstream default is false. Mons restart on upgrades, node reboots and
+      # reschedules, so this gives a periodic full reclaim without paying for
+      # one on every trim. Cheap here — the store is ~106 MB.
+      mon_compact_on_start: "true"
    mgr:
      mgr/volumes/max_concurrent_clones: "16"
    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
@@ -5,8 +5,31 @@ metadata:
  namespace: rook-ceph
 spec:
  failureDomain: host
+  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # every drive in the cluster: per-drive write volume equalled total cluster
+  # write volume, and all three drives wore at an identical rate and would have
+  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that failure mode, because
+  # wear-out here is correlated by construction rather than independent.
+  #
+  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  #
+  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # blocks IO for its PGs and hangs every RBD volume in the cluster.
+  #
+  # The trade this accepts: while the pool is degraded (one OSD down), a second
+  # drive failing loses data, and with all three drives at 80%+ endurance that
+  # window is not hypothetical. Mitigations that already exist: CNPG barman and
+  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
+  # exposure is the RPO of those backups, not total loss. Keep node reboots
+  # serialised and wait for HEALTH_OK between them.
  replicated:
-    size: 3
+    size: 2
+  parameters:
+    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -64,13 +87,23 @@ metadata:
  name: cephfs
  namespace: rook-ceph
 spec:
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: data0
+      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # the comment there for the full rationale and the accepted trade.
      replicated:
-        size: 3
+        size: 2
+      parameters:
+        min_size: "1"
  metadataServer:
    activeCount: 1
    activeStandby: true
@@ -38,7 +38,7 @@ spec:
      serviceAccountName: rook-ceph-default
      containers:
        - name: rook-ceph-tools
-          image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
+          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
          command:
            - /bin/bash
            - -c
@@ -164,7 +164,9 @@ spec:
            name: rook-config-override
            optional: true
        - name: ceph-config
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
@@ -85,4 +85,6 @@ spec:
                name: restore-test-script
                defaultMode: 0755
            - name: tmp
-              emptyDir: {}
+              emptyDir:
+                medium: Memory
+                sizeLimit: 16Mi