feat(monitoring): alert on CNPG WAL archiving and backup failure

Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.

CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.

CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".

DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.

The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.

Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.

Refs #522

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -0,0 +1,123 @@
+# Home Assistant recorder: SQLite → Postgres
+
+## Why
+
+The recorder's SQLite database is the single largest client writer in the
+cluster. Measured 2026-08-19 (cgroup `io.stat` on the home-assistant container,
+corroborated by kernel diskstats and Ceph pool counters):
+
+| | |
+|---|---|
+| HA writes to its RBD volume | **40 GB/day** |
+| share of all client writes | **~26%** |
+| actual row data underneath | **~5 GB/day** |
+
+The 8× gap is the write *pattern*, not the volume:
+
+```
+~30 state changes/sec  (107k rows/hour, 2.57M/day)
+  ×2   SQLite WAL + checkpoint — every dirtied page written twice
+  ×4   ext4 journal + commit, because synchronous=FULL fsyncs every commit
+       (measured: 819 KB of 4 KiB writes → 3.28 MB at the block device)
+  = ~15 KB written per recorded state change
+```
+
+Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
+This attacks the multiplier rather than trimming what gets recorded — excluding
+entities would hide the problem, not fix it.
+
+**Context:** this is one of three write-reduction changes. The other two are
+merged and verified — `size 3→2` on the Ceph pools (#519) and
+`mon_compact_on_trim=false` (#520, which alone removed ~228 GB/day). See #434.
+
+## What is at risk
+
+`states` is disposable — `purge_keep_days: 7`, so it regenerates in a week.
+
+**`statistics` is not.** As of 2026-08-19:
+
+| table | rows | note |
+|---|---|---|
+| `statistics` | 10,793,712 | **spans 2024-03-13 → today, 2.4 years** |
+| `statistics_short_term` | 1,902,624 | 5-minute buckets |
+| `statistics_meta` | 1,448 | entity ↔ id mapping, required by both |
+| `states` | 18,289,190 | disposable, 7-day retention |
+| `events` | 483,989 | disposable |
+
+**Home Assistant does not migrate data between database engines.** Repointing
+`db_url` at an empty Postgres silently starts from zero and the energy dashboard
+loses its entire history. The long-term statistics are irreplaceable — they are
+not in InfluxDB, which HA does not read for the energy dashboard.
+
+## Phase 1 — deploy the cluster (this PR)
+
+`ha-pg` comes up **empty** and nothing consumes it. HA stays on SQLite, so a
+failure here is inert and there is nothing to roll back.
+
+Verify before going further:
+
+```sh
+kubectl -n home-assistant get cluster ha-pg          # Cluster in healthy state, 1/1
+kubectl -n home-assistant get backup                 # ha-pg-daily Completed (immediate: true)
+kubectl -n home-assistant logs ha-pg-1 -c postgres | grep -i archive   # WAL archiving, no errors
+```
+
+Confirm the backup actually landed in the bucket rather than trusting the CR
+status — the WordPress backup failed nightly for three days against a bucket
+that was declared but never created (see `opentofu/object_storage.tf`).
+
+## Phase 2 — migrate the data (NOT in this PR)
+
+Do not start until phase 1 is verified.
+
+**Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
+HA outage, and buys a table that is discarded within 7 days anyway.
+
+1. **Snapshot first.** Velero backup of the HA PVC, plus a `.backup` copy of
+   `home-assistant_v2.db`. This is the rollback.
+2. **Stop the recorder** — scale `deploy/home-assistant` to 0. The database must
+   be quiescent; copying a live SQLite file gives a torn read.
+3. **Create the schema.** Start HA once against the empty Postgres so it runs
+   its own migrations. Never hand-create the tables — the schema version must
+   match the HA version exactly or HA will refuse to start or silently migrate.
+4. **Copy** `statistics_meta` → `statistics` → `statistics_short_term`, in that
+   order (the other two carry `metadata_id` foreign keys into the first).
+   Sequences must be reset afterwards, or the first insert collides on a
+   duplicate primary key:
+   ```sql
+   SELECT setval('statistics_id_seq', (SELECT MAX(id) FROM statistics));
+   SELECT setval('statistics_meta_id_seq', (SELECT MAX(id) FROM statistics_meta));
+   SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
+   ```
+5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
+   there, not here:
+   ```yaml
+   recorder:
+     db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
+   ```
+   The password is in the CNPG-generated `ha-pg-app` Secret. It must not be
+   committed — render it into `secrets.yaml` the same way the Z2M MQTT password
+   is handled in `zigbee2mqtt-deployment.yaml`.
+6. **Verify** the energy dashboard renders history back to 2024, not just today.
+   That is the check that proves the statistics migration worked; "HA starts and
+   records" does not.
+
+### Rollback
+
+Revert `db_url` to the SQLite path and restore the `.backup` copy. Keep the
+SQLite file until the energy dashboard has been confirmed over a full day —
+deleting it early removes the only rollback that preserves the 2.4 years.
+
+## After the migration
+
+- The HA config PVC shrinks from 5.5 GB to a few hundred MB. Do not shrink the
+  PVC itself: `ceph-block` allows expansion, not reduction.
+- Re-measure. `kubectl -n home-assistant exec deploy/home-assistant -c
+  home-assistant -- cat /sys/fs/cgroup/io.stat` was 40 GB/day before. If it does
+  not drop substantially, the migration did not achieve its purpose and that
+  should be recorded rather than assumed.
+- `synchronous_commit: off` in `ha-pg-cluster.yaml` is the single biggest
+  contributor to the reduction. It is safe for telemetry (Postgres stays crash
+  consistent; at most a few hundred ms of commits are lost on unclean shutdown)
+  but it is a deliberate trade and should not be "corrected" later without
+  understanding why it is set.

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -0,0 +1,90 @@
+---
+# Postgres backend for the Home Assistant recorder.
+#
+# WHY THIS EXISTS
+# The recorder's SQLite database is the single largest client writer in the
+# cluster. Measured 2026-08-19 (cgroup io.stat on the home-assistant container,
+# corroborated by kernel diskstats): 40 GB/day to its RBD volume, ~26% of all
+# client writes, against only ~5 GB/day of actual row data. The gap is the write
+# PATTERN, not the volume:
+#
+#   ~30 state changes/sec (107k rows/hour, 2.57M/day)
+#     x2  SQLite WAL + checkpoint — every dirtied page is written twice
+#     x4  ext4 journal + commit, because synchronous=FULL fsyncs every commit
+#         (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
+#     = ~15 KB written per recorded state change
+#
+# Postgres replaces scattered fsync'd page rewrites with sequential, batched WAL.
+# That is the point of the move — it attacks the multiplier rather than trimming
+# what gets recorded, which would only hide the problem.
+#
+# STAGING
+# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
+# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
+# to Hetzner, and a successful base backup before going further.
+# Phase 2: migrate the statistics tables and repoint recorder.db_url.
+# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: ha-pg
+  namespace: home-assistant
+spec:
+  instances: 1
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  storage:
+    storageClass: ceph-block
+    # Sized for the migrated statistics plus 7 days of states at the current
+    # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with
+    # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window
+    # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn.
+    # ceph-block has allowVolumeExpansion: true, so this can grow in place.
+    size: 20Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: ha-pg-store
+  bootstrap:
+    initdb:
+      # Explicit, not left to CNPG's `app` default: the generated ha-pg-app
+      # Secret and the -app Service point at whatever is named here, and the
+      # recorder db_url in phase 2 must match. Same trap that made the wp-pg
+      # namespace migration point credentials at an empty database (#433).
+      database: homeassistant
+      owner: homeassistant
+  postgresql:
+    parameters:
+      shared_buffers: "256MB"
+      max_connections: "50"
+      # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
+      # entire reason for the migration. Defaults would give a correct database
+      # that writes far more than it needs to on replicated Ceph storage.
+      #
+      # synchronous_commit off: the recorder is telemetry, not transactional
+      # record-keeping. This does NOT risk corruption — Postgres stays crash
+      # consistent; it only means the last few hundred ms of commits can be lost
+      # on an unclean shutdown, which for state history is a non-event. In
+      # exchange every commit stops forcing an fsync, which is precisely the
+      # cost that makes SQLite expensive here.
+      synchronous_commit: "off"
+      # Compress full-page images in WAL. Recorder writes are many small row
+      # inserts scattered across pages, so full-page writes after each
+      # checkpoint dominate WAL volume; compression cuts that directly.
+      wal_compression: "on"
+      # Fewer checkpoints => fewer full-page writes, since a page is only
+      # fully-logged on its first modification after a checkpoint. 15min with a
+      # 2GB ceiling keeps recovery time sane while cutting repeat full-page
+      # images for the hot rows the recorder rewrites constantly.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      min_wal_size: "512MB"
+      # full_page_writes is deliberately LEFT ON (the default). RBD can tear a
+      # page across a 4 MiB object boundary, and turning this off would trade
+      # write volume for the risk of an unrecoverable database. Not worth it.
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1Gi

kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml

@@ -0,0 +1,38 @@
+---
+# Barman S3 credentials for ha-pg.
+#
+# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
+# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
+# problem and the same note. The values are the same today, so this is correct
+# but coupled: rotating paperless's Hetzner credentials would silently change
+# the backup credentials for Home Assistant and Woodpecker too.
+#
+# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
+# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
+#
+#   bao kv put homelab/seed/home-assistant/hetzner-creds \
+#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
+#
+# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
+# 403 on `bao kv list seed`, which was a wrong-path error rather than a
+# permissions problem.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace or ESO silently never syncs (#505). home-assistant is already
+# allowlisted — it has several ExternalSecrets in this directory already.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: home-assistant
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/paperless/hetzner-creds

kubernetes/apps/home-assistant/ha-pg-objectstore.yaml

@@ -0,0 +1,35 @@
+---
+# Barman object store for ha-pg: base backups + continuous WAL archiving to
+# Hetzner Object Storage, same target bucket as the other CNPG clusters.
+#
+# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
+# their streams and corrupt both — see the note in
+# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: ha-pg-store
+  namespace: home-assistant
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  # Matches the 30d retention the other CNPG clusters use and the
+  # renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
+  # opentofu/object_storage.tf, so old S3 versions don't outlive the backups
+  # that superseded them.
+  retentionPolicy: "30d"

kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml

@@ -0,0 +1,23 @@
+---
+# Daily base backups for ha-pg. Continuous WAL archiving runs separately
+# (ha-pg-cluster.yaml spec.plugins[].isWALArchiver); without periodic base
+# backups the prefix would have no recovery point of its own, and
+# ha-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:50 UTC, offset from the other CNPG clusters (02:30, wp-pg at 02:40) so they
+# do not all hit the Hetzner bucket at once.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: ha-pg-daily
+  namespace: home-assistant
+spec:
+  schedule: "0 50 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: ha-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/home-assistant/kustomization.yaml

@@ -52,6 +52,14 @@ resources:
   # CNP ingress rule.
   - zigbee2mqtt-gate-service.yaml
   - zigbee2mqtt-ingress.yaml
+  # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
+  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
+  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
+  # tables and repoints recorder.db_url in the homeassistant repo.
+  - ha-pg-hetzner-creds-externalsecret.yaml
+  - ha-pg-objectstore.yaml
+  - ha-pg-cluster.yaml
+  - ha-pg-scheduled-backup.yaml
 configMapGenerator:
   # nginx config for the reverse-proxy sidecar. Hash suffix disabled so
   # the cm name stays stable; editing proxy.conf does not auto-roll the

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -99,3 +99,26 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# ha-pg — Postgres backend for the Home Assistant recorder
+# (kubernetes/apps/home-assistant/ha-pg-cluster.yaml). Added with the cluster
+# rather than after it, so the migration has metrics from the first moment
+# instead of discovering the gap later, which is how wp-pg's scrape hole was
+# found (#433).
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-ha-pg
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -13,6 +13,10 @@ resources:
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
+  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
+  # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
+  # endpoint, which proves reachability, not that anything was written.
+  - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - home-assistant-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -0,0 +1,78 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cnpg-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cnpg-backup.rules
+      rules:
+        # WHY THIS FILE EXISTS
+        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
+        # stopped. The only related rule was HetznerBackupEndpointDown, which
+        # probes the S3 endpoint — it proves the bucket is reachable, not that
+        # anything was written to it. Same failure shape as the WordPress backup
+        # that failed nightly for three days against a bucket that was declared
+        # in tofu but never applied: the status surface looked fine because
+        # nothing checked the thing that mattered.
+
+        - alert: CNPGWalArchiveBacklog
+          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
+          # to be archived. Steady state is 0 across every cluster (verified
+          # 2026-08-19). A sustained backlog means the archiver is failing:
+          # point-in-time recovery is silently broken from that moment, and
+          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          #
+          # This is the load-bearing alert of the two. It depends only on the
+          # instance-level collector, so unlike the backup-age metrics below it
+          # is unaffected by the missing Backup CRs (#522).
+          #
+          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
+          # 80 MiB of un-archived WAL, well below anything that threatens the
+          # volume but far above idle noise.
+          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGBackupFailed
+          # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
+          # when no failure is recorded. Guard on > 0 so the epoch-zero default
+          # is not read as "failed in 1970".
+          expr: |
+            cnpg_collector_last_failed_backup_timestamp > 0
+            and
+            (time() - cnpg_collector_last_failed_backup_timestamp) < 86400
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
+
+        # DELIBERATELY ABSENT: a backup-age alert, which is the one you would
+        # most want here.
+        #
+        # The natural expression is
+        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
+        # and it CANNOT be used today: that metric reads exactly 0 on every
+        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
+        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
+        # successful backup anywhere, because the Backup CRs it derives that
+        # from are being deleted within a minute of completing — see #522.
+        #
+        # Backups themselves DO work; that was verified independently by
+        # listing the bucket with barman-cloud-backup-list and restoring a WAL
+        # segment back out of S3. The gap is in the record, not the data.
+        #
+        # Adding the rule now would fire permanently for all seven clusters and
+        # be silenced within a day, which is worse than not having it. Add it as
+        # part of resolving #522, once the metric reflects reality.

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -358,3 +358,67 @@ spec:
         - ports:
             - port: "8123"
               protocol: TCP
+---
+# ha-pg (CNPG) — instance-manager access for the CloudNativePG operator.
+#
+# DEDICATED policy with its own endpointSelector, NOT another rule on the
+# namespace-wide default-deny above. That policy has endpointSelector:{}, so any
+# rule added there applies to EVERY pod in the namespace — and :8000 is served
+# both by the CNPG instance manager and by other workloads, so a namespace-wide
+# :8000 grant would expose far more than intended. Same reasoning as
+# wp-pg-allow-cnpg-operator in woodpecker-ns.yaml, which was corrected during
+# review for exactly this (#433).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-cnpg-operator
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP
+    # ICMP to the ha-pg pods. REQUIRED because the L4 rules in this policy
+    # select those endpoints, and Cilium drops ICMP to an endpoint selected by
+    # any layer 4 policy unless the traffic is CT-related. Without this, PMTUD
+    # Destination Unreachable / Packet Too Big toward ha-pg are dropped
+    # silently, and the symptom is stalled transfers rather than a policy
+    # denial — the failure mode documented in woodpecker-ns.yaml.
+    - fromEntities:
+        - cluster
+      icmps:
+        - fields:
+            - { type: 3,   family: IPv4 }   # Destination Unreachable — PMTUD
+            - { type: 1,   family: IPv6 }   # Destination Unreachable
+            - { type: 2,   family: IPv6 }   # Packet Too Big — PMTUD, mandatory
+---
+# ha-pg — Postgres access for the Home Assistant recorder.
+#
+# Scoped to the HA pod on both ends: only home-assistant reaches ha-pg, and only
+# on 5432. Nothing else in this namespace has any business in the recorder
+# database.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-home-assistant
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app.kubernetes.io/name: home-assistant
+      toPorts:
+        - ports:
+            - port: "5432"
+              protocol: TCP

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -45,6 +45,49 @@ spec:
       # would be circular, and Rook only applies mon.volumeClaimTemplate to
       # newly created mons, never migrating existing ones.
       mon_data_avail_warn: "20"
+      # Stop compacting the whole RocksDB prefix on every paxos trim.
+      #
+      # Upstream default is true — "Compact a certain prefix (including paxos)
+      # when we trim its old states" — which is fine at low paxos rates and
+      # pathological here. Measured 2026-08-19:
+      #
+      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
+      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
+      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
+      #                                        (cgroup io.stat: 2.15 TB/27 d)
+      #   => ~98x write amplification, for a ~106 MB database
+      #
+      # That was ~62% of all OS-disk writes on these nodes and roughly a
+      # quarter of everything the cluster writes — more than every client
+      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
+      # means hundreds of forced range compactions per day, each rewriting the
+      # store.
+      #
+      # The churn itself is 99% cluster log and mgr stats, not real cluster
+      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
+      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
+      # report / osd dump — 23 audit entries per minute per mon) becomes a
+      # cluster-log entry, hence a paxos transaction, replicated to all three
+      # mons. Reducing that polling is the complementary fix and is tracked
+      # separately; this setting removes the amplification regardless of rate.
+      #
+      # This does NOT disable compaction. RocksDB's own levelled compaction
+      # still runs and still reclaims trimmed keys; what stops is the extra
+      # forced compact_range on every trim. Store growth is therefore bounded
+      # by normal compaction, and there are two backstops if that assumption is
+      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
+      # CephHealthWarn in monitoring-rules already alerts on, and
+      # mon_compact_on_start below compacts fully on every mon restart.
+      #
+      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
+      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
+      # not come back down, revisit rather than leaving it.
+      mon_compact_on_trim: "false"
+      # Compensating control for the above: a full compaction on mon start.
+      # Upstream default is false. Mons restart on upgrades, node reboots and
+      # reschedules, so this gives a periodic full reclaim without paying for
+      # one on every trim. Cheap here — the store is ~106 MB.
+      mon_compact_on_start: "true"
     mgr:
       mgr/volumes/max_concurrent_clones: "16"
     # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —