feat(monitoring): alert on CNPG WAL archiving and backup failure

Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.

CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.

CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".

DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.

The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.

Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.

Refs #522

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/audiobookshelf/deployment.yaml

@@ -121,9 +121,7 @@ spec:
           persistentVolumeClaim:
             claimName: audiobookshelf-books
         - name: le-ca
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -66,52 +66,7 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
 status — the WordPress backup failed nightly for three days against a bucket
 that was declared but never created (see `opentofu/object_storage.tf`).
 
-## Phase 2b — bulk statistics copy: DONE 2026-08-19
-
-Completed with **HA running** — the SQLite file was read through a read-only
-mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
-was needed for the bulk copy.
-
-```
-statistics_meta:        sqlite=1449       pg=1449       OK
-statistics:             sqlite=10799904   pg=10799904   OK   (153.9s)
-statistics_short_term:  sqlite=1969804    pg=1969804    OK   ( 27.0s)
-
-statistics span in Postgres: 2024-03-13 -> 2026-08-19
-entities with long-term history: 1449
-rows carrying sum (energy dashboard): 2627859
-database size: 2252 MB   (PVC is 20Gi)
-```
-
-`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
-`COPY` the planner has no statistics, and HA queries these tables constantly for
-history and energy views, so without it every view seq-scans 10.8M rows.
-
-Two schema details that cost a failed first attempt, recorded so the top-up does
-not repeat them:
-
-- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
-  will not coerce; convert explicitly and preserve NULL.
-- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
-  both bulk tables (superseded by the `_ts` variants), which is what makes a
-  tab-separated `COPY` safe — nothing in those tables needs escaping.
-
-The schema itself was created with HA's own models
-(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
-left EMPTY on purpose. HA's `_initialize_database` detects a new database by
-looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
-itself. Pre-stamping that row would skip HA's own initialisation path.
-
-### High-water marks for the cutover top-up
-
-Rows written after the bulk copy are not yet in Postgres. Resume from:
-
-```
-statistics             max_id=13323904    max_start_ts=1787162400.0
-statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
-```
-
-## Phase 2c — cutover (REMAINING)
+## Phase 2 — migrate the data (NOT in this PR)
 
 Do not start until phase 1 is verified.
 
@@ -135,18 +90,7 @@ HA outage, and buys a table that is discarded within 7 days anyway.
    SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
    ```
 5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
-   there, not here. The `recorder_db_url` secret it references is already
-   rendered into `secrets.yaml` by the `render-db-url` init container in
-   `deployment.yaml` (phase 2a), so this order is mandatory: **the init
-   container must be deployed first**, or HA refuses to start on a missing
-   secret.
-
-   The init container percent-encodes the credentials and validates the
-   resulting YAML rather than counting lines. That is not incidental — the CNPG
-   password is random, and a naive `"postgresql://user:pass@host/db"` string
-   makes `secrets.yaml` unparseable the moment the password contains a quote,
-   which takes out *every* secret HA loads, not just this one. Verified against
-   a deliberately hostile password before merge.
+   there, not here:
    ```yaml
    recorder:
      db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant

kubernetes/apps/home-assistant/deployment.yaml

@@ -36,64 +36,6 @@ spec:
       # (sml2mqtt bridge owns the USB device on edge), so it no longer needs
       # the edge USB device and schedules on the general cluster pool. This
       # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
-      initContainers:
-        # Puts CNPG's connection URI into /config/secrets.yaml as
-        # `recorder_db_url`, for the recorder's move off SQLite
-        # (MIGRATION-recorder-postgres.md).
-        #
-        # HA has no way to read a db_url from the environment — `!env_var` was
-        # removed from its YAML loader — so `!secret` is the only mechanism, and
-        # secrets.yaml is PVC state that config-sync deliberately does not
-        # manage. Hence a small init container rather than a env: entry.
-        #
-        # The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
-        # percent-encodes it, so there is nothing to escape here: the value is
-        # restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
-        # An earlier version of this assembled the URI from user/password/dbname
-        # and needed percent-encoding plus YAML round-trip validation to be
-        # safe. That was solving a problem CNPG had already solved.
-        #
-        # Re-rendered on every pod start, so a CNPG password rotation takes
-        # effect rather than silently leaving a stale URI behind.
-        #
-        # INERT until the recorder is repointed: nothing reads this key until
-        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
-        # homeassistant repo. That ordering is required — HA refuses to start
-        # when a referenced secret is missing.
-        - name: render-db-url
-          image: busybox:1.36
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              S=/config/secrets.yaml
-              T=/config/.secrets.render.tmp
-              [ -f "$S" ] || : > "$S"
-              # Filter-then-append, never rewrite in place: this file holds
-              # every other HA secret (influxdb_token and friends).
-              grep -v '^recorder_db_url:' "$S" > "$T" || true
-              printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
-              # Refuse to shrink the file. Cheap guard against the one failure
-              # that would actually hurt: losing the other keys.
-              [ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
-                echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
-              chmod 600 "$T"
-              mv "$T" "$S"
-              echo "recorder_db_url rendered"
-          env:
-            - name: DB_URI
-              valueFrom:
-                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
-          volumeMounts:
-            - name: config
-              mountPath: /config
-          resources:
-            requests:
-              cpu: 10m
-              memory: 16Mi
-            limits:
-              memory: 32Mi
       containers:
         - name: home-assistant
           image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
@@ -279,13 +221,9 @@ spec:
         # Writable emptyDir for the nginx entrypoint's envsubst output
         # (/etc/nginx/conf.d/default.conf rendered from the .template).
         - name: proxy-conf-d
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
         # code-server's settings/extensions cache. Not on the config PVC
         # so installed extensions don't pollute HA's /config listing.
         # Trade-off: extensions reset across pod restarts.
         - name: vscode-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}

kubernetes/apps/home-assistant/mosquitto-deployment.yaml

@@ -313,9 +313,7 @@ spec:
           configMap:
             name: mosquitto-config
         - name: auth
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
         - name: tls
           secret:
             secretName: mosquitto-tls

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -344,13 +344,9 @@ spec:
           configMap:
             name: zigbee2mqtt-interview-watch-script
         - name: interview-watch-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
         - name: discovery-guard-script
           configMap:
             name: zigbee2mqtt-discovery-guard-script
         - name: discovery-guard-state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}

kubernetes/apps/immich/server.yaml

@@ -147,9 +147,7 @@ spec:
               - key: immich-config.json
                 path: immich-config.json
         - name: le-ca
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -13,6 +13,10 @@ resources:
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
+  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
+  # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
+  # endpoint, which proves reachability, not that anything was written.
+  - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - home-assistant-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -0,0 +1,78 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cnpg-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cnpg-backup.rules
+      rules:
+        # WHY THIS FILE EXISTS
+        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
+        # stopped. The only related rule was HetznerBackupEndpointDown, which
+        # probes the S3 endpoint — it proves the bucket is reachable, not that
+        # anything was written to it. Same failure shape as the WordPress backup
+        # that failed nightly for three days against a bucket that was declared
+        # in tofu but never applied: the status surface looked fine because
+        # nothing checked the thing that mattered.
+
+        - alert: CNPGWalArchiveBacklog
+          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
+          # to be archived. Steady state is 0 across every cluster (verified
+          # 2026-08-19). A sustained backlog means the archiver is failing:
+          # point-in-time recovery is silently broken from that moment, and
+          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          #
+          # This is the load-bearing alert of the two. It depends only on the
+          # instance-level collector, so unlike the backup-age metrics below it
+          # is unaffected by the missing Backup CRs (#522).
+          #
+          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
+          # 80 MiB of un-archived WAL, well below anything that threatens the
+          # volume but far above idle noise.
+          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGBackupFailed
+          # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
+          # when no failure is recorded. Guard on > 0 so the epoch-zero default
+          # is not read as "failed in 1970".
+          expr: |
+            cnpg_collector_last_failed_backup_timestamp > 0
+            and
+            (time() - cnpg_collector_last_failed_backup_timestamp) < 86400
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
+
+        # DELIBERATELY ABSENT: a backup-age alert, which is the one you would
+        # most want here.
+        #
+        # The natural expression is
+        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
+        # and it CANNOT be used today: that metric reads exactly 0 on every
+        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
+        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
+        # successful backup anywhere, because the Backup CRs it derives that
+        # from are being deleted within a minute of completing — see #522.
+        #
+        # Backups themselves DO work; that was verified independently by
+        # listing the bucket with barman-cloud-backup-list and restoring a WAL
+        # segment back out of S3. The gap is in the record, not the data.
+        #
+        # Adding the rule now would fire permanently for all seven clusters and
+        # be silenced within a day, which is worse than not having it. Add it as
+        # part of resolving #522, once the metric reflects reality.

kubernetes/apps/monitoring/pve-exporter-deployment.yaml

@@ -64,9 +64,7 @@ spec:
           secret:
             secretName: pve-exporter-config
         - name: tmp
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/paperless/paperless.yaml

@@ -218,9 +218,7 @@ spec:
           persistentVolumeClaim:
             claimName: paperless-consume
         - name: bridge-cert
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service

kubernetes/infrastructure/netbird-ingress/daemonset.yaml

@@ -156,6 +156,4 @@ spec:
         # each pod enrolls its own peer and a restart re-enrolls cleanly
         # (the old peer is reaped).
         - name: state
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
   mon:

kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml

@@ -38,7 +38,7 @@ spec:
       serviceAccountName: rook-ceph-default
       containers:
         - name: rook-ceph-tools
-          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
+          image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
           command:
             - /bin/bash
             - -c
@@ -164,9 +164,7 @@ spec:
             name: rook-config-override
             optional: true
         - name: ceph-config
-          emptyDir:
-            medium: Memory
-            sizeLimit: 16Mi
+          emptyDir: {}
       tolerations:
         - key: "node.kubernetes.io/unreachable"
           operator: "Exists"

kubernetes/infrastructure/velero/restore-test-cronjob.yaml

@@ -85,6 +85,4 @@ spec:
                 name: restore-test-script
                 defaultMode: 0755
             - name: tmp
-              emptyDir:
-                medium: Memory
-                sizeLimit: 16Mi
+              emptyDir: {}