From 196ebda1889cae1daf76df6dc5369d6be3be56a9 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 19 Aug 2026 15:22:21 +0200 Subject: [PATCH] feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster) The recorder's SQLite database is the largest client writer in the cluster. Measured 2026-08-19 via cgroup io.stat on the home-assistant container, corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD volume, ~26% of all client writes, against only ~5 GB/day of actual row data. The 8x gap is the write PATTERN, not the volume: ~30 state changes/sec (107k rows/hour, 2.57M/day) x2 SQLite WAL + checkpoint -- every dirtied page written twice x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device) = ~15 KB written per recorded state change Postgres replaces scattered fsync'd page rewrites with sequential batched WAL. This attacks the multiplier rather than trimming what gets recorded, which would hide the problem instead of fixing it. PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves. Phase 2 migrates the statistics tables and repoints recorder.db_url in the homeassistant repo; MIGRATION-recorder-postgres.md has the sequence. What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT migrate data between database engines. `statistics` holds 10,793,712 rows spanning 2024-03-13 to today -- 2.4 years of energy history that is not in InfluxDB and cannot be reconstructed. Repointing db_url at an empty database silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7. Postgres is tuned to minimise write volume, which is the point of the move: synchronous_commit off -- the recorder is telemetry, not transactional record-keeping. Postgres stays crash consistent; only the last few hundred ms of commits are at risk on unclean shutdown. In exchange, commits stop forcing an fsync -- the exact cost that makes SQLite expensive here. wal_compression on -- recorder writes are small inserts scattered across pages, so post-checkpoint full-page writes dominate WAL volume. checkpoint_timeout 15min / max_wal_size 2GB -- fewer checkpoints means fewer repeat full-page images for the hot rows the recorder rewrites. full_page_writes -- deliberately LEFT ON. RBD can tear a page across a 4 MiB object boundary; trading that for write volume risks an unrecoverable database. Also included: - ObjectStore on its own s3 prefix (two clusters sharing a destinationPath interleave WAL streams and corrupt both). - ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters. - PodMonitor added WITH the cluster rather than after it, so the migration has metrics from the start instead of discovering the gap later (#433). - Cilium policies as DEDICATED policies with their own endpointSelector, not rules on the namespace-wide default-deny -- that policy is endpointSelector:{} so any rule there applies to every pod in the namespace, and :8000 is served by more than the CNPG instance manager. Includes the ICMP allowance that attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD. The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same coupling #397 tracks for woodpecker. Noted in the file with the exact bao command to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed` was a wrong-path error rather than missing permissions. Refs #434, #397 Co-Authored-By: Claude Opus 5 (1M context) --- .../MIGRATION-recorder-postgres.md | 123 ++++++++++++++++++ .../apps/home-assistant/ha-pg-cluster.yaml | 90 +++++++++++++ .../ha-pg-hetzner-creds-externalsecret.yaml | 38 ++++++ .../home-assistant/ha-pg-objectstore.yaml | 35 +++++ .../ha-pg-scheduled-backup.yaml | 23 ++++ .../apps/home-assistant/kustomization.yaml | 8 ++ .../monitoring-rules/cnpg-podmonitors.yaml | 23 ++++ .../network-policies/home-assistant-ns.yaml | 64 +++++++++ 8 files changed, 404 insertions(+) create mode 100644 kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md create mode 100644 kubernetes/apps/home-assistant/ha-pg-cluster.yaml create mode 100644 kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml create mode 100644 kubernetes/apps/home-assistant/ha-pg-objectstore.yaml create mode 100644 kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml diff --git a/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md b/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md new file mode 100644 index 00000000..27e18630 --- /dev/null +++ b/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md @@ -0,0 +1,123 @@ +# Home Assistant recorder: SQLite → Postgres + +## Why + +The recorder's SQLite database is the single largest client writer in the +cluster. Measured 2026-08-19 (cgroup `io.stat` on the home-assistant container, +corroborated by kernel diskstats and Ceph pool counters): + +| | | +|---|---| +| HA writes to its RBD volume | **40 GB/day** | +| share of all client writes | **~26%** | +| actual row data underneath | **~5 GB/day** | + +The 8× gap is the write *pattern*, not the volume: + +``` +~30 state changes/sec (107k rows/hour, 2.57M/day) + ×2 SQLite WAL + checkpoint — every dirtied page written twice + ×4 ext4 journal + commit, because synchronous=FULL fsyncs every commit + (measured: 819 KB of 4 KiB writes → 3.28 MB at the block device) + = ~15 KB written per recorded state change +``` + +Postgres replaces scattered fsync'd page rewrites with sequential batched WAL. +This attacks the multiplier rather than trimming what gets recorded — excluding +entities would hide the problem, not fix it. + +**Context:** this is one of three write-reduction changes. The other two are +merged and verified — `size 3→2` on the Ceph pools (#519) and +`mon_compact_on_trim=false` (#520, which alone removed ~228 GB/day). See #434. + +## What is at risk + +`states` is disposable — `purge_keep_days: 7`, so it regenerates in a week. + +**`statistics` is not.** As of 2026-08-19: + +| table | rows | note | +|---|---|---| +| `statistics` | 10,793,712 | **spans 2024-03-13 → today, 2.4 years** | +| `statistics_short_term` | 1,902,624 | 5-minute buckets | +| `statistics_meta` | 1,448 | entity ↔ id mapping, required by both | +| `states` | 18,289,190 | disposable, 7-day retention | +| `events` | 483,989 | disposable | + +**Home Assistant does not migrate data between database engines.** Repointing +`db_url` at an empty Postgres silently starts from zero and the energy dashboard +loses its entire history. The long-term statistics are irreplaceable — they are +not in InfluxDB, which HA does not read for the energy dashboard. + +## Phase 1 — deploy the cluster (this PR) + +`ha-pg` comes up **empty** and nothing consumes it. HA stays on SQLite, so a +failure here is inert and there is nothing to roll back. + +Verify before going further: + +```sh +kubectl -n home-assistant get cluster ha-pg # Cluster in healthy state, 1/1 +kubectl -n home-assistant get backup # ha-pg-daily Completed (immediate: true) +kubectl -n home-assistant logs ha-pg-1 -c postgres | grep -i archive # WAL archiving, no errors +``` + +Confirm the backup actually landed in the bucket rather than trusting the CR +status — the WordPress backup failed nightly for three days against a bucket +that was declared but never created (see `opentofu/object_storage.tf`). + +## Phase 2 — migrate the data (NOT in this PR) + +Do not start until phase 1 is verified. + +**Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long +HA outage, and buys a table that is discarded within 7 days anyway. + +1. **Snapshot first.** Velero backup of the HA PVC, plus a `.backup` copy of + `home-assistant_v2.db`. This is the rollback. +2. **Stop the recorder** — scale `deploy/home-assistant` to 0. The database must + be quiescent; copying a live SQLite file gives a torn read. +3. **Create the schema.** Start HA once against the empty Postgres so it runs + its own migrations. Never hand-create the tables — the schema version must + match the HA version exactly or HA will refuse to start or silently migrate. +4. **Copy** `statistics_meta` → `statistics` → `statistics_short_term`, in that + order (the other two carry `metadata_id` foreign keys into the first). + Sequences must be reset afterwards, or the first insert collides on a + duplicate primary key: + ```sql + SELECT setval('statistics_id_seq', (SELECT MAX(id) FROM statistics)); + SELECT setval('statistics_meta_id_seq', (SELECT MAX(id) FROM statistics_meta)); + SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term)); + ``` +5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives + there, not here: + ```yaml + recorder: + db_url: !secret recorder_db_url # postgresql://homeassistant:@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant + ``` + The password is in the CNPG-generated `ha-pg-app` Secret. It must not be + committed — render it into `secrets.yaml` the same way the Z2M MQTT password + is handled in `zigbee2mqtt-deployment.yaml`. +6. **Verify** the energy dashboard renders history back to 2024, not just today. + That is the check that proves the statistics migration worked; "HA starts and + records" does not. + +### Rollback + +Revert `db_url` to the SQLite path and restore the `.backup` copy. Keep the +SQLite file until the energy dashboard has been confirmed over a full day — +deleting it early removes the only rollback that preserves the 2.4 years. + +## After the migration + +- The HA config PVC shrinks from 5.5 GB to a few hundred MB. Do not shrink the + PVC itself: `ceph-block` allows expansion, not reduction. +- Re-measure. `kubectl -n home-assistant exec deploy/home-assistant -c + home-assistant -- cat /sys/fs/cgroup/io.stat` was 40 GB/day before. If it does + not drop substantially, the migration did not achieve its purpose and that + should be recorded rather than assumed. +- `synchronous_commit: off` in `ha-pg-cluster.yaml` is the single biggest + contributor to the reduction. It is safe for telemetry (Postgres stays crash + consistent; at most a few hundred ms of commits are lost on unclean shutdown) + but it is a deliberate trade and should not be "corrected" later without + understanding why it is set. diff --git a/kubernetes/apps/home-assistant/ha-pg-cluster.yaml b/kubernetes/apps/home-assistant/ha-pg-cluster.yaml new file mode 100644 index 00000000..15848e87 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-cluster.yaml @@ -0,0 +1,90 @@ +--- +# Postgres backend for the Home Assistant recorder. +# +# WHY THIS EXISTS +# The recorder's SQLite database is the single largest client writer in the +# cluster. Measured 2026-08-19 (cgroup io.stat on the home-assistant container, +# corroborated by kernel diskstats): 40 GB/day to its RBD volume, ~26% of all +# client writes, against only ~5 GB/day of actual row data. The gap is the write +# PATTERN, not the volume: +# +# ~30 state changes/sec (107k rows/hour, 2.57M/day) +# x2 SQLite WAL + checkpoint — every dirtied page is written twice +# x4 ext4 journal + commit, because synchronous=FULL fsyncs every commit +# (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device) +# = ~15 KB written per recorded state change +# +# Postgres replaces scattered fsync'd page rewrites with sequential, batched WAL. +# That is the point of the move — it attacks the multiplier rather than trimming +# what gets recorded, which would only hide the problem. +# +# STAGING +# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite. +# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving +# to Hetzner, and a successful base backup before going further. +# Phase 2: migrate the statistics tables and repoint recorder.db_url. +# See MIGRATION-recorder-postgres.md for the sequence and the rollback. +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: ha-pg + namespace: home-assistant +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:16 + storage: + storageClass: ceph-block + # Sized for the migrated statistics plus 7 days of states at the current + # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with + # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window + # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn. + # ceph-block has allowVolumeExpansion: true, so this can grow in place. + size: 20Gi + plugins: + - name: barman-cloud.cloudnative-pg.io + isWALArchiver: true + parameters: + barmanObjectName: ha-pg-store + bootstrap: + initdb: + # Explicit, not left to CNPG's `app` default: the generated ha-pg-app + # Secret and the -app Service point at whatever is named here, and the + # recorder db_url in phase 2 must match. Same trap that made the wp-pg + # namespace migration point credentials at an empty database (#433). + database: homeassistant + owner: homeassistant + postgresql: + parameters: + shared_buffers: "256MB" + max_connections: "50" + # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the + # entire reason for the migration. Defaults would give a correct database + # that writes far more than it needs to on replicated Ceph storage. + # + # synchronous_commit off: the recorder is telemetry, not transactional + # record-keeping. This does NOT risk corruption — Postgres stays crash + # consistent; it only means the last few hundred ms of commits can be lost + # on an unclean shutdown, which for state history is a non-event. In + # exchange every commit stops forcing an fsync, which is precisely the + # cost that makes SQLite expensive here. + synchronous_commit: "off" + # Compress full-page images in WAL. Recorder writes are many small row + # inserts scattered across pages, so full-page writes after each + # checkpoint dominate WAL volume; compression cuts that directly. + wal_compression: "on" + # Fewer checkpoints => fewer full-page writes, since a page is only + # fully-logged on its first modification after a checkpoint. 15min with a + # 2GB ceiling keeps recovery time sane while cutting repeat full-page + # images for the hot rows the recorder rewrites constantly. + checkpoint_timeout: "15min" + max_wal_size: "2GB" + min_wal_size: "512MB" + # full_page_writes is deliberately LEFT ON (the default). RBD can tear a + # page across a 4 MiB object boundary, and turning this off would trade + # write volume for the risk of an unrecoverable database. Not worth it. + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + memory: 1Gi diff --git a/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml b/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml new file mode 100644 index 00000000..2f666186 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml @@ -0,0 +1,38 @@ +--- +# Barman S3 credentials for ha-pg. +# +# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where +# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical +# problem and the same note. The values are the same today, so this is correct +# but coupled: rotating paperless's Hetzner credentials would silently change +# the backup credentials for Home Assistant and Woodpecker too. +# +# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds) +# and a one-line change to the `key:` below. Creating it needs an OpenBao write: +# +# bao kv put homelab/seed/home-assistant/hetzner-creds \ +# @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data') +# +# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a +# 403 on `bao kv list seed`, which was a wrong-path error rather than a +# permissions problem. +# +# The namespace allowlist on the openbao ClusterSecretStore +# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this +# namespace or ESO silently never syncs (#505). home-assistant is already +# allowlisted — it has several ExternalSecrets in this directory already. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: hetzner-s3-creds + namespace: home-assistant +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao + target: + name: hetzner-s3-creds + dataFrom: + - extract: + key: seed/paperless/hetzner-creds diff --git a/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml b/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml new file mode 100644 index 00000000..444ec01a --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml @@ -0,0 +1,35 @@ +--- +# Barman object store for ha-pg: base backups + continuous WAL archiving to +# Hetzner Object Storage, same target bucket as the other CNPG clusters. +# +# Its own prefix. Two clusters archiving WAL to one destinationPath interleave +# their streams and corrupt both — see the note in +# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way. +apiVersion: barmancloud.cnpg.io/v1 +kind: ObjectStore +metadata: + name: ha-pg-store + namespace: home-assistant +spec: + configuration: + destinationPath: s3://renner-uno-k8s-cnpg/ha-pg + endpointURL: https://fsn1.your-objectstorage.com + s3Credentials: + accessKeyId: + name: hetzner-s3-creds + key: ACCESS_KEY_ID + secretAccessKey: + name: hetzner-s3-creds + key: ACCESS_SECRET_KEY + region: + name: hetzner-s3-creds + key: REGION + wal: + compression: gzip + data: + compression: gzip + # Matches the 30d retention the other CNPG clusters use and the + # renner-uno-k8s-cnpg bucket's noncurrent-version expiry in + # opentofu/object_storage.tf, so old S3 versions don't outlive the backups + # that superseded them. + retentionPolicy: "30d" diff --git a/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml b/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml new file mode 100644 index 00000000..fcdbc7e3 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml @@ -0,0 +1,23 @@ +--- +# Daily base backups for ha-pg. Continuous WAL archiving runs separately +# (ha-pg-cluster.yaml spec.plugins[].isWALArchiver); without periodic base +# backups the prefix would have no recovery point of its own, and +# ha-pg-store's retentionPolicy: 30d would have nothing to prune against, so the +# WAL chain would grow without bound. +# +# 02:50 UTC, offset from the other CNPG clusters (02:30, wp-pg at 02:40) so they +# do not all hit the Hetzner bucket at once. +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: ha-pg-daily + namespace: home-assistant +spec: + schedule: "0 50 2 * * *" + immediate: true + backupOwnerReference: self + cluster: + name: ha-pg + method: plugin + pluginConfiguration: + name: barman-cloud.cloudnative-pg.io diff --git a/kubernetes/apps/home-assistant/kustomization.yaml b/kubernetes/apps/home-assistant/kustomization.yaml index 197c1c2c..8f129c5c 100644 --- a/kubernetes/apps/home-assistant/kustomization.yaml +++ b/kubernetes/apps/home-assistant/kustomization.yaml @@ -52,6 +52,14 @@ resources: # CNP ingress rule. - zigbee2mqtt-gate-service.yaml - zigbee2mqtt-ingress.yaml + # Postgres backend for the recorder (MIGRATION-recorder-postgres.md). + # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still + # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics + # tables and repoints recorder.db_url in the homeassistant repo. + - ha-pg-hetzner-creds-externalsecret.yaml + - ha-pg-objectstore.yaml + - ha-pg-cluster.yaml + - ha-pg-scheduled-backup.yaml configMapGenerator: # nginx config for the reverse-proxy sidecar. Hash suffix disabled so # the cm name stays stable; editing proxy.conf does not auto-roll the diff --git a/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml b/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml index 3e059817..e65671f9 100644 --- a/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml +++ b/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml @@ -99,3 +99,26 @@ spec: interval: 30s podTargetLabels: - role +--- +# ha-pg — Postgres backend for the Home Assistant recorder +# (kubernetes/apps/home-assistant/ha-pg-cluster.yaml). Added with the cluster +# rather than after it, so the migration has metrics from the first moment +# instead of discovering the gap later, which is how wp-pg's scrape hole was +# found (#433). +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: cnpg-ha-pg + namespace: home-assistant + labels: + app.kubernetes.io/name: cloudnative-pg + release: kube-prometheus-stack +spec: + selector: + matchLabels: + cnpg.io/cluster: ha-pg + podMetricsEndpoints: + - port: metrics + interval: 30s + podTargetLabels: + - role diff --git a/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml b/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml index b1144fcc..b7033982 100644 --- a/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml +++ b/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml @@ -358,3 +358,67 @@ spec: - ports: - port: "8123" protocol: TCP +--- +# ha-pg (CNPG) — instance-manager access for the CloudNativePG operator. +# +# DEDICATED policy with its own endpointSelector, NOT another rule on the +# namespace-wide default-deny above. That policy has endpointSelector:{}, so any +# rule added there applies to EVERY pod in the namespace — and :8000 is served +# both by the CNPG instance manager and by other workloads, so a namespace-wide +# :8000 grant would expose far more than intended. Same reasoning as +# wp-pg-allow-cnpg-operator in woodpecker-ns.yaml, which was corrected during +# review for exactly this (#433). +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: ha-pg-allow-cnpg-operator + namespace: home-assistant +spec: + endpointSelector: + matchLabels: + cnpg.io/cluster: ha-pg + ingress: + - fromEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": database + app.kubernetes.io/name: cloudnative-pg + toPorts: + - ports: + - port: "8000" + protocol: TCP + # ICMP to the ha-pg pods. REQUIRED because the L4 rules in this policy + # select those endpoints, and Cilium drops ICMP to an endpoint selected by + # any layer 4 policy unless the traffic is CT-related. Without this, PMTUD + # Destination Unreachable / Packet Too Big toward ha-pg are dropped + # silently, and the symptom is stalled transfers rather than a policy + # denial — the failure mode documented in woodpecker-ns.yaml. + - fromEntities: + - cluster + icmps: + - fields: + - { type: 3, family: IPv4 } # Destination Unreachable — PMTUD + - { type: 1, family: IPv6 } # Destination Unreachable + - { type: 2, family: IPv6 } # Packet Too Big — PMTUD, mandatory +--- +# ha-pg — Postgres access for the Home Assistant recorder. +# +# Scoped to the HA pod on both ends: only home-assistant reaches ha-pg, and only +# on 5432. Nothing else in this namespace has any business in the recorder +# database. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: ha-pg-allow-home-assistant + namespace: home-assistant +spec: + endpointSelector: + matchLabels: + cnpg.io/cluster: ha-pg + ingress: + - fromEndpoints: + - matchLabels: + app.kubernetes.io/name: home-assistant + toPorts: + - ports: + - port: "5432" + protocol: TCP -- 2.54.0