diff --git a/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md b/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md new file mode 100644 index 00000000..27e18630 --- /dev/null +++ b/kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md @@ -0,0 +1,123 @@ +# Home Assistant recorder: SQLite → Postgres + +## Why + +The recorder's SQLite database is the single largest client writer in the +cluster. Measured 2026-08-19 (cgroup `io.stat` on the home-assistant container, +corroborated by kernel diskstats and Ceph pool counters): + +| | | +|---|---| +| HA writes to its RBD volume | **40 GB/day** | +| share of all client writes | **~26%** | +| actual row data underneath | **~5 GB/day** | + +The 8× gap is the write *pattern*, not the volume: + +``` +~30 state changes/sec (107k rows/hour, 2.57M/day) + ×2 SQLite WAL + checkpoint — every dirtied page written twice + ×4 ext4 journal + commit, because synchronous=FULL fsyncs every commit + (measured: 819 KB of 4 KiB writes → 3.28 MB at the block device) + = ~15 KB written per recorded state change +``` + +Postgres replaces scattered fsync'd page rewrites with sequential batched WAL. +This attacks the multiplier rather than trimming what gets recorded — excluding +entities would hide the problem, not fix it. + +**Context:** this is one of three write-reduction changes. The other two are +merged and verified — `size 3→2` on the Ceph pools (#519) and +`mon_compact_on_trim=false` (#520, which alone removed ~228 GB/day). See #434. + +## What is at risk + +`states` is disposable — `purge_keep_days: 7`, so it regenerates in a week. + +**`statistics` is not.** As of 2026-08-19: + +| table | rows | note | +|---|---|---| +| `statistics` | 10,793,712 | **spans 2024-03-13 → today, 2.4 years** | +| `statistics_short_term` | 1,902,624 | 5-minute buckets | +| `statistics_meta` | 1,448 | entity ↔ id mapping, required by both | +| `states` | 18,289,190 | disposable, 7-day retention | +| `events` | 483,989 | disposable | + +**Home Assistant does not migrate data between database engines.** Repointing +`db_url` at an empty Postgres silently starts from zero and the energy dashboard +loses its entire history. The long-term statistics are irreplaceable — they are +not in InfluxDB, which HA does not read for the energy dashboard. + +## Phase 1 — deploy the cluster (this PR) + +`ha-pg` comes up **empty** and nothing consumes it. HA stays on SQLite, so a +failure here is inert and there is nothing to roll back. + +Verify before going further: + +```sh +kubectl -n home-assistant get cluster ha-pg # Cluster in healthy state, 1/1 +kubectl -n home-assistant get backup # ha-pg-daily Completed (immediate: true) +kubectl -n home-assistant logs ha-pg-1 -c postgres | grep -i archive # WAL archiving, no errors +``` + +Confirm the backup actually landed in the bucket rather than trusting the CR +status — the WordPress backup failed nightly for three days against a bucket +that was declared but never created (see `opentofu/object_storage.tf`). + +## Phase 2 — migrate the data (NOT in this PR) + +Do not start until phase 1 is verified. + +**Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long +HA outage, and buys a table that is discarded within 7 days anyway. + +1. **Snapshot first.** Velero backup of the HA PVC, plus a `.backup` copy of + `home-assistant_v2.db`. This is the rollback. +2. **Stop the recorder** — scale `deploy/home-assistant` to 0. The database must + be quiescent; copying a live SQLite file gives a torn read. +3. **Create the schema.** Start HA once against the empty Postgres so it runs + its own migrations. Never hand-create the tables — the schema version must + match the HA version exactly or HA will refuse to start or silently migrate. +4. **Copy** `statistics_meta` → `statistics` → `statistics_short_term`, in that + order (the other two carry `metadata_id` foreign keys into the first). + Sequences must be reset afterwards, or the first insert collides on a + duplicate primary key: + ```sql + SELECT setval('statistics_id_seq', (SELECT MAX(id) FROM statistics)); + SELECT setval('statistics_meta_id_seq', (SELECT MAX(id) FROM statistics_meta)); + SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term)); + ``` +5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives + there, not here: + ```yaml + recorder: + db_url: !secret recorder_db_url # postgresql://homeassistant:@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant + ``` + The password is in the CNPG-generated `ha-pg-app` Secret. It must not be + committed — render it into `secrets.yaml` the same way the Z2M MQTT password + is handled in `zigbee2mqtt-deployment.yaml`. +6. **Verify** the energy dashboard renders history back to 2024, not just today. + That is the check that proves the statistics migration worked; "HA starts and + records" does not. + +### Rollback + +Revert `db_url` to the SQLite path and restore the `.backup` copy. Keep the +SQLite file until the energy dashboard has been confirmed over a full day — +deleting it early removes the only rollback that preserves the 2.4 years. + +## After the migration + +- The HA config PVC shrinks from 5.5 GB to a few hundred MB. Do not shrink the + PVC itself: `ceph-block` allows expansion, not reduction. +- Re-measure. `kubectl -n home-assistant exec deploy/home-assistant -c + home-assistant -- cat /sys/fs/cgroup/io.stat` was 40 GB/day before. If it does + not drop substantially, the migration did not achieve its purpose and that + should be recorded rather than assumed. +- `synchronous_commit: off` in `ha-pg-cluster.yaml` is the single biggest + contributor to the reduction. It is safe for telemetry (Postgres stays crash + consistent; at most a few hundred ms of commits are lost on unclean shutdown) + but it is a deliberate trade and should not be "corrected" later without + understanding why it is set. diff --git a/kubernetes/apps/home-assistant/ha-pg-cluster.yaml b/kubernetes/apps/home-assistant/ha-pg-cluster.yaml new file mode 100644 index 00000000..15848e87 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-cluster.yaml @@ -0,0 +1,90 @@ +--- +# Postgres backend for the Home Assistant recorder. +# +# WHY THIS EXISTS +# The recorder's SQLite database is the single largest client writer in the +# cluster. Measured 2026-08-19 (cgroup io.stat on the home-assistant container, +# corroborated by kernel diskstats): 40 GB/day to its RBD volume, ~26% of all +# client writes, against only ~5 GB/day of actual row data. The gap is the write +# PATTERN, not the volume: +# +# ~30 state changes/sec (107k rows/hour, 2.57M/day) +# x2 SQLite WAL + checkpoint — every dirtied page is written twice +# x4 ext4 journal + commit, because synchronous=FULL fsyncs every commit +# (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device) +# = ~15 KB written per recorded state change +# +# Postgres replaces scattered fsync'd page rewrites with sequential, batched WAL. +# That is the point of the move — it attacks the multiplier rather than trimming +# what gets recorded, which would only hide the problem. +# +# STAGING +# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite. +# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving +# to Hetzner, and a successful base backup before going further. +# Phase 2: migrate the statistics tables and repoint recorder.db_url. +# See MIGRATION-recorder-postgres.md for the sequence and the rollback. +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: ha-pg + namespace: home-assistant +spec: + instances: 1 + imageName: ghcr.io/cloudnative-pg/postgresql:16 + storage: + storageClass: ceph-block + # Sized for the migrated statistics plus 7 days of states at the current + # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with + # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window + # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn. + # ceph-block has allowVolumeExpansion: true, so this can grow in place. + size: 20Gi + plugins: + - name: barman-cloud.cloudnative-pg.io + isWALArchiver: true + parameters: + barmanObjectName: ha-pg-store + bootstrap: + initdb: + # Explicit, not left to CNPG's `app` default: the generated ha-pg-app + # Secret and the -app Service point at whatever is named here, and the + # recorder db_url in phase 2 must match. Same trap that made the wp-pg + # namespace migration point credentials at an empty database (#433). + database: homeassistant + owner: homeassistant + postgresql: + parameters: + shared_buffers: "256MB" + max_connections: "50" + # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the + # entire reason for the migration. Defaults would give a correct database + # that writes far more than it needs to on replicated Ceph storage. + # + # synchronous_commit off: the recorder is telemetry, not transactional + # record-keeping. This does NOT risk corruption — Postgres stays crash + # consistent; it only means the last few hundred ms of commits can be lost + # on an unclean shutdown, which for state history is a non-event. In + # exchange every commit stops forcing an fsync, which is precisely the + # cost that makes SQLite expensive here. + synchronous_commit: "off" + # Compress full-page images in WAL. Recorder writes are many small row + # inserts scattered across pages, so full-page writes after each + # checkpoint dominate WAL volume; compression cuts that directly. + wal_compression: "on" + # Fewer checkpoints => fewer full-page writes, since a page is only + # fully-logged on its first modification after a checkpoint. 15min with a + # 2GB ceiling keeps recovery time sane while cutting repeat full-page + # images for the hot rows the recorder rewrites constantly. + checkpoint_timeout: "15min" + max_wal_size: "2GB" + min_wal_size: "512MB" + # full_page_writes is deliberately LEFT ON (the default). RBD can tear a + # page across a 4 MiB object boundary, and turning this off would trade + # write volume for the risk of an unrecoverable database. Not worth it. + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + memory: 1Gi diff --git a/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml b/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml new file mode 100644 index 00000000..2f666186 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml @@ -0,0 +1,38 @@ +--- +# Barman S3 credentials for ha-pg. +# +# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where +# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical +# problem and the same note. The values are the same today, so this is correct +# but coupled: rotating paperless's Hetzner credentials would silently change +# the backup credentials for Home Assistant and Woodpecker too. +# +# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds) +# and a one-line change to the `key:` below. Creating it needs an OpenBao write: +# +# bao kv put homelab/seed/home-assistant/hetzner-creds \ +# @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data') +# +# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a +# 403 on `bao kv list seed`, which was a wrong-path error rather than a +# permissions problem. +# +# The namespace allowlist on the openbao ClusterSecretStore +# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this +# namespace or ESO silently never syncs (#505). home-assistant is already +# allowlisted — it has several ExternalSecrets in this directory already. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: hetzner-s3-creds + namespace: home-assistant +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao + target: + name: hetzner-s3-creds + dataFrom: + - extract: + key: seed/paperless/hetzner-creds diff --git a/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml b/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml new file mode 100644 index 00000000..444ec01a --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-objectstore.yaml @@ -0,0 +1,35 @@ +--- +# Barman object store for ha-pg: base backups + continuous WAL archiving to +# Hetzner Object Storage, same target bucket as the other CNPG clusters. +# +# Its own prefix. Two clusters archiving WAL to one destinationPath interleave +# their streams and corrupt both — see the note in +# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way. +apiVersion: barmancloud.cnpg.io/v1 +kind: ObjectStore +metadata: + name: ha-pg-store + namespace: home-assistant +spec: + configuration: + destinationPath: s3://renner-uno-k8s-cnpg/ha-pg + endpointURL: https://fsn1.your-objectstorage.com + s3Credentials: + accessKeyId: + name: hetzner-s3-creds + key: ACCESS_KEY_ID + secretAccessKey: + name: hetzner-s3-creds + key: ACCESS_SECRET_KEY + region: + name: hetzner-s3-creds + key: REGION + wal: + compression: gzip + data: + compression: gzip + # Matches the 30d retention the other CNPG clusters use and the + # renner-uno-k8s-cnpg bucket's noncurrent-version expiry in + # opentofu/object_storage.tf, so old S3 versions don't outlive the backups + # that superseded them. + retentionPolicy: "30d" diff --git a/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml b/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml new file mode 100644 index 00000000..fcdbc7e3 --- /dev/null +++ b/kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml @@ -0,0 +1,23 @@ +--- +# Daily base backups for ha-pg. Continuous WAL archiving runs separately +# (ha-pg-cluster.yaml spec.plugins[].isWALArchiver); without periodic base +# backups the prefix would have no recovery point of its own, and +# ha-pg-store's retentionPolicy: 30d would have nothing to prune against, so the +# WAL chain would grow without bound. +# +# 02:50 UTC, offset from the other CNPG clusters (02:30, wp-pg at 02:40) so they +# do not all hit the Hetzner bucket at once. +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: ha-pg-daily + namespace: home-assistant +spec: + schedule: "0 50 2 * * *" + immediate: true + backupOwnerReference: self + cluster: + name: ha-pg + method: plugin + pluginConfiguration: + name: barman-cloud.cloudnative-pg.io diff --git a/kubernetes/apps/home-assistant/kustomization.yaml b/kubernetes/apps/home-assistant/kustomization.yaml index 197c1c2c..8f129c5c 100644 --- a/kubernetes/apps/home-assistant/kustomization.yaml +++ b/kubernetes/apps/home-assistant/kustomization.yaml @@ -52,6 +52,14 @@ resources: # CNP ingress rule. - zigbee2mqtt-gate-service.yaml - zigbee2mqtt-ingress.yaml + # Postgres backend for the recorder (MIGRATION-recorder-postgres.md). + # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still + # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics + # tables and repoints recorder.db_url in the homeassistant repo. + - ha-pg-hetzner-creds-externalsecret.yaml + - ha-pg-objectstore.yaml + - ha-pg-cluster.yaml + - ha-pg-scheduled-backup.yaml configMapGenerator: # nginx config for the reverse-proxy sidecar. Hash suffix disabled so # the cm name stays stable; editing proxy.conf does not auto-roll the diff --git a/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml b/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml index 3e059817..e65671f9 100644 --- a/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml +++ b/kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml @@ -99,3 +99,26 @@ spec: interval: 30s podTargetLabels: - role +--- +# ha-pg — Postgres backend for the Home Assistant recorder +# (kubernetes/apps/home-assistant/ha-pg-cluster.yaml). Added with the cluster +# rather than after it, so the migration has metrics from the first moment +# instead of discovering the gap later, which is how wp-pg's scrape hole was +# found (#433). +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: cnpg-ha-pg + namespace: home-assistant + labels: + app.kubernetes.io/name: cloudnative-pg + release: kube-prometheus-stack +spec: + selector: + matchLabels: + cnpg.io/cluster: ha-pg + podMetricsEndpoints: + - port: metrics + interval: 30s + podTargetLabels: + - role diff --git a/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml b/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml index b1144fcc..b7033982 100644 --- a/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml +++ b/kubernetes/infrastructure/network-policies/home-assistant-ns.yaml @@ -358,3 +358,67 @@ spec: - ports: - port: "8123" protocol: TCP +--- +# ha-pg (CNPG) — instance-manager access for the CloudNativePG operator. +# +# DEDICATED policy with its own endpointSelector, NOT another rule on the +# namespace-wide default-deny above. That policy has endpointSelector:{}, so any +# rule added there applies to EVERY pod in the namespace — and :8000 is served +# both by the CNPG instance manager and by other workloads, so a namespace-wide +# :8000 grant would expose far more than intended. Same reasoning as +# wp-pg-allow-cnpg-operator in woodpecker-ns.yaml, which was corrected during +# review for exactly this (#433). +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: ha-pg-allow-cnpg-operator + namespace: home-assistant +spec: + endpointSelector: + matchLabels: + cnpg.io/cluster: ha-pg + ingress: + - fromEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": database + app.kubernetes.io/name: cloudnative-pg + toPorts: + - ports: + - port: "8000" + protocol: TCP + # ICMP to the ha-pg pods. REQUIRED because the L4 rules in this policy + # select those endpoints, and Cilium drops ICMP to an endpoint selected by + # any layer 4 policy unless the traffic is CT-related. Without this, PMTUD + # Destination Unreachable / Packet Too Big toward ha-pg are dropped + # silently, and the symptom is stalled transfers rather than a policy + # denial — the failure mode documented in woodpecker-ns.yaml. + - fromEntities: + - cluster + icmps: + - fields: + - { type: 3, family: IPv4 } # Destination Unreachable — PMTUD + - { type: 1, family: IPv6 } # Destination Unreachable + - { type: 2, family: IPv6 } # Packet Too Big — PMTUD, mandatory +--- +# ha-pg — Postgres access for the Home Assistant recorder. +# +# Scoped to the HA pod on both ends: only home-assistant reaches ha-pg, and only +# on 5432. Nothing else in this namespace has any business in the recorder +# database. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: ha-pg-allow-home-assistant + namespace: home-assistant +spec: + endpointSelector: + matchLabels: + cnpg.io/cluster: ha-pg + ingress: + - fromEndpoints: + - matchLabels: + app.kubernetes.io/name: home-assistant + toPorts: + - ports: + - port: "5432" + protocol: TCP