feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster) #521
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#521
Reference in New Issue
Block a user
Refs #434, #397. Third of the write-reduction changes, after #519 (size 2) and #520 (mon compaction, which alone removed ~228 GB/day).
Why
The recorder SQLite database is the largest single client writer in the cluster. Measured 2026-08-19 via cgroup
io.staton the home-assistant container, corroborated by kernel diskstats and Ceph pool counters:The 8x gap is the write pattern, not the volume:
Postgres replaces scattered fsync-ed page rewrites with sequential batched WAL. This attacks the multiplier rather than trimming what gets recorded — excluding entities would hide the problem, not fix it.
Phase 1 only — this is inert
The cluster comes up empty and nothing consumes it. HA stays on SQLite. There is nothing to roll back if it misbehaves.
Verify before phase 2:
…and confirm the objects actually landed in the bucket rather than trusting the CR status — the WordPress backup failed nightly for three days against a bucket that was declared but never created.
What phase 2 must not get wrong
HA does not migrate data between database engines. Recorded now while it is fresh:
statisticsstatistics_short_termstatistics_metastatespurge_keep_days: 7Repointing
db_urlat an empty Postgres silently discards the energy history. It is not in InfluxDB — HA does not read InfluxDB for the energy dashboard.MIGRATION-recorder-postgres.mdhas the full sequence, sequence-reset SQL, and rollback.Postgres tuning
Deliberately tuned to minimise write volume, since that is the point:
synchronous_commit: offwal_compression: oncheckpoint_timeout: 15min,max_wal_size: 2GBfull_page_writesAlso included
destinationPathinterleave WAL streams and corrupt both.endpointSelector, not rules on the namespace-wide default-deny — that one isendpointSelector: {}, so any rule there applies to every pod in the namespace, and:8000is served by more than the CNPG instance manager. Includes the ICMP allowance that attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD.Note on #397
The hetzner-creds ExternalSecret reuses
seed/paperless/hetzner-creds— the same coupling #397 tracks for woodpecker, so this adds a third consumer rather than a new problem. The file carries the exactbaocommand to fix it.Worth correcting in #397 itself: the recorded blocker ("
bao kv list seed-> 403") was a wrong-path error, not missing permissions. The KV mount ishomelab, so the path ishomelab/seed/..., which lists fine.🤖 Generated with Claude Code
The recorder's SQLite database is the largest client writer in the cluster. Measured 2026-08-19 via cgroup io.stat on the home-assistant container, corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD volume, ~26% of all client writes, against only ~5 GB/day of actual row data. The 8x gap is the write PATTERN, not the volume: ~30 state changes/sec (107k rows/hour, 2.57M/day) x2 SQLite WAL + checkpoint -- every dirtied page written twice x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device) = ~15 KB written per recorded state change Postgres replaces scattered fsync'd page rewrites with sequential batched WAL. This attacks the multiplier rather than trimming what gets recorded, which would hide the problem instead of fixing it. PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves. Phase 2 migrates the statistics tables and repoints recorder.db_url in the homeassistant repo; MIGRATION-recorder-postgres.md has the sequence. What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT migrate data between database engines. `statistics` holds 10,793,712 rows spanning 2024-03-13 to today -- 2.4 years of energy history that is not in InfluxDB and cannot be reconstructed. Repointing db_url at an empty database silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7. Postgres is tuned to minimise write volume, which is the point of the move: synchronous_commit off -- the recorder is telemetry, not transactional record-keeping. Postgres stays crash consistent; only the last few hundred ms of commits are at risk on unclean shutdown. In exchange, commits stop forcing an fsync -- the exact cost that makes SQLite expensive here. wal_compression on -- recorder writes are small inserts scattered across pages, so post-checkpoint full-page writes dominate WAL volume. checkpoint_timeout 15min / max_wal_size 2GB -- fewer checkpoints means fewer repeat full-page images for the hot rows the recorder rewrites. full_page_writes -- deliberately LEFT ON. RBD can tear a page across a 4 MiB object boundary; trading that for write volume risks an unrecoverable database. Also included: - ObjectStore on its own s3 prefix (two clusters sharing a destinationPath interleave WAL streams and corrupt both). - ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters. - PodMonitor added WITH the cluster rather than after it, so the migration has metrics from the start instead of discovering the gap later (#433). - Cilium policies as DEDICATED policies with their own endpointSelector, not rules on the namespace-wide default-deny -- that policy is endpointSelector:{} so any rule there applies to every pod in the namespace, and :8000 is served by more than the CNPG instance manager. Includes the ICMP allowance that attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD. The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same coupling #397 tracks for woodpecker. Noted in the file with the exact bao command to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed` was a wrong-path error rather than missing permissions. Refs #434, #397 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit196ebda1889cae1daf76df6dc5369d6be3be56a9.Summary
Phase 1 of the HA recorder SQLite→Postgres migration: a new, empty CNPG cluster (
ha-pg) with its own S3 backup prefix, scheduled backup, PodMonitor, and dedicated Cilium policies. Nothing in the cluster consumes it yet (HA stays on SQLite), so the change is inert as claimed. It closely mirrors the existingwp-pg/pprl-pgclusters in structure, resource sizing, and backup conventions, and I could not find a defect that would make this phase misbehave. One network-policy claim in the diff doesn't hold given pre-existing rules elsewhere in the namespace — see Suggestions. No blocking issues.🟡 Suggestions
kubernetes/infrastructure/network-policies/home-assistant-ns.yaml:403-424(ha-pg-allow-home-assistant) — the comment states this scopes Postgres access to "only home-assistant... Nothing else in this namespace has any business in the recorder database." That isn't actually enforced: the namespace'sdefault-deny-ingresspolicy (same file, lines 9-14) already grants every pod inhome-assistant(mosquitto, whisper, matter-server, speech-to-phrase, openwakeword, zwave-js, zone-configurator, code-server, esphome, zigbee2mqtt, shelly-manager, …) ingress to any port on any other pod in the namespace, with notoPortsrestriction. Sinceha-pg's pods will match thatendpointSelector: {}policy too, they're already reachable on 5432 (and every other port) by every sibling pod, regardless of the new dedicated policy — Cilium allow-rules are additive/OR'd, so a narrower rule elsewhere can't subtract from a broader one. Same reasoning applies to the namespace-wide "Prometheus scrape" rule (lines 105-108 of the same file), which similarly grantsmonitoringunrestricted (not just:metrics) access toha-pg.This doesn't block phase 1 (cluster is empty, and network reachability alone doesn't grant DB auth — an attacker would still need the CNPG-generated app credentials), but the isolation the comment describes is not real, and that gap matters more once phase 2 lands 2.4 years of statistics behind it. Worth tightening the sibling-wide rule with a
toPortsallowlist (ashome-assistant-ns.yamlalready does for the various:8080gate-outpost rules) before relying onha-pg-allow-home-assistantas an actual boundary — or at least correct the comment so it doesn't overstate the isolation.kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml— reusesseed/paperless/hetzner-creds, openly documented as a known, tracked coupling (#397) shared with woodpecker. Fine for now per the PR's own reasoning, just flagging that this is the third consumer of a credential rotation that isn't scoped to any of the three.Tests
No automated tests apply to this repo's YAML manifests; verification is the
kubectlchecklist in the PR body andMIGRATION-recorder-postgres.md(Cluster healthy, backup completed, WAL archiving present, bucket contents actually checked). That's consistent with how the siblingwp-pg/pprl-pgclusters were verified in this repo's history, and is reasonable for a phase that stops at "empty cluster exists." I did not have cluster access to run these checks myself, so I verified the manifests structurally (API fields, resource sizing, cross-references between the Cluster/ObjectStore/ExternalSecret/ScheduledBackup/PodMonitor/CiliumNetworkPolicy) against the equivalent livewp-pgandpprl-pgresources and found them consistent — sameplugins/bootstrap.initdb/resourcesshape, matchingbarmanObjectNamereferences, distinct S3 prefix (s3://renner-uno-k8s-cnpg/ha-pg, avoiding the WAL-interleaving bug the PR calls out), and a cron offset (02:50 UTC) that doesn't collide withwp-pg(02:40).Recommendation
Looks reasonable — no blocking issues; the network-policy comment above is worth a follow-up before phase 2 introduces real data, but doesn't block this phase.