feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster) #521
Merged
k.renner
merged 1 commits from 2026-08-19 13:24:33 +00:00
feat-ha-recorder-postgres into master
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  claude-botandClaude Opus 5
|
196ebda188 |
feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster)
The recorder's SQLite database is the largest client writer in the cluster.
Measured 2026-08-19 via cgroup io.stat on the home-assistant container,
corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD
volume, ~26% of all client writes, against only ~5 GB/day of actual row data.
The 8x gap is the write PATTERN, not the volume:
~30 state changes/sec (107k rows/hour, 2.57M/day)
x2 SQLite WAL + checkpoint -- every dirtied page written twice
x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit
(measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
= ~15 KB written per recorded state change
Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
This attacks the multiplier rather than trimming what gets recorded, which would
hide the problem instead of fixing it.
PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on
SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves.
Phase 2 migrates the statistics tables and repoints recorder.db_url in the
homeassistant repo; MIGRATION-recorder-postgres.md has the sequence.
What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT
migrate data between database engines. `statistics` holds 10,793,712 rows
spanning 2024-03-13 to today -- 2.4 years of energy history that is not in
InfluxDB and cannot be reconstructed. Repointing db_url at an empty database
silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7.
Postgres is tuned to minimise write volume, which is the point of the move:
synchronous_commit off -- the recorder is telemetry, not transactional
record-keeping. Postgres stays crash consistent;
only the last few hundred ms of commits are at risk
on unclean shutdown. In exchange, commits stop
forcing an fsync -- the exact cost that makes SQLite
expensive here.
wal_compression on -- recorder writes are small inserts scattered across
pages, so post-checkpoint full-page writes dominate
WAL volume.
checkpoint_timeout 15min / max_wal_size 2GB
-- fewer checkpoints means fewer repeat full-page
images for the hot rows the recorder rewrites.
full_page_writes -- deliberately LEFT ON. RBD can tear a page across a
4 MiB object boundary; trading that for write volume
risks an unrecoverable database.
Also included:
- ObjectStore on its own s3 prefix (two clusters sharing a destinationPath
interleave WAL streams and corrupt both).
- ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters.
- PodMonitor added WITH the cluster rather than after it, so the migration has
metrics from the start instead of discovering the gap later (#433).
- Cilium policies as DEDICATED policies with their own endpointSelector, not
rules on the namespace-wide default-deny -- that policy is endpointSelector:{}
so any rule there applies to every pod in the namespace, and :8000 is served
by more than the CNPG instance manager. Includes the ICMP allowance that
attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD.
The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same
coupling #397 tracks for woodpecker. Noted in the file with the exact bao command
to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed`
was a wrong-path error rather than missing permissions.
Refs #434, #397
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|