Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
b34450b22e | ||
 k.renner
|
3faddcadcd | ||
|
claude-botandClaude Opus 5
|
6ae734eef4 | ||
|
claude-botandClaude Opus 5
|
39885b9400 | ||
 security-bot
|
694389b569 | ||
|
claude-botandClaude Opus 5
|
57bb39e7ee | ||
|
claude-botandClaude Opus 5
|
17f913ba2f |
@@ -121,7 +121,9 @@ spec:
|
||||
persistentVolumeClaim:
|
||||
claimName: audiobookshelf-books
|
||||
- name: le-ca
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -66,7 +66,52 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
|
||||
status — the WordPress backup failed nightly for three days against a bucket
|
||||
that was declared but never created (see `opentofu/object_storage.tf`).
|
||||
|
||||
## Phase 2 — migrate the data (NOT in this PR)
|
||||
## Phase 2b — bulk statistics copy: DONE 2026-08-19
|
||||
|
||||
Completed with **HA running** — the SQLite file was read through a read-only
|
||||
mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
|
||||
was needed for the bulk copy.
|
||||
|
||||
```
|
||||
statistics_meta: sqlite=1449 pg=1449 OK
|
||||
statistics: sqlite=10799904 pg=10799904 OK (153.9s)
|
||||
statistics_short_term: sqlite=1969804 pg=1969804 OK ( 27.0s)
|
||||
|
||||
statistics span in Postgres: 2024-03-13 -> 2026-08-19
|
||||
entities with long-term history: 1449
|
||||
rows carrying sum (energy dashboard): 2627859
|
||||
database size: 2252 MB (PVC is 20Gi)
|
||||
```
|
||||
|
||||
`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
|
||||
`COPY` the planner has no statistics, and HA queries these tables constantly for
|
||||
history and energy views, so without it every view seq-scans 10.8M rows.
|
||||
|
||||
Two schema details that cost a failed first attempt, recorded so the top-up does
|
||||
not repeat them:
|
||||
|
||||
- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
|
||||
will not coerce; convert explicitly and preserve NULL.
|
||||
- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
|
||||
both bulk tables (superseded by the `_ts` variants), which is what makes a
|
||||
tab-separated `COPY` safe — nothing in those tables needs escaping.
|
||||
|
||||
The schema itself was created with HA's own models
|
||||
(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
|
||||
left EMPTY on purpose. HA's `_initialize_database` detects a new database by
|
||||
looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
|
||||
itself. Pre-stamping that row would skip HA's own initialisation path.
|
||||
|
||||
### High-water marks for the cutover top-up
|
||||
|
||||
Rows written after the bulk copy are not yet in Postgres. Resume from:
|
||||
|
||||
```
|
||||
statistics max_id=13323904 max_start_ts=1787162400.0
|
||||
statistics_short_term max_id=159205762 max_start_ts=1787166000.0
|
||||
```
|
||||
|
||||
## Phase 2c — cutover (REMAINING)
|
||||
|
||||
Do not start until phase 1 is verified.
|
||||
|
||||
@@ -90,7 +135,18 @@ HA outage, and buys a table that is discarded within 7 days anyway.
|
||||
SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
|
||||
```
|
||||
5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
|
||||
there, not here:
|
||||
there, not here. The `recorder_db_url` secret it references is already
|
||||
rendered into `secrets.yaml` by the `render-db-url` init container in
|
||||
`deployment.yaml` (phase 2a), so this order is mandatory: **the init
|
||||
container must be deployed first**, or HA refuses to start on a missing
|
||||
secret.
|
||||
|
||||
The init container percent-encodes the credentials and validates the
|
||||
resulting YAML rather than counting lines. That is not incidental — the CNPG
|
||||
password is random, and a naive `"postgresql://user:pass@host/db"` string
|
||||
makes `secrets.yaml` unparseable the moment the password contains a quote,
|
||||
which takes out *every* secret HA loads, not just this one. Verified against
|
||||
a deliberately hostile password before merge.
|
||||
```yaml
|
||||
recorder:
|
||||
db_url: !secret recorder_db_url # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
|
||||
|
||||
@@ -36,6 +36,64 @@ spec:
|
||||
# (sml2mqtt bridge owns the USB device on edge), so it no longer needs
|
||||
# the edge USB device and schedules on the general cluster pool. This
|
||||
# gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
|
||||
initContainers:
|
||||
# Puts CNPG's connection URI into /config/secrets.yaml as
|
||||
# `recorder_db_url`, for the recorder's move off SQLite
|
||||
# (MIGRATION-recorder-postgres.md).
|
||||
#
|
||||
# HA has no way to read a db_url from the environment — `!env_var` was
|
||||
# removed from its YAML loader — so `!secret` is the only mechanism, and
|
||||
# secrets.yaml is PVC state that config-sync deliberately does not
|
||||
# manage. Hence a small init container rather than a env: entry.
|
||||
#
|
||||
# The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
|
||||
# percent-encodes it, so there is nothing to escape here: the value is
|
||||
# restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
|
||||
# An earlier version of this assembled the URI from user/password/dbname
|
||||
# and needed percent-encoding plus YAML round-trip validation to be
|
||||
# safe. That was solving a problem CNPG had already solved.
|
||||
#
|
||||
# Re-rendered on every pod start, so a CNPG password rotation takes
|
||||
# effect rather than silently leaving a stale URI behind.
|
||||
#
|
||||
# INERT until the recorder is repointed: nothing reads this key until
|
||||
# `db_url: !secret recorder_db_url` lands in configuration.yaml in the
|
||||
# homeassistant repo. That ordering is required — HA refuses to start
|
||||
# when a referenced secret is missing.
|
||||
- name: render-db-url
|
||||
image: busybox:1.36
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
S=/config/secrets.yaml
|
||||
T=/config/.secrets.render.tmp
|
||||
[ -f "$S" ] || : > "$S"
|
||||
# Filter-then-append, never rewrite in place: this file holds
|
||||
# every other HA secret (influxdb_token and friends).
|
||||
grep -v '^recorder_db_url:' "$S" > "$T" || true
|
||||
printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
|
||||
# Refuse to shrink the file. Cheap guard against the one failure
|
||||
# that would actually hurt: losing the other keys.
|
||||
[ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
|
||||
echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
|
||||
chmod 600 "$T"
|
||||
mv "$T" "$S"
|
||||
echo "recorder_db_url rendered"
|
||||
env:
|
||||
- name: DB_URI
|
||||
valueFrom:
|
||||
secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 32Mi
|
||||
containers:
|
||||
- name: home-assistant
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
@@ -221,9 +279,13 @@ spec:
|
||||
# Writable emptyDir for the nginx entrypoint's envsubst output
|
||||
# (/etc/nginx/conf.d/default.conf rendered from the .template).
|
||||
- name: proxy-conf-d
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
# code-server's settings/extensions cache. Not on the config PVC
|
||||
# so installed extensions don't pollute HA's /config listing.
|
||||
# Trade-off: extensions reset across pod restarts.
|
||||
- name: vscode-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -313,7 +313,9 @@ spec:
|
||||
configMap:
|
||||
name: mosquitto-config
|
||||
- name: auth
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
- name: tls
|
||||
secret:
|
||||
secretName: mosquitto-tls
|
||||
|
||||
@@ -344,9 +344,13 @@ spec:
|
||||
configMap:
|
||||
name: zigbee2mqtt-interview-watch-script
|
||||
- name: interview-watch-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
- name: discovery-guard-script
|
||||
configMap:
|
||||
name: zigbee2mqtt-discovery-guard-script
|
||||
- name: discovery-guard-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -147,7 +147,9 @@ spec:
|
||||
- key: immich-config.json
|
||||
path: immich-config.json
|
||||
- name: le-ca
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -13,10 +13,6 @@ resources:
|
||||
- prometheusrule-netbird-ingress.yaml
|
||||
- prometheusrule-velero.yaml
|
||||
- prometheusrule-woodpecker-rotation.yaml
|
||||
# CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
|
||||
# alerted if that stopped — HetznerBackupEndpointDown only probes the S3
|
||||
# endpoint, which proves reachability, not that anything was written.
|
||||
- prometheusrule-cnpg-backup.yaml
|
||||
- alloy-servicemonitor.yaml
|
||||
- cnpg-podmonitors.yaml
|
||||
- home-assistant-servicemonitor.yaml
|
||||
|
||||
@@ -1,78 +0,0 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: cnpg-backup
|
||||
namespace: monitoring
|
||||
labels:
|
||||
# kube-prometheus-stack's Prometheus resource selects on this label;
|
||||
# without it the rule is created but never loaded into Prometheus.
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
groups:
|
||||
- name: cnpg-backup.rules
|
||||
rules:
|
||||
# WHY THIS FILE EXISTS
|
||||
# Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
|
||||
# stopped. The only related rule was HetznerBackupEndpointDown, which
|
||||
# probes the S3 endpoint — it proves the bucket is reachable, not that
|
||||
# anything was written to it. Same failure shape as the WordPress backup
|
||||
# that failed nightly for three days against a bucket that was declared
|
||||
# in tofu but never applied: the status surface looked fine because
|
||||
# nothing checked the thing that mattered.
|
||||
|
||||
- alert: CNPGWalArchiveBacklog
|
||||
# `ready` counts WAL segments sitting in pg_wal/archive_status waiting
|
||||
# to be archived. Steady state is 0 across every cluster (verified
|
||||
# 2026-08-19). A sustained backlog means the archiver is failing:
|
||||
# point-in-time recovery is silently broken from that moment, and
|
||||
# pg_wal keeps growing until it fills the PVC and takes Postgres down.
|
||||
#
|
||||
# This is the load-bearing alert of the two. It depends only on the
|
||||
# instance-level collector, so unlike the backup-age metrics below it
|
||||
# is unaffected by the missing Backup CRs (#522).
|
||||
#
|
||||
# Threshold: >5 segments sustained for 30m. Brief non-zero values are
|
||||
# normal under write bursts — a segment is 16 MiB, so 5 is roughly
|
||||
# 80 MiB of un-archived WAL, well below anything that threatens the
|
||||
# volume but far above idle noise.
|
||||
expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
|
||||
|
||||
- alert: CNPGBackupFailed
|
||||
# cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
|
||||
# when no failure is recorded. Guard on > 0 so the epoch-zero default
|
||||
# is not read as "failed in 1970".
|
||||
expr: |
|
||||
cnpg_collector_last_failed_backup_timestamp > 0
|
||||
and
|
||||
(time() - cnpg_collector_last_failed_backup_timestamp) < 86400
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
|
||||
|
||||
# DELIBERATELY ABSENT: a backup-age alert, which is the one you would
|
||||
# most want here.
|
||||
#
|
||||
# The natural expression is
|
||||
# (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
|
||||
# and it CANNOT be used today: that metric reads exactly 0 on every
|
||||
# instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
|
||||
# pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
|
||||
# successful backup anywhere, because the Backup CRs it derives that
|
||||
# from are being deleted within a minute of completing — see #522.
|
||||
#
|
||||
# Backups themselves DO work; that was verified independently by
|
||||
# listing the bucket with barman-cloud-backup-list and restoring a WAL
|
||||
# segment back out of S3. The gap is in the record, not the data.
|
||||
#
|
||||
# Adding the rule now would fire permanently for all seven clusters and
|
||||
# be silenced within a day, which is worse than not having it. Add it as
|
||||
# part of resolving #522, once the metric reflects reality.
|
||||
@@ -64,7 +64,9 @@ spec:
|
||||
secret:
|
||||
secretName: pve-exporter-config
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -218,7 +218,9 @@ spec:
|
||||
persistentVolumeClaim:
|
||||
claimName: paperless-consume
|
||||
- name: bridge-cert
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -156,4 +156,6 @@ spec:
|
||||
# each pod enrolls its own peer and a restart re-enrolls cleanly
|
||||
# (the old peer is reaped).
|
||||
- name: state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -5,7 +5,7 @@ metadata:
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
cephVersion:
|
||||
image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
|
||||
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
|
||||
allowUnsupported: false
|
||||
dataDirHostPath: /var/lib/rook
|
||||
mon:
|
||||
|
||||
@@ -38,7 +38,7 @@ spec:
|
||||
serviceAccountName: rook-ceph-default
|
||||
containers:
|
||||
- name: rook-ceph-tools
|
||||
image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
|
||||
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
@@ -164,7 +164,9 @@ spec:
|
||||
name: rook-config-override
|
||||
optional: true
|
||||
- name: ceph-config
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
tolerations:
|
||||
- key: "node.kubernetes.io/unreachable"
|
||||
operator: "Exists"
|
||||
|
||||
@@ -85,4 +85,6 @@ spec:
|
||||
name: restore-test-script
|
||||
defaultMode: 0755
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
Reference in New Issue
Block a user