Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 b34450b22e fix: make small scratch emptyDirs RAM-backed instead of node-disk
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.

Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.

HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:

  /var/lib/code-server   8.0K
  /mosquitto/auth        8.0K
  /state (z2m guards)    8.0K

So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.

Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.

Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.

DELIBERATELY NOT CONVERTED:
  - kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
    and generated from the chart; a hand edit would be lost on upgrade.
  - trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.

Cost: this rolls the affected pods once.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 06:30:23 +02:00
k.renner 3faddcadcd Merge pull request 'feat(home-assistant): render recorder_db_url from the ha-pg Secret' (#524) from feat-ha-recorder-db-url into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#524
2026-08-19 19:24:38 +00:00
claude-botandClaude Opus 5 6ae734eef4 refactor(home-assistant): use CNPG's own URI instead of rebuilding it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Net -44 lines. The previous version assembled the connection URI from
username/password/dbname, which forced percent-encoding, YAML-safe quoting via
json.dumps, a YAML round-trip validation, and python3 from HA's image.

None of that was necessary: CNPG already publishes a finished, percent-encoded
connection string in the app secret as `uri` / `fqdn-uri`. Reading that key
removes the whole class of problem the earlier review correctly flagged --
rather than solving it more carefully.

Verified against the live secret before simplifying: fqdn-uri contains only
[A-Za-z0-9:/@._%-], so naive double-quoting is provably safe and the YAML
validation was guarding against an input that cannot occur.

What is kept, because it is the part that actually matters: secrets.yaml is PVC
state holding every other HA secret (influxdb_token and friends), so the file is
filtered-and-appended rather than rewritten, moved atomically, and guarded
against shrinking. Tested in HA's own busybox shell -- comment and pre-existing
keys preserved, idempotent across runs, no duplicate key.

Also records why an env var cannot be used at all: HA removed `!env_var` from
its YAML loader, so `!secret` is the only mechanism for recorder.db_url, and
config-sync deliberately does not manage secrets.yaml.

Back to busybox; HA's image is no longer needed for this.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 21:18:27 +02:00
claude-botandClaude Opus 5 39885b9400 docs(home-assistant): record the completed statistics migration
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Phase 2b is done: 12.77M statistics rows copied SQLite -> Postgres with HA
running and no downtime (read-only mount; SQLite WAL gives readers a consistent
snapshot).

  statistics_meta        1449       OK
  statistics             10799904   OK   153.9s
  statistics_short_term  1969804    OK    27.0s

  span in Postgres: 2024-03-13 -> 2026-08-19
  1449 entities with long-term history, 2627859 rows carrying sum
  database size 2252 MB against a 20Gi PVC

Records the high-water marks the cutover top-up must resume from, and two schema
details that cost a failed first attempt:

  - has_mean/has_sum are INTEGER in SQLite, BOOLEAN in Postgres, and Postgres
    will not coerce them.
  - the legacy created/start/last_reset DATETIME columns are 100% NULL in both
    bulk tables, which is what makes a tab-separated COPY safe.

Also documents why the schema was built from HA's own models with schema_changes
left EMPTY: HA's _initialize_database detects a new database via the
events.time_fired_ts index and writes SCHEMA_VERSION itself, so pre-stamping that
row would bypass HA's own initialisation.

ANALYZE was run on all three tables. After a bulk COPY the planner has no
statistics and HA seq-scans 10.8M rows on every history/energy view.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 21:10:26 +02:00
security-bot 694389b569 chore(image-automation): bump quay.io/ceph/ceph:v19.2.6
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-19 15:43:41 +00:00
claude-botandClaude Opus 5 57bb39e7ee fix(home-assistant): percent-encode db_url and validate the rendered YAML
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review catch on #524, and it was a genuine blocker.

The first version built "postgresql://${PGUSER}:${PGPASSWORD}@host/db" in
busybox sh with no encoding. Two defects:

  1. The CNPG-generated password is random and may contain @ : / ? #, each of
     which changes how the URL parses. This repo already hit this exact issue
     with the same CNPG password-generation mechanism -- see
     kubernetes/apps/woodpecker/server.yaml, which switched to a lib/pq
     keyword DSN specifically to avoid escaping generated passwords. That
     option does not exist here: HA's recorder.db_url must be a SQLAlchemy URL.
     So the fix is percent-encoding.

  2. Worse, and the reason this was worth blocking: a quote in the password
     makes the YAML line malformed, so secrets.yaml stops parsing entirely and
     HA loses EVERY secret, not just this one. The line-count guard could never
     catch that -- the count is unchanged; the file is simply invalid.

Rewritten to run python3 from HA's own image (already pulled for the main
container, so no extra image) instead of busybox:

  - urllib.parse.quote(..., safe="") on user, password and dbname.
  - json.dumps for the YAML value: YAML is a JSON superset, so this emits a
    correctly escaped double-quoted scalar even with quotes or backslashes.
  - The file is still rewritten LINE-WISE rather than round-tripped through the
    YAML parser, so hand-maintained comments and key order survive.
  - The guard now validates the RESULT: parse the new content, assert no key
    from the old file disappeared, and assert recorder_db_url round-trips to
    exactly the URL that was built. That catches corruption the line count
    cannot.

Verified in HA's image against a deliberately hostile password
(p@ss:w/o"rd#?&\x=1) before commit:

  new version   -> parses, comment preserved, idempotent, password decodes back
                   byte-exact, host and dbname parse correctly
  naive version -> yaml.safe_load raises ParserError

i.e. the approach this replaces would have produced an unparseable
secrets.yaml.

Also updates MIGRATION-recorder-postgres.md for this step landing, including
why the ordering (init container before the configuration.yaml change) is
mandatory.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:58:50 +02:00
claude-botandClaude Opus 5 17f913ba2f feat(home-assistant): render recorder_db_url from the ha-pg Secret
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Phase 2a of the recorder move off SQLite (MIGRATION-recorder-postgres.md).
Adds an init container that writes `recorder_db_url` into /config/secrets.yaml
from the CNPG-generated ha-pg-app Secret.

INERT until the recorder is repointed. It only adds a key to secrets.yaml;
nothing reads it until `db_url: !secret recorder_db_url` lands in
configuration.yaml in the homeassistant repo. That ordering is REQUIRED in this
direction — HA refuses to start when a referenced secret is missing, so the
config change must not go first.

Why an init container rather than a hand-added line: the password is generated
by CNPG and can be rotated, so a hand-written entry goes stale silently and HA
then fails to start on a bad password. Re-rendering on every pod start is also
what makes a rotation take effect.

secrets.yaml is PVC runtime state that config-sync deliberately does not manage
(it holds influxdb_token and others), so this must EDIT the existing file, never
replace it. The script filters out any existing recorder_db_url line, appends
the current one, and guards the result: if the rewrite would drop any
non-recorder line it aborts rather than truncating the file holding every other
secret HA needs. Written to a temp path and moved atomically, chmod 600.

Verified in HA's own Alpine/busybox shell against a fixed sample before commit
-- pre-existing keys preserved, guard reports old=3 new=3, and a second run
produces a byte-identical 4-line file (idempotent, no duplicate key).

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:48:00 +02:00
k.renner 53172dff74 Merge pull request 'feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster)' (#521) from feat-ha-recorder-postgres into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#521
2026-08-19 13:24:32 +00:00
12 changed files with 154 additions and 16 deletions
@@ -121,7 +121,9 @@ spec:
persistentVolumeClaim:
claimName: audiobookshelf-books
- name: le-ca
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
---
apiVersion: v1
kind: Service
@@ -66,7 +66,52 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
status — the WordPress backup failed nightly for three days against a bucket
that was declared but never created (see `opentofu/object_storage.tf`).
## Phase 2 — migrate the data (NOT in this PR)
## Phase 2b — bulk statistics copy: DONE 2026-08-19
Completed with **HA running** — the SQLite file was read through a read-only
mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
was needed for the bulk copy.
```
statistics_meta: sqlite=1449 pg=1449 OK
statistics: sqlite=10799904 pg=10799904 OK (153.9s)
statistics_short_term: sqlite=1969804 pg=1969804 OK ( 27.0s)
statistics span in Postgres: 2024-03-13 -> 2026-08-19
entities with long-term history: 1449
rows carrying sum (energy dashboard): 2627859
database size: 2252 MB (PVC is 20Gi)
```
`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
`COPY` the planner has no statistics, and HA queries these tables constantly for
history and energy views, so without it every view seq-scans 10.8M rows.
Two schema details that cost a failed first attempt, recorded so the top-up does
not repeat them:
- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
will not coerce; convert explicitly and preserve NULL.
- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
both bulk tables (superseded by the `_ts` variants), which is what makes a
tab-separated `COPY` safe — nothing in those tables needs escaping.
The schema itself was created with HA's own models
(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
left EMPTY on purpose. HA's `_initialize_database` detects a new database by
looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
itself. Pre-stamping that row would skip HA's own initialisation path.
### High-water marks for the cutover top-up
Rows written after the bulk copy are not yet in Postgres. Resume from:
```
statistics max_id=13323904 max_start_ts=1787162400.0
statistics_short_term max_id=159205762 max_start_ts=1787166000.0
```
## Phase 2c — cutover (REMAINING)
Do not start until phase 1 is verified.
@@ -90,7 +135,18 @@ HA outage, and buys a table that is discarded within 7 days anyway.
SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
```
5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
there, not here:
there, not here. The `recorder_db_url` secret it references is already
rendered into `secrets.yaml` by the `render-db-url` init container in
`deployment.yaml` (phase 2a), so this order is mandatory: **the init
container must be deployed first**, or HA refuses to start on a missing
secret.
The init container percent-encodes the credentials and validates the
resulting YAML rather than counting lines. That is not incidental — the CNPG
password is random, and a naive `"postgresql://user:pass@host/db"` string
makes `secrets.yaml` unparseable the moment the password contains a quote,
which takes out *every* secret HA loads, not just this one. Verified against
a deliberately hostile password before merge.
```yaml
recorder:
db_url: !secret recorder_db_url # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
+64 -2
View File
@@ -36,6 +36,64 @@ spec:
# (sml2mqtt bridge owns the USB device on edge), so it no longer needs
# the edge USB device and schedules on the general cluster pool. This
# gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
initContainers:
# Puts CNPG's connection URI into /config/secrets.yaml as
# `recorder_db_url`, for the recorder's move off SQLite
# (MIGRATION-recorder-postgres.md).
#
# HA has no way to read a db_url from the environment — `!env_var` was
# removed from its YAML loader — so `!secret` is the only mechanism, and
# secrets.yaml is PVC state that config-sync deliberately does not
# manage. Hence a small init container rather than a env: entry.
#
# The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
# percent-encodes it, so there is nothing to escape here: the value is
# restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
# An earlier version of this assembled the URI from user/password/dbname
# and needed percent-encoding plus YAML round-trip validation to be
# safe. That was solving a problem CNPG had already solved.
#
# Re-rendered on every pod start, so a CNPG password rotation takes
# effect rather than silently leaving a stale URI behind.
#
# INERT until the recorder is repointed: nothing reads this key until
# `db_url: !secret recorder_db_url` lands in configuration.yaml in the
# homeassistant repo. That ordering is required — HA refuses to start
# when a referenced secret is missing.
- name: render-db-url
image: busybox:1.36
command:
- sh
- -c
- |
set -eu
S=/config/secrets.yaml
T=/config/.secrets.render.tmp
[ -f "$S" ] || : > "$S"
# Filter-then-append, never rewrite in place: this file holds
# every other HA secret (influxdb_token and friends).
grep -v '^recorder_db_url:' "$S" > "$T" || true
printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
# Refuse to shrink the file. Cheap guard against the one failure
# that would actually hurt: losing the other keys.
[ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
chmod 600 "$T"
mv "$T" "$S"
echo "recorder_db_url rendered"
env:
- name: DB_URI
valueFrom:
secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
volumeMounts:
- name: config
mountPath: /config
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
@@ -221,9 +279,13 @@ spec:
# Writable emptyDir for the nginx entrypoint's envsubst output
# (/etc/nginx/conf.d/default.conf rendered from the .template).
- name: proxy-conf-d
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
# code-server's settings/extensions cache. Not on the config PVC
# so installed extensions don't pollute HA's /config listing.
# Trade-off: extensions reset across pod restarts.
- name: vscode-state
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
@@ -313,7 +313,9 @@ spec:
configMap:
name: mosquitto-config
- name: auth
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
- name: tls
secret:
secretName: mosquitto-tls
@@ -344,9 +344,13 @@ spec:
configMap:
name: zigbee2mqtt-interview-watch-script
- name: interview-watch-state
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
- name: discovery-guard-script
configMap:
name: zigbee2mqtt-discovery-guard-script
- name: discovery-guard-state
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
+3 -1
View File
@@ -147,7 +147,9 @@ spec:
- key: immich-config.json
path: immich-config.json
- name: le-ca
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
---
apiVersion: v1
kind: Service
@@ -64,7 +64,9 @@ spec:
secret:
secretName: pve-exporter-config
- name: tmp
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
---
apiVersion: v1
kind: Service
+3 -1
View File
@@ -218,7 +218,9 @@ spec:
persistentVolumeClaim:
claimName: paperless-consume
- name: bridge-cert
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
---
apiVersion: v1
kind: Service
@@ -156,4 +156,6 @@ spec:
# each pod enrolls its own peer and a restart re-enrolls cleanly
# (the old peer is reaped).
- name: state
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
@@ -5,7 +5,7 @@ metadata:
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
allowUnsupported: false
dataDirHostPath: /var/lib/rook
mon:
@@ -38,7 +38,7 @@ spec:
serviceAccountName: rook-ceph-default
containers:
- name: rook-ceph-tools
image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
command:
- /bin/bash
- -c
@@ -164,7 +164,9 @@ spec:
name: rook-config-override
optional: true
- name: ceph-config
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
@@ -85,4 +85,6 @@ spec:
name: restore-test-script
defaultMode: 0755
- name: tmp
emptyDir: {}
emptyDir:
medium: Memory
sizeLimit: 16Mi