4354 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
k.renner	1d2248139d	Merge pull request 'perf(grafana-db): raise archive_timeout to 30min on gf-pg (re-target of #539)' (#540) from archive-timeout-gf-pg into master ... Reviewed-on: k.renner/homelab#540	2026-08-20 12:49:41 +00:00
claude-bot	0a9d19a7c6	perf(grafana-db): raise archive_timeout to 30min on gf-pg ... A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment switch on a timer so the archive never trails reality by more than that interval, and the switched-out segment is written and archived at its full 16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to 5min, which permits up to 288 segments/day = 4.6 GB/day per cluster. Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted from 12 switches of 16 MiB. Essentially every byte written to this volume is padding, roughly 1100x amplification. This resolves an anomaly that has been open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it. Segment padding does, exactly. Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster and gives the cleanest signal -- its real WAL is small enough that any change in volume is unambiguously attributable to this parameter. The same default affects all seven clusters and is worth an estimated ~24 GB/day of logical writes fleet-wide, but that only goes ahead once this one is verified. The cost is RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences, changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated separately rather than inheriting this value by default. archive_timeout is in CNPG's GlobalDefaultSettings, not its FixedConfigurationParameters, so it is user-overridable -- verified against pkg/postgres/configuration.go rather than assumed. Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.	2026-08-20 14:37:46 +02:00
k.renner	082092c4ba	Merge pull request 'perf(cnpg): compress WAL with zstd instead of pglz' (#538) from reduce-pg-wal-fpi into master ... Reviewed-on: k.renner/homelab#538	2026-08-20 12:30:18 +00:00
claude-bot	591c76fb86	perf(cnpg): compress WAL with zstd instead of pglz ... `wal_compression: "on"` resolves to pglz, PostgreSQL's original 2016-era algorithm. Every CNPG cluster here was set that way, so none of them were getting the compression actually available to them: all seven builds accept lz4 and zstd (verified via pg_settings.enumvals on each, including immich's custom pgvector image and trmnl's PostgreSQL 18). This matters because of what dominates recorder-style write volume. A page is written to WAL in full -- all 8 KiB -- on its first modification after a checkpoint, for torn-page protection. Measured on ha-pg, those full-page images are 62.4% of WAL bytes. Compressing them is therefore the single largest lever available on WAL volume, and pglz was the weakest option for the job. ha-pg additionally moves checkpoint_timeout 15min -> 30min. Full-page images are emitted once per page per checkpoint interval, so doubling the interval roughly halves how often each hot page pays the 8 KiB toll. max_wal_size stays at 2GB deliberately: it must remain well above what one interval generates, or checkpoints would trigger on size and the interval change would do nothing. Recovery replays at most one interval of WAL -- under 200MB at the measured rate, i.e. seconds. The other six clusters keep checkpoint_timeout at 15min. Their measured write rates do not justify trading recovery time: gf-pg generates 4.5 MB/day of WAL and wp-pg 62 MB/day, against ak-pg's 604 MB/day. full_page_writes stays on everywhere. Disabling it would remove the overhead entirely but RBD can tear a page across an object boundary, and the trade is write volume against an unrecoverable database. Both parameters are sighup-reloadable, so this does not restart any cluster. Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.	2026-08-20 14:03:18 +02:00
k.renner	f26a98688c	Merge pull request 'fix(monitoring): drop apiserver high-cardinality histograms' (#537) from fix-prometheus-cardinality into master ... Reviewed-on: k.renner/homelab#537	2026-08-20 11:28:21 +00:00
claude-botandClaude Bot	3c770f2a8d	fix(monitoring): drop apiserver high-cardinality histograms ... Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x verb x resource x scope, so a single metric was 23,810 series on its own. Every Home Assistant metric combined is ~4,100 for comparison. That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of TSDB + WAL, doubled again by Ceph replication) and none of it is observable value on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs and dozens of nodes. Dropped, all verified to have ZERO rule dependencies by checking each metric against the live alert and recording rules: apiserver_request_duration_seconds_bucket rules=0 23,810 series apiserver_request_body_size_bytes_bucket rules=0 23,200 etcd_request_duration_seconds_bucket rules=0 15,840 apiserver_watch_list_duration_seconds_bucket rules=0 11,538 apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232 apiserver_response_sizes_bucket rules=0 7,592 apiserver_watch_events_sizes_bucket rules=0 5,220 ------- 95,432 KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series). 23 recording rules depend on it, including the apiserver availability SLO and the KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that alerting -- the manifest says so inline, so a later "tidy up" of the regex does not remove it by accident. Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so request rates and mean latencies still work; what is lost is per-bucket quantiles. etcd_* is included even though kubeEtcd is disabled: those are the apiserver's own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape. Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus writes, ~8 GB/day off the drives once replication is counted. Will verify against prometheus_tsdb_head_series rather than assume. Refs #434 Co-Authored-By: Claude Bot <noreply@anthropic.com>	2026-08-20 13:25:49 +02:00
k.renner	e8ad1f704e	Merge pull request 'fix(cnpg-barman): take the sidecar tag out of base64 and track it' (#536) from fix-barman-sidecar-visible into master ... Reviewed-on: k.renner/homelab#536	2026-08-20 10:59:27 +00:00
claude-botandClaude Bot	154c933177	fix(cnpg-barman): take the sidecar tag out of base64 and track it ... An image reference is not a secret. Encoding it as base64 `data` had two real costs: no reviewer sees a version change in a diff, and no $imagepolicy marker can attach to a base64 blob. The result was a two-minor version skew in the component that performs every base backup and WAL archive for seven Postgres clusters (#535): controller plugin-barman-cloud:v0.14.0 <- auto-bumped by Flux sidecar plugin-barman-cloud-sidecar:v0.12.0 <- invisible, never moved Not stale pods: ha-pg-1 was created hours ago during the recorder migration and still received v0.12.0. Three changes: 1. The Secret now uses stringData with the tag in plaintext, carrying an $imagepolicy marker. The Secret KIND is kept because the controller reads it via secretKeyRef; switching to a ConfigMap would need a matching edit to that reference and diverge further from upstream. stringData is accepted identically by the API server. 2. ImageRepository + ImagePolicy added for the sidecar image, with the same filter and 0.x range as the controller, so the two cannot drift apart again. Only the controller was tracked before. 3. Tag aligned to v0.14.0 to match the controller. The Secret name carries a kustomize hash suffix from upstream's secretGenerator but is static here, so editing the value does not change the name and the deployment's secretKeyRef stays valid — verified in the render, which still shows the name twice (Secret + reference). Verified the tag EXISTS before pinning it: pulling plugin-barman-cloud-sidecar:v0.14.0 succeeds (the pod fails later on a missing `true` binary in that minimal image, which is the container command, not the pull). Pinning a non-existent tag would have put all seven Postgres clusters into ImagePullBackOff. EXPECTED IMPACT: bumping the sidecar changes every Postgres pod spec, so all seven CNPG clusters roll. CNPG handles this as a normal rolling restart. Backups should be confirmed working afterwards — 113 Backup CRs exist today, newest ~3h old and completed. Note for future re-imports: upstream ships this Secret as base64 `data` with no marker, so a wholesale re-import silently reintroduces the skew. The manifest now says so inline. Refs #535 Co-Authored-By: Claude Bot <noreply@anthropic.com>	2026-08-20 12:50:31 +02:00
k.renner	eafd99faca	Merge pull request 'feat(paperless): repoint data volume to RBD (phase 3)' (#534) from feat-paperless-data-repoint into master ... Reviewed-on: k.renner/homelab#534	2026-08-20 05:45:25 +00:00
claude-botandClaude Bot	3a279d2dcd	feat(paperless): repoint data volume to RBD (phase 3) ... Switches the Deployment's `data` volume from paperless-data (RWX, ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media` stay on CephFS -- the ftp-scanner writes to those from another node, so they genuinely need RWX. The copy is already done and verified byte-for-byte with paperless stopped: src files: 75 dst files: 75 IDENTICAL: name, size and mtime match for every file index segments: src 3, dst 3 mtime parity matters specifically because the Whoosh index segments carry their own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines in the copy log were for the mount root /dst/. itself, not the files.) Why: only this Deployment ever mounted the volume, and its contents are single-writer state. Being on CephFS meant every access crossed the MDS -- measured with zero CI running, 22 requests/sec sustained and ~15 journal events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather than file writes. That accounted for most of the constant 7-8 GB/day on cephfs-metadata, which is a size 3 pool. cephfs-metadata stays at size 3 deliberately: losing it destroys the entire filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0 depends on it. paperless-data is retained (prune: disabled) as the rollback. It must NOT be deleted until search has been confirmed returning results -- that is what proves the index survived, not merely that paperless starts. Refs #434 Co-Authored-By: Claude Bot <noreply@anthropic.com>	2026-08-20 07:44:07 +02:00
k.renner	efb552f043	Merge pull request 'feat(paperless): provision RBD volume to move data off CephFS (phase 1)' (#533) from feat-paperless-data-block into master ... Reviewed-on: k.renner/homelab#533	2026-08-20 05:38:43 +00:00
k.renner	1e79bc6d2f	Merge pull request 'fix(cnpg): checkpoint tuning across all Postgres clusters' (#532) from fix-cnpg-write-tuning into master ... Reviewed-on: k.renner/homelab#532	2026-08-20 05:21:25 +00:00
claude-botandClaude Opus 5	d2d06d8532	feat(paperless): provision RBD volume to move data off CephFS (phase 1) ... paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod mounts it -- the paperless Deployment itself. Its contents are all single-writer state: the 50 MB Whoosh index, classification_model.pickle, celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only `consume` and `media` genuinely need RWX, because the ftp-scanner writes to those from another node. It is on CephFS by default, not by requirement, and every access goes through the MDS. Measured 2026-08-20 with ZERO CI pipelines running: paperless-data client request_load_avg 22 (all other sessions 0-1) MDS over 180s 4,052 client requests = 22.5/s 2,718 journal events = 15/s 2 dir commits <- not file writes Cap and inode churn from repeated access, not data being written. That accounts for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is size 3, roughly 21-24 GB/day reaches the drives. cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0 rides on it. Cutting writes at source is the fix; cutting replication is not. PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data copy and the deployment repoint are separate steps, documented in MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the Whoosh index and celerybeat schedule are both live-written, so copying under an active writer gives a torn read). Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so about 5%. This is a correctness fix -- a single-writer volume does not belong on a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS load, which benefits Woodpecker CI workspaces sharing the same MDS. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 07:18:31 +02:00
claude-botandClaude Opus 5	68c8c86046	fix(cnpg): checkpoint tuning across all Postgres clusters ... Ten Postgres instances across seven CNPG clusters are ~48 GB/day of the cluster's ~114 GB/day of client writes, and only ha-pg carried any write-reduction tuning. The rest ran Postgres defaults, which favour throughput over storage that wears out. MEASURED, not assumed. ak-pg since 2026-05-17: checkpoints_timed 27,624 = 291/day, i.e. one every 5 min (the default) buffers_checkpoint 25,801,628 = ~206 GB flushed database size 1,046 MB That is the whole database rewritten roughly twice a day. The pathology is specific to idle databases: sessions, tokens and job state re-dirty the same few pages constantly, and checkpoint_timeout=5min forces each of them to disk 288 times a day instead of letting them coalesce. At 15min they are written a third as often. max_wal_size is raised alongside it. Without that, checkpoints trigger on size instead and the timeout change does nothing. It also bounds crash recovery to at most 2GB of WAL replay. Applied to ak-pg, pprl-pg, tr-pg, wp-pg, im-pg, gf-pg. ha-pg already had it. synchronous_commit is deliberately NOT changed here. It is off for ha-pg because a state recorder is telemetry; these clusters hold credentials, documents and job state, where losing the last few hundred ms of commits on an unclean shutdown is not an acceptable trade for write volume. Separately, ha-pg shared_buffers 256MB -> 512MB. After 15h on Postgres its buffers_backend was 629,704 of 939,025 total page writes -- 67% written by backends rather than the checkpointer or bgwriter, which means the working set does not fit in the buffer pool and pages are written out repeatedly instead of coalescing. Memory limit raised 1Gi -> 1536Mi to keep the same headroom (measured 383Mi in use, of which 256MB was shared_buffers). Verified before commit by rendering every cluster and asserting the tuning landed under spec.postgresql.parameters with the barman plugin block untouched. That check exists because the first attempt at this matched the wrong `parameters:` key -- the barman plugin's, at 6-space indent under plugins: -- and corrupted wp-pg's barmanObjectName. Reverted and redone with an indent-aware match. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 06:55:42 +02:00
k.renner	a2239515e9	Merge pull request 'feat(monitoring): alert on CNPG WAL archiving and backup failure' (#523) from fix-cnpg-backup-alerting into master ... Reviewed-on: k.renner/homelab#523	2026-08-20 04:39:20 +00:00
k.renner	0d65fb0982	Merge pull request 'fix: make small scratch emptyDirs RAM-backed instead of node-disk' (#530) from fix-emptydir-tmpfs into master ... Reviewed-on: k.renner/homelab#530	2026-08-20 04:38:21 +00:00
k.renner	ece3c93ad5	Merge pull request 'fix(trmnl): make the renderer's /tmp RAM-backed (34 GB/day of node writes)' (#529) from fix-trmnl-renderer-tmpfs into master ... Reviewed-on: k.renner/homelab#529	2026-08-20 04:33:37 +00:00
claude-botandClaude Opus 5	b34450b22e	fix: make small scratch emptyDirs RAM-backed instead of node-disk ... Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory` is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is ephemeral, but every byte still hits the node's disk. Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts 12 of the remaining ones. HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then nothing emptyDir-driven -- the rest of the top of that list is Ceph mons (hostPath, not emptyDir), kube-apiserver, and container logs from databases. Every volume converted here measured 8.0K in the running pod: /var/lib/code-server 8.0K /mosquitto/auth 8.0K /state (z2m guards) 8.0K So the value is correctness and, for mosquitto, keeping credential material off the node's filesystem: that volume holds the generated passwd/acl files. Each converted volume gets an explicit sizeLimit (16Mi where none existed), so tmpfs cannot silently consume a container's memory limit. All are orders of magnitude above measured usage. Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on either medium, so the mosquitto shim's assumption that the passwd file survives a container restart still holds. DELIBERATELY NOT CONVERTED: - kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored and generated from the chart; a hand edit would be lost on upgrade. - trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was. Cost: this rolls the affected pods once. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 06:30:23 +02:00
claude-botandClaude Opus 5	b4f44b2002	fix(trmnl): make the renderer's /tmp RAM-backed (34 GB/day of node writes) ... The renderer was the single largest writer in the cluster -- more than every Ceph OSD and every database combined -- and none of it needed to touch disk. Measured 2026-08-20 from the container's own cgroup io.stat: 253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day 252:48 (its own PVC) 340 MB total = negligible So the writes were not going to its volume at all; they were going to the node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without `medium: Memory` is node storage, which is easy to read as "ephemeral, so free". The cause is cadence rather than volume. schedules.template.json renders on "* * * * *", so Chrome is launched 1440 times a day and each launch creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp holds under 8 MB -- it is pure churn for data that never outlives the render. tmpfs pages count against the container's memory limit, so the 512Mi cap is kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample headroom even if the cap were fully consumed. Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a 4h window. This is a bigger cluster-wide reduction than the Home Assistant recorder migration achieved, for a one-line change. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 06:26:38 +02:00
k.renner	3faddcadcd	Merge pull request 'feat(home-assistant): render recorder_db_url from the ha-pg Secret' (#524) from feat-ha-recorder-db-url into master ... Reviewed-on: k.renner/homelab#524	2026-08-19 19:24:38 +00:00
claude-botandClaude Opus 5	6ae734eef4	refactor(home-assistant): use CNPG's own URI instead of rebuilding it ... Net -44 lines. The previous version assembled the connection URI from username/password/dbname, which forced percent-encoding, YAML-safe quoting via json.dumps, a YAML round-trip validation, and python3 from HA's image. None of that was necessary: CNPG already publishes a finished, percent-encoded connection string in the app secret as `uri` / `fqdn-uri`. Reading that key removes the whole class of problem the earlier review correctly flagged -- rather than solving it more carefully. Verified against the live secret before simplifying: fqdn-uri contains only [A-Za-z0-9:/@._%-], so naive double-quoting is provably safe and the YAML validation was guarding against an input that cannot occur. What is kept, because it is the part that actually matters: secrets.yaml is PVC state holding every other HA secret (influxdb_token and friends), so the file is filtered-and-appended rather than rewritten, moved atomically, and guarded against shrinking. Tested in HA's own busybox shell -- comment and pre-existing keys preserved, idempotent across runs, no duplicate key. Also records why an env var cannot be used at all: HA removed `!env_var` from its YAML loader, so `!secret` is the only mechanism for recorder.db_url, and config-sync deliberately does not manage secrets.yaml. Back to busybox; HA's image is no longer needed for this. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 21:18:27 +02:00
claude-botandClaude Opus 5	39885b9400	docs(home-assistant): record the completed statistics migration ... Phase 2b is done: 12.77M statistics rows copied SQLite -> Postgres with HA running and no downtime (read-only mount; SQLite WAL gives readers a consistent snapshot). statistics_meta 1449 OK statistics 10799904 OK 153.9s statistics_short_term 1969804 OK 27.0s span in Postgres: 2024-03-13 -> 2026-08-19 1449 entities with long-term history, 2627859 rows carrying sum database size 2252 MB against a 20Gi PVC Records the high-water marks the cutover top-up must resume from, and two schema details that cost a failed first attempt: - has_mean/has_sum are INTEGER in SQLite, BOOLEAN in Postgres, and Postgres will not coerce them. - the legacy created/start/last_reset DATETIME columns are 100% NULL in both bulk tables, which is what makes a tab-separated COPY safe. Also documents why the schema was built from HA's own models with schema_changes left EMPTY: HA's _initialize_database detects a new database via the events.time_fired_ts index and writes SCHEMA_VERSION itself, so pre-stamping that row would bypass HA's own initialisation. ANALYZE was run on all three tables. After a bulk COPY the planner has no statistics and HA seq-scans 10.8M rows on every history/energy view. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 21:10:26 +02:00
security-bot	694389b569	chore(image-automation): bump quay.io/ceph/ceph:v19.2.6 ... Updates managed by image-automation-controller.	2026-08-19 15:43:41 +00:00
claude-botandClaude Opus 5	57bb39e7ee	fix(home-assistant): percent-encode db_url and validate the rendered YAML ... Review catch on #524, and it was a genuine blocker. The first version built "postgresql://${PGUSER}:${PGPASSWORD}@host/db" in busybox sh with no encoding. Two defects: 1. The CNPG-generated password is random and may contain @ : / ? #, each of which changes how the URL parses. This repo already hit this exact issue with the same CNPG password-generation mechanism -- see kubernetes/apps/woodpecker/server.yaml, which switched to a lib/pq keyword DSN specifically to avoid escaping generated passwords. That option does not exist here: HA's recorder.db_url must be a SQLAlchemy URL. So the fix is percent-encoding. 2. Worse, and the reason this was worth blocking: a quote in the password makes the YAML line malformed, so secrets.yaml stops parsing entirely and HA loses EVERY secret, not just this one. The line-count guard could never catch that -- the count is unchanged; the file is simply invalid. Rewritten to run python3 from HA's own image (already pulled for the main container, so no extra image) instead of busybox: - urllib.parse.quote(..., safe="") on user, password and dbname. - json.dumps for the YAML value: YAML is a JSON superset, so this emits a correctly escaped double-quoted scalar even with quotes or backslashes. - The file is still rewritten LINE-WISE rather than round-tripped through the YAML parser, so hand-maintained comments and key order survive. - The guard now validates the RESULT: parse the new content, assert no key from the old file disappeared, and assert recorder_db_url round-trips to exactly the URL that was built. That catches corruption the line count cannot. Verified in HA's image against a deliberately hostile password (p@ss:w/o"rd#?&\x=1) before commit: new version -> parses, comment preserved, idempotent, password decodes back byte-exact, host and dbname parse correctly naive version -> yaml.safe_load raises ParserError i.e. the approach this replaces would have produced an unparseable secrets.yaml. Also updates MIGRATION-recorder-postgres.md for this step landing, including why the ordering (init container before the configuration.yaml change) is mandatory. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 15:58:50 +02:00
claude-botandClaude Opus 5	17f913ba2f	feat(home-assistant): render recorder_db_url from the ha-pg Secret ... Phase 2a of the recorder move off SQLite (MIGRATION-recorder-postgres.md). Adds an init container that writes `recorder_db_url` into /config/secrets.yaml from the CNPG-generated ha-pg-app Secret. INERT until the recorder is repointed. It only adds a key to secrets.yaml; nothing reads it until `db_url: !secret recorder_db_url` lands in configuration.yaml in the homeassistant repo. That ordering is REQUIRED in this direction — HA refuses to start when a referenced secret is missing, so the config change must not go first. Why an init container rather than a hand-added line: the password is generated by CNPG and can be rotated, so a hand-written entry goes stale silently and HA then fails to start on a bad password. Re-rendering on every pod start is also what makes a rotation take effect. secrets.yaml is PVC runtime state that config-sync deliberately does not manage (it holds influxdb_token and others), so this must EDIT the existing file, never replace it. The script filters out any existing recorder_db_url line, appends the current one, and guards the result: if the rewrite would drop any non-recorder line it aborts rather than truncating the file holding every other secret HA needs. Written to a temp path and moved atomically, chmod 600. Verified in HA's own Alpine/busybox shell against a fixed sample before commit -- pre-existing keys preserved, guard reports old=3 new=3, and a second run produces a byte-identical 4-line file (idempotent, no duplicate key). Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 15:48:00 +02:00
claude-botandClaude Opus 5	1260165af1	feat(monitoring): alert on CNPG WAL archiving and backup failure ... Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped. The only related rule was HetznerBackupEndpointDown, which probes the S3 endpoint — it proves the bucket is reachable, not that anything was written to it. Same failure shape as the WordPress backup that failed nightly for three days against a bucket declared in tofu but never applied: the status surface looked fine because nothing checked the thing that mattered. CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments waiting in pg_wal/archive_status; steady state is 0 on every cluster. A sustained backlog means the archiver is failing, so point-in-time recovery is silently broken from that moment and pg_wal grows until it fills the PVC and takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so that is ~80 MiB of un-archived WAL — well below anything that threatens the volume, far above idle noise. It reads only the instance-level collector, so it works despite #522. CNPGBackupFailed fires on a base backup failure within 24h, guarded on timestamp > 0 so the epoch-zero default is not read as "failed in 1970". DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot be used today because that metric reads exactly 0 on every instance in the cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any successful backup anywhere, because the Backup CRs it derives that from are deleted within a minute of completing — #522. Adding it now would fire permanently for all seven clusters and be silenced within a day, which is worse than not having it. The backups themselves DO work; verified independently by listing the bucket with barman-cloud-backup-list and restoring a WAL segment back out of S3. The gap is in the record, not the data. Both expressions were validated against live Prometheus before commit rather than written from the metric names: the WAL selector matches 8 real series (a zero match would mean a dead rule, the defect fixed in prometheusrule-woodpecker-rotation.yaml), and neither expression fires currently, which is correct since archiving is healthy and the only recorded backup failure is ~91 days old. Refs #522 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 15:43:07 +02:00
k.renner	53172dff74	Merge pull request 'feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster)' (#521) from feat-ha-recorder-postgres into master ... Reviewed-on: k.renner/homelab#521	2026-08-19 13:24:32 +00:00
claude-botandClaude Opus 5	196ebda188	feat(home-assistant): add ha-pg for the recorder (phase 1, empty cluster) ... The recorder's SQLite database is the largest client writer in the cluster. Measured 2026-08-19 via cgroup io.stat on the home-assistant container, corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD volume, ~26% of all client writes, against only ~5 GB/day of actual row data. The 8x gap is the write PATTERN, not the volume: ~30 state changes/sec (107k rows/hour, 2.57M/day) x2 SQLite WAL + checkpoint -- every dirtied page written twice x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device) = ~15 KB written per recorded state change Postgres replaces scattered fsync'd page rewrites with sequential batched WAL. This attacks the multiplier rather than trimming what gets recorded, which would hide the problem instead of fixing it. PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves. Phase 2 migrates the statistics tables and repoints recorder.db_url in the homeassistant repo; MIGRATION-recorder-postgres.md has the sequence. What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT migrate data between database engines. `statistics` holds 10,793,712 rows spanning 2024-03-13 to today -- 2.4 years of energy history that is not in InfluxDB and cannot be reconstructed. Repointing db_url at an empty database silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7. Postgres is tuned to minimise write volume, which is the point of the move: synchronous_commit off -- the recorder is telemetry, not transactional record-keeping. Postgres stays crash consistent; only the last few hundred ms of commits are at risk on unclean shutdown. In exchange, commits stop forcing an fsync -- the exact cost that makes SQLite expensive here. wal_compression on -- recorder writes are small inserts scattered across pages, so post-checkpoint full-page writes dominate WAL volume. checkpoint_timeout 15min / max_wal_size 2GB -- fewer checkpoints means fewer repeat full-page images for the hot rows the recorder rewrites. full_page_writes -- deliberately LEFT ON. RBD can tear a page across a 4 MiB object boundary; trading that for write volume risks an unrecoverable database. Also included: - ObjectStore on its own s3 prefix (two clusters sharing a destinationPath interleave WAL streams and corrupt both). - ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters. - PodMonitor added WITH the cluster rather than after it, so the migration has metrics from the start instead of discovering the gap later (#433). - Cilium policies as DEDICATED policies with their own endpointSelector, not rules on the namespace-wide default-deny -- that policy is endpointSelector:{} so any rule there applies to every pod in the namespace, and :8000 is served by more than the CNPG instance manager. Includes the ICMP allowance that attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD. The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same coupling #397 tracks for woodpecker. Noted in the file with the exact bao command to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed` was a wrong-path error rather than missing permissions. Refs #434, #397 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 15:22:21 +02:00
k.renner	1d76dba474	Merge pull request 'fix(ceph): stop mon compacting on every paxos trim (240 GB/day of writes)' (#520) from fix-ceph-mon-write-amplification into master ... Reviewed-on: k.renner/homelab#520	2026-08-19 12:58:24 +00:00
claude-botandClaude Opus 5	63f25c1511	fix(ceph): stop mon compacting on every paxos trim (240 GB/day of writes) ... The three Ceph monitors were writing 240 GB/day between them for a ~106 MB database. That was ~62% of all OS-disk writes on the nodes and roughly a quarter of everything the cluster writes -- more than every client workload combined. Measured 2026-08-19: paxos transactions ~99,000/day (1.15/s, sampled over 60 s) paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime) ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three (cgroup io.stat: 2.15 TB / 27 d) => ~98x write amplification Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix (including paxos) when we trim its old states". With paxos_trim_min at 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store. The churn driving it is not cluster activity. 99% of paxos versions are cluster log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook operator poll (mgr stat / versions / status / report / osd dump; 23 audit entries per minute per mon) becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons. Reducing that polling is the complementary fix and is tracked separately; this change removes the amplification regardless of the rate. This does NOT disable compaction. RocksDB's own levelled compaction still runs and still reclaims trimmed keys; what stops is the extra forced compact_range on every trim. Two backstops if that assumption is wrong: - mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn in monitoring-rules already alerts on. - mon_compact_on_start is set to true here (upstream default false), so every mon restart -- upgrade, node reboot, reschedule -- does a full compaction. Cheap on a ~106 MB store. Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z: wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z. Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212, and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank writers; every figure above comes from cgroup io.stat or kernel diskstats, which reconcile with each other and with the drives' SMART counters. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 14:56:47 +02:00
k.renner	3be75f535e	Merge pull request 'fix(ceph): replicapool and cephfs data to size 2, min_size 1' (#519) from fix-ceph-replica-size-2 into master ... Reviewed-on: k.renner/homelab#519	2026-08-19 12:40:01 +00:00
claude-botandClaude Opus 5	db460079f8	fix(ceph): replicapool and cephfs data to size 2, min_size 1 ... With 3 OSDs on 3 hosts, size=3 meant every write landed on every drive in the cluster. Per-drive write volume equalled TOTAL cluster write volume, so all three drives wore at an identical rate and would have reached end-of-rated-life together: measured 2026-08-19 from the drives' own SMART counters, 80/82/87% endurance used, advancing ~0.13 points/day, exhausted between Nov 2026 and Jan 2027 (#434). Replication gives no protection there — the wear-out is correlated by construction, not independent. size 2 cuts per-drive write volume by a third and desynchronises the wear. min_size 1 is required, not incidental. Ceph does NOT lower min_size when size is reduced; it was 2, and leaving it would make replicapool 2/2, where a single OSD down blocks IO for its PGs. That would hang every RBD volume in the cluster on every rolling node reboot and every OSD restart. Accepted trade, stated explicitly: while degraded (one OSD down), a second drive failure loses data, and at 80%+ endurance that window is not hypothetical. What limits the blast radius is that CNPG barman and Velero both ship offsite to Hetzner, so exposure is the RPO of those backups rather than total loss. Node reboots must stay serialised with HEALTH_OK between them. CephFS metadataPool deliberately stays at size 3: 467 MiB against data0's 275 GiB, so it contributes nothing measurable to wear, while losing it destroys the entire filesystem rather than the objects on one OSD. The wear saving comes from data0. The RGW pools in objectstore.yaml were already size 2. One-time cost on apply: Ceph deletes the third replica of everything, which is background IO, and raw used drops from ~1.4 TiB to ~0.95 TiB. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 14:35:58 +02:00
k.renner	ef40c68b1f	Merge pull request 'fix(ceph): stop deferring small writes on SSD (halves write amplification)' (#518) from fix-ceph-deferred-write-amplification into master ... Reviewed-on: k.renner/homelab#518	2026-08-19 12:30:21 +00:00
claude-botandClaude Opus 5	6c31b552d1	fix(ceph): stop deferring small writes on SSD (halves write amplification) ... bluestore_prefer_deferred_size_ssd was set to 65536 and bluestore_deferred_batch_ops_ssd to 64 as a QLC throughput tune. Ceph's SSD defaults are 0 and 16. Reverted to the defaults, set explicitly so the stored `ceph config` value is overwritten rather than merely un-declared. prefer_deferred_size defers every write SMALLER than the threshold: it lands in RocksDB's WAL first and is applied to the object store afterwards. That is a double write, and at 64 KiB it caught essentially every write this cluster makes -- SQLite pages are 4 KiB, Postgres pages 8 KiB. Ceph disables deferral on SSD by default because solid state does not need the random-write batching that helps spinning media. Why the original tune was the wrong trade: - The benchmark justifying it cannot have exercised it. The 2026-05-05 baseline used 4 MiB writes, 64x above the 64 KiB cutoff, so none of those writes were ever deferred. The setting was validated by a workload it does not affect. - It doubles writes onto QLC, the lowest-endurance NAND type. Measured 2026-08-19 from the drives' own SMART counters: all three OSD devices at 80/82/87% endurance used, advancing ~0.13 points/day, i.e. rated life exhausted between Nov 2026 and Jan 2027. Because size=3 on 3 OSDs, every drive absorbs 100% of cluster writes, so they wear in lockstep and expire together (see #434). Measured amplification before this change: 640 GB/day written to the three NVMe devices against 117 GB/day at the RBD client layer -- 5.5x, of which 3x is replication and ~1.8x BlueStore. Removing the double write should take the BlueStore factor toward ~1.1x. The cluster is write-light, so throughput was never the binding constraint; endurance is. Both parameters are runtime-updatable, so no OSD restart. Also corrects the drive model in the comment: the OSD devices are CT1000P3PSSD8 (Crucial P3 Plus), not the P2 named previously. bdev_enable_discard / bdev_async_discard are unrelated and kept as-is. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 14:27:01 +02:00
k.renner	d437775319	Merge pull request 'fix(alert-monitor): stop over-suppressing alerts in the noise log (#326)' (#517) from fix-326-noise-log-suppression into master ... Reviewed-on: k.renner/homelab#517	2026-08-19 10:54:27 +00:00
claude-botandClaude Opus 5	8a8145011a	fix(alert-monitor): stop over-suppressing alerts in the noise log ... Two defects in the noise-log dedup, both of which silently mute real alerts. 1. Prose-mentioned fingerprints became suppressions. The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons routinely cite another alert's fingerprint in prose ("see the issue filed for fingerprint <fp>"), so those citations were read as suppression entries. Verified against the live noise log: the old extractor yields 22 fingerprints for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert that HAD been filed as a genuine issue, permanently muted purely because two other entries mention it. Now only lines in the exact shape the write-back step emits count: - <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason> Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among them. 2. Suppression never expired. The fingerprint is sha256(alertname|scope) and is deliberately stable across recurrences, so a single blip verdict muted that alertname+scope pair forever. 21 pairs were permanently muted, including openbao's KubePodNotReady, KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real openbao outage would have filed no issue at all. A noise verdict means "this occurrence was benign", not "this alert can never be interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted from the date already present in each line. Re-triaging a recurrence weeks later is cheap; missing an outage because a July blip muted it is not. If it is still noise, triage re-adds it automatically. Rollout is gentle by construction: every current entry is dated 2026-07-21 or later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge -- the oldest entries lapse a day later, and only re-triage if they actually fire again. Unparseable lines are dropped, i.e. they stop suppressing: this fails toward more alerting, never less. Verified end-to-end by running the shipped snippet inside the real ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20, 21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch arithmetic with `date -d @<epoch>`, which was confirmed working in that image. Refs #326 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 10:11:45 +02:00
k.renner	2df54d302b	Merge pull request 'fix(zigbee2mqtt): restore HA discovery after a mosquitto restart (#381)' (#516) from fix-381-z2m-discovery into master ... Reviewed-on: k.renner/homelab#516	2026-08-19 08:05:30 +00:00
claude-botandClaude Opus 5	da0f895b6d	fix(zigbee2mqtt): bound the guard's MQTT calls with timeout ... Review catch on #516: the restart-triggering mosquitto_pub was unbounded, unlike the sibling call in zigbee2mqtt-interview-watch.sh. Neither client bounds a TCP connect that HANGS rather than being refused, and a stalled connect is most likely precisely when this guard runs -- around a broker restart. Unbounded, the loop would block forever and stop logging, which is the silent-failure mode the guard exists to prevent. Applied to the reads as well, not just the publish: mosquitto_sub -W bounds the wait for a message once CONNECTED, so it does not cover the connect phase either. The timeout wrapper is set above -W so the protocol-level wait still wins in the normal case. Not mosquitto_pub -W: that flag does not exist for mosquitto_pub and makes every publish fail on a usage error. The cooldown stamp was already written only on a successful publish, so a failed restart request still retries on the next cycle rather than burning the cooldown. `timeout` is confirmed present in eclipse-mosquitto:2.0.22 (already used by interview-watch, and exercised directly in this image). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 09:51:17 +02:00
claude-botandClaude Opus 5	5c856af578	feat(zigbee2mqtt): restore HA discovery after a broker restart ... mosquitto runs `persistence false` with no PVC (deliberate and settled), so every broker restart wipes the retained store. Zigbee2MQTT publishes its HA discovery configs retained and has no way to restore them itself, so a broker restart not followed by a Z2M restart leaves every Z2M entity unavailable in HA indefinitely -- ~14 h on 2026-07-23. Adds a discovery-guard sidecar to the Z2M pod that detects the wiped state and asks Z2M to restart, which re-runs discover(). Detection. bridge/state and bridge/{info,devices,groups,definitions} are all retained, but only bridge/state is republished on reconnect (mqtt.ts onConnect). So "bridge/state present + bridge/groups absent" means Z2M is talking to this broker but nothing has re-run start() since the store was wiped. bridge/groups specifically, because the absent-half partner must not be re-publishable by ordinary runtime events or it self-heals and the guard goes blind while HA is still broken. bridge/devices, bridge/definitions and bridge/info all reach inbound radio traffic via devicesChanged (receive.ts hands deviceExposesChanged to every fromZigbee converter); bridge/health self-heals on its own 10 min timer. bridge/groups is only republished by deliberate admin actions, and publishGroups() always publishes, emitting `[]` when no groups exist -- confirmed live: it currently holds a 2-byte payload, so presence is testable. Not AND-ed with bridge/info being absent. That would cut false positives at the cost of false negatives, and the costs are asymmetric: a false positive is one ~60s Z2M restart, a false negative is every Zigbee entity dead until a human notices. Tuned for sensitivity. Spurious positives are handled by a 90s confirm re-check, which covers both the startup race (bridge/state is published at mqtt.connect(), bridge/groups only at the end of BridgeExtension.start()) and a dropped publish (these topics go out at QoS 0). A 30 min cooldown keeps a Z2M that will not come up healthy from being restarted in a loop. Action is bridge/request/restart, which is a full in-process restart -- index.js restart() -> stop(true) tears down every extension, then start() builds a new Controller and re-runs HomeAssistant.start(). discover() then republishes only missing or changed configs, which after a wipe is all of them. Reuses the existing zigbee2mqtt broker credential (ACL `readwrite zigbee2mqtt/#`), so no new broker user and no read grant on homeassistant/#, which the ACL denies on purpose. Verified in the eclipse-mosquitto:2.0.22 image the sidecar actually runs: mosquitto_sub -C/-W, mosquitto_pub and `date -u +%s` all present; `sh -n` clean; startup run logs correctly and enters the loop. Detection was exercised against the live broker and correctly reports healthy (bridge/state online, bridge/groups present) with no action. NOTE: this rolls mosquitto, because the corrected ACL comment lives inside the container command. That roll wipes the retained store, which the guard then repairs within ~6 min if the Z2M roll does not already do so. Closes #381 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 09:35:58 +02:00
claude-botandClaude Opus 5	9d69eb1a29	docs(mosquitto): correct the disproven "discovery self-heals" claim ... Four comments across the broker manifests asserted that retained state is re-populated by all clients on reconnect, and specifically that Zigbee2MQTT re-publishes HA discovery when it sees the homeassistant/status birth message. Both are false, and the second is the premise that kept issue #381 open. Verified against Zigbee2MQTT 2.12.1 source: - discover() runs only from HomeAssistant.start(); it is never called from the reconnect path. - lib/mqtt.ts onConnect() republishes bridge/state and resubscribes, nothing more. republishRetainedTimer is armed only inside connect() and is never re-armed on reconnect. - lib/extension/homeassistant.ts onMQTTMessage(), on status_topic == "online", republishes bridge/state plus cached device states after a hardcoded 30s delay -- but never the discovery configs. - Discovery is published retain:true, and the only knob is the global mqtt.force_disable_retain, which upstream documents as breaking the HA integration. Broker retention is the sole mechanism Z2M relies on. - No bridge/request endpoint forces a rediscovery; upstream declined to add one (Koenkk/zigbee2mqtt #27458, #22977, #16950, all closed NOT_PLANNED). The mosquitto ACL was never the blocker: `read homeassistant/status` is already granted. That grant is still correct -- the cached-state republish is what makes HA show current values after an HA restart -- so it is kept and the comment now says what it actually buys. Also records that a RollingUpdate roll is equivalent to a broker restart for retention: maxUnavailable:0/maxSurge:1 preserves connectivity, but the new pod starts with an empty retained store. Comment-only. The two deployment comments are plain YAML comments outside the container command, so the rendered Deployment is byte-identical and the broker does not roll; the ConfigMap carries no reloader annotation, so it updates in place without restarting mosquitto. Refs #381 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 09:27:29 +02:00
k.renner	3d05d47239	Merge pull request 'feat(opentofu): destroy the rustfs LXCs 800/801/802' (#515) from decommission-rustfs-lxc into master ... Reviewed-on: k.renner/homelab#515	2026-08-19 06:55:17 +00:00
claude-bot	fd35f5f21b	feat(opentofu): destroy the rustfs LXCs 800/801/802 ... Stage 2 of the decommission the shim was written for. Deleting the file makes the next apply destroy the three containers. The containers were taken out of service months ago — they are stopped (pve_up = 0 for all three) and rustfs-{1,2,3}.srv.renner.uno no longer resolve — but the LXC objects and their disks still exist on saturn/jupiter/neptune. Two earlier destroy attempts (#270) died instantly with "context deadline exceeded", because the bpg provider reads timeout_delete from STATE and the imported resources had stored 0. That is now fixed: apply 5675 landed timeout_delete = 1800 and the rest of the timeouts in state, in place, with 0 destroyed. The destroy has a real deadline to work with for the first time. Checked before removing, since the containers are tagged `state-backend` and `terraform-backend`: - The tofu backend is Hetzner S3 (renner-uno-tfstate at fsn1.your-objectstorage.com), not rustfs. Destroying these cannot pull the state out from under the apply. The tags are historical, from before that migration. - No Kubernetes or Ansible resource references rustfs. - scripts/restore-authentik-db.sh mentions rustfs only as an `mc` alias name; its endpoint is s3.renner.uno:9000, the Ceph RGW `migration` store. - DNS for the three names is already gone. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 08:51:38 +02:00
k.renner	d208c17509	Merge pull request 'docs(opentofu): record the hairpin _id diff, retract the PUT 400 warning' (#514) from document-hairpin-id-diff into master ... Reviewed-on: k.renner/homelab#514	2026-08-19 06:50:24 +00:00
claude-bot	853928b6b1	docs(opentofu): record the hairpin _id diff and retract the PUT 400 warning ... Both of these were learned the expensive way today and existed only in commit messages and PR descriptions, which is not where anyone looks. providers.tf carried a warning that a UniFi PUT may require _id in the body, which restapi does not send, and that a 400 should be answered by tainting the object so CI recreates it. That is wrong on this controller. Tested against a live rule with exactly the body tofu sends: HTTP 200, object returned intact, rule byte-identical before and after, 60 rules before and after. Confirmed at scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and immich.renner.uno still 200. That wrong warning had a cost. A note saying an apply might break firewall rules is one nobody argues with, so the apply did not get run, and the renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days while the WordPress backup failed nightly with "The specified bucket does not exist". The warning is replaced with the evidence that disproves it. unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and cosmetic, and lists the four attempts to remove it with the plan number that disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662), restapi 3.x ignore_server_additions (5667), and create_returns_object (5677). The first three cannot work in principle — `data` is a configured attribute, so Terraform core diffs it against state before provider-side ignore logic runs. The fourth was the better theory, that PUT responses re-polluted `data`, and it is wrong too: the diff returned on the plan immediately after the apply. That means the response is written into `data` on the read path, and settling it needs the provider source rather than more guesses at its schema. No behaviour change. Documentation only, so the next person inherits the findings instead of the dead ends. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 08:34:42 +02:00
k.renner	3fac750430	Merge pull request 'fix(opentofu): make apply safe — no container replacement, no permanent NAT diff' (#513) from fix-rustfs-shim-replace into master ... Reviewed-on: k.renner/homelab#513	2026-08-19 06:15:35 +00:00
claude-bot	49cc880169	fix(unifi): capture the API response on create only, not on every write ... Root cause of the 60 permanently-dirty hairpin NAT rules. The provider was configured write_returns_object = true, which makes it store the API response into `data` after POST *and* PUT. The UDM assigns `_id`; unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write put a field into state that the config does not have, and all 60 rules planned as "will be updated in-place" with `- _id` as the entire diff — re-dirtying themselves on the next write, indefinitely. No provider-side ignore can suppress this, because `data` is a configured attribute and Terraform compares config to state in core, before provider logic runs. Three were tried and disproven against real plans rather than argued about: ignore_changes_to = ["_id"] plan 5661 — 60 remained ignore_all_server_changes = true plan 5662 — 60 remained restapi 3.x + ignore_server_additions = true plan 5667 — 60 remained, plus a new api_data diff create_returns_object keeps the part the provider genuinely needs — the POST response is how it learns `_id` for id_attribute = "_id" — while no longer letting updates rewrite `data` from the response. This does not clear the existing pollution by itself: state still carries `_id` in `data` for all 60 rules, so the plan will keep showing them until one apply rewrites `data` to the configured value. What it changes is that the apply then sticks, instead of the next write undoing it. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 07:55:44 +02:00
claude-bot	08b3a39878	fix(opentofu): ignore operating_system instead of declaring it ... Plan 5657 rejected the previous commit: Error: Missing required argument on lxc_800_rustfs_decommission.tf line 98, in resource "proxmox_virtual_environment_container" "rustfs_decommission": The argument "template_file_id" is required, but no definition was found. The plan's own state dump showed only `type = "debian"` inside operating_system, so mirroring it looked complete. It is not: template_file_id is create-time-only, PVE does not report the original template for an existing container, and nothing else in opentofu/ declares one to copy — these three are the only container resources in the repo. Guessing a template would be a lie that stays invisible until something recreates these containers, which is precisely the event this file exists to prevent. So operating_system is left undeclared and ignored via lifecycle instead, which stops it planning as a removal without asserting a value that is not known. The timeouts remain outside ignore_changes on purpose — they are the one thing this shim needs to actually apply. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 07:11:39 +02:00
claude-bot	a8c1af1502	fix(opentofu): make the rustfs decommission shim plan in-place, not replace ... The v2 shim's own instructions say the plan MUST show an in-place update and no replace before applying. It does not. Pipeline 5656 planned all three containers as `must be replaced`, which would destroy and recreate LXCs tagged critical / state-backend / terraform-backend instead of nudging a timeout value into state. Cause: v2 declared only node_name, vm_id, started, unprivileged and the timeouts, on the stated assumption that "the other attributes mirror the real containers so the plan stays in-place". They were not declared at all, so every absent attribute planned as a removal — and two of those removals force replacement: - mount_point { ... } # forces replacement (size, volume) - operating_system { ... } # forces replacement Declaring the full attribute set, mirrored from the values the plan itself reported out of state: cpu, memory, disk, mount_point, network_interface, operating_system and initialization. hostname and mac_address differ per container, so the local map carries them per key rather than being repeated or guessed. tags is declared for a separate reason: it was planning as ["critical","rustfs","state-backend","terraform-backend"] -> null, which does not force replacement but would have stripped the tags off the live containers on apply. This does not unblock the WordPress backup on its own. That needs renner-uno-k8s-mariadb, which the same plan wants to create — but the bucket cannot be applied while the plan also carries three container replacements, and a -target apply is not an acceptable way around it: it diverges state from config and leaves the real problem in place. Verification is the file's own gate — a plan showing in-place only. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 07:08:47 +02:00
k.renner	38eb6b6503	Merge pull request 'fix(woodpecker): revive the rotation alert rules after the namespace move' (#512) from fix-rotator-woodpecker-url into master ... Reviewed-on: k.renner/homelab#512	2026-08-19 04:53:19 +00:00
claude-bot	5d542af431	fix(woodpecker): revive the rotation alert rules after the namespace move ... Rebased onto master now that #511 landed the WOODPECKER_URL fix for both rotators — that part is dropped from here rather than duplicated. What remains is the half nothing else covered. All four rules in prometheusrule-woodpecker-rotation.yaml were selecting series that do not exist, so the whole group was silent: - WoodpeckerServerNotReady queried kube_statefulset_status_replicas_ready. The server became a Deployment when the sqlite->CNPG migration dropped its PVC, so that metric has no series at all — a `critical` "all CI is down" alert that could never fire, and had been dead well before the namespace move. Now kube_deployment_status_replicas_ready in the woodpecker namespace. - WoodpeckerAgentNotReady selected namespace="apps". Broken by the namespace migration; mine. - The two woodpecker-secret-rotate rules selected namespace="apps", but that rotation runs from the openbao namespace. Broken since that move, which the woodpecker kustomization documents. Verified against live Prometheus instead of reasoned about: server ready=1, agent available=1, cronjob 1 series. The originals returned nothing, which is exactly why they never fired rather than firing wrongly. Two stale claims in the same file corrected while here: it pointed at kubernetes/apps/woodpecker/rotate-agent-secret.yaml, which no longer exists, and asserted the CronJob/Job series "stay empty until the rotation feature is merged" — that rotation has since landed in openbao. Also finishes the netpol comments #511 left mid-update, which still credited woodpecker-allow-rotators to `apps/*`. gitea.apps.svc:3000 in review-bot.yaml is deliberately untouched: gitea really is still in apps. Co-Authored-By: Claude Bot <claude-bot@renner.uno>	2026-08-19 06:50:42 +02:00