Comments should say what the setting does and what constrains it. The
blocks added in #542, #544, #545 and #547 had grown into incident
write-ups: measurement runs, prior values, how a problem was found,
what a check used to be. That belongs in commit messages and the wiki,
both of which already carry it.
Trimmed, no behaviour change:
cephcluster.yaml 22 lines -> 6 (cephx rotation)
cephfs-ci.yaml 26 lines -> 7 (why size 2)
prometheusrule-ceph.yaml (CephHealthWarn, 43 -> 13;
CephPoolLostRedundancy, 15 -> 7)
The operative facts are kept: which checks are excluded and why they
cannot clear, that POOL_NO_REDUNDANCY reports blocked maintenance and
not only data risk, and that size 1 needs dedicated OSDs.
The unreplicated CI pools introduced in #542 make every OSD on the
cluster permanently un-restartable.
`ceph osd ok-to-stop` refuses while stopping an OSD would take PGs
offline. A size-1 pool puts every PG on exactly one OSD, so the check
fails for all three:
Error EBUSY: unsafe to stop osd(s) at this time
(20 PGs are or would become offline)
bad_become_inactive: 13.6 13.9 ... 14.1c 14.1d
Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2
pools are not implicated — they run min_size 1 and appear under
ok_become_degraded, which is allowed.
Rook gates OSD restarts on that check, so this blocks CephX key
rotation, Ceph upgrades, node drains and reboots, and drive
replacement. It surfaced when #545 deadlocked: Rook retried
ok-to-stop against all three OSDs several times a second and could
never succeed, while `ceph -s` reported 217 pgs active+clean
throughout. Nothing is unhealthy; the restarts are simply refused.
The reasoning in #542 was that unreplicated CI data only risks
in-flight pipelines, since every byte is a fresh git clone. That part
holds. What it missed is that the cost is not scoped to the pool: one
unreplicated pool freezes maintenance for the entire cluster.
Most of #542's saving came from keeping CI churn off `cephfs`, whose
metadataPool is size 3 — that survives this change. Only the second-
order gain from dropping the replica is given up, and cephfs-ci-data0
stores 0 B with cephfs-ci-metadata at 182 MiB, so the backfill is
trivial.
size 1 becomes viable again once these pools sit on OSDs of their own,
where the permanently un-stoppable OSD holds nothing but throwaway
data. That is planned with the retired QLC drives.
Alerting follows the same change:
- POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was
excluded because cephfs-ci was expected to keep it permanently
active; no pool is unreplicated now, so the exclusion would only
hide a real one losing its replicas.
- CephPoolLostRedundancy drops its cephfs-ci exemption for the same
reason, and its description now states the maintenance consequence,
not just the data-loss one.
- The AUTH_INSECURE_* exclusion is reworded. Calling it
"known-accepted" reads as a decision that the risk is fine; it is a
kernel-blocked remediation (aes256k over the CSI mount path needs
kernel 7.0+, nodes run 6.18).
Refs #542, #545, #527
Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.
The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.
Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.
Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.
CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.
Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.
Refs #527, #528
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CephHealthWarn alerted on `ceph_health_status == 1`. It has been FIRING since
at least 2026-08-21T00:26Z and cannot clear: four AUTH_INSECURE_* checks are
permanently active from CVE-2025-30156 (#527) and stay that way until Rook
>=1.20.5 lands. A permanently-firing alert reports nothing -- every Ceph
warning since then has been invisible behind it.
This is a pre-existing gap, not a consequence of the cephfs-ci work, though
that work would have added a second permanent cause (POOL_NO_REDUNDANCY, since
the CI pools are size 1 on purpose).
ceph_health_detail exposes each check as its own series, so the alert now fires
on any check that is NOT known-accepted, and names it. Verified against live
data: 5 checks active, 0 unexpected, so it is quiet now and fires the moment
something genuinely new appears.
Excluding POOL_NO_REDUNDANCY wholesale would give something up: that check is
cluster-wide and does not say WHICH pool, so accepting it for cephfs-ci would
equally hide a production pool dropping to one replica. CephPoolLostRedundancy
restores that. There is no pool-size metric, but bytes_used/stored IS the
replication factor -- verified live as replicapool 2, cephfs-data0 2,
cephfs-metadata 3, cephfs-ci-metadata 1.
Both alerts were tested against live data before committing, in BOTH
directions. The positive control matters as much as the negative one: an early
draft used a >1e9 floor which silently excluded the CI pools entirely, so the
expression could not have matched anything. It looked correct precisely because
it returned no results. The control caught it; the floor is now 1e6.
The floor is not what suppresses the near-empty rgw pools. Their ratios are
9-36 because per-object overhead dominates, so they never match `< 1.5`.
6h on CephHealthWarn is unchanged -- Ceph warns for self-clearing conditions
and paging on those trains people to ignore the alert.
Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.
Measured 2026-08-21, polling both CephFS pools every 15s:
CI active 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB
CI idle 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB
CI share of writes: data0 100%, metadata 61%
data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.
Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.
RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.
Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.
The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.
cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.
Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.
one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
every interval with no CI pod: data0 ~0 metadata ~0
So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.
size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.
`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.
Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.
Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.
The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.
gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.
Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.
ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.
ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.
Also corrects two comments in ha-pg-cluster.yaml that this work falsified.
wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.
checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.
Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.
This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.
Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.
The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.
archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
`wal_compression: "on"` resolves to pglz, PostgreSQL's original 2016-era
algorithm. Every CNPG cluster here was set that way, so none of them were
getting the compression actually available to them: all seven builds accept
lz4 and zstd (verified via pg_settings.enumvals on each, including immich's
custom pgvector image and trmnl's PostgreSQL 18).
This matters because of what dominates recorder-style write volume. A page
is written to WAL in full -- all 8 KiB -- on its first modification after a
checkpoint, for torn-page protection. Measured on ha-pg, those full-page
images are 62.4% of WAL bytes. Compressing them is therefore the single
largest lever available on WAL volume, and pglz was the weakest option for
the job.
ha-pg additionally moves checkpoint_timeout 15min -> 30min. Full-page images
are emitted once per page per checkpoint interval, so doubling the interval
roughly halves how often each hot page pays the 8 KiB toll. max_wal_size
stays at 2GB deliberately: it must remain well above what one interval
generates, or checkpoints would trigger on size and the interval change
would do nothing. Recovery replays at most one interval of WAL -- under
200MB at the measured rate, i.e. seconds.
The other six clusters keep checkpoint_timeout at 15min. Their measured
write rates do not justify trading recovery time: gf-pg generates 4.5 MB/day
of WAL and wp-pg 62 MB/day, against ak-pg's 604 MB/day.
full_page_writes stays on everywhere. Disabling it would remove the
overhead entirely but RBD can tear a page across an object boundary, and
the trade is write volume against an unrecoverable database.
Both parameters are sighup-reloadable, so this does not restart any cluster.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
An image reference is not a secret. Encoding it as base64 `data` had two real
costs: no reviewer sees a version change in a diff, and no $imagepolicy marker
can attach to a base64 blob.
The result was a two-minor version skew in the component that performs every
base backup and WAL archive for seven Postgres clusters (#535):
controller plugin-barman-cloud:v0.14.0 <- auto-bumped by Flux
sidecar plugin-barman-cloud-sidecar:v0.12.0 <- invisible, never moved
Not stale pods: ha-pg-1 was created hours ago during the recorder migration and
still received v0.12.0.
Three changes:
1. The Secret now uses stringData with the tag in plaintext, carrying an
$imagepolicy marker. The Secret KIND is kept because the controller reads it
via secretKeyRef; switching to a ConfigMap would need a matching edit to that
reference and diverge further from upstream. stringData is accepted
identically by the API server.
2. ImageRepository + ImagePolicy added for the sidecar image, with the same
filter and 0.x range as the controller, so the two cannot drift apart again.
Only the controller was tracked before.
3. Tag aligned to v0.14.0 to match the controller.
The Secret name carries a kustomize hash suffix from upstream's secretGenerator
but is static here, so editing the value does not change the name and the
deployment's secretKeyRef stays valid — verified in the render, which still shows
the name twice (Secret + reference).
Verified the tag EXISTS before pinning it: pulling
plugin-barman-cloud-sidecar:v0.14.0 succeeds (the pod fails later on a missing
`true` binary in that minimal image, which is the container command, not the
pull). Pinning a non-existent tag would have put all seven Postgres clusters into
ImagePullBackOff.
EXPECTED IMPACT: bumping the sidecar changes every Postgres pod spec, so all
seven CNPG clusters roll. CNPG handles this as a normal rolling restart. Backups
should be confirmed working afterwards — 113 Backup CRs exist today, newest ~3h
old and completed.
Note for future re-imports: upstream ships this Secret as base64 `data` with no
marker, so a wholesale re-import silently reintroduces the skew. The manifest now
says so inline.
Refs #535
Co-Authored-By: Claude Bot <noreply@anthropic.com>
Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.
The copy is already done and verified byte-for-byte with paperless stopped:
src files: 75
dst files: 75
IDENTICAL: name, size and mtime match for every file
index segments: src 3, dst 3
mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)
Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.
cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.
paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.
It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:
paperless-data client request_load_avg 22 (all other sessions 0-1)
MDS over 180s 4,052 client requests = 22.5/s
2,718 journal events = 15/s
2 dir commits <- not file writes
Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.
cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.
PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).
Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten Postgres instances across seven CNPG clusters are ~48 GB/day of the
cluster's ~114 GB/day of client writes, and only ha-pg carried any
write-reduction tuning. The rest ran Postgres defaults, which favour throughput
over storage that wears out.
MEASURED, not assumed. ak-pg since 2026-05-17:
checkpoints_timed 27,624 = 291/day, i.e. one every 5 min (the default)
buffers_checkpoint 25,801,628 = ~206 GB flushed
database size 1,046 MB
That is the whole database rewritten roughly twice a day. The pathology is
specific to idle databases: sessions, tokens and job state re-dirty the same few
pages constantly, and checkpoint_timeout=5min forces each of them to disk 288
times a day instead of letting them coalesce. At 15min they are written a third
as often.
max_wal_size is raised alongside it. Without that, checkpoints trigger on size
instead and the timeout change does nothing. It also bounds crash recovery to at
most 2GB of WAL replay.
Applied to ak-pg, pprl-pg, tr-pg, wp-pg, im-pg, gf-pg. ha-pg already had it.
synchronous_commit is deliberately NOT changed here. It is off for ha-pg because
a state recorder is telemetry; these clusters hold credentials, documents and job
state, where losing the last few hundred ms of commits on an unclean shutdown is
not an acceptable trade for write volume.
Separately, ha-pg shared_buffers 256MB -> 512MB. After 15h on Postgres its
buffers_backend was 629,704 of 939,025 total page writes -- 67% written by
backends rather than the checkpointer or bgwriter, which means the working set
does not fit in the buffer pool and pages are written out repeatedly instead of
coalescing. Memory limit raised 1Gi -> 1536Mi to keep the same headroom (measured
383Mi in use, of which 256MB was shared_buffers).
Verified before commit by rendering every cluster and asserting the tuning landed
under spec.postgresql.parameters with the barman plugin block untouched. That
check exists because the first attempt at this matched the wrong `parameters:`
key -- the barman plugin's, at 6-space indent under plugins: -- and corrupted
wp-pg's barmanObjectName. Reverted and redone with an indent-aware match.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.
Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.
HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:
/var/lib/code-server 8.0K
/mosquitto/auth 8.0K
/state (z2m guards) 8.0K
So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.
Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.
Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.
DELIBERATELY NOT CONVERTED:
- kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
and generated from the chart; a hand edit would be lost on upgrade.
- trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.
Cost: this rolls the affected pods once.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer was the single largest writer in the cluster -- more than every
Ceph OSD and every database combined -- and none of it needed to touch disk.
Measured 2026-08-20 from the container's own cgroup io.stat:
253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day
252:48 (its own PVC) 340 MB total = negligible
So the writes were not going to its volume at all; they were going to the
node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without
`medium: Memory` is node storage, which is easy to read as "ephemeral, so
free".
The cause is cadence rather than volume. schedules.template.json renders on
"* * * * *", so Chrome is launched 1440 times a day and each launch creates,
fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp
holds under 8 MB -- it is pure churn for data that never outlives the render.
tmpfs pages count against the container's memory limit, so the 512Mi cap is
kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample
headroom even if the cap were fully consumed.
Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a
4h window. This is a bigger cluster-wide reduction than the Home Assistant
recorder migration achieved, for a one-line change.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Net -44 lines. The previous version assembled the connection URI from
username/password/dbname, which forced percent-encoding, YAML-safe quoting via
json.dumps, a YAML round-trip validation, and python3 from HA's image.
None of that was necessary: CNPG already publishes a finished, percent-encoded
connection string in the app secret as `uri` / `fqdn-uri`. Reading that key
removes the whole class of problem the earlier review correctly flagged --
rather than solving it more carefully.
Verified against the live secret before simplifying: fqdn-uri contains only
[A-Za-z0-9:/@._%-], so naive double-quoting is provably safe and the YAML
validation was guarding against an input that cannot occur.
What is kept, because it is the part that actually matters: secrets.yaml is PVC
state holding every other HA secret (influxdb_token and friends), so the file is
filtered-and-appended rather than rewritten, moved atomically, and guarded
against shrinking. Tested in HA's own busybox shell -- comment and pre-existing
keys preserved, idempotent across runs, no duplicate key.
Also records why an env var cannot be used at all: HA removed `!env_var` from
its YAML loader, so `!secret` is the only mechanism for recorder.db_url, and
config-sync deliberately does not manage secrets.yaml.
Back to busybox; HA's image is no longer needed for this.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2b is done: 12.77M statistics rows copied SQLite -> Postgres with HA
running and no downtime (read-only mount; SQLite WAL gives readers a consistent
snapshot).
statistics_meta 1449 OK
statistics 10799904 OK 153.9s
statistics_short_term 1969804 OK 27.0s
span in Postgres: 2024-03-13 -> 2026-08-19
1449 entities with long-term history, 2627859 rows carrying sum
database size 2252 MB against a 20Gi PVC
Records the high-water marks the cutover top-up must resume from, and two schema
details that cost a failed first attempt:
- has_mean/has_sum are INTEGER in SQLite, BOOLEAN in Postgres, and Postgres
will not coerce them.
- the legacy created/start/last_reset DATETIME columns are 100% NULL in both
bulk tables, which is what makes a tab-separated COPY safe.
Also documents why the schema was built from HA's own models with schema_changes
left EMPTY: HA's _initialize_database detects a new database via the
events.time_fired_ts index and writes SCHEMA_VERSION itself, so pre-stamping that
row would bypass HA's own initialisation.
ANALYZE was run on all three tables. After a bulk COPY the planner has no
statistics and HA seq-scans 10.8M rows on every history/energy view.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #524, and it was a genuine blocker.
The first version built "postgresql://${PGUSER}:${PGPASSWORD}@host/db" in
busybox sh with no encoding. Two defects:
1. The CNPG-generated password is random and may contain @ : / ? #, each of
which changes how the URL parses. This repo already hit this exact issue
with the same CNPG password-generation mechanism -- see
kubernetes/apps/woodpecker/server.yaml, which switched to a lib/pq
keyword DSN specifically to avoid escaping generated passwords. That
option does not exist here: HA's recorder.db_url must be a SQLAlchemy URL.
So the fix is percent-encoding.
2. Worse, and the reason this was worth blocking: a quote in the password
makes the YAML line malformed, so secrets.yaml stops parsing entirely and
HA loses EVERY secret, not just this one. The line-count guard could never
catch that -- the count is unchanged; the file is simply invalid.
Rewritten to run python3 from HA's own image (already pulled for the main
container, so no extra image) instead of busybox:
- urllib.parse.quote(..., safe="") on user, password and dbname.
- json.dumps for the YAML value: YAML is a JSON superset, so this emits a
correctly escaped double-quoted scalar even with quotes or backslashes.
- The file is still rewritten LINE-WISE rather than round-tripped through the
YAML parser, so hand-maintained comments and key order survive.
- The guard now validates the RESULT: parse the new content, assert no key
from the old file disappeared, and assert recorder_db_url round-trips to
exactly the URL that was built. That catches corruption the line count
cannot.
Verified in HA's image against a deliberately hostile password
(p@ss:w/o"rd#?&\x=1) before commit:
new version -> parses, comment preserved, idempotent, password decodes back
byte-exact, host and dbname parse correctly
naive version -> yaml.safe_load raises ParserError
i.e. the approach this replaces would have produced an unparseable
secrets.yaml.
Also updates MIGRATION-recorder-postgres.md for this step landing, including
why the ordering (init container before the configuration.yaml change) is
mandatory.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2a of the recorder move off SQLite (MIGRATION-recorder-postgres.md).
Adds an init container that writes `recorder_db_url` into /config/secrets.yaml
from the CNPG-generated ha-pg-app Secret.
INERT until the recorder is repointed. It only adds a key to secrets.yaml;
nothing reads it until `db_url: !secret recorder_db_url` lands in
configuration.yaml in the homeassistant repo. That ordering is REQUIRED in this
direction — HA refuses to start when a referenced secret is missing, so the
config change must not go first.
Why an init container rather than a hand-added line: the password is generated
by CNPG and can be rotated, so a hand-written entry goes stale silently and HA
then fails to start on a bad password. Re-rendering on every pod start is also
what makes a rotation take effect.
secrets.yaml is PVC runtime state that config-sync deliberately does not manage
(it holds influxdb_token and others), so this must EDIT the existing file, never
replace it. The script filters out any existing recorder_db_url line, appends
the current one, and guards the result: if the rewrite would drop any
non-recorder line it aborts rather than truncating the file holding every other
secret HA needs. Written to a temp path and moved atomically, chmod 600.
Verified in HA's own Alpine/busybox shell against a fixed sample before commit
-- pre-existing keys preserved, guard reports old=3 new=3, and a second run
produces a byte-identical 4-line file (idempotent, no duplicate key).
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.
CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.
CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".
DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.
The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.
Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.
Refs #522
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recorder's SQLite database is the largest client writer in the cluster.
Measured 2026-08-19 via cgroup io.stat on the home-assistant container,
corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD
volume, ~26% of all client writes, against only ~5 GB/day of actual row data.
The 8x gap is the write PATTERN, not the volume:
~30 state changes/sec (107k rows/hour, 2.57M/day)
x2 SQLite WAL + checkpoint -- every dirtied page written twice
x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit
(measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
= ~15 KB written per recorded state change
Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
This attacks the multiplier rather than trimming what gets recorded, which would
hide the problem instead of fixing it.
PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on
SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves.
Phase 2 migrates the statistics tables and repoints recorder.db_url in the
homeassistant repo; MIGRATION-recorder-postgres.md has the sequence.
What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT
migrate data between database engines. `statistics` holds 10,793,712 rows
spanning 2024-03-13 to today -- 2.4 years of energy history that is not in
InfluxDB and cannot be reconstructed. Repointing db_url at an empty database
silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7.
Postgres is tuned to minimise write volume, which is the point of the move:
synchronous_commit off -- the recorder is telemetry, not transactional
record-keeping. Postgres stays crash consistent;
only the last few hundred ms of commits are at risk
on unclean shutdown. In exchange, commits stop
forcing an fsync -- the exact cost that makes SQLite
expensive here.
wal_compression on -- recorder writes are small inserts scattered across
pages, so post-checkpoint full-page writes dominate
WAL volume.
checkpoint_timeout 15min / max_wal_size 2GB
-- fewer checkpoints means fewer repeat full-page
images for the hot rows the recorder rewrites.
full_page_writes -- deliberately LEFT ON. RBD can tear a page across a
4 MiB object boundary; trading that for write volume
risks an unrecoverable database.
Also included:
- ObjectStore on its own s3 prefix (two clusters sharing a destinationPath
interleave WAL streams and corrupt both).
- ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters.
- PodMonitor added WITH the cluster rather than after it, so the migration has
metrics from the start instead of discovering the gap later (#433).
- Cilium policies as DEDICATED policies with their own endpointSelector, not
rules on the namespace-wide default-deny -- that policy is endpointSelector:{}
so any rule there applies to every pod in the namespace, and :8000 is served
by more than the CNPG instance manager. Includes the ICMP allowance that
attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD.
The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same
coupling #397 tracks for woodpecker. Noted in the file with the exact bao command
to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed`
was a wrong-path error rather than missing permissions.
Refs #434, #397
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three Ceph monitors were writing 240 GB/day between them for a ~106 MB
database. That was ~62% of all OS-disk writes on the nodes and roughly a
quarter of everything the cluster writes -- more than every client workload
combined.
Measured 2026-08-19:
paxos transactions ~99,000/day (1.15/s, sampled over 60 s)
paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime)
ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three
(cgroup io.stat: 2.15 TB / 27 d)
=> ~98x write amplification
Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix
(including paxos) when we trim its old states". With paxos_trim_min at 250 and
~99k transactions/day, that is hundreds of forced range compactions per day,
each rewriting the store.
The churn driving it is not cluster activity. 99% of paxos versions are cluster
log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook
operator poll (mgr stat / versions / status / report / osd dump; 23 audit
entries per minute per mon) becomes a cluster-log entry, hence a paxos
transaction, replicated to all three mons. Reducing that polling is the
complementary fix and is tracked separately; this change removes the
amplification regardless of the rate.
This does NOT disable compaction. RocksDB's own levelled compaction still runs
and still reclaims trimmed keys; what stops is the extra forced compact_range
on every trim. Two backstops if that assumption is wrong:
- mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn
in monitoring-rules already alerts on.
- mon_compact_on_start is set to true here (upstream default false), so every
mon restart -- upgrade, node reboot, reschedule -- does a full compaction.
Cheap on a ~106 MB store.
Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z:
wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z.
Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly
double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212,
and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank
writers; every figure above comes from cgroup io.stat or kernel diskstats, which
reconcile with each other and with the drives' SMART counters.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With 3 OSDs on 3 hosts, size=3 meant every write landed on every drive in the
cluster. Per-drive write volume equalled TOTAL cluster write volume, so all
three drives wore at an identical rate and would have reached end-of-rated-life
together: measured 2026-08-19 from the drives' own SMART counters, 80/82/87%
endurance used, advancing ~0.13 points/day, exhausted between Nov 2026 and
Jan 2027 (#434). Replication gives no protection there — the wear-out is
correlated by construction, not independent.
size 2 cuts per-drive write volume by a third and desynchronises the wear.
min_size 1 is required, not incidental. Ceph does NOT lower min_size when size
is reduced; it was 2, and leaving it would make replicapool 2/2, where a single
OSD down blocks IO for its PGs. That would hang every RBD volume in the cluster
on every rolling node reboot and every OSD restart.
Accepted trade, stated explicitly: while degraded (one OSD down), a second
drive failure loses data, and at 80%+ endurance that window is not
hypothetical. What limits the blast radius is that CNPG barman and Velero both
ship offsite to Hetzner, so exposure is the RPO of those backups rather than
total loss. Node reboots must stay serialised with HEALTH_OK between them.
CephFS metadataPool deliberately stays at size 3: 467 MiB against data0's
275 GiB, so it contributes nothing measurable to wear, while losing it destroys
the entire filesystem rather than the objects on one OSD. The wear saving comes
from data0. The RGW pools in objectstore.yaml were already size 2.
One-time cost on apply: Ceph deletes the third replica of everything, which is
background IO, and raw used drops from ~1.4 TiB to ~0.95 TiB.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bluestore_prefer_deferred_size_ssd was set to 65536 and
bluestore_deferred_batch_ops_ssd to 64 as a QLC throughput tune. Ceph's SSD
defaults are 0 and 16. Reverted to the defaults, set explicitly so the stored
`ceph config` value is overwritten rather than merely un-declared.
prefer_deferred_size defers every write SMALLER than the threshold: it lands in
RocksDB's WAL first and is applied to the object store afterwards. That is a
double write, and at 64 KiB it caught essentially every write this cluster
makes -- SQLite pages are 4 KiB, Postgres pages 8 KiB. Ceph disables deferral on
SSD by default because solid state does not need the random-write batching that
helps spinning media.
Why the original tune was the wrong trade:
- The benchmark justifying it cannot have exercised it. The 2026-05-05
baseline used 4 MiB writes, 64x above the 64 KiB cutoff, so none of those
writes were ever deferred. The setting was validated by a workload it does
not affect.
- It doubles writes onto QLC, the lowest-endurance NAND type. Measured
2026-08-19 from the drives' own SMART counters: all three OSD devices at
80/82/87% endurance used, advancing ~0.13 points/day, i.e. rated life
exhausted between Nov 2026 and Jan 2027. Because size=3 on 3 OSDs, every
drive absorbs 100% of cluster writes, so they wear in lockstep and expire
together (see #434).
Measured amplification before this change: 640 GB/day written to the three NVMe
devices against 117 GB/day at the RBD client layer -- 5.5x, of which 3x is
replication and ~1.8x BlueStore. Removing the double write should take the
BlueStore factor toward ~1.1x.
The cluster is write-light, so throughput was never the binding constraint;
endurance is. Both parameters are runtime-updatable, so no OSD restart.
Also corrects the drive model in the comment: the OSD devices are
CT1000P3PSSD8 (Crucial P3 Plus), not the P2 named previously.
bdev_enable_discard / bdev_async_discard are unrelated and kept as-is.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the noise-log dedup, both of which silently mute real alerts.
1. Prose-mentioned fingerprints became suppressions.
The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons
routinely cite another alert's fingerprint in prose ("see the issue filed for
fingerprint <fp>"), so those citations were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints
for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert
that HAD been filed as a genuine issue, permanently muted purely because two
other entries mention it. Now only lines in the exact shape the write-back step
emits count:
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among
them.
2. Suppression never expired.
The fingerprint is sha256(alertname|scope) and is deliberately stable across
recurrences, so a single blip verdict muted that alertname+scope pair forever.
21 pairs were permanently muted, including openbao's KubePodNotReady,
KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real
openbao outage would have filed no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be
interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted
from the date already present in each line. Re-triaging a recurrence weeks later
is cheap; missing an outage because a July blip muted it is not. If it is still
noise, triage re-adds it automatically.
Rollout is gentle by construction: every current entry is dated 2026-07-21 or
later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge --
the oldest entries lapse a day later, and only re-triage if they actually fire
again.
Unparseable lines are dropped, i.e. they stop suppressing: this fails toward
more alerting, never less.
Verified end-to-end by running the shipped snippet inside the real
ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20,
21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox
date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch
arithmetic with `date -d @<epoch>`, which was confirmed working in that image.
Refs #326
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #516: the restart-triggering mosquitto_pub was unbounded,
unlike the sibling call in zigbee2mqtt-interview-watch.sh.
Neither client bounds a TCP connect that HANGS rather than being refused, and a
stalled connect is most likely precisely when this guard runs -- around a broker
restart. Unbounded, the loop would block forever and stop logging, which is the
silent-failure mode the guard exists to prevent.
Applied to the reads as well, not just the publish: mosquitto_sub -W bounds the
wait for a message once CONNECTED, so it does not cover the connect phase
either. The timeout wrapper is set above -W so the protocol-level wait still
wins in the normal case.
Not mosquitto_pub -W: that flag does not exist for mosquitto_pub and makes every
publish fail on a usage error.
The cooldown stamp was already written only on a successful publish, so a failed
restart request still retries on the next cycle rather than burning the cooldown.
`timeout` is confirmed present in eclipse-mosquitto:2.0.22 (already used by
interview-watch, and exercised directly in this image).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mosquitto runs `persistence false` with no PVC (deliberate and settled), so
every broker restart wipes the retained store. Zigbee2MQTT publishes its HA
discovery configs retained and has no way to restore them itself, so a broker
restart not followed by a Z2M restart leaves every Z2M entity unavailable in HA
indefinitely -- ~14 h on 2026-07-23.
Adds a discovery-guard sidecar to the Z2M pod that detects the wiped state and
asks Z2M to restart, which re-runs discover().
Detection. bridge/state and bridge/{info,devices,groups,definitions} are all
retained, but only bridge/state is republished on reconnect (mqtt.ts onConnect).
So "bridge/state present + bridge/groups absent" means Z2M is talking to this
broker but nothing has re-run start() since the store was wiped.
bridge/groups specifically, because the absent-half partner must not be
re-publishable by ordinary runtime events or it self-heals and the guard goes
blind while HA is still broken. bridge/devices, bridge/definitions and
bridge/info all reach inbound radio traffic via devicesChanged (receive.ts hands
deviceExposesChanged to every fromZigbee converter); bridge/health self-heals on
its own 10 min timer. bridge/groups is only republished by deliberate admin
actions, and publishGroups() always publishes, emitting `[]` when no groups
exist -- confirmed live: it currently holds a 2-byte payload, so presence is
testable.
Not AND-ed with bridge/info being absent. That would cut false positives at the
cost of false negatives, and the costs are asymmetric: a false positive is one
~60s Z2M restart, a false negative is every Zigbee entity dead until a human
notices. Tuned for sensitivity.
Spurious positives are handled by a 90s confirm re-check, which covers both the
startup race (bridge/state is published at mqtt.connect(), bridge/groups only at
the end of BridgeExtension.start()) and a dropped publish (these topics go out at
QoS 0). A 30 min cooldown keeps a Z2M that will not come up healthy from being
restarted in a loop.
Action is bridge/request/restart, which is a full in-process restart -- index.js
restart() -> stop(true) tears down every extension, then start() builds a new
Controller and re-runs HomeAssistant.start(). discover() then republishes only
missing or changed configs, which after a wipe is all of them.
Reuses the existing zigbee2mqtt broker credential (ACL `readwrite
zigbee2mqtt/#`), so no new broker user and no read grant on homeassistant/#,
which the ACL denies on purpose.
Verified in the eclipse-mosquitto:2.0.22 image the sidecar actually runs:
mosquitto_sub -C/-W, mosquitto_pub and `date -u +%s` all present; `sh -n` clean;
startup run logs correctly and enters the loop. Detection was exercised against
the live broker and correctly reports healthy (bridge/state online,
bridge/groups present) with no action.
NOTE: this rolls mosquitto, because the corrected ACL comment lives inside the
container command. That roll wipes the retained store, which the guard then
repairs within ~6 min if the Z2M roll does not already do so.
Closes#381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four comments across the broker manifests asserted that retained state is
re-populated by all clients on reconnect, and specifically that Zigbee2MQTT
re-publishes HA discovery when it sees the homeassistant/status birth message.
Both are false, and the second is the premise that kept issue #381 open.
Verified against Zigbee2MQTT 2.12.1 source:
- discover() runs only from HomeAssistant.start(); it is never called from
the reconnect path.
- lib/mqtt.ts onConnect() republishes bridge/state and resubscribes, nothing
more. republishRetainedTimer is armed only inside connect() and is never
re-armed on reconnect.
- lib/extension/homeassistant.ts onMQTTMessage(), on status_topic == "online",
republishes bridge/state plus cached device states after a hardcoded 30s
delay -- but never the discovery configs.
- Discovery is published retain:true, and the only knob is the global
mqtt.force_disable_retain, which upstream documents as breaking the HA
integration. Broker retention is the sole mechanism Z2M relies on.
- No bridge/request endpoint forces a rediscovery; upstream declined to add
one (Koenkk/zigbee2mqtt #27458, #22977, #16950, all closed NOT_PLANNED).
The mosquitto ACL was never the blocker: `read homeassistant/status` is already
granted. That grant is still correct -- the cached-state republish is what makes
HA show current values after an HA restart -- so it is kept and the comment now
says what it actually buys.
Also records that a RollingUpdate roll is equivalent to a broker restart for
retention: maxUnavailable:0/maxSurge:1 preserves connectivity, but the new pod
starts with an empty retained store.
Comment-only. The two deployment comments are plain YAML comments outside the
container command, so the rendered Deployment is byte-identical and the broker
does not roll; the ConfigMap carries no reloader annotation, so it updates in
place without restarting mosquitto.
Refs #381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stage 2 of the decommission the shim was written for. Deleting the file makes
the next apply destroy the three containers.
The containers were taken out of service months ago — they are stopped
(pve_up = 0 for all three) and rustfs-{1,2,3}.srv.renner.uno no longer resolve
— but the LXC objects and their disks still exist on saturn/jupiter/neptune.
Two earlier destroy attempts (#270) died instantly with "context deadline
exceeded", because the bpg provider reads timeout_delete from STATE and the
imported resources had stored 0.
That is now fixed: apply 5675 landed timeout_delete = 1800 and the rest of the
timeouts in state, in place, with 0 destroyed. The destroy has a real deadline
to work with for the first time.
Checked before removing, since the containers are tagged `state-backend` and
`terraform-backend`:
- The tofu backend is Hetzner S3 (renner-uno-tfstate at
fsn1.your-objectstorage.com), not rustfs. Destroying these cannot pull the
state out from under the apply. The tags are historical, from before that
migration.
- No Kubernetes or Ansible resource references rustfs.
- scripts/restore-authentik-db.sh mentions rustfs only as an `mc` alias name;
its endpoint is s3.renner.uno:9000, the Ceph RGW `migration` store.
- DNS for the three names is already gone.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Both of these were learned the expensive way today and existed only in commit
messages and PR descriptions, which is not where anyone looks.
providers.tf carried a warning that a UniFi PUT may require _id in the body,
which restapi does not send, and that a 400 should be answered by tainting the
object so CI recreates it. That is wrong on this controller. Tested against a
live rule with exactly the body tofu sends: HTTP 200, object returned intact,
rule byte-identical before and after, 60 rules before and after. Confirmed at
scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and
immich.renner.uno still 200.
That wrong warning had a cost. A note saying an apply might break firewall
rules is one nobody argues with, so the apply did not get run, and the
renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
while the WordPress backup failed nightly with "The specified bucket does not
exist". The warning is replaced with the evidence that disproves it.
unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and
cosmetic, and lists the four attempts to remove it with the plan number that
disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662),
restapi 3.x ignore_server_additions (5667), and create_returns_object (5677).
The first three cannot work in principle — `data` is a configured attribute, so
Terraform core diffs it against state before provider-side ignore logic runs.
The fourth was the better theory, that PUT responses re-polluted `data`, and it
is wrong too: the diff returned on the plan immediately after the apply. That
means the response is written into `data` on the read path, and settling it
needs the provider source rather than more guesses at its schema.
No behaviour change. Documentation only, so the next person inherits the
findings instead of the dead ends.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Root cause of the 60 permanently-dirty hairpin NAT rules.
The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.
No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:
ignore_changes_to = ["_id"] plan 5661 — 60 remained
ignore_all_server_changes = true plan 5662 — 60 remained
restapi 3.x + ignore_server_additions = true plan 5667 — 60 remained,
plus a new api_data diff
create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.
This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Plan 5657 rejected the previous commit:
Error: Missing required argument
on lxc_800_rustfs_decommission.tf line 98, in resource
"proxmox_virtual_environment_container" "rustfs_decommission":
The argument "template_file_id" is required, but no definition was found.
The plan's own state dump showed only `type = "debian"` inside
operating_system, so mirroring it looked complete. It is not:
template_file_id is create-time-only, PVE does not report the original
template for an existing container, and nothing else in opentofu/ declares
one to copy — these three are the only container resources in the repo.
Guessing a template would be a lie that stays invisible until something
recreates these containers, which is precisely the event this file exists to
prevent. So operating_system is left undeclared and ignored via lifecycle
instead, which stops it planning as a removal without asserting a value that
is not known.
The timeouts remain outside ignore_changes on purpose — they are the one
thing this shim needs to actually apply.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The v2 shim's own instructions say the plan MUST show an in-place update and
no replace before applying. It does not. Pipeline 5656 planned all three
containers as `must be replaced`, which would destroy and recreate LXCs
tagged critical / state-backend / terraform-backend instead of nudging a
timeout value into state.
Cause: v2 declared only node_name, vm_id, started, unprivileged and the
timeouts, on the stated assumption that "the other attributes mirror the real
containers so the plan stays in-place". They were not declared at all, so
every absent attribute planned as a removal — and two of those removals force
replacement:
- mount_point { ... } # forces replacement (size, volume)
- operating_system { ... } # forces replacement
Declaring the full attribute set, mirrored from the values the plan itself
reported out of state: cpu, memory, disk, mount_point, network_interface,
operating_system and initialization. hostname and mac_address differ per
container, so the local map carries them per key rather than being repeated
or guessed.
tags is declared for a separate reason: it was planning as
["critical","rustfs","state-backend","terraform-backend"] -> null, which does
not force replacement but would have stripped the tags off the live
containers on apply.
This does not unblock the WordPress backup on its own. That needs
renner-uno-k8s-mariadb, which the same plan wants to create — but the bucket
cannot be applied while the plan also carries three container replacements,
and a -target apply is not an acceptable way around it: it diverges state
from config and leaves the real problem in place.
Verification is the file's own gate — a plan showing in-place only.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Rebased onto master now that #511 landed the WOODPECKER_URL fix for both
rotators — that part is dropped from here rather than duplicated. What remains
is the half nothing else covered.
All four rules in prometheusrule-woodpecker-rotation.yaml were selecting
series that do not exist, so the whole group was silent:
- WoodpeckerServerNotReady queried kube_statefulset_status_replicas_ready.
The server became a Deployment when the sqlite->CNPG migration dropped its
PVC, so that metric has no series at all — a `critical` "all CI is down"
alert that could never fire, and had been dead well before the namespace
move. Now kube_deployment_status_replicas_ready in the woodpecker namespace.
- WoodpeckerAgentNotReady selected namespace="apps". Broken by the namespace
migration; mine.
- The two woodpecker-secret-rotate rules selected namespace="apps", but that
rotation runs from the openbao namespace. Broken since that move, which the
woodpecker kustomization documents.
Verified against live Prometheus instead of reasoned about: server ready=1,
agent available=1, cronjob 1 series. The originals returned nothing, which is
exactly why they never fired rather than firing wrongly.
Two stale claims in the same file corrected while here: it pointed at
kubernetes/apps/woodpecker/rotate-agent-secret.yaml, which no longer exists,
and asserted the CronJob/Job series "stay empty until the rotation feature is
merged" — that rotation has since landed in openbao.
Also finishes the netpol comments #511 left mid-update, which still credited
woodpecker-allow-rotators to `apps/*`. gitea.apps.svc:3000 in review-bot.yaml
is deliberately untouched: gitea really is still in apps.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
From the update audit (#426), which flags this as the one item warranting
prompt action. Confirmed live: woodpecker-server runs v3.17.0 while the lint
step still pinned the v3.16.0 CLI.
That gap is the whole point of the step. The CLI compiles pipeline configs
against its own schema, so one behind the server can pass a config the server
then rejects — the gate stops predicting what the server will do, which is the
only thing it exists to do.
Verified the bump does not trade one problem for another: pulled
v3.17.0-alpine and ran the step's own lint loop over all 18 files in
.woodpecker/. Every one passes, so v3.17's schema does not reject anything
currently committed.
The drift is structural, not an oversight, and the comment now says so instead
of implying the two versions track each other. The server is a Kubernetes
manifest and image-automation bumps it; this pipeline file is plain YAML with
no apiVersion/kind, which Flux cannot patch (fluxcd/flux2#2826, recorded in
infrastructure/image-automation/README.md). So the server always moves first
and this always trails until someone bumps it by hand. Caught here by the
audit rather than by CI, which is the arrangement working as designed.
Also drops two stale "v3.16.0" references in the surrounding commentary that
would have aged into the same confusion.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#433.
Deletes the four apps-namespace manifests, promotes the migration-* set onto
their names, and gives the surviving Cluster the prune: disabled data guard it
was deliberately denied while a rollback still existed. After this the
kustomization renders nothing into `apps` at all.
Merging this deletes nothing. The old resources carry prune: disabled, so
dropping them from git only stops Flux managing them — they keep running and
keep their PVC. The teardown is a separate, deliberate act, and the runbook
carries the commands plus the precondition to check first.
That precondition is already satisfied: the new cluster took its own base
backup to the wp-pg-woodpecker prefix (wp-pg-daily-20260818205831, completed)
and reports Initialized, ConsistentSystemID, Ready, ContinuousArchiving and
LastBackupSucceeded all True. Recovery no longer depends on the retired
cluster's data.
Two things kept rather than cleaned up, both deliberate:
- bootstrap.recovery and its externalCluster stay on the Cluster. CNPG honours
bootstrap only when PGDATA is absent, so it is inert on a live cluster — and
keeping it means a cold rebuild restores from backup instead of coming up
empty, which is the gap RECOVERY.md exists to close for the other clusters.
- wp-pg-store-source stays, pointing read-only at the old prefix. Nothing
writes to it, and it is a second independent restore point predating the
move — worth having until the new prefix builds real backup history. It is
also the actual safety net for the teardown, rather than the running cluster.
The hetzner-creds comment now records the openbao ClusterSecretStore allowlist
requirement that cost stage 1 a failed restore (#505), so the next namespace
move does not rediscover it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The stage-1 verification section said row counts "will not match exactly if
builds ran after the backup the restore used — that is expected, and stage 2
quiesces the server before the final cutover".
Stage 2 does not quiesce. Flux prunes the apps workloads and creates the
woodpecker ones; nothing stops writes to the old database in between. The
sentence described an intent that was never implemented, which is worse than
saying nothing.
Measured on the live cutover rather than estimated: the restore captured
through pipeline 9488; 9491, 9492, 9494 and 9496 landed in apps/wp-pg before
the swap; the new cluster resumed at 9521. Four pipeline records and their log
entries exist only in the old cluster. No ID collision and no functional
breakage — cluster.yaml itself calls this database small and non-critical with
largely reconstructable state — but it is real history loss, and it should be a
decision rather than something discovered later from a row count.
Documents how to avoid it next time (suspend the KS, scale the old server and
agent to zero, re-run the restore) and notes the rows stay recoverable while
the old cluster keeps its prune guard, which is until stage 3.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Follow-up to #506, which merged before this landed. Addresses the review's
stale-comment suggestion, and a real over-grant found while checking it.
The comment above woodpecker-allow-rotators still explained the rule in terms
of the `apps` baseline and allow-ingress.yaml. Chasing that down showed
allow-ingress.yaml is hard-scoped to `namespace: apps` and matches traefik by
label — so it does not apply in the new namespace, and the traefik admit I put
in the woodpecker baseline instead was broader than the thing it replaced:
endpointSelector:{} plus a whole-namespace source, where the `ingress`
namespace also holds netbird-ingress. That let those pods reach wp-pg and the
agent, not just the server.
Replaced with woodpecker-allow-traefik, scoped to traefik and to the server
pod, matching how the rotator and CI-step rules are already written.
It lives in network-policies/woodpecker-ns.yaml rather than beside the other
workload policies, deliberately: the baseline admit is withdrawn in the same
file, so both halves ship in the infrastructure-crds Kustomization and swap in
one reconcile. Split across Kustomizations they would not — infrastructure-crds
reconciles before the `woodpecker` KS, so the admit would be withdrawn minutes
before its replacement arrived and ci.renner.uno would 503 in the gap.
Confirmed safe against certificate renewal: letsencrypt-cluster solves DNS-01
through Cloudflare, so there is no in-cluster ACME solver pod needing ingress.
With an HTTP-01 issuer this scoping would have broken renewals silently, 90
days later.
Runbook: records that the stage-2 outage outlasts pods becoming Ready, because
certificates are namespace-scoped and cert-manager must reissue woodpecker-tls.
Observed during the live cutover as a TLS SNI failure with no HTTP status at
all, which reads like a routing fault and is not one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. This is the downtime window: Flux prunes the apps copies and
creates the woodpecker ones, so ci.renner.uno is unavailable in between and
in-flight pipelines die with their agent.
Moved into `woodpecker`: server, agent, service, ingress, pdb, the agent
ServiceAccount, the woodpecker-secret ExternalSecret, and the rotator ingress
policy. The server's DSN now points at wp-pg-rw.woodpecker.svc and the agent's
gRPC at woodpecker.woodpecker.svc:9000.
Stage 1's restore is verified before doing this, not assumed: the new cluster
reports Ready=True and ConsistentSystemID=True (both False on the apps copy for
three weeks), its row counts match the live database exactly — 6 repos, 6125
pipelines, 2 users, 659 MB — and cnpg_collector_up now has a woodpecker series
for the first time.
Cross-namespace edges that had to move with it:
- rbac.yaml's RoleBinding subject. The Role and RoleBinding stay in
woodpecker-ci, because that is where they grant pod create/delete; only the
subject namespace changes.
- The openbao rotators' egress, which targeted `namespace: apps` + name
woodpecker on :8000.
New: woodpecker-allow-ci-steps grants woodpecker-ci → server :8000, scoped to
the SERVER pod. Stage 1 left this out on purpose after review found the same
rule on the namespace baseline would have handed arbitrary build steps a route
to wp-pg, since the CNPG instance manager serves its admin API on :8000 too.
Post-stage-1 the datapath confirms step-pod identities reach wp-pg on ICMP types
only; this rule must not change that.
Deliberately NOT touched: cluster.yaml, objectstore.yaml, scheduled-backup.yaml
and hetzner-creds-externalsecret.yaml still describe the original apps cluster.
They are the rollback, and they keep their prune guards. Stage 3 retires them
and renames the migration-* set onto their names.
Also corrects the runbook. It told the next person to verify policy semantics
with `cilium-dbg policy trace`, which does not exist in Cilium 1.20 — the
subcommand was removed, leaving only get/selectors/subject-selectors/wait.
Replaced with reading the realized BPF policy map via `bpf policy get
<endpoint-id>`, which is better evidence anyway: it is the datapath as
programmed rather than a simulation, and it carries live byte/packet counters.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Stage 1 of #433 is failing in the cluster. The wp-pg restore job errors
with:
can't get credentials for cluster wp-pg: while getting secret
hetzner-s3-creds: secrets "hetzner-s3-creds" not found
and the ExternalSecret behind it reports SecretSyncedError:
using cluster store "openbao" is not allowed from namespace
"woodpecker": denied by spec.condition
The ClusterSecretStore carries an explicit namespace allowlist, and the new
namespace was not on it. Its own header says so — "adding an ExternalSecret
in a NEW namespace requires adding that namespace here first, or ESO
reports 'not allowed to reference ClusterSecretStore' and the secret never
syncs" — so the requirement was documented and simply missed.
The reference sweep for #504 looked for everything that referenced
woodpecker or resolved *.apps.svc. It never asked the other question a
namespace move raises: what allowlists must the NEW namespace join? This
allowlist is exactly that shape, and it is the kind of thing kustomize
build and dry-run cannot surface, because the manifests are all valid — the
denial happens at ESO's admission of a cross-namespace reference.
Added in sorted position, keeping the list's stated ordering.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
From the fourth review on #504 — no blocking issues, but this one is a real
gap rather than a style point.
The woodpecker baseline was the only one of the repo's namespace policies
without an ICMP allow. That matters here specifically: wp-pg-allow-cnpg-operator
attaches an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP
to an endpoint once any layer 4 policy selects it unless the traffic is
CT-related. So PMTUD Destination Unreachable and Packet Too Big toward wp-pg
would have been dropped silently, surfacing as stalled transfers rather than
as a policy denial.
Uses the 16-type set the other baselines now carry after #495 trimmed them
from 39, rather than reintroducing the old list.
Granting this namespace-wide does not contradict the caution comment above it.
The :8000 case had to be per-workload because that port is shared between the
woodpecker server and the CNPG instance manager's admin API, so a blanket rule
handed CI step pods a route to the database. ICMP echo and path-MTU messages
expose no API surface, so a namespace-wide grant concedes nothing a scoped one
would withhold — which is why every other baseline does it this way too.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the third blocking review finding on #504.
The PR claimed the namespace move fixes wp-pg's absent metrics. It does not,
on its own. Prometheus requires the `release: kube-prometheus-stack` label
that CNPG's own enablePodMonitor omits, so every scraped cluster has a
hand-written PodMonitor in apps/monitoring-rules/cnpg-podmonitors.yaml —
ak-pg, im-pg, gf-pg, pprl-pg. There has never been one for wp-pg.
So wp-pg was missing both halves: no scrape target configured, and no
monitoring ingress allow in the apps baseline. Fixing the policy half was
necessary but not sufficient, and the runbook's cnpg_collector_up check
would have returned empty either way — unable to confirm the thing it
existed to confirm. Adds the PodMonitor in the woodpecker namespace.
Also from the review:
- Added a ScheduledBackup for the restored cluster. Every other CNPG
cluster here pairs isWALArchiver with one. Without it the new
wp-pg-woodpecker prefix takes continuous WAL but never a base backup, so
it has no independent recovery point and wp-pg-store's 30d retention has
nothing to prune against — the WAL chain grows unbounded for as long as
stage 1 sits before cutover. Scheduled 02:40, clear of the apps cluster's
02:35 and the other clusters' 02:30.
- Documented that the `woodpecker` KS goes not-Ready while the restore runs.
Its healthCheckExprs matches Cluster by GVK with no name qualifier, so
Flux evaluates it against both the live apps/wp-pg and the restoring
woodpecker/wp-pg, and wait: true makes the KS that owns production
Woodpecker report unhealthy. Expected, does not cascade since nothing
dependsOn it, but it would read as an outage without a note.
- Dropped the app.kubernetes.io/name label from the new Namespace; no other
namespace in that file carries one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the second blocking review finding on #504.
bootstrap.recovery set only `source`. The live CRD documents the defaults:
`database` defaults to `app`, and `owner` defaults to whatever `database`
is. Neither name exists in the physically restored data, which carries
woodpecker/woodpecker from the backup.
So CNPG would have restored the real data, then provisioned a separate
empty `app` database and pointed the generated wp-pg-app Secret and the
-app Service at that instead — a cluster reporting healthy, holding the
right bytes, with the credentials aimed at the wrong database. The runbook's
own verification (psql -U woodpecker -d woodpecker using the wp-pg-app
password) would not have authenticated, so the failure would have surfaced
as a confusing auth error rather than as what it was.
kubernetes/RECOVERY.md sets both fields on all three recovery examples it
documents (ak-pg, im-pg, pprl-pg) for exactly this reason; this one did not
follow the pattern.
Added a runbook step to read dbname/username back out of the wp-pg-app
Secret before the row-count checks, so the mismatch is caught directly
rather than inferred from a psql failure.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #504.
The "step pods call the server back on :8000" rule was attached to the
namespace baseline, which uses endpointSelector: {}. A toPorts block
narrows the port, not the destination workload — that is governed solely by
the enclosing policy's endpointSelector. So the rule granted woodpecker-ci
access to :8000 on every pod in the namespace, and the CNPG instance
manager serves its status API on :8000 (server.yaml:94 uses the same port).
Arbitrary repo-supplied build steps would have had a route to wp-pg's
instance manager.
That is the same endpointSelector:{}-means-blanket-grant failure this PR
exists to escape, reproduced inside the fix.
Removed rather than rescoped: there is no woodpecker server in this
namespace until stage 2, so the rule was premature as well as wrong. It
lands there as its own policy selecting the server pod, the way
woodpecker-allow-rotators already scopes its :8000 rule.
The baseline now carries no port-scoped rules at all, so nothing in it can
be mistaken for workload scoping, and a comment warns why that must stay
true.
Also from the review:
- Corrected the KS attribution. The network policies ship with
infrastructure-crds (infrastructure/post-crds includes
../network-policies/), not the top-level infrastructure KS. The ordering
conclusion is unchanged — infrastructure-crds dependsOn infrastructure,
and woodpecker dependsOn both — but the comments named the wrong one.
- Documented why the baseline has no egress section, instead of leaving it
to read as an oversight. Writing that ruleset in stage 2, with the server
and agent present, beats guessing it now and breaking the WAL archiving
this stage exists to verify.
- Dropped the monitoring -> :9187 rule from wp-pg-allow-cnpg-operator. The
baseline already allows the monitoring namespace unrestricted, so it added
no coverage while implying scoping it did not provide.
- Added a cilium-dbg policy trace step to the runbook. dry-run validates
manifest shape, not Cilium semantics, and would not have caught this bug.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. No cutover, no CI impact: this stands up the `woodpecker`
namespace and restores wp-pg into it from barman, running in parallel with
the live apps/wp-pg. Nothing points at it yet.
Why a namespace move rather than a policy fix. wp-pg has been Ready=False
with "Instance Status Extraction Error" for three weeks because the CNPG
operator in `database` cannot reach https://<pod-ip>:8000/pg/status. The
apps default-deny-ingress uses endpointSelector:{}, so the only
namespace-level remedy is to allow the whole database namespace in — which
would expose gitea, gatus, external-dns and ical-proxy along with wp-pg, to
fix one pod on one port. The same gap explains why cnpg_collector_up has
series for authentik, immich, monitoring and paperless but not wp-pg: apps
never allowed the monitoring namespace either.
woodpecker-ci is deliberately not the target. It enforces pod-security:
privileged and exists for ephemeral step pods running repo-supplied code;
the database has no business sharing it.
Two ObjectStores, deliberately. wp-pg-store-source reads the existing
s3://renner-uno-k8s-cnpg/wp-pg prefix; wp-pg-store archives to a new
wp-pg-woodpecker prefix. Two clusters writing one destinationPath would
interleave their WAL streams and corrupt both, and splitting them keeps the
pre-migration backup intact as rollback.
The Namespace is declared in infrastructure/namespaces rather than beside
these workloads: its network policies ship with the `infrastructure` KS,
which the `woodpecker` KS dependsOn, so creating it in the workload KS would
invert that order and deadlock a cold rebuild.
The restored cluster deliberately carries no prune: disabled guard. The
original has one because it holds the only copy of the data; this one is
reconstructable, and a prune guard on a migration artifact is how orphans
outlive an abandoned migration. It gains the guard in stage 3.
Barman creds still resolve from seed/paperless/hetzner-creds. #397 wants a
woodpecker-owned path and this was the natural moment, but creating it needs
OpenBao write access that is not available here (bao kv list seed -> 403).
Pointing at a path that may not exist would leave the restored cluster
unable to archive WAL, which is what this stage exists to prove.
Runbook, including how to verify the restore holds real data rather than
merely coming up healthy: MIGRATION-namespace.md.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Ceph's 30% default assumes the mon owns its filesystem, where 30% free is
a real reserve. Our mons sit on dataDirHostPath /var/lib/rook — the node's
/var, shared with ~29 GB of containerd storage — and the mon DB is ~106 MB.
The default therefore measures container churn rather than Ceph state, and
k8s-2 trips MON_DISK_LOW whenever its image cache fills.
Tuning image GC does not help: measured on k8s-2, dropping the GC high
threshold to 60 deleted 28 images and moved free space by 0.02 percentage
points, because most of /var/lib/containerd is overlayfs snapshots held by
running containers, which image GC cannot reclaim.
20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database.
mon_data_avail_crit is untouched at 5%, and CephMonDiskLow watches /var
directly for the real risk — the node filling, not the mon growing.
Set through cephConfig so it is reconciled from git rather than applied as
a live `ceph config set`.
The comment records that this override should be revisited, and dropped,
when mons get their own volume at bare-metal migration. That is not
possible on the current cluster: every StorageClass is Ceph-backed, so a
mon PVC would be circular, and Rook applies mon.volumeClaimTemplate only
to newly created mons.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The 60/50 change in #500 was based on a wrong premise. Reverting the
values and correcting the reasoning recorded around them.
Applied to k8s-2 alone and measured. GC did fire: 28 images deleted,
content store 9.93 -> 8.62 GB. Free space moved 31.49% -> 31.47% — no
effect.
The premise was that ~19 GB of overlayfs snapshots would be reclaimed
along with their images, at roughly 2.9x the compressed size. That is
wrong. /var/lib/containerd is 20.5 GB of snapshots against 8.6 GB of
content, and those snapshots are unpacked layers shared with running
containers. Image GC cannot touch them, so deleting unused image
references frees only their unshared blobs. Remaining reclaimable
content is now ~0.7 GB; further sweeps free nothing.
70/60 stands on its own merits as a bound on the image cache — it is
simply not a lever on MON_DISK_LOW, and the comment no longer claims it
is. The mon's own data is ~106 MB, so a 30%-free threshold on a 60 GB
filesystem it merely shares says nothing about Ceph.
The CephMonDiskLow runbook text carried the same wrong claim and is
corrected to point at finding the real consumer instead.
k8s-2's live kubelet is already back to 70/60 via talosctl patch; node
Ready, Ceph HEALTH_OK with mon quorum a,b,c intact. k8s-1 and k8s-3 were
never changed.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Gap 2 — no admin path. There is still no rook-ceph-tools deployment, so
ceph status, ceph health detail, ceph osd df tree, ceph osd perf and ceph
device ls are all unreachable during an incident: mons, OSDs and mgr
mount no admin keyring and the operator's /etc/ceph is empty. Adds the
upstream toolbox from rook v1.20.4 (matching the running operator image)
with the image pinned to the cluster's own quay.io/ceph/ceph:v19.2.5 and
carrying the same imagepolicy marker as cephcluster.yaml, so the client
cannot drift ahead of the mons. Prerequisites verified present: the
rook-ceph-default ServiceAccount, the rook-ceph-mon-endpoints ConfigMap,
and ceph-secret/ceph-username in the rook-ceph-mon secret.
Image GC. The thresholds were already lowered once from the kubelet
defaults, to 70/60, explicitly to stay under Ceph's mon_data_avail_warn.
But that line is <=30% free and a high of 70 fires at exactly 30% free —
on the threshold, not below it. So the cache fills to precisely the point
where the mon warns, and MON_DISK_LOW trips before kubelet reclaims
anything. Observed on k8s-2: 29.81% free and HEALTH_WARN MON_DISK_LOW,
clearing to 31.64% once GC ran. 60/50 fires at 40% free and reclaims to
50%, leaving 10 points of margin.
There is plenty to reclaim, so the lower threshold will bite: containerd
holds ~29 GB of k8s-2's 60 GB /var and caches 179 images against 60 in
use, some dating to May. (The ~309 GB talosctl attributes to
/var/lib/kubelet is mounted Ceph PVCs counted through mount points, not
local disk.)
Alerting, the follow-up #434 parked until ceph_* series existed — they do
now, via #432. CephHealthWarn deliberately waits 6h: Ceph warns for
self-clearing conditions and paging on those teaches people to ignore it,
so what is worth waking for is a warning that does not clear.
CephHealthError fires at 5m. CephMonDiskLow watches /var free directly
rather than going through Ceph, because this fires for a full image cache
rather than for Ceph growth, and naming that keeps the next person from
debugging the wrong subsystem.
NOTE: cluster.yaml is a bootstrap patch. Merging changes nothing on
running nodes — the live kubelet value needs talosctl patch machineconfig.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The earlier commits in this branch raised policyMapMax and added an alert.
Neither addresses why the map fills. This does.
Measured on k8s-1 with cilium-dbg: 8 endpoint policy maps sit at exactly
16384 entries — full — and the rest cluster at ~8,500. The split tracks
policy direction, so one direction costs ~8,500 entries. Dumping velero's
endpoint 2328 (8,499 entries, ingress only) shows 217 identities x 39
port/proto values, of which 38 are ICMP types. The map is ~99% ICMP, and
3,744 of its entries (44%) belong to woodpecker-ci-step identities.
Two multiplying causes, both fixed here.
The ICMP rule in every default-deny-ingress allowed 39 enumerated types
from `namespace: Exists` — every identity in the cluster. Cilium expands
that to one entry per (identity, type), per endpoint, per direction, in
19 namespaces. The rule's own comment records it was widened "from the
error class to the full standard type set"; that widening is the cost.
Trimmed to 16: the error types that motivated the rule (v4 3/11/12, v6
1/2/3/4 — PMTUD is the documented need), echo, and the IPv6 ND/RA/MLDv2
types observed in flight via Hubble (133, 135, 143). Dropped types are
deprecated (v4 4 Source Quench, RFC 6633), router-only (v4 5, v6 137),
obsolete (v4 13-18, 30), superseded (v6 130-132 by 143), or never
deployed (v6 138, 141/142, 148/149, 151-153). openbao-ns.yaml already
runs an error-only set of 7 in production, so 16 is the conservative side
of existing practice.
Cilium had no `labels` filter, so identity came from all pod labels.
Woodpecker stamps each step pod with woodpecker-ci.org/task-uuid and
step=<name>, making every build step its own security identity — 96 of
them minted in the last hour, against zero pods in the namespace. The GC
is healthy and simply outpaced. Excluding those two label patterns
collapses 78 current identities to one. No CiliumNetworkPolicy selects on
them, so they bought churn and nothing else.
Together these take a direction from ~8,500 entries to roughly 140
identities x 17 values ~= 2,400 — about 4,800 for both directions, well
inside even the old 16384 ceiling. policyMapMax stays raised as headroom,
not as the fix.
Per Cilium's docs, existing identities are not recomputed by a `labels`
change; new ones use the filter and the old ones age out through normal
GC, so the map drains over roughly an hour rather than instantly.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
renner.uno and www.renner.uno were pinned to 159.69.42.237, the VM that hosted
WordPress before it moved into the cluster. Neither record is live on the UDM
any more (verified: a LAN query for www.renner.uno returns the public answer,
proxy.renner.uno), so the file was drift waiting to be re-applied — a tofu
apply would have pointed the house at a host that is being decommissioned,
while the rest of the world already reaches the cluster.
Their absence is the correct end state, not an omission: www.renner.uno
carries dns.renner.uno/public, so external-dns-unifi writes no UDM record by
design and LAN clients take the public path and hairpin back in. The apex
stays with Cloudflare, which now redirects renner.uno/* to www.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Two optional points from the second review on #495.
The PrometheusRule alerts on cilium_bpf_map_pressure without filtering on
map_name, so it covers conntrack, NAT and LB maps as well as the policy
map. That is intended — running out of any of them degrades the datapath
— but the PR is framed around the policy map, so a reader could take the
narrow reading. Say so in the file.
The cross-references between the three copies of bpf-policy-map-max were
one-directional: helmrelease.yaml and patches/cilium-cni.yaml each named
only one other site, while talos/cilium.yaml named both. Each now names
the other two.
Not taken: the suggestion to quote policyMapMax as "65536". The chart
declares it `type: [null, integer]` in its values schema
(helm show values cilium --version 1.20.0), so an integer is correct and
a quoted string would be the schema violation.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #495.
Raising bpf.policyMapMax only in the HelmRelease left the bootstrap path
on the old value. kubernetes/bootstrap.sh:126 passes
kubernetes/talos/patches/cilium-cni.yaml to talosctl apply-config for
every node, and that manifest hardcoded bpf-policy-map-max: "16384" — so
a cold rebuild would install Cilium with the map size this PR exists to
fix, and stay there until Flux reconciled the HelmRelease.
The sibling bpf-lb-map-max is "65536" in both files, matching the live
agent config, so keeping these in sync is the repo's existing practice.
kubernetes/talos/cilium.yaml carries the same key and is updated too.
Nothing in the repo references that file — only patches/cilium-cni.yaml
is used by bootstrap.sh — so it looks like an orphaned rendered snapshot.
Syncing it rather than deleting it: whether it should exist at all is a
separate question from this fix.
Each of the three sites now names the others, so the next change to this
value has the lockstep written down where someone will hit it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#451. Closes#454.
Both issues are the same fault: a pod's dedicated cilium_policy_v3_<id>
BPF map ran out of space, endpoint regeneration failed with "no space
left on device", the CNI ADD call never returned, and the pod sat in
sandbox creation. For paperless — replicas: 1 with strategy Recreate —
that was a 41-minute outage on 2026-07-30.
It is not over. Prometheus (metrics landed via #452) puts
cilium_bpf_map_pressure{map_name="cilium_policy_v3_*"} at 0.867 on all
three nodes right now, and max_over_time(...[30d]) at 1.03 — the map has
been driven past capacity. Loki has the overflow recurring as recently
as 2026-08-12 17:46Z, hitting authentik and esphome endpoints.
Three changes:
- bpf.policyMapMax 16384 (chart default) -> 65536, matching the cluster's
existing bpf-lb-map-max. Takes steady-state pressure from ~87% to ~22%.
The map is sized at creation, so this only takes effect as agents
restart.
- The chart range ">=1.19.0" is what let Flux take 1.19.6 -> 1.20.0
unattended 45 minutes before the first outage. Narrowed to
">=1.20.0 <1.21.0" so a CNI minor is a reviewed diff, not a surprise.
- A PrometheusRule on cilium_bpf_map_pressure: warning over 80% for 15m,
critical at 100%. The metric was already scraped but nothing alerted on
it, so the first signal was a downstream replica mismatch.
Not addressed: why the busiest endpoint needs ~14k entries. Only 178
CiliumIdentities exist and no CiliumNetworkPolicy declares more than 9
ports, so the entry count is not obviously explained by policy shape.
Establishing that needs `cilium-dbg bpf policy get` on an agent.
MERGING THIS ROLLS THE CNI. The DaemonSet uses
updateStrategy.rollingUpdate.maxUnavailable: 2, so two of three agents
restart at once.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Verified against the deployed cluster: GET /wp-admin/ correctly 302s to
https://www.renner.uno/outpost.goauthentik.io/start?rd=... — and that URL
returned WordPress's 404 page, because /outpost.goauthentik.io fell through to
the public '/' prefix router.
The effect was that /wp-admin was unreachable rather than protected. An
attacker gains nothing, but neither does the editor the gate exists for.
paperless needs the same route and solves it with a separate Ingress on
auth.renner.uno (apps/authentik/paperless-gate-ingress.yaml). Here the whole
flow stays on www.renner.uno, so the path belongs in this rule set.
These endpoints are the login flow itself and are served by the outpost, so
routing them to it is not an authentication hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Closes#461.
Five pipeline steps pinned ci-k8s at two stale tags: e2e-test.yml,
e2e.yml, alert-monitor.yml and issue-fix.yml at 2026-05-01, and
ci-manifests-test.yml at 2026-07-30. 2026-08-10 is the newest dated tag
in gitea_admin/ci-k8s and shares a digest with :latest
(sha256:0019f264...), so all five now run the same image the build
pipeline last produced.
A stale pin eventually points at a garbage-collected tag, and the split
had no deliberate reason behind it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The first deploy returned 500 on every request:
PHP Fatal error: Uncaught Error: Call to undefined function add_filter()
in /var/www/html/wp-config.php(128) : eval()'d code
WORDPRESS_CONFIG_EXTRA is eval'd inside wp-config.php, which runs long before
wp-settings.php loads the plugin API. add_filter() does not exist yet at that
point, so the call was fatal on every request.
Removing it exposed the second, worse half: the comment justified the filter by
claiming xmlrpc.php was 'blocked at the ingress'. It was not. The public router
matches '/' as a prefix, so xmlrpc.php was served like any other path — the
claim was written without being checked.
An Ingress has no deny verb, so xmlrpc.php now routes to the Authentik gate
alongside /wp-admin and /wp-login.php: anonymous callers meet Authentik instead
of the XML-RPC endpoint. Nothing on this site uses it — no Jetpack, no mobile
app, no pingbacks worth keeping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Addresses the automated review on #492.
Blocking: two comments claimed the public/admin split was kept honest by
kubernetes/apps/wordpress/network-policy.yaml. That file does not exist — the
comment was written against the paperless template and never updated once the
namespace policy proved sufficient. The deeper point in the review is the real
one: a paperless-style workload-scoped policy would be a no-op here, because
traefik must reach the pod directly for every public page. The split is
enforced by the Ingress path rules and Authentik, not at the network layer,
and the comment now says so — an auditor looking for a network control would
otherwise have gone hunting for a file that was never there.
Velero: the schedule excluded the database PVC by app.kubernetes.io/instance,
a label the mariadb-operator was only assumed to set. A wrong assumption there
fails silently and sweeps the live database volume into the snapshot. Inverted
to select the content PVC by labels this repo sets on it, the same direction
openbao-daily already uses.
The Service-name assumption flagged as a suggestion is kept: the CRD's separate
primaryService/secondaryService fields indicate the base Service carries the
resource name, and a mismatch fails loudly at pod startup. Written down as a
first-deploy check rather than left implicit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Migrates the Hetzner VM 'wordpress-hochzeit' into the cluster. The site is not
the frozen wedding archive it looks like: it carries an unpublished, fully
composed birth announcement from 2026-05-14, so wp-admin has to keep working
for a human author rather than being replaced by a Git workflow.
Split routing on one host: '/' goes straight to WordPress so the blog stays
publicly readable, while /wp-admin and /wp-login.php route through a dedicated
wordpress-gate outpost. The inverse -- gate the host and skip_path_regex the
public paths, as paperless-gate does -- would need a negative lookahead, and
the outpost's RE2 has none.
Gating all of /wp-admin is only safe because the dead wpgiftregistry plugin is
dropped in the migration: it was the sole emitter of ajaxurl on this site, and
of 121 admin-ajax hits in the current log window not one returned 200.
Core lives in the pinned image, not on the volume, so upgrading is a tag bump
in Git. Only wp-content is persisted.
Backups are split the same way CNPG/Barman and Velero are elsewhere: the
operator takes a logical dump to Hetzner Object Storage at 02:45, Velero takes
the wp-content PVC at 08:45 and skips the database volume -- a crash-consistent
snapshot of a running database is not a backup.
Group membership for the gate is owned by Git, as with the Home Assistant,
Shelly and zone-configurator gates. s.renner is the Authentik account for the
site's WordPress Editor, whose login there is still the pre-2019 s.birzer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
WordPress speaks MySQL natively. The PostgreSQL route would mean pg4wp, an
SQL-rewriting shim whose README lists WordPress 6.5.3 as its newest tested
version while renner.uno runs 7.0.4 — every core update would become a
compatibility gamble.
Charts are OCI artifacts, so this follows the OCIRepository pattern from
spegel rather than the HelmRepository pattern used for CNPG. CRDs ship as a
separate chart that upstream requires first, hence the dependsOn. The version
range is capped at the year boundary: the project uses calendar versioning and
ships an upgrade guide with each minor.
renner-uno-k8s-mariadb carries the same 30d noncurrent-version expiry as the
CNPG and Velero buckets, matched to the Backup resource's maxRetention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
The passkey apps were reachable only with a WebAuthn credential:
passkey-passwordless-authentication denies at its validation stage, and
passkey-stepup-authorization demands a passkey on every authorize. An
account with no passkey had no way in.
Branch both flows on enrollment with an expression policy,
passkey-user-has-webauthn. Enrolled accounts keep the mandatory passkey
at login and at every authorize. Accounts without one skip the WebAuthn
stage and get default-authentication-password followed by
default-authentication-mfa-validation — the same pair, in the same
order, that default-authentication-flow already runs, so the fallback is
not a weaker way in than auth.renner.uno's own login.
The policy runs against pending_user, so the branch follows the
identified account's enrollment and is not selectable by the requester.
Bindings keep evaluate_on_plan false and re_evaluate_policies true;
on-plan evaluation would run before a user is resolved.
The negated bindings carry failure_result: true. UserLoginStage defaults
a missing authentication backend to BACKEND_INBUILT rather than
refusing, so a flow that dropped both branches would log the user in on
a username alone. A PolicyException resolves to failure_result before
negate is applied, so a broken expression degrades to password + MFA
instead of to no factor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every fresh login to a passkey app costs two touches: one in
passkey-passwordless-authentication, a second immediately after in
passkey-stepup-authorization.
last_auth_threshold is enforced with a single browser cookie
(authentik_mfa) whose JWT is signed with, and carries, the validation
stage's PK. A stage only honours a cookie it wrote itself, and both
stages write to the same cookie name — so two stage objects each demand
their own touch and clobber each other's window.
Collapse them into one passkey-webauthn-validation stage bound into both
flows. not_configured_action is deny, since the authentication flow
reaches the stage with the user only identified and no password stage in
front of it. Passkey-less users enroll from user settings after a
password login; default-authenticator-webauthn-setup has a
configure_flow, so that path is live.
The 8h cookie now also covers the login flow, so a re-login in the same
browser inside the window is satisfied by the cookie alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
:latest never rolls. The tag moves when a build publishes, but nothing
tells the Deployment: the pod template is unchanged, so Kubernetes has
no reason to replace anything.
Nothing else covers it either. This image is deliberately not enrolled in
Flux image automation — kubernetes/infrastructure/image-automation/README.md
lists ghcr.io/konairius/ical-proxy as having "only main, latest, and
main-<sha> commit builds; no release tags", so no ImagePolicy watches it.
The two pods have been running a 21-day-old image while :latest moved
underneath them, and the only way to pick up a new build was a manual
restart, which leaves no record anywhere.
Pinning makes the rollout a git change. Flux applies the new pod
template, Kubernetes rolls it, and what is RUNNING becomes recoverable
from this repo instead of "whatever :latest meant the last time a pod
happened to restart".
main-99a912b is the merge of konairius/ical-proxy#12, which adds the
summary_match and summary_replace parameters. The waste calendar needs
them to strip "| Abfuhrkalender - Landkreis Amberg-Sulzbach" from every
collection — the feed repeats the whole calendar's name on each event,
and no consumer can shorten it after the fact.
Bumping this tag is now a manual, reviewed edit. Format is
main-<short-sha>, from docker/metadata-action type=sha with the branch
as prefix.
The renderer's init container writes /data/schedules.json ONCE, at pod
start, and nothing re-reads either source afterwards. So a ConfigMap or
Secret change lands in git, updates the object, and the running
renderer keeps the old values indefinitely.
The app re-reads schedules.json every 60 seconds, which makes it look
self-updating. It is not — it re-reads the FILE, and only the init
container writes the file.
This was hit three times in practice, each needing `kubectl delete pod`
by hand:
* switching the theme to Graphite E-ink Light — the pod kept
rendering with a theme name that no longer existed
* setting the schedule's lang field to "de"
* every re-mint of the BYOS tokens, which is the Secret half
The theme case is the instructive one: nothing errored. HA silently
served the default theme for a name that did not resolve, so the panel
just looked wrong, and I was about to debug the design.
Annotates both sources. Named per-resource rather than the blanket
reloader.stakater.com/auto: "true", matching terminus.yaml and
mosquitto-deployment.yaml.
Terminus got this annotation earlier; the renderer did not, despite
having the same init-container pattern and hitting the same failure in
the same session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lang was null, so the screenshot session rendered in English — and the
dashboard grew a German weekday array, a German month array and a
fifteen-entry weather-condition dict to translate around it by hand.
The field is documented as "UI language code (HA mode only)" and calls
Home Assistant's own _selectLanguage() on the page before capture;
upstream's tests use lang: 'de'. So localisation is HA's job and the
field was sitting in the schedule template, already written out, set to
null because I copied the shape without asking what it did.
Pairs with homeassistant#220, which deletes both hand-written mappings:
the weather badge points at weather.garten directly and the date is a
device_class: timestamp the frontend formats.
The instance stays language "en". This affects only the render session,
so the panel is German without changing anyone's UI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer stopped pushing and the panel preview froze. Every webhook:
HTTP 401: Unauthorized - This session has expired, please login again
Terminus has TWO independent expiries and the earlier fix only addressed
one. API_ACCESS_TOKEN_PERIOD governs the JWT's own lifetime and is now
10 years. SESSION_EXPIRATION_ENABLED / SESSION_INACTIVITY_LIMIT govern
the SESSION BEHIND that token and were left at their defaults, so a
valid, decade-long token stopped being accepted after about half an
hour of session inactivity.
Worth being precise, because the two look alike and are not: the first
failure was 400 "expired JWT access token", the token itself. This is
401 "This session has expired", the session. Different mechanism,
different error, same symptom — a panel that silently stops updating.
I noticed both expiries when diagnosing the first one, wrote that
Terminus "has two independent expiries", and then fixed one and moved
on. That is the actual mistake here, not the missing variable.
Disabling session expiry is the state the renderer's own code already
assumes ("stock Terminus disables session expiration"), which is false
by default and has to be made true here — the same shape as the token
period. And it is the deciding constraint for an unattended appliance:
a session that expires needs a human with a password to restore it, and
a hallway panel has neither.
Reloader will roll the pod; the ConfigMap annotation added earlier
covers this. Existing tokens must be re-minted afterwards — the current
session is already dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer's own docs recommend Graphite by name — "Use e-ink theme
(Graphite recommended)", with theme=Graphite in the example — and the
Home Assistant community converges on it independently. The dashboard
now ships graphite-eink-light.yaml (homeassistant repo) in place of the
theme I hand-wrote from a spec sheet.
Pairs with homeassistant#216. The theme name must match the key inside
the vendored file exactly ("Graphite E-ink Light"); a name that does not
resolve fails silently — HA serves the default theme and the panel just
looks wrong rather than erroring.
Note upstream's example also passes `dark`. Not used here: the e-ink
LIGHT variant is the one that gives black-on-white, which is what an
e-paper panel wants. `dark: false` stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The setting was treated as an unverified assumption in review, and I
repeated that framing. It was not one.
Upstream's doc/configuration.adoc documents the variable and its
1800-second default in as many words, and this instance corroborated it
empirically: a token minted here was rejected roughly thirty minutes
later, which is the documented default behaving exactly as written.
Documentation plus matching observed behaviour is not an assumption.
The reviewer's uncertainty was reasonable — Terminus's source is not
vendored in this repo, so from the checkout alone the claim is
unsupported. The failing was mine for citing nothing at the point of
use, which is the same gap as the /health probe earlier: the evidence
existed and was not written where a reader could find it. Quotes the
documentation inline so the question is answerable from the manifest.
Note this does not make the post-merge check redundant. Whether Terminus
honours the variable is settled here; whether this ConfigMap edit
actually reached the running process via Reloader is a separate
question, and that is what decoding the new token's exp claim answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review's remaining point: with API_ACCESS_TOKEN_PERIOD at 10 years,
nothing will ever expire the renderer's BYOS token or prompt anyone to
rotate it. Left undocumented, "never rotated" is indistinguishable from
"forgotten".
Records it where it is discoverable — the rotation directory itself,
whose purpose is that unrotated credentials are a visible choice rather
than an omission. Notes why there is no rotator: minting a Terminus JWT
requires the account password, which is precisely the secret this design
declined to hold in-cluster, so a rotator would have to store the thing
the long-lived token exists to avoid. Rotation is therefore manual, tied
to the Terminus account password changing.
Also flags that HA_ACCESS_TOKEN in the same OpenBao path is a different
case: secret-rotator already has an `ha-token` mode (home-assistant.yaml
uses it), so enrolling that one is a real follow-up rather than a
limitation, and doing so would leave only the BYOS pair unrotated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review caught that this PR, as it stood, would not have fixed the
outage it describes.
terminus-config is consumed via envFrom, and env vars are read once at
container start — Kubernetes does not hot-reload them. The ConfigMap is
a plain resource rather than a configMapGenerator, so editing it does
not change the pod template hash and Flux will not roll the pods
either. Stakater Reloader is deployed but acts only on annotated
workloads, and this Deployment carried no annotation.
Net effect would have been a silent no-op: git and the ConfigMap object
would show a 10-year token period while the running app kept issuing
30-minute tokens, and the re-mint step in the PR plan would have
produced another short-lived token that looked correct and expired
anyway. The manifest and reality would disagree with nothing to
indicate it.
Annotates for terminus-config specifically rather than the blanket
reloader.stakater.com/auto: "true", matching the reasoning already
recorded on mosquitto-deployment.yaml — enrolling anything further
stays a visible, reviewable edit to that line.
Verified Reloader is actually running (kube-system/reloader-reloader).
This also removes a manual step rather than documenting one, which is
the point: a fix that needs a human to remember `kubectl rollout
restart` is not deployed, it is a runbook entry waiting to be missed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer renders correctly and pushes nothing. Every webhook is
rejected:
[webhook] Webhook failed: 400 {"error":"expired JWT access token"}
[webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."}
Terminus issues API JWTs with API_ACCESS_TOKEN_PERIOD, default 1800s.
Minting a replacement via POST /api/jwt requires a still-VALID access
token in the Authorization header, so an expired token cannot refresh
itself — the second line above is the refresh attempt being rejected by
the very expiry it was trying to repair. There is no recovery path once
the window closes.
That makes "mint once, store in OpenBao" unworkable for an unattended
appliance, which is what this stack is. The renderer's own code assumes
"stock Terminus disables session expiration and issues effectively
permanent tokens"; that is not true of this deployment. Rather than work
around the assumption in the client, this makes it true: sets
API_ACCESS_TOKEN_PERIOD to 10 years.
The trade is deliberate. A long-lived JWT with full Terminus API rights
now sits in OpenBao. The alternative was putting the Terminus account
PASSWORD into the cluster so an init container could mint a short-lived
token on every pod start — a strictly worse secret to hold, for a
LAN-only service whose token is revocable by rotating the account.
Also corrects the obtained_at comment on the renderer, which justified
stamping "now" by repeating that same false permanence assumption.
Stamping now is safe ONLY because of this setting; against the 30-minute
default it is actively harmful, because the app would believe a stale
token had a fresh window, skip its proactive refresh, and strand itself.
The comment now says so, and says what to do if this setting is reverted.
The period is fixed at mint time, so this does not revive the existing
tokens — they must be re-minted after it deploys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the HOME=/tmp fix in this PR, from review.
Pointing HOME at /tmp was correct but left it on the container's
writable layer, which was harmless while Chromium crashed on launch and
wrote nothing. With the browser actually starting once a minute for the
life of the pod, crashpad state and profile data now accumulate there
with nothing to bound them — resources.limits sets memory only, so
there is no ephemeral-storage ceiling to catch it either.
Backs it with an emptyDir at a 512Mi sizeLimit, matching the /tmp mount
convention already used in pve-exporter-deployment.yaml. The point of
the limit is the failure mode, not the size: unbounded it degrades into
node disk pressure, which surfaces far from this pod and hurts whatever
else lands on that node; bounded it is an eviction of the renderer,
attributable on sight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two runtime faults from the first deploy. Neither is visible statically
— both needed the pod running to find.
1. The browser never launched. Every render died with
chrome_crashpad_handler: --database is required
Chromium cannot start without a writable HOME: crashpad builds its
database path from it. Diagnosed on the pod rather than guessed —
HOME=/, uid=1000, / is root-owned 0755 and not writable, /tmp is.
The cause is OUR securityContext, not the image: forcing
runAsUser 1000 with no matching passwd entry makes HOME fall back
to "/". Sets HOME=/tmp; crash dumps are throwaway, so they do not
belong on the PVC.
Worth being precise about, because review had flagged the dropped
capabilities as the likely Chromium risk and the comment I added
said they were fine. The capabilities were fine. Something adjacent
to that securityContext broke it anyway, and the comment now says
to check HOME before loosening the capability block.
2. Even once the browser works, nothing would have been pushed. The
scheduler logged
BYOS tokens ... expired beyond the refresh window
and skipped the webhook entirely. It derives token age from
obtained_at, and the template shipped a literal 0, so
freshly-minted tokens read as dating from 1970. The init container
now stamps it with jq's `now * 1000 | floor` at write time.
Verified the filter produces a 13-digit epoch-ms value. Stock
Terminus disables session expiry and issues effectively permanent
tokens, so a write-time stamp is honest enough.
Failure mode of the pair was quiet in the way this stack keeps being
quiet: pod Running and Ready, /health passing, schedule firing once a
minute, zero screens reaching Terminus.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were verified against the upstream image while writing this, but
the evidence never made it into the manifest, so review flagged them as
unverified assumptions. Fair — the reasoning was invisible.
/health is not a guess: the image's own Dockerfile declares
`HEALTHCHECK ... curl -f http://localhost:10000/health`, so the route is
part of its published contract. That is the difference from
terminus.yaml, which got TCP-only probes precisely because Terminus
publishes no health endpoint.
Dropping ALL capabilities is safe despite the container running headless
Chromium because the app launches it with --no-sandbox
--disable-setuid-sandbox --no-zygote --disable-dev-shm-usage, baked into
its own PUPPETEER_ARGS. That last flag is also why no /dev/shm emptyDir
is mounted. Both hold only for the pinned tag, so the comment says to
re-check on a bump.
No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review caught a blocking bug, correctly. HOME_ASSISTANT_URL pointed at
http://ha.renner.uno:8123, which is unusable from inside a pod.
That hostname is served by the home-assistant-bgp LoadBalancer with
externalTrafficPolicy: Local, advertised to the LAN over BGP. It is
not backed by an Ingress, so CoreDNS's k8s_gateway delegation does not
intercept it — a pod resolving it gets the real external address
(10.2.99.87) and hairpins off the LB.
This cluster has hit precisely this before with precisely this
Service. kubernetes/infrastructure/openbao/rotation/home-assistant.yaml
reaches HA from a pod for token rotation and says so outright: "The
public name is unusable from a pod (ingress-LB hairpin)." The same
language recurs in the gitea and woodpecker network policies. I took
the hostname from the repo's own "internal http://ha.renner.uno:8123"
note, which is true for a workstation and false from a pod.
Now uses home-assistant-bgp.home-assistant.svc.cluster.local:8123,
verified to exist with ClusterIP 10.96.79.213 on 8123/TCP.
Fixing the URL alone was not enough. The home-assistant namespace's
default-deny-ingress had no rule admitting trmnl pods on 8123 — the
only 8123 rule is scoped to the openbao secret-rotator — so traffic
would still have been dropped. Adds a companion ingress rule modelled
on the rotator's, scoped to the renderer's pod label rather than the
whole trmnl namespace, since Terminus shares that namespace and has no
business reaching Home Assistant. Only the ingress half is needed:
trmnl has no default-deny egress selecting the renderer, unlike
openbao.
Worth naming the failure mode, because it is the fourth "fails
quietly" case in this stack: nothing crashes. The schedule fires every
minute, each render errors on connect, and the panel simply never
updates.
Also updates the trmnl-daily Velero comment, which enumerated PVCs and
so undersold itself once trmnl-renderer-data existed — the schedule
selects the namespace minus CNPG-labelled PVCs, so new PVCs are
already covered without editing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves the one field flagged as unverified on the renderer PR, now
that a Terminus API token exists to check it with.
GET /api/screens on the running instance returns model_id as a NUMBER
(an existing welcome screen carries model_id 11), and GET /api/models
gives TRMNL X as id 1, name "v2", 1872x1404, 16 colours. Upstream's
example value "og_1bit" matches no model on this server at all — the
2-bit OG model here is id 11, name "og_plus" — so that example was
illustrative rather than a real identifier.
Would have surfaced as Terminus rejecting every pushed screen with a
422, i.e. a permanently blank panel with the renderer looking healthy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 4. Screenshots the Home Assistant e-ink dashboard, dithers it to
16 greys and hands it to Terminus, which the panel then polls.
Upstream ships this as a Home Assistant add-on. Ours is a plain HA
container with no Supervisor, so it runs standalone — viable because
the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env vars over
/data/options.json and authenticates with an ordinary long-lived
token. SUPERVISOR_TOKEN is deliberately left unset: it only flips an
internal isAddOn flag and would switch the app to add-on behaviour.
The non-obvious part is the direction of travel. With the byos-hanami
webhook in `uri` delivery mode the renderer does NOT push image bytes
— it POSTs a URL pointing at its own /output/<file> endpoint and
TERMINUS FETCHES THE IMAGE BACK. That is why this lives in the trmnl
namespace: the fetch is intra-namespace and already permitted by
default-deny-ingress. Splitting the two across namespaces would have
needed a new policy for a dependency that is easy to miss.
Fully declarative, no clicks. Schedules are a JSON file at
/data/schedules.json which the app re-reads every 60s, so the whole
schedule — viewport 1872x1404, floyd-steinberg at bitDepth 4, the
trmnl-eink theme, the dashboard path — is reviewable in git rather
than typed into the app's web UI. An init container splices the BYOS
tokens into the ConfigMap template with jq; jq rather than sed because
the tokens are JWTs going into a JSON string and sed would corrupt on
any character it treats as special (verified with a token containing
/ + &).
No HTTP in the init container on purpose. The app exposes a
/api/byos/login proxy that trades a Terminus username+password for
these tokens, but using it would mean shipping the Terminus PASSWORD
into the cluster and making every pod start depend on a network call.
Stock Terminus disables session expiry and issues effectively
permanent tokens, so they are minted once out of band and kept in
OpenBao; the password never enters the cluster.
Three things that are load-bearing and would fail quietly otherwise:
* The PVC at /data is REQUIRED. The app picks its data directory by
testing whether /data exists — with nothing mounted it silently
falls back to /app/data in the container, so schedules and rendered
output vanish on restart and the panel goes blank.
* The Service is ClusterIP with NO Ingress. The app's web UI on this
port has no authentication whatsoever, per upstream, so it must not
be routable from the LAN.
* The memory limit is 2Gi, not the 512Mi upstream's troubleshooting
note suggests. The app's own watchdog restarts it past 900MB but
only measures its OWN rss — Chromium is a separate process, and it
is rendering a 1872x1404 page.
The image is arch-suffixed (trmnl-ha-amd64); upstream publishes no
multi-arch manifest. Pinned and not enrolled in image automation,
since a bad bump here surfaces as a corrupted or blank panel rather
than a failed rollout — rationale recorded in the image-automation
README alongside Terminus's.
PREREQUISITE: homelab/apps/trmnl/renderer must be seeded in OpenBao
with HA_ACCESS_TOKEN, BYOS_ACCESS_TOKEN and BYOS_REFRESH_TOKEN before
this merges, or the ExternalSecret never syncs, the init container
blocks, and the trmnl Kustomization fails its wait.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things the first reconcile surfaced.
Cilium IPAM allocated 10.2.99.0 for the device LoadBalancer. Nothing
requested it — the Service carried no lbipam.cilium.io/ips annotation, so
IPAM handed out the first free address in the pool block, which is the base
address of the 10.2.99.0-10.2.99.255 range. Each LB address is advertised
to the UDM as its own BGP route, so it is routable in principle, but a .0
host address is exactly the sort of thing constrained embedded IP stacks
get wrong, and the TRMNL firmware is the one client that has to work here.
Requesting .89 explicitly also brings this in line with every other LB
Service in the repo, all of which pin their address, and matches the
fd99::89 IPAM had already chosen for v6.
Omitting the annotation was my call on the original PR, on the reasoning
that the device is provisioned with a hostname so the address could move
freely. That reasoning still holds; it just is not worth spending it on an
address nobody picked.
Second, the sidekiq container exits 1 once on a fresh deploy with
`PG::UndefinedTable: relation "account" does not exist` — it loses the race
against the web container's migration, then comes up healthy on the
restart. Observed on the first deploy, converged in about a second. There
is no upstream ordering hook, and gating it behind an initContainer that
probes for a named table would couple us to Terminus's schema, so this is
left to Kubernetes' restart deliberately. Documented in place so the next
person reading `RESTARTS 1` can tell the benign case from a real one.
Verified on the live cluster before this commit: Kustomization Ready, tr-pg
"Cluster in healthy state" on PostgreSQL 18, both ExternalSecrets
SecretSynced, terminus and valkey Running.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 1 of the TRMNL X dashboard plan: stand up the self-hosted Terminus
(BYOS) server the panel will poll. The Home Assistant dashboard, the e-ink
theme and the screenshot renderer are later phases and are not in here.
Phase 0 verification, all confirmed against upstream before writing any of
this:
* TRMNL X IS a first-class model in Terminus (id v2 / label "TRMNL X",
1872x1404, 16 colours, 4-bit) — the stop-and-report condition in the
plan does not trigger. Note it is NOT in config/db/seeds.rb: the model
list is populated by the MODEL_SYNCHRONIZER job pulling trmnl.com, so
the Models list is empty until that job has run once.
* The HA add-on can run standalone. Our HA is plain HA Core, not
Supervised, but the add-on reads env vars in preference to
/data/options.json and authenticates with an ordinary long-lived
token — SUPERVISOR_TOKEN only flips an isAddOn boolean. So the
fallback path in the plan (upstream container, not add-on) is viable.
That deployment lands with Phase 4.
Deviations from the repo's house patterns, each deliberate:
* tr-pg runs PostgreSQL 18, not the standard 16. Terminus ships a
committed config/db/structure.sql containing `SET transaction_timeout
= 0;`, a GUC that does not exist before PG 17, so schema load against
a 16 server fails outright. This is the first non-16 CNPG cluster
here. The existing 16 pin is inherited from 739a67e9 (Apr 2026) and
has no documented rationale.
* The device endpoint is cleartext HTTP on a Cilium BGP LoadBalancer
(trmnl-device.renner.uno) rather than an Ingress. The firmware cannot
be given the internal CA, and a TLS-less Ingress would be a new
pattern for this repo — a dedicated LB address is not. The admin UI
keeps a normal TLS Ingress at trmnl.renner.uno; both front the same
Service. Same split as home.renner.uno / ha.renner.uno.
* ghcr.io/usetrmnl/terminus is NOT enrolled in image automation despite
publishing a usable tag stream. It serves the firmware and
/api/display contract to a wall-mounted device, upstream is pre-1.0
with no releases or changelog, and an unattended bump that breaks
either leaves the panel dark. Bumps are manual and reviewed. The
Valkey sidecar is enrolled — it touches no device-facing contract.
* Egress is an FQDN allowlist (trmnl.com, plus the S3 host that serves
firmware binaries), not toEntities: world. The S3 host is returned by
the API at request time rather than being a source constant, so it can
drift; the policy comment says where to look when firmware sync
starts failing. This also means Terminus's Extensions/Poll feature,
which fetches admin-configured URLs, will not work — accepted, since
screen content is pushed in by the renderer.
* The CNPG pods are deliberately left OUT of that egress policy so the
barman WAL archive path is not behind a rule that fails closed.
Images verified to exist in their registries before pinning:
ghcr.io/usetrmnl/terminus:0.68.0, valkey/valkey:9.1-alpine,
ghcr.io/cloudnative-pg/postgresql:18.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prometheus and Loki run in Kubernetes -- statefulsets
prometheus-kube-prometheus-stack-prometheus and the Loki release, both up 93
days. The standalone VMs these plays targeted are gone: neither
prometheus.srv.renner.uno nor loki.srv.renner.uno resolves at all, and the
proxmox_logging group install-log-aggregation.yml targets is not in the
inventory.
So these rendered scrape configs, exporter installs and log shippers for
hosts that do not exist. Removed:
plays/install-prometheus.yml -> prometheus.srv.renner.uno (gone)
plays/install-blackbox-exporter.yml -> prometheus.srv.renner.uno (gone)
plays/install-loki.yml -> loki.srv.renner.uno (gone)
plays/install-log-aggregation.yml -> proxmox_logging (no such group)
roles/prometheus-server, snmp-exporter, pve-exporter, unifi-poller,
roles/blackbox-exporter, loki-server, promtail
Each role was consumed only by the plays above; promtail was referenced by
nothing at all. Verified no dangling references remain.
plays/update.yml imported install-log-aggregation.yml, so deleting it alone
would have broken the fleet update play. That import is removed and
update.yml re-syntax-checked.
This also retires #473, which changed IPv4 literals to FQDNs in
prometheus.yml.j2 and nodes.yml.j2. That was a change to a template nothing
renders. Its PR body described the instance-label consequences for
dashboards and alert rules, which was fiction -- no dashboard reads those
labels. When it would not apply I reported it as blocked on Proxmox API
credentials; the actual reason was that the target host had not existed for
months, and `--list-hosts` returning 0 was the evidence I misread.
Worth checking separately, not assumed here: whether anything scrapes the
PVE node_exporter and chrony_exporter today. install-node-exporter-pve.yml
still installs them on pve_cluster, but the config that pointed a Prometheus
at them was in the templates just deleted. If that was the only reference,
those metrics have been dark since the VM went away -- and the FQDN change
in #473 would have masked the question rather than raising it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image shipped without [unix_http_server], [rpcinterface:supervisor] or
[supervisorctl], so supervisorctl could not connect:
Error: .ini file does not include supervisorctl section
That means no way to query or restart an individual service inside the
container -- docker logs was the only introspection. Per-service lifecycle
management was the whole argument for running a supervisor instead of one
process per container, and the first revision did not actually deliver it.
Unix socket only, inside the container. Nothing outside needs it, and this
container runs with host networking, so a TCP listener would be exposed on
the host.
The build-time self-test now asserts the section exists and supervisorctl
runs, so this cannot regress silently the way it shipped.
NOT fixed here, because my diagnosis was wrong: the repeated
"mqtt broker disconnected. reason_code = Unspecified error" is not a client
ID collision. E3onCAN has no -mcid option at all -- verified against the
pinned source -- and generates its own ID; the native broker log showed
E3onCANclient.py-e380-7f847281 alongside open3e-edge, so the two never
collided. I nearly "fixed" it by passing a flag that does not exist.
Telemetry is flowing (each reconnect succeeds), so this is churn rather than
an outage, but the cause is still unidentified. The next step is the broker's
own log on edge-1 -- mosquitto records the disconnect reason, which the
client side reports only as "Unspecified error".
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The migration did not stop anything. It ran stop + disable + mask in one
call with failed_when: false, then deleted the unit files. Masking a unit
whose file still exists conflicts, so the call failed, failed_when hid the
failure, nothing was stopped -- and the next task deleted the unit files out
from under two running processes.
That left systemctl is-enabled reporting "not-found", which looks clean,
while is-active still said "active". I reported the cutover successful on
the strength of the former.
On 2026-08-10 the result was two open3e instances issuing UDS requests on
the same CAN bus from 19:28 to 19:37, taking over each other's
open3e-edge MQTT session -- the exact failure the block's own comment said
it existed to prevent.
Fixed in three parts:
- Stop and disable FIRST, and do not swallow the result. Only units that
actually exist are touched (stat first), so a fresh host does not need
an error hidden to get through.
- No masking. It writes a symlink at the same path the next task removes,
so the two cancel out. With the unit files gone there is nothing to
start.
- Assert the invariant that actually matters -- no native process on the
CAN bus -- rather than inferring it from systemd reporting success.
pgrep, then fail loudly with the surviving command lines and why it
matters.
Verified against the live host: ok=15 changed=1 failed=0, the assert passing
with "No native CAN processes remain; safe to start the container", and the
run now idempotent (the single change is the compose up, not the teardown).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#476 merged first, which removed edge_open3e_pkg, edge_e3oncan_repo and
edge_e3oncan_version in favour of the image. The audit entry added here still
told the agent to read those, so it would have gone looking for variables
that no longer exist and reported nothing -- the same silent non-auditing the
entry was written to prevent.
Repointed at where the pins actually live now: OPEN3E_REF and E3ONCAN_REF as
ARGs in ci/images/heating-can/Containerfile. The awkward specifics carry over
unchanged, since they are properties of the upstreams rather than of how we
package them: open3e publishes no tags so the comparison must be against
commits, and E3onCAN's pin is an untagged commit later than its newest tag.
Added a note the previous version could not have made: heating_can_image is
:latest, so the node follows whatever CI last built. The Containerfile ARGs
are what actually fix the versions, and bumping one is what requires a
rebuild -- otherwise a reader could reasonably conclude the tag is the pin.
The pin commit itself is now a no-op against master and survives only as the
audit registration; its conflict on heating-can/defaults resolved in favour of
the image.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both from review on #475.
The rule "a host in a capability group must also be in [edge_host]" was
enforced by a comment, in a file that already uses :children a few lines
below for ntp_cluster. The three capability groups are now children of
edge_host, so the relation is structural: a host added only to [heating_can]
still resolves group_vars/edge_host, where the local MQTT contract lives, and
cannot silently come up without edge_mqtt_host and friends.
That matters more than it looks. The MQTT contract only moved to group_vars
in this PR because splitting the plays exposed that the roles could not
render without each other's defaults -- so an edge host outside [edge_host]
would hit exactly the failure that change was made to fix.
Verified all four groups still resolve to edge-1, and the redundant
ansible_user on the capability entries is dropped: it is inherited from the
[edge_host] declaration rather than repeated three times.
Also unstaled a Day-2 note that still read "the pip task tracks the git
default branch (pin edge_open3e_pkg to a tag if that ever bites)". Both
services are pinned to commits now, deliberately, and pinning to a tag is not
available for open3e -- it publishes none. Replaced with the actual workflow:
the weekly update-audit reports drift, bump the pin and re-run.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
open3e was installed from an unpinned git URL, so the heating bridge
silently tracked upstream main: whatever HEAD happened to be on the day a
converge ran was what ended up driving the heat pump. Now pinned to
6553c104, which was simultaneously upstream HEAD and the copy already
installed on edge-1, so the pin froze the running state and changed nothing.
A commit rather than a tag because open3e publishes none -- its tags API
returns an empty array. The package self-reports 0.7.5 at that commit, which
is useful context but is not a tag.
Honest about what this does NOT fix: pip still re-clones and rebuilds a VCS
requirement even when the SHA is pinned, because it cannot compare an
installed VCS dist against a requested URL without fetching it first.
Verified directly on the host:
pip install --dry-run "git+...@6553c104"
-> Cloning ... Resolved ... to commit 6553c104
Installing build dependencies: started
So the task still reports changed and still restarts open3e on every
converge. Pinning buys deliberate versioning, not idempotence.
Pinning without an update path only converts churn into silent rot, so both
services are now registered with the weekly update-audit -- which had been
auditing neither, despite E3onCAN having been pinned since July. The audit
prompt warns that anything not listed is silently not audited, and these two
were exactly that.
The new scope entry records the awkward specifics so the agent does not
report normal states as problems: open3e has no tags at all, so it must
compare commits against the default branch; E3onCAN does tag, but sparsely,
and its pin sits between the newest tag and HEAD.
Also corrected while there: edge_e3oncan_version was commented "# V0.5.2",
which does not exist upstream. The repo's only tag is v0.5.1 (6f43b3fb), and
461dcbdf is a later untagged commit from 2026-06-21, "Fixed: MQTT client ID
collision caused stale retained values; added publish error logging".
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Splitting edge-host into single-concern roles left the coupling one layer
up: install-edge-host.yml was still one play applying all eleven roles to
one group. A host in [edge_host] got the Z-Wave gateway and the CAN services
whether or not it had the hardware -- the monolith again, in the play.
Inventory groups now state which hardware a host physically has, and each
playbook targets the group for its own concern:
[edge_host] install-edge-host.yml base: flash-wear, log
shipper, MQTT bridge, VLAN,
docker
[heating_can] install-heating-can.yml Innomaker USB2CAN
[zwave_gateway] install-zwave-gateway.yml Nabu Casa ZWA-2
[power_meter] install-power-meter.yml FTDI meter head
No flags and no `when:` on the roles. Moving a stick to another box is a
one-line inventory change, which is what it should be. A host with nothing
attached is simply absent from the capability groups -- which is what lets a
second thin client join as an NTP peer without being handed peripherals it
does not have.
The split exposed a circular dependency the single play had been hiding: the
local MQTT contract was scattered across two roles' defaults, in both
directions. mqtt-bridge's config needs edge_open3e_mqtt_user, because the
bridge authenticates to the cluster broker AS that user; heating-can and
power-meter-sml need edge_mqtt_host, because that is where they publish.
Neither role could render without the other's defaults, and it only ever
worked because one play loaded every role at once.
Those four values are the interface between the broker and its clients, not
the property of either side, so they move to group_vars/edge_host -- where
the matching secrets (edge_open3e_mqtt_password, edge_sml2mqtt_mqtt_password)
already lived. Removed from the role defaults so there is one source of
truth. Each role keeps what is genuinely its own: mqtt-bridge the cluster
endpoint and CA path, heating-can the datapoints and poll interval.
Also dropped: the legacy shared-compose teardown pre_tasks. That migration
completed on the 2026-08-08 apply (verified: /opt/edge-host/docker-compose.yml
is gone), and carrying migration cruft in a base play would be the same
mixing this commit is undoing.
Verified by --check --diff against the live host, all four plays:
install-edge-host ok=33 changed=0 failed=0
install-zwave-gateway ok=8 changed=0 failed=0
install-power-meter ok=4 changed=0 failed=0
install-heating-can ok=18 changed=4 failed=0
Three of four are fully idempotent. heating-can's four are the known open3e
pip churn plus the E3onCAN venv, which reports changed only under check mode
(it was ok in the real apply) -- and their two restart handlers.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #476 found a real regression plus three things worth fixing before
this reaches edge-1.
Blocking: edge_open3e_devices.vx3.dpList still pointed at the old host path
/opt/edge-host/open3e/vx3plus.py. The compose file mounts that overlay at
/config/vx3plus.py and nothing mounts anything under /opt/edge-host inside
the container, so open3e would have looked for it at a path that does not
exist -- silently falling back to the stock vx3 datapoint list and dropping
the battery-module DIDs (vx3.1577-1582 and friends) the overlay exists to
add. Now /config/vx3plus.py.
Worth noting the build-time self-test could not have caught this: it proves
the binaries execute, not that devices.json resolves against the mounted
files at runtime.
Writable working directory. WORKDIR was /opt, which the COPY layers create
root-owned, while the process runs as open3e. udsoncan writes its log file
into the CWD -- the native install gave open3e's directory `owner: open3e`
for exactly that reason, and I dropped that when containerising. Now a
dedicated /var/lib/open3e, owned by the service user.
Secret mode. MQTT_PASSWORD was inline in docker-compose.yml, which is
rendered 0644. The env file this replaced was 0600, and the sibling roles on
this host keep secrets in a 0600 file beside a world-readable compose file.
Restored: credentials move to heating-can.env at 0600, referenced via
env_file.
edge_e3oncan_mqtt_format had become dead -- both services were fed
MQTT_FORMAT from edge_open3e_mqtt_format. The two default to the same string
so behaviour was unchanged, but overriding it did nothing. Wired through as
E3ONCAN_MQTT_FORMAT, which supervisord passes to e3oncan only.
Also from review: dropped the registered-but-unread heating_can_native_stopped,
and the migration now removes the emptied {{ edge_base_dir }}/open3e and
/e3oncan directories rather than leaving them behind.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The smoke-test step I added errored the whole pipeline before any step ran.
It used the freshly built image as its own step image, but the Gitea registry
is private: the build steps authenticate kaniko for PUSH, nothing gives the
runner credentials to PULL. So the step could never have worked.
Moved into the Containerfile as a build-time RUN, which is better anyway --
it needs no registry credentials, and a broken image is never published
rather than published and then flagged.
It asserts what the previous revision got wrong: both services actually
execute, and supervisord's config carries both programs. That is exactly the
failure that shipped green -- supervisord pointed at
/opt/e3oncan/src/E3onCANcollect.py, which does not exist (the file is at the
repo root), so the image would have run open3e while e3oncan crash-looped
and the E380 meter went silently missing.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image shipped a path that does not exist. supervisord ran
`python /opt/e3oncan/src/E3onCANcollect.py`, but that file is at the repo
ROOT -- there is no src/ directory. The `src` came from the old Ansible task,
which cloned into `{{ edge_base_dir }}/e3oncan/src`; that was the clone
DESTINATION, not part of the upstream layout, and I carried it into the
Containerfile without checking. Confirmed against the upstream tree at the
pinned commit: E3onCANcollect.py sits at the root alongside requirements.txt,
which is why the build succeeded while the runtime path was wrong.
So the image built green and would have started open3e and failed e3oncan on
every restart -- the E380 meter silently absent, with supervisord retrying
forever.
Nothing caught that, because nothing ran the image. Now CI does: a smoke-test
step uses the freshly built image as its own step image and asserts both
services execute and supervisord's config carries both programs.
It cannot exercise the CAN bus or the broker, neither of which exists on a
runner, so it verifies exactly what is verifiable without hardware. That is
still the difference between "built" and "works" -- and it belongs on a
throwaway runner, not on the production edge host, which is where I wrongly
went looking for this the first time.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts my own reinterpretation. The instruction was to put both in one
container because they belong together; I argued against it on crash-isolation
grounds, then read the correction "two services, one image" as agreement with
my version rather than as a restatement of the original. It was not.
A container runs one PID 1. With two processes that PID 1 is a supervisor,
which is how a container handles multiple services. supervisord now runs both
readers: it restarts either on its own if it dies, reaps children and forwards
shutdown -- none of which a bare process as PID 1 can do for a sibling, which
was the actual answer to the concern I raised.
Also gone: the dispatch entrypoint and the argv in the compose file. Both were
the same mistake in different places -- the invocation living outside the
artifact. open3e's and E3onCAN's command lines are an implementation detail of
this image now; supervisord builds them from environment variables it expands
itself. Nothing outside the image knows either CLI, so an upstream flag rename
cannot break the compose file or an Ansible template.
The compose file supplies configuration and nothing else: one service, one
container, environment only. The credential env file is dropped -- the
credentials are environment already.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing is installed on the host any more. The image is built in CI with
both sources pinned, published to the registry, and edge-1 pulls and runs
it. That is the whole point of building it -- installation happens at build
time, once, not on every converge.
Removed from the node: both venvs, the E3onCAN checkout, and the two systemd
units. Kept and still rendered: devices.json, the vx3 datapoint overlay and
the broker credentials, now bind-mounted into the containers rather than
read from a venv working directory.
Two services from one image, as before: open3e drives the Vitocal/VX3 over
UDS, e3oncan listens passively for the E380 meter. Separate containers so
either can crash and restart without a supervisor as PID 1.
The migration stops AND masks the old units before the containers start.
Masked rather than disabled because two open3e instances issuing UDS
requests on one CAN bus is the failure mode to avoid, and a stray
`systemctl start` or leftover dependency should not be able to bring the
native one back alongside the container.
No missing-device tolerance here, unlike the zwave and meter roles: can0 is
this host's purpose. If it is absent the heating telemetry is down and the
run should say so rather than pass quietly.
heating_can_image replaces edge_open3e_pkg / edge_e3oncan_repo /
edge_e3oncan_version. The source pins move into the Containerfile, so what
is deployed here is a tag -- and `docker compose up -d` on an unchanged tag
is a genuine no-op, which is what removes the restart-per-converge that
pinning the pip URL could not.
Verification is limited and worth stating: plays/install-heating-can.yml
lives in #475 and is not on this branch, so only the master-side play
syntax-checks here. A --check run cannot exercise this at all -- command
tasks are skipped under check mode, so neither the pull nor the compose up
would run. It cannot be meaningfully verified until the image has been built
and #475 is merged.
Depends on #475 (the capability-split playbooks, and the MQTT contract
moving to group_vars). This branch is cut from master, so heating-can's
defaults will conflict there; #475 should merge first and this be rebased
onto it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image had no ENTRYPOINT and no CMD, with a comment claiming that was
deliberate because the commands are rendered by Ansible. That was
rationalising a defect.
An image with nothing runnable is not a self-contained artifact. The
knowledge of how to START each service leaked into Ansible, and the two were
not even symmetric: open3e installs a console script via pip, while E3onCAN
is an unpackaged script in a checkout, so the compose file would have had to
carry `python /opt/e3oncan/src/E3onCANcollect.py` -- an implementation detail
of this image's internal layout, in a file that has no business knowing it.
Move the checkout and every caller breaks.
E3onCAN now gets a wrapper on PATH, so both services are invoked the same
way and `docker run <image> e3oncan --help` works.
CMD prints what the image provides and exits 64 (EX_USAGE) rather than
silently doing nothing, so `docker run <image>` is self-describing. Which
service to run, and its arguments, stays the caller's choice -- the compose
file renders those, because the datapoint list and broker settings live
there.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds one image containing open3e (Vitocal heat pump, Vitocharge VX3) and
E3onCAN (E380 grid energy meter), both pinned in the Containerfile.
Why: both were pip-installed into venvs on edge-1, open3e straight from an
unpinned git URL. pip cannot compare an installed VCS dist against a
requested one without fetching it, so `pip install` re-cloned and rebuilt on
every Ansible converge -- reporting changed, firing the restart handler, and
putting a gap in heating telemetry each time. Pinning the SHA does not fix
that; verified directly on the host that pip still clones and rebuilds a
pinned VCS requirement. A tagged image does: `docker compose up -d` on an
unchanged tag is a genuine no-op.
It also gives the weekly update-audit a normal artifact instead of two
awkward special cases. open3e publishes no tags at all, and E3onCAN is
pinned to an untagged commit sitting between its newest tag and HEAD -- both
of which the audit currently has to be told about in prose.
One image, two services. They read the same physical installation over the
same bus and are versioned together, so they are one artifact; but they will
run as two containers from it, so either can crash and restart on its own
without a supervisor as PID 1.
Fixes a long-standing blind spot in passing: e3oncan.service set
PYTHONUNBUFFERED=1 and open3e.service did not, so open3e's stdout was
block-buffered and effectively never reached the journal or Loki. The image
sets it for both.
Build follows the existing kaniko pattern exactly -- path-scoped `when:` so
only this image rebuilds, date tag plus latest, pushed to the Gitea registry
alongside the ci-* images.
Deliberately artifact-only. The heating-can role still deploys venvs and
systemd units; moving it to a compose project is a separate change, once
this image exists and has been pulled and run once. Rewriting the role that
drives the heat pump against an image that has never been built would be the
wrong order.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #474 caught that the previous commit put the daemon.json task in
docker-install, which is shared by ten plays: install-netbird-server,
install-netbird-proxy, install-n8n, install-authentik-ha, install-owncloud,
install-gatus, install-ical-proxy, install-grist and
install-music-assistant.
None of those hosts has an /etc/docker/daemon.json today -- the template and
the default were both new here -- so the next routine run of any of those
plays would have created one, reported changed, fired the Restart docker
handler, and bounced every container on that host. Unannounced restarts
across authentik, n8n, owncloud, grist, netbird, gatus, music-assistant and
ical-proxy, from a change about edge-host bootstrapping.
The --check verification could not have caught it: it only ever exercised
install-edge-host.yml, and the damage was in the other nine plays.
The comment I wrote there was the tell. It said "the services that use this
host run with network_mode: host" -- true of edge-1, false of most consumers
of that role. A justification that only held for the one host I was looking
at, written into a shared file.
Moved to edge-flash-wear, which is edge-host-only and already owns this
rationale: routing container stdout to the journal is flash-wear handling,
not generic docker setup. The networking aside is dropped -- in a flash-wear
role the only thing worth saying about daemon.json is why logging goes to
the journal.
docker-install is now byte-identical to master (git diff origin/master --
roles/docker-install/ is empty).
Two details the move needed: /etc/docker is created explicitly, because
edge-flash-wear runs before docker-install on a first bootstrap and the
directory will not exist yet; and the Restart docker handler fires at
end-of-play, by which time docker is installed.
Verified: ok=67 changed=25. The changed count is unchanged from before the
move, and no docker task appears in it -- daemon.json renders byte-identical
to what the old role produced, so it reports ok and the handler never fires
on edge-1.
Also from review: noted on the play that heating-can, zwave-gateway and
power-meter-sml each declare edge_base_dir in their own defaults so they
stay usable standalone, that play vars win, and that the two must be kept in
step.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #474 found two blocking defects, both in exactly the place the
--check verification is blind: command and shell tasks are skipped, not
simulated, under check mode. So "verified with --check --diff" covered
everything except the two things that would actually have broken.
1. Ordering: a real run would have aborted.
The legacy-project teardown lived in power-meter-sml, which the play runs
last. zwave-gateway runs before it and does `docker compose up -d` for a
container named zwave-js -- while the old shared project still owns that
name. Docker enforces container names globally, not per project, so the
command fails with "Conflict: container name already in use". That is not
the missing-USB-device string the task tolerates, so the task fails and
the play aborts before ever reaching the cleanup.
Moved to pre_tasks, which is its correct home anyway: it is a migration
step for both services, and putting it inside either one meant whichever
ran first hit the conflict.
The play now also declares edge_base_dir and edge_legacy_compose in its
own vars rather than borrowing them from a role's defaults, since
pre_tasks must not depend on role default resolution. And it removes the
now-orphaned zwavejs.env left at the old shared project root.
2. Dangling handler.
The libeatmydata task still notified "Restart edge compose services", a
handler that was deleted with the old role. Masked today only because
`creates:` makes the task report ok -- but the role's own comment says to
delete that .so when the Alpine base bumps, and the next time it does,
Ansible would fatally error with "handler not found". Now notifies
"Restart zwave-js".
Swept every notify in the new roles against every defined handler; that
was the only unresolved one.
Also from review: kubernetes/bootstrap.sh referenced the deleted role.
Repointed to ansible/docs/edge-host.md.
Re-verified: ok=66 changed=25. The teardown moved from the power-meter-sml
group into pre_tasks and gained the orphaned-env removal, so the count goes
5 + 5 + 14 -> 5 pre-existing + 6 migration + 14 header repointing. Nothing
else moved.
The limit is worth stating plainly: this re-run still cannot prove the
teardown works, for the same reason it missed the bug. The first real apply
is the test.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edge-host was a single role doing eight unrelated jobs: flash-wear
mitigation, log shipping, VLAN networking, an MQTT bridge, a CAN interface,
the heating CAN services, a Z-Wave gateway and a power-meter reader.
That is not just untidy. It is what blocked a second thin client from
joining as an NTP peer: the host-generic work (journald to RAM, Alloy, the
VLAN sub-interface) could not be applied without also handing the box a
Z-Wave radio and a CAN bus it does not have. The role bound its services to
specific USB device paths, so a second host would have tried to start
containers for hardware that is not there.
Now:
edge-flash-wear journald -> RAM, noatime/lazytime, TRIM, eMMC health
edge-log-shipper Alloy: journal -> Loki
edge-vlan tagged VLAN sub-interface
mqtt-bridge loopback mosquitto -> cluster broker over TLS
socketcan can0 only
heating-can open3e (Vitocal/VX3) + e3oncan (E380 meter)
zwave-gateway zwave-js-ui, own compose project
power-meter-sml sml2mqtt, own compose project
The E380 reader lives with open3e rather than in its own role: it is part
of the same heating installation -- it feeds the heat pump's own energy
management -- not a separate system that happens to share a bus. can0 does
stay separate, because an interface is not the same concern as the services
consuming it.
Each role installs its own packages, so it stands alone. An edge host with
no peripherals takes the first five and none of the rest.
docker-daemon.json moves to docker-install with its own docker_log_driver
variable and Restart docker handler, since configuring the docker daemon was
never an edge-host concern. Handlers split ten ways; defaults split per
role; all sixteen templates and files moved with git mv.
Two deliberate behaviour changes on the node, both accepted:
1. Compose split. zwave-js and sml2mqtt shared a compose file, so touching
either recreated both. They are now separate projects in their own
directories. container_name is fixed, so the old project must release the
names first -- a guarded one-time task downs it and removes the file.
First apply costs a brief down/up on both. Data paths are unchanged; the
zwave store is not migrated.
2. Header repointing. Templates and scripts carried "Managed by
roles/edge-host" headers, which render onto the host -- they would have
sent an operator to a role that no longer exists. Updating them changes
file content, so it restarts the services that consume those files.
Verified by --check --diff against the live host. 24 changed items, and
every one is accounted for:
5 pre-existing, unchanged from the baseline before this work:
common timesyncd; open3e venv; E3onCAN venv; and the open3e /
e3oncan restart handlers. The venv tasks install from a git URL,
which pip cannot verify, so they report changed on every run --
a separate defect, not introduced here.
5 the compose split: zwave env + compose and sml compose to their
new paths, plus the two legacy-project retirement tasks.
14 the header repointing: journald conf, eMMC health script, alloy
config, mosquitto bridge config, 80-can0.network, open3e overlay /
env / unit, e3oncan unit, zwave backup script + env, sml2mqtt
config -- and the resulting restarts of journald, alloy, mosquitto,
open3e and e3oncan.
Nothing changed outside those three groups.
One consequence check mode cannot show: 80-can0.network.j2 changed, so the
"Reload networkd" handler fires on a real run and reconfigures can0,
briefly dropping the bus under open3e and e3oncan. It is absent from the
check-mode output only because shell handlers are skipped there. The VLAN
templates did NOT change, so the riskier reconfigure of the parent uplink
on this headless host is not triggered.
Also fixed in passing: the keepalived-ntp guard told operators to recover
with `--limit <host> --tags vlan40`, and referred to the edge-host role.
Both corrected. The role README becomes ansible/docs/edge-host.md -- it
documents the host, not any single role now.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #474 caught that the previous commit fixed the procedure in
ansible/roles/edge-host/README.md and left the same two defects in the two
places an operator actually reads them first.
That happened because my grep for the stale patterns ran with the working
directory set to ansible/, so `images/edge-host-iso/README.md` was a path
that did not exist. grep returned nothing and I read that as clean. A false
negative from a bad path, not a clean sweep.
Fixed now, verified from the repository root:
airootfs/usr/local/bin/edge-install
Printed the superseded two-command sequence at the console immediately
after imaging -- the very first instructions anyone sees, telling them
to run common.yml with -l edge_host. Now prints the single unrestricted
run, plus the two steps that precede it (confirm the host resolves by
name, add it to [edge_host]).
README.md
Described the image as being for `edge-1.srv.renner.uno`, a name that
does not resolve, and repeated the same two-command bootstrap. Both
replaced. This file is the link target of step 2 in the ansible-side
README, so following the procedure end to end landed straight back on
the defect #474 claims to close.
Also corrected while there: the ISO README called the target "the bare-metal
N100". edge-1 is a Wyse 3040 (Atom x5-Z8350), and the installer text further
down already says it "covers the Wyse 3040". Rewritten to describe the image
as not tied to one box or model, since the installer autodetects the target
disk -- which is what makes it usable for the second thin client.
Also from review: a comment on why `become: true` is set when the connection
is already root. Not taken -- ignore_unreachable, which common.yml sets. The
review's own reasoning applies: Ansible already skips an unreachable host and
continues with the others, so it adds nothing here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The documented base-install procedure had three defects, two of which are
standing house rules and one of which simply does not work.
1. It required a DHCP reservation.
"Add the DHCP reservation/DNS entry for <host> on the UDM" was step 3 --
a reservation as a documented precondition. Reservations are off the
table here. The UDM registers DHCP leases in DNS on its own, so the
record appears without one, and the host is identified by name
everywhere: inventory, plays, README. Its address is never recorded.
2. It required --limit, because the plays were split wrong.
Step 4 was:
ansible-playbook plays/common.yml -l edge_host
ansible-playbook plays/install-edge-host.yml
The --limit was load-bearing, not sloppiness: common.yml is
`hosts: all:!...`, so running it unrestricted while bootstrapping a
single new box would reconfigure the entire fleet. The split left no
other option. But --limit narrows the target list rather than staging
anything, and leaves every other host outside convergence with no report
that they were skipped.
Fixed structurally rather than by deleting the flag: install-edge-host.yml
now carries `certificates` and `common` itself -- the parts of common.yml
that apply to an edge host -- so the bootstrap is one unrestricted run
over the edge_host group. No second command, no limit.
Not carried over: qemu-guest-agent (these are bare metal, not VMs) and
monitoring-agent (Debian-oriented; edge hosts ship logs via edge-alloy).
common.yml's loki/prometheus vars go with monitoring-agent and are not
needed -- verified that the common role does not reference them.
3. It named a host that does not exist.
The README said `edge-1.srv.renner.uno` in both the base-install steps
and the Z-Wave cutover checklist. The host is `edge-1.iot.renner.uno`;
the .srv name does not resolve, so anyone following the runbook hits a
dead name at step 3.
Verified idempotent against the live host before committing:
`ansible-playbook plays/install-edge-host.yml --check --diff` over the whole
edge_host group -> ok=53 changed=5 failed=0, with all five changes inside
the edge-host role (open3e venv, E3onCAN checkout/venv), which was already
in this play. certificates and common contributed no changes at all.
Also checked the common role branches only on Debian/RedHat os_family, so
its OS-specific tasks skip cleanly on Arch.
Prompted by preparing a second Wyse 3040 as an NTP peer -- this is the
procedure that would have been followed to build it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to #472, which changed the inventory but left the same three IPv4
literals in the monitoring config and a runbook. Raised in review there.
prometheus.yml.j2:
- pve job targets -> FQDNs
- chrony job targets -> FQDNs, and restructured to one static_config per
host so each carries its own hostname label
The chrony restructure is the substantive part. The previous form listed all
three targets together and recovered the hostname in relabel_configs by
regex-matching __address__ against each node's IP:
- source_labels: [__address__]
regex: '10\.2\.40\.201:.*'
target_label: hostname
replacement: 'jupiter'
That silently coupled the labels to the addressing. Swapping a target for a
name would have left every regex unmatched and the hostname label simply
absent, with the scrape still succeeding and nothing failing -- a
find-and-replace on the addresses alone would have broken the labels
quietly. Per-host static_configs remove the coupling entirely.
nodes.yml.j2: targets -> FQDNs. Already one entry per host with an explicit
hostname label, so nothing else to change.
claude-workstation/RUNBOOK.md: the kubeconfig command pointed --server at
https://10.2.40.201:6443. That is jupiter, a Proxmox host, not a control
plane node -- stale from when k8s ran as VMs on it. The cluster's endpoint
is https://k8s.renner.uno:6443 (kubernetes/talos/patches/cluster.yaml), and
the name is also what the API server certificate carries as a SAN, so an
address there fails TLS verification even when reachable. The documented
command was very likely broken as written.
Deliberately NOT changed: kubernetes/apps/pve/endpointslice.yaml. An
EndpointSlice takes literal addresses; hostnames are not valid in that
field, so those three are correct as they are.
Note this changes the `instance` label on the pve job from an address to a
name, since the relabel chain feeds __address__ into __param_target and on
to instance. Anything selecting on the old value needs updating.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter, neptune and saturn were the last entries in pve_hosts.yml still
reached via ansible_host=<IPv4 literal>. Every other host here --
authentik, netbird, proxy, nas-1, grafana, edge-1 -- has always used its
FQDN, so these three were the outlier rather than the convention.
An address in an inventory is allocator state standing in for identity, and
it pins access to the legacy family: the names resolve IPv6-first, which is
the primary path on this network. DNS is authoritative for where a host is;
the inventory should say which host.
The short inventory_hostname is kept deliberately -- plays key on it
(ntp_servers[inventory_hostname] in install-ntp-ha.yml, node names in
opentofu, the prometheus targets) -- so only ansible_host changes.
No claim that this fixes the connection stalls seen while applying the NTP
work on 2026-08-07. Those were the IPS reacting to the connection pattern I
was generating (a port sweep, then rapid repeated SSH across hosts
interleaved with Ansible's own churn), not an address-family problem. This
change stands on convention and on not encoding a lease as identity.
claude-ws is left alone: its comment documents that obg.renner.uno is
split-horizon and resolves to the public proxy from home. That explains why
that particular name is unusable, not why a literal is right -- a name that
resolves to the wrong target calls for one that resolves correctly. Out of
scope here, and I do not know whether such a record exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chrony_allowed_networks contained only 10.2.0.0/16, so the rendered
chrony.conf permitted IPv4 clients and nothing else. IPv6 is the primary
address family here: ntp.srv.renner.uno resolves to fd40::123, so an
IPv4-only allow list refuses every client that uses the documented name.
This was live but invisible. The hosts carried `allow fd00::/8` in
chrony.conf for a long time without it ever being in this repository.
Applying the role on 2026-08-07 rendered the template over the live config
and dropped the line on all four NTP servers at once, taking internal NTP
down over IPv6 until it was added here.
Same class of untracked host state as the anycast service address, and the
same failure: config that exists only on the box survives right up until
something re-renders it. Reading the drifted file is not enough -- the rest
of it has to be diffed against what the template would produce.
The outage was not caught by the play, which reported the v6 master as
"serving ntp.srv.renner.uno" throughout. It was caught by querying the
service:
chronyd -Q "server ntp.srv.renner.uno iburst"
-> No suitable source for synchronisation
and after the fix:
chronyd -Q "server ntp.srv.renner.uno iburst"
-> System clock wrong by 0.001999 seconds (ignored)
with a control query against ptbtime1.ptb.de to confirm the method rather
than the target was sound.
fe80::/10 is included as well: a client on the same segment may source its
query from a link-local address, and without it chrony drops the request,
which presents as the server being unreachable rather than as a policy
refusal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applying #469 to edge-1 exposed two defects that only show up on real
hardware.
1. The VRRP address was unusable.
keepalived took MASTER and assigned fd40::123, but the kernel left it:
inet6 fd40::123/128 scope global dadfailed deprecated tentative
proto keepalived
valid_lft forever preferred_lft 0sec
`-nodad` did not take effect -- no nodad flag on the address, and
net.ipv6.conf.enp1s0/40.accept_dad = 1 -- so DAD ran and found a real
duplicate: jupiter, saturn and neptune still held the address statically.
edge-1 believed it was the v6 master while serving nothing.
The flag is dropped rather than fixed. It was inherited from the previous
design, where every NTP server carried the address at once and duplicate
detection had to be suppressed for that to work. Under single-master VRRP
the opposite holds: exactly one host should have it, so a duplicate is a
genuine fault and detecting it is the point.
What that costs is that the migration can no longer be done host by host.
A static copy anywhere and a VRRP copy elsewhere cannot coexist, and the
dadfailed state is sticky -- the kernel does not retry. install-ntp-ha.yml
is therefore split in two: a first play withdraws every static assignment
across the whole group, then the second hands the address to keepalived.
Deliberately not serial, since it must finish everywhere before any host
claims the address. There is a gap between the phases where nobody holds
it, bounded by one play's run time; clients retry.
The withdrawal tasks move to roles/keepalived-ntp/tasks/withdraw-static.yml
for the first play to include. main.yml keeps the detect-and-assert guards,
which still run every time and fail loudly if any static assignment
survives.
2. keepalived warned:
(NTP_VIP6) the first IPv6 VIP address should be link local
RFC 5798 gives an IPv6 virtual router a link-local primary. keepalived
carried on regardless, but the instance was sourcing VRRPv3 from something
the protocol does not expect. fe80::123 is now the first
virtual_ipaddress: it mirrors the service address and cannot collide with
a NIC's EUI-64 link-local.
ntp_vip6 and ntp_vip6_linklocal move to group_vars/ntp_cluster.yml so both
plays read one definition rather than duplicating a service address.
Verified: `keepalived -t` on edge-1 (keepalived 2.4.1) parses the two-address
form clean, and the playbook passes --syntax-check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #469 found the Debian path still fails open, and was right that
my previous commit message was wrong to claim otherwise.
The whole Debian chain -- detect, withdraw, confirm, assert -- was gated on
`keepalived_ntp_static6_removed.changed`, which is only true when the
lineinfile regex matches exactly, prefix length included. The hook it
targets is hand-written and was never in version control, so any deviation
(a different mask, unusual whitespace, a second assignment elsewhere)
leaves .changed false and silently skips every following task. The static
assignment then coexists with the VRRP-managed one, which is the
double-assignment failure this PR exists to prevent -- just relocated from
edge-1 to a PVE host.
Same fail-open class as the Arch grep fixed in 5e01e2f1. The argument in
that commit message -- that the Debian path was safe because
`ip -6 addr show ... to <prefix>` matches by range -- is true of the runtime
checks but says nothing about the lineinfile text match, which is a
different mechanism and was exactly as fragile as the pre-fix Arch grep.
Fixed by checking the CONFIG FILE unconditionally, prefix-agnostic and
\b-anchored, mirroring the networkd check.
Note this is deliberately not the fix the review proposed. It suggested
running the runtime `ip -6 addr show ... to <addr>` check and its assert
unconditionally. That would break the PR: after migration the address is
legitimately present on whichever host is the elected v6 master, so an
unconditional "must be absent" assert fails on the master on every run. The
config file is the right thing to check every time precisely because
keepalived never writes there -- anything found is by definition static,
whereas the live address is expected on the master. The runtime withdrawal
stays gated on .changed, which remains correct for it.
Verified on jupiter:
grep -rlE 'fd40::123\b' /etc/network/interfaces /etc/network/interfaces.d/
-> /etc/network/interfaces
Sequencing holds: on the migration run the lineinfile removes the hook
first, so the grep finds nothing and the assert passes; if the regex does
not match, the grep finds it and the assert fails loudly instead of
skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Arch guard added in the previous commit grepped for the literal
"fd40::123/128" -- the full address/prefixlen from ntp_vip6. A hand-written
[Address] using any other mask (/64, say) would not have matched, the
assert would have passed, and keepalived would have been handed an address
networkd was still assigning.
That fails open, which is the wrong direction for this particular check.
It is detect-only: there is no delete afterwards whose result gets
re-verified, so a miss is silent, and the outcome is the dual-ownership
state the guard exists to prevent.
The Debian path does not share the problem. `ip -6 addr show ... to
<prefix>` matches by range regardless of the prefix length the address was
actually assigned with, and its `ip addr del` -- which IS prefix-sensitive
-- is re-verified by a following assert, so a mismatch there fails loudly.
Now matches the bare address, anchored with \b so it cannot instead hit a
longer address sharing the prefix. Verified both directions:
grep -rlE 'fd40::123\b' /etc/systemd/network/ on edge-1
-> /etc/systemd/network/10-enp1s0.40.network (still detected)
grep -cE 'fd40::123\b' against 'Address=fd40::1234/64'
-> 0 (no false positive)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second review round on #469 found two ways the migration could silently
leave the address outside VRRP control -- both of which would defeat the
health gating while the play reported success.
1. The one-shot migration had no verification.
Gating the withdrawal on `keepalived_ntp_static6_removed.changed` is what
makes it idempotent, but it also means it gets exactly one attempt and
never retries. Combined with `failed_when: false` on the delete, a
mismatch would pass silently: `ip addr del` matches on address AND prefix
length, so a host whose static assignment used a different mask or
interface than ntp_vip6 / keepalived_ntp_interface would see the delete
no-op, keep the address permanently outside VRRP control, and still
report success.
The idempotency fix from the previous commit created this: making the
task run once removed the retry that would otherwise have masked it.
Now asserted explicitly after the delete, with a message naming the
likely cause and the commands to inspect it.
2. edge-1 -- the host the incident actually happened on -- was not covered.
The removal tasks are gated to Debian, so they never touch edge-1 (Arch).
Its static [Address] block belongs to the edge-host role, and
install-edge-host.yml is a separate play with no ordering relationship to
this one. Nothing guaranteed it had been re-applied.
edge_host already sets ntp_keepalived_enabled: true, so the next run of
install-ntp-ha.yml gives edge-1 the NTP_VIP6 instance immediately. With
the [Address] block still present it would hold fd40::123 statically AND
run VRRP for it: keepalived would believe it had withdrawn the address
while networkd kept it assigned, and the host would go on answering NTP
with a clock check_ntp had already rejected. That is the failure of
2026-08-04, on the same host.
This is not hypothetical -- edge-1 still has the block right now:
grep -rl fd40::123/128 /etc/systemd/network/
-> /etc/systemd/network/10-enp1s0.40.network
The role now detects this and fails fast, naming the play to run first,
rather than proceeding into a split-ownership state.
Also documents why the post-deploy presence check is meaningful: presence
alone cannot distinguish a VRRP-held address from a static leftover, and
only means what it claims because the role asserts no static assignment
survives on either distro.
Not taken: narrowing serial: 1 so routine runs are not serialised through
the chrony settle pause. Real cost, but correctness of the address handover
comes first at four hosts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #469 caught a real defect. The withdrawal task used
ansible.builtin.command with no changed_when, so it reported changed and
fired `notify: restart keepalived` on every run where the address was
present.
That is fine exactly once. After this lands, fd40::123 is legitimately
present on whichever host is the elected v6 master, put there by keepalived
-- and an ungated delete cannot tell that apart from a static assignment,
because the address looks identical either way. So every later run of this
play (adding a host, changing a priority, plain re-convergence) would have
deleted the master's address and restarted keepalived, dropping the service
address and forcing a re-election each time. That is precisely the flapping
this change exists to prevent.
Both the check and the delete are now gated on the ifupdown hook having just
been removed, so they run once during the migration off static config and
never again. Steady-state runs leave the master's address alone.
Also from review:
- failed_when: false on the delete. The desired end state is "address
absent". If check_ntp fails in the window between the check and the
delete, keepalived withdraws the address first and `ip addr del` errors
on an address that is already gone -- the outcome the task wants, and no
reason to abort the play for that host.
- The play's post-deploy verification only ever checked the IPv4 VIP. The
v6 address is the path clients actually use, so a broken v6 rollout --
address never claimed, or -nodad not behaving as expected on a given
kernel -- would not have surfaced in the play at all. It now gets the
same check, and the message points at check_ntp and the health log for
the case where no host holds it.
Not taken: the suggestion to narrow serial: 1 so it does not also serialise
chrony. With four hosts it costs nothing, and scoping it per-role would mean
splitting the play. Worth revisiting if the cluster grows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this the health gating is defeated on three of the four NTP
servers. fd40::123 was assigned statically on every one of them, by hand
and in two different ways, none of it in version control:
PVE nodes `up ip -6 addr replace fd40::123/128 dev vmbr1.40 nodad`
in /etc/network/interfaces
edge-1 an [Address] block in /etc/systemd/network/
A statically assigned copy does not disappear when keepalived decides the
host is unfit to serve time. Leaving it in place means the host keeps
answering NTP on an address VRRP believes it has withdrawn -- exactly the
failure this whole change exists to prevent, with the added confusion of
keepalived reporting the address as released.
The ifupdown hook is removed here. edge-1 needs nothing: the edge-host
networkd template has never carried the address, so applying that role
drops it.
The live address is deleted as well, because removing the config line alone
does not take effect until the next network reload, and a reload on a PVE
node interrupts guest connectivity including the Talos VM and its Ceph OSD.
Deleting it is safe where keepalived already holds it: the deletion
notifies a keepalived restart, and the elected master re-adds it on the way
back up. Doing it unconditionally avoids having to tell "assigned
statically" from "assigned by keepalived", which the address alone does not
reveal.
The play now runs serial: 1. Withdrawing the address and restarting
keepalived on every NTP server at once would leave the fleet with nobody
holding fd40::123 until the elections settle; serialised, the remaining
members answer throughout.
Verified against the live hosts rather than assumed:
- all three PVE nodes carry the identical hook (jupiter:54, saturn:55,
neptune:56), so one task covers them
- the removal regexp matches exactly one line per host
- `ip -6 addr show dev <if> to <addr>` prints the matching address when
present and nothing at all when absent, so the guard condition is sound
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit set keepalived_ntp_vip6 in group_vars/edge_host, which
would have given the v6 VRRP instance to edge-1 alone. The other NTP
servers would have rendered no NTP_VIP6 instance at all and stayed silently
outside the election -- so the address could never fail over to them, and
the health gating would have amounted to "edge-1 withdraws and nobody takes
over".
Moves it alongside ntp_vip in plays/install-ntp-ha.yml, which is already
where the service-wide address lives, and passes it to the role for every
host in ntp_cluster.
Verified that edge-1 is currently BACKUP and does not hold the IPv4 VIP: a
VRRP BACKUP receiving no advertisements promotes itself to MASTER, so the
other members are alive and one of them holds it. They are real
participants, not retired hosts.
Not verified: whether the other members also carry fd40::123 statically, as
edge-1 did. They have no DNS records -- jupiter/saturn/neptune resolve only
through the *.renner.uno wildcard to the proxy -- so there is no hostname
path to them to check. If they do carry it statically, that address must be
removed at the same time this lands, or two hosts will hold it outside VRRP
control.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ntp.srv.renner.uno resolves to fd40::123 (IPv6) and to the keepalived VIP
(IPv4, legacy). IPv6 is the primary family here, so the v6 address is the
path clients actually use -- and it was the one with no health gating at
all.
It was assigned statically in /etc/systemd/network/ on edge-1, so networkd
carried it unconditionally from boot. That is how the host answered NTP
queries on 2026-08-04 while its clock was ~16 days in the past: chronyd was
running, udp/123 was listening, the address was up, and nothing anywhere
was checking whether the clock was right. Fixing the keepalived health
check does not help, because keepalived was not holding that address.
This moves it into keepalived as a second VRRP instance tracking the same
check_ntp script, so the address exists only while the host is fit to serve
time.
Three details that are load-bearing rather than stylistic:
- A separate vrrp_instance is mandatory. keepalived requires that "all
addresses in virtual_ipaddress must be of the same family", so the v6
address cannot join NTP_VIP.
- No authentication block. VRRPv3 removed authentication (RFC 5798) and
keepalived always uses version 3 for IPv6 instances.
- track_script uses `weight 0`, overriding the script's -20. A non-zero
weight only subtracts from priority on failure, so the last remaining
host -- having nobody to lose the election to -- would stay MASTER and
keep serving a clock it had already reported as wrong. Weight 0
transitions the instance to FAULT instead, releasing the address
unconditionally. That is the required behaviour: a host unfit to serve
time must stop answering whether or not a replacement exists.
Semantic change, deliberate and worth stating: VRRP is a single-master
unicast VIP, not anycast. Previously every NTP server carried fd40::123 and
clients reached whichever answered neighbour discovery first. Now exactly
one host holds it. For NTP this is the better trade -- query volume makes
load spreading pointless, and on failover keepalived sends unsolicited
Neighbour Advertisements so client neighbour caches update in seconds,
where the static address had no withdrawal signal at all and clients kept
using a dead holder until their own neighbour unreachability detection
expired. True routed anycast with health withdrawal would mean announcing
the /128 via an IGP, which the gateway would have to speak.
The edge-host role is deliberately untouched: its template already has no
[Address] block, so applying it removes the hand-placed address, and
keepalived then adds it back under health control. The service address is
NTP-role configuration and now lives there.
Validated with `keepalived -t` against keepalived 2.4.1 on edge-1 using the
config as it renders for that host; parses clean apart from the pre-existing
auth_pass truncation warning on the IPv4 instance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The NTP health check asked "is chronyd alive", never "is this host safe to
take time from". Those came apart on 2026-08-04.
edge-1's CMOS battery is dead. After a power loss its clock came up ~16
days in the past and, with no makestep directive, chronyd never corrected
it. All three health checks passed continuously for ~18h:
- chronyd was active
- `chronyc tracking` responded
- udp/123 was listening
and `local stratum 10` meant chronyd advertised itself to clients as a
valid reference the whole time. keepalived would have kept the VIP on a
host ready to serve a clock from July.
The sharpest part: check 2 ran `chronyc tracking >/dev/null 2>&1` and
discarded the output. The one command that knew the answer was executed
and its answer thrown away. During the incident that output read:
System time : 1376235.375000000 seconds slow of NTP time
This captures the output and adds two assertions:
- offset above keepalived_ntp_max_offset (default 1.0s) fails, so a host
that knows its clock is wrong drops the VIP
- no upstream source selected in `chronyc sources` fails, because with
`local stratum N` chronyd advertises itself as a usable reference even
when it has never synchronised to anything -- upstream chrony docs are
explicit that the local reference activates immediately in that case
Both predicates were validated against live output from edge-1 rather than
assumed: the offset comparison against the real incident value, and the
`chronyc sources` selected-source column against a real run.
Failing the check costs the configured weight (-20). When the WAN is down
every host fails together and loses the same weight, so relative priority
is unchanged and the VIP stays put -- the local-reference fallback still
works. What this prevents is one host with a broken clock winning.
Scope: this hardens the keepalived path, which is legacy. It does not
address the documented end state in plays/install-ntp-ha.yml, where edge-1
becomes the SOLE NTP server -- there, a failing check means no host serves
time at all, converting "serves the wrong time" into "serves no time".
That needs an architectural answer, not a better health check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#465 landed the makestep directive with the wrong root cause in its
comment. It claimed edge-1 "rebooted with an RTC ~16 days in the past".
That is false, and the real cause matters for what to do next.
edge-1's CMOS battery is dead. After a power loss the RTC came up reading
2018-08-20 -- roughly eight years stale, not 16 days:
PM: RTC time: 09:18:56, date: 2018-08-20
rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC
The clock was then reconstructed entirely in software. systemd floored it
forward to its own build date:
systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC
and it settled at ~2026-07-19, about 16 days behind. So the 16 days is
incidental -- it is just where the flooring landed. There is no 16-day
mechanism, and nothing "drifted". The two real faults are a dead RTC
battery and a chrony config that never steps. #465 fixed the second.
Also collapses the duplicated rationale in chrony.conf.j2 to a pointer at
defaults/main.yml, as suggested in review on #465, so the explanation
lives in exactly one place.
No behaviour change: the makestep directive and its defaults are
untouched. Comments only.
The commit message on 7fcaa4b0 still carries the wrong explanation and is
left alone rather than rewriting merged history.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.
On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:
tls: failed to verify certificate: x509: certificate has expired or is
not yet valid: current time 2026-07-20T16:15:36+02:00 is before
2026-07-23T06:15:01Z
That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.
Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.
The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
security-bot
k.renner
claude-bot
