Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.
crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.
Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the
MS-01's only U.2-capable slot means moving the boot drive anyway. Doing the
bare-metal conversion in the same window avoids opening the machine twice and
throws away no work: the osd-ssd PCI mapping that a Proxmox-side swap would
need to be re-pointed at ceases to exist.
Networking uses the v1.12 multi-document config rather than the deprecated
machine.network.interfaces. The docs do not define what happens when one link
is configured by both schemas, so k8s-1 keeps every link in one of them. This
also buys VLANConfig.name, which is free-form: naming the VLAN 30 link ens19
keeps the four v30iot macvlan NADs working unchanged across VM and bare-metal
nodes, so the cluster can stay mixed without pinning those workloads.
Physical NICs are matched by permanent MAC, not driver — both X710 ports bind
i40e here, unlike the single passed-through port k8s-2/k8s-3 see.
cephfs-metadata drops to size 2 / min_size 1 for the duration. At size 3 on
two remaining hosts every PG sits undersized with min_size 2, so one further
OSD loss would block IO on every CephFS mount cluster-wide. This must go back
to size 3 once k8s-1 rejoins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sweep restores every discovered Velero schedule, immich's ~180 GiB
library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and
measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per
week — ~47% of the endurance budget, against a ~65 GB/day weekday
baseline.
That baseline is itself the result of the 2026-08-18..21 write-reduction
work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs,
paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive
writes ~70% from ~195 GB/day. With the application tier down, the restore
test is now the single largest consumer of remaining QLC endurance; the
three OSD drives are at 81/82/87% used.
Monthly still exercises every restore path and cuts the sweep's share of
drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.
Cadence-only change:
- 0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the
03:15-07:15 daily backup window (see the comment in the manifest).
- ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still
inspectable when the next one fires.
- Comments and docs that state the cadence updated to match, including
the BLUESTORE_SLOW_OP_ALERT runbook text, which tells the reader to
check whether a stall fell inside the restore-test window.
No day-of-week form is available: Kubernetes parses cron with
robfig/cron's standard parser, whose day-of-week field allows only
`* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields
ORs them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Wetterstation now carries a non-default reporting configuration: min
interval 5s on all ten reportable attributes, and tighter reportable-change
deadbands on temperature, humidity and pressure. None of it is expressible
in configuration.yaml - Zigbee2MQTT has no settings-schema field for
reporting configuration - so it lives only on the device and in Z2M's
database.db. This documents it and how to restore it.
min was 10s against an 8.8s transmit interval. Since min is a hold-off, every
change-triggered report was held past a transmit cycle and landed on the
next, so attributes reported at best every 17.6s and drifted out of phase.
The published stream showed no 8.8s periodicity at all. At min 5 each
attribute reports on its own transmit and the stream carries one publish per
cycle, measured at a mean gap of 8.69s. Publish volume fell from ~631/h to
~445/h because each publish is now a whole cycle rather than a fragment.
The deadbands come from the shared modernExtend helpers, which the WS90
definition composes without overrides. They are anti-chatter values sized for
battery-powered indoor sensors. Temperature was 1.00 C, giving a staircase
that moved about twice an hour; it is now 0.1 C, matching the resolution the
device actually reports. Pressure was 5 kPa - 50 hPa, which real barometric
pressure never traverses, so it only ever updated on the hourly heartbeat
that pressure_trend and weather_condition derive from. Note the same library
configures the identical ZCL attribute with a 5 hPa deadband in its older
reporting.ts helper, so the two code paths disagree by 10x.
Anything that makes Z2M re-run configure on the device reverts all of it, and
a fresh PVC will not have it. The restore commands are written out literally
rather than generated by a shell helper, because the nested quoting through
`kubectl exec -- sh -c` is where this goes wrong; each was executed and
verified as written, including the read-back, and re-applying is a no-op.
No behaviour change from this commit itself - the device was reconfigured at
runtime, this only records it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#560 claimed a merged episode "surfaces as a single all-zero frame". It
does not, and the difference decides whether a downstream filter is even
possible.
The zeros propagate attribute by attribute through the burst, so the
merged frame carries whatever subset had zeroed when that burst ended.
Across three episodes captured at nanosecond resolution the propagation
happened to complete, which is where the wrong generalisation came from.
Nothing guarantees it. A burst ending mid-propagation publishes a frame
with temperature at zero while humidity and pressure still read normally,
which is a physically consistent measurement - Z2M recomputes dew_point
from the false temperature - and is not separable from a genuine winter
reading.
The zeros also begin inside an otherwise normal transmit cycle, roughly
20ms after a valid reading in the same burst (measured: 27.7ms, 19.8ms,
22.0ms). They occupy a regular reporting slot, so no timing property
distinguishes them either.
Since the merge is last-wins and the zeros arrive at the tail of the
burst, it resolves that burst to the zeros and discards the valid reading
that led it. So debounce trades six bad samples per episode for one, at
the cost of one good sample. That is a bound on the corruption, not a
filter, and the comment now says so.
No behaviour change; the debounce value and device stanza are untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The value is unchanged and correct; the comment committed alongside it in
#559 was neither accurate nor the right place for most of what it said.
Accuracy. It cited the device's ZCL reporting configuration as the bound,
reading minRepIntval 10 as "the station updates every 10s". minRepIntval
is not a cadence: with maxRepIntval 3600 and a reportable change set, the
ZCL reporting model gives periodic reports at the MAXIMUM interval — once
an hour — plus change-driven reports each held back so it lands no sooner
than 10s after the previous. It is a hold-off on change-triggered reports
and says nothing about the gap between reports of different attributes,
which is what a debounce window sees.
The bound that applies is the documented transmit interval — readings
every 2s, transmitted every 8.8s — against Z2M's documented rule not to
set debounce above the sensor's update interval. Shelly's knowledge base
rounds the same figure to 9s.
It also claimed the window had to exceed 1s because "the recovery burst
straddles a second boundary", inferred from log lines stamped 11:42:11
and 11:42:12. Those are second-resolution timestamps and cannot
distinguish 10ms from 900ms. No lower bound was ever established.
Length. The rest was commit-message material sitting in a config file:
justification for the chosen value, gap-distribution and run-length
measurements, and notes reconciling my own earlier misreadings. Comments
state facts about the system, not the history of arriving at them. What
remains is what the device does, what debounce does to it, and the
constraint on the value — 41 lines down to 13.
Verified before writing: post-debounce frames arrive complete (all 22
fields, no partials) at a settled ~10s cadence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes to the Wetterstation (Ecowitt WS90, 0xc02cedfffe958f40), both
aimed at the station's recurring zero-drop glitch.
2.13.0 carries zigbee-herdsman-converters#12748, which discards the ZCL
non-value markers (0xffff / 0xff / 0xffffff) the WS90 emits when a reading
is unavailable. Without it those markers are scaled and published as
measurements — the 6553 m/s gust readings in Koenkk/zigbee2mqtt#31048. The
release also recovers the rain rate after a counter reset, which is the
same lifetime-counter dip that packages/wetterstation.yaml compensates for
on the Home Assistant side. Its only breaking change is dropping Node.js 20
for bare-metal installs; the container ships its own runtime. The amd64
manifest was verified pullable before this commit.
debounce: 2 addresses the part 2.13.0 does not fix. Koenkk/zigbee2mqtt#31048
is still open, and the author of #12748 states explicitly that it does not
cover the zero drops: "a zero can be a real reading for these attributes,
only the all-bits-set markers are provably invalid."
The station reports as a burst of seven per-cluster ZCL frames and Z2M
publishes its full cumulative state once per frame, so one device report
becomes seven MQTT messages. During a glitch the zeros appear progressively
across those seven, which is what makes the corruption unrecognisable
downstream: an intermediate frame carrying {"temperature":0,"humidity":92,
"pressure":96.9} is internally consistent (Z2M recomputes dew_point from the
false temperature) and cannot be told apart from a legitimate winter
reading. Merging the burst yields a single frame with the all-zero
signature, which a consumer can reject.
The window is bounded on both sides. Above: the device's own reporting
configuration sets minRepIntval 10 / maxRepIntval 3600 on all seven WS90
clusters (1024/1026/1027/1029 and 0xfc01/0xfc02/0xfc03), so no attribute
repeats inside 10s. Below: the recovery burst straddles a second boundary,
so 1s splits it and emits a partial frame.
Staggering was checked rather than assumed — the 10s floor is per attribute,
and nine attributes report across those clusters, so a trailing-edge timer
could in principle keep resetting and never flush. Over 6h the longest
unbroken chain of sub-2s gaps was 5, bounding worst-case deferral under
~10s.
The device is also absent from the seeded device list, which predates it
joining, so this adds the friendly_name too.
NOTE: this file is seed-only. A running Z2M owns its configuration.yaml on
the PVC, so this does not change the live instance — it only makes a fresh
PVC come up correctly. debounce is requiresRestart, so applying it live
needs the option set on the running instance and a restart; the image bump
supplies the restart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From the #426 audit. Tag verified present before bumping.
The kaniko bump that was originally in this branch is dropped. It would
have pinned a new `-debug` image, and debug variants are not to run in
the production cluster. kaniko is archived upstream, so the fix is
migration rather than a version bump; tracked in #552 along with the
`-debug` requirement.
Comments should say what the setting does and what constrains it. The
blocks added in #542, #544, #545 and #547 had grown into incident
write-ups: measurement runs, prior values, how a problem was found,
what a check used to be. That belongs in commit messages and the wiki,
both of which already carry it.
Trimmed, no behaviour change:
cephcluster.yaml 22 lines -> 6 (cephx rotation)
cephfs-ci.yaml 26 lines -> 7 (why size 2)
prometheusrule-ceph.yaml (CephHealthWarn, 43 -> 13;
CephPoolLostRedundancy, 15 -> 7)
The operative facts are kept: which checks are excluded and why they
cannot clear, that POOL_NO_REDUNDANCY reports blocked maintenance and
not only data risk, and that size 1 needs dedicated OSDs.
The unreplicated CI pools introduced in #542 make every OSD on the
cluster permanently un-restartable.
`ceph osd ok-to-stop` refuses while stopping an OSD would take PGs
offline. A size-1 pool puts every PG on exactly one OSD, so the check
fails for all three:
Error EBUSY: unsafe to stop osd(s) at this time
(20 PGs are or would become offline)
bad_become_inactive: 13.6 13.9 ... 14.1c 14.1d
Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2
pools are not implicated — they run min_size 1 and appear under
ok_become_degraded, which is allowed.
Rook gates OSD restarts on that check, so this blocks CephX key
rotation, Ceph upgrades, node drains and reboots, and drive
replacement. It surfaced when #545 deadlocked: Rook retried
ok-to-stop against all three OSDs several times a second and could
never succeed, while `ceph -s` reported 217 pgs active+clean
throughout. Nothing is unhealthy; the restarts are simply refused.
The reasoning in #542 was that unreplicated CI data only risks
in-flight pipelines, since every byte is a fresh git clone. That part
holds. What it missed is that the cost is not scoped to the pool: one
unreplicated pool freezes maintenance for the entire cluster.
Most of #542's saving came from keeping CI churn off `cephfs`, whose
metadataPool is size 3 — that survives this change. Only the second-
order gain from dropping the replica is given up, and cephfs-ci-data0
stores 0 B with cephfs-ci-metadata at 182 MiB, so the backfill is
trivial.
size 1 becomes viable again once these pools sit on OSDs of their own,
where the permanently un-stoppable OSD holds nothing but throwaway
data. That is planned with the retired QLC drives.
Alerting follows the same change:
- POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was
excluded because cephfs-ci was expected to keep it permanently
active; no pool is unreplicated now, so the exclusion would only
hide a real one losing its replicas.
- CephPoolLostRedundancy drops its cephfs-ci exemption for the same
reason, and its description now states the maintenance consequence,
not just the data-loss one.
- The AUTH_INSECURE_* exclusion is reworded. Calling it
"known-accepted" reads as a decision that the risk is fine; it is a
kernel-blocked remediation (aes256k over the CSI mount path needs
kernel 7.0+, nodes run 6.18).
Refs #542, #545, #527
Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.
The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.
Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.
Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.
CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.
Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.
Refs #527, #528
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CephHealthWarn alerted on `ceph_health_status == 1`. It has been FIRING since
at least 2026-08-21T00:26Z and cannot clear: four AUTH_INSECURE_* checks are
permanently active from CVE-2025-30156 (#527) and stay that way until Rook
>=1.20.5 lands. A permanently-firing alert reports nothing -- every Ceph
warning since then has been invisible behind it.
This is a pre-existing gap, not a consequence of the cephfs-ci work, though
that work would have added a second permanent cause (POOL_NO_REDUNDANCY, since
the CI pools are size 1 on purpose).
ceph_health_detail exposes each check as its own series, so the alert now fires
on any check that is NOT known-accepted, and names it. Verified against live
data: 5 checks active, 0 unexpected, so it is quiet now and fires the moment
something genuinely new appears.
Excluding POOL_NO_REDUNDANCY wholesale would give something up: that check is
cluster-wide and does not say WHICH pool, so accepting it for cephfs-ci would
equally hide a production pool dropping to one replica. CephPoolLostRedundancy
restores that. There is no pool-size metric, but bytes_used/stored IS the
replication factor -- verified live as replicapool 2, cephfs-data0 2,
cephfs-metadata 3, cephfs-ci-metadata 1.
Both alerts were tested against live data before committing, in BOTH
directions. The positive control matters as much as the negative one: an early
draft used a >1e9 floor which silently excluded the CI pools entirely, so the
expression could not have matched anything. It looked correct precisely because
it returned no results. The control caught it; the floor is now 1e6.
The floor is not what suppresses the near-empty rgw pools. Their ratios are
9-36 because per-object overhead dominates, so they never match `< 1.5`.
6h on CephHealthWarn is unchanged -- Ceph warns for self-clearing conditions
and paging on those trains people to ignore the alert.
Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.
Measured 2026-08-21, polling both CephFS pools every 15s:
CI active 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB
CI idle 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB
CI share of writes: data0 100%, metadata 61%
data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.
Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.
RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.
Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.
The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.
cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.
Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.
one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
every interval with no CI pod: data0 ~0 metadata ~0
So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.
size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.
`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.
Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.
Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.
The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.
gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.
Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.
ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.
ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.
Also corrects two comments in ha-pg-cluster.yaml that this work falsified.
wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.
checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.
Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.
This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.
Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.
The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.
archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
`wal_compression: "on"` resolves to pglz, PostgreSQL's original 2016-era
algorithm. Every CNPG cluster here was set that way, so none of them were
getting the compression actually available to them: all seven builds accept
lz4 and zstd (verified via pg_settings.enumvals on each, including immich's
custom pgvector image and trmnl's PostgreSQL 18).
This matters because of what dominates recorder-style write volume. A page
is written to WAL in full -- all 8 KiB -- on its first modification after a
checkpoint, for torn-page protection. Measured on ha-pg, those full-page
images are 62.4% of WAL bytes. Compressing them is therefore the single
largest lever available on WAL volume, and pglz was the weakest option for
the job.
ha-pg additionally moves checkpoint_timeout 15min -> 30min. Full-page images
are emitted once per page per checkpoint interval, so doubling the interval
roughly halves how often each hot page pays the 8 KiB toll. max_wal_size
stays at 2GB deliberately: it must remain well above what one interval
generates, or checkpoints would trigger on size and the interval change
would do nothing. Recovery replays at most one interval of WAL -- under
200MB at the measured rate, i.e. seconds.
The other six clusters keep checkpoint_timeout at 15min. Their measured
write rates do not justify trading recovery time: gf-pg generates 4.5 MB/day
of WAL and wp-pg 62 MB/day, against ak-pg's 604 MB/day.
full_page_writes stays on everywhere. Disabling it would remove the
overhead entirely but RBD can tear a page across an object boundary, and
the trade is write volume against an unrecoverable database.
Both parameters are sighup-reloadable, so this does not restart any cluster.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
An image reference is not a secret. Encoding it as base64 `data` had two real
costs: no reviewer sees a version change in a diff, and no $imagepolicy marker
can attach to a base64 blob.
The result was a two-minor version skew in the component that performs every
base backup and WAL archive for seven Postgres clusters (#535):
controller plugin-barman-cloud:v0.14.0 <- auto-bumped by Flux
sidecar plugin-barman-cloud-sidecar:v0.12.0 <- invisible, never moved
Not stale pods: ha-pg-1 was created hours ago during the recorder migration and
still received v0.12.0.
Three changes:
1. The Secret now uses stringData with the tag in plaintext, carrying an
$imagepolicy marker. The Secret KIND is kept because the controller reads it
via secretKeyRef; switching to a ConfigMap would need a matching edit to that
reference and diverge further from upstream. stringData is accepted
identically by the API server.
2. ImageRepository + ImagePolicy added for the sidecar image, with the same
filter and 0.x range as the controller, so the two cannot drift apart again.
Only the controller was tracked before.
3. Tag aligned to v0.14.0 to match the controller.
The Secret name carries a kustomize hash suffix from upstream's secretGenerator
but is static here, so editing the value does not change the name and the
deployment's secretKeyRef stays valid — verified in the render, which still shows
the name twice (Secret + reference).
Verified the tag EXISTS before pinning it: pulling
plugin-barman-cloud-sidecar:v0.14.0 succeeds (the pod fails later on a missing
`true` binary in that minimal image, which is the container command, not the
pull). Pinning a non-existent tag would have put all seven Postgres clusters into
ImagePullBackOff.
EXPECTED IMPACT: bumping the sidecar changes every Postgres pod spec, so all
seven CNPG clusters roll. CNPG handles this as a normal rolling restart. Backups
should be confirmed working afterwards — 113 Backup CRs exist today, newest ~3h
old and completed.
Note for future re-imports: upstream ships this Secret as base64 `data` with no
marker, so a wholesale re-import silently reintroduces the skew. The manifest now
says so inline.
Refs #535
Co-Authored-By: Claude Bot <noreply@anthropic.com>
Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.
The copy is already done and verified byte-for-byte with paperless stopped:
src files: 75
dst files: 75
IDENTICAL: name, size and mtime match for every file
index segments: src 3, dst 3
mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)
Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.
cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.
paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.
It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:
paperless-data client request_load_avg 22 (all other sessions 0-1)
MDS over 180s 4,052 client requests = 22.5/s
2,718 journal events = 15/s
2 dir commits <- not file writes
Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.
cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.
PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).
Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten Postgres instances across seven CNPG clusters are ~48 GB/day of the
cluster's ~114 GB/day of client writes, and only ha-pg carried any
write-reduction tuning. The rest ran Postgres defaults, which favour throughput
over storage that wears out.
MEASURED, not assumed. ak-pg since 2026-05-17:
checkpoints_timed 27,624 = 291/day, i.e. one every 5 min (the default)
buffers_checkpoint 25,801,628 = ~206 GB flushed
database size 1,046 MB
That is the whole database rewritten roughly twice a day. The pathology is
specific to idle databases: sessions, tokens and job state re-dirty the same few
pages constantly, and checkpoint_timeout=5min forces each of them to disk 288
times a day instead of letting them coalesce. At 15min they are written a third
as often.
max_wal_size is raised alongside it. Without that, checkpoints trigger on size
instead and the timeout change does nothing. It also bounds crash recovery to at
most 2GB of WAL replay.
Applied to ak-pg, pprl-pg, tr-pg, wp-pg, im-pg, gf-pg. ha-pg already had it.
synchronous_commit is deliberately NOT changed here. It is off for ha-pg because
a state recorder is telemetry; these clusters hold credentials, documents and job
state, where losing the last few hundred ms of commits on an unclean shutdown is
not an acceptable trade for write volume.
Separately, ha-pg shared_buffers 256MB -> 512MB. After 15h on Postgres its
buffers_backend was 629,704 of 939,025 total page writes -- 67% written by
backends rather than the checkpointer or bgwriter, which means the working set
does not fit in the buffer pool and pages are written out repeatedly instead of
coalescing. Memory limit raised 1Gi -> 1536Mi to keep the same headroom (measured
383Mi in use, of which 256MB was shared_buffers).
Verified before commit by rendering every cluster and asserting the tuning landed
under spec.postgresql.parameters with the barman plugin block untouched. That
check exists because the first attempt at this matched the wrong `parameters:`
key -- the barman plugin's, at 6-space indent under plugins: -- and corrupted
wp-pg's barmanObjectName. Reverted and redone with an indent-aware match.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.
Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.
HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:
/var/lib/code-server 8.0K
/mosquitto/auth 8.0K
/state (z2m guards) 8.0K
So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.
Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.
Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.
DELIBERATELY NOT CONVERTED:
- kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
and generated from the chart; a hand edit would be lost on upgrade.
- trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.
Cost: this rolls the affected pods once.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer was the single largest writer in the cluster -- more than every
Ceph OSD and every database combined -- and none of it needed to touch disk.
Measured 2026-08-20 from the container's own cgroup io.stat:
253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day
252:48 (its own PVC) 340 MB total = negligible
So the writes were not going to its volume at all; they were going to the
node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without
`medium: Memory` is node storage, which is easy to read as "ephemeral, so
free".
The cause is cadence rather than volume. schedules.template.json renders on
"* * * * *", so Chrome is launched 1440 times a day and each launch creates,
fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp
holds under 8 MB -- it is pure churn for data that never outlives the render.
tmpfs pages count against the container's memory limit, so the 512Mi cap is
kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample
headroom even if the cap were fully consumed.
Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a
4h window. This is a bigger cluster-wide reduction than the Home Assistant
recorder migration achieved, for a one-line change.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Net -44 lines. The previous version assembled the connection URI from
username/password/dbname, which forced percent-encoding, YAML-safe quoting via
json.dumps, a YAML round-trip validation, and python3 from HA's image.
None of that was necessary: CNPG already publishes a finished, percent-encoded
connection string in the app secret as `uri` / `fqdn-uri`. Reading that key
removes the whole class of problem the earlier review correctly flagged --
rather than solving it more carefully.
Verified against the live secret before simplifying: fqdn-uri contains only
[A-Za-z0-9:/@._%-], so naive double-quoting is provably safe and the YAML
validation was guarding against an input that cannot occur.
What is kept, because it is the part that actually matters: secrets.yaml is PVC
state holding every other HA secret (influxdb_token and friends), so the file is
filtered-and-appended rather than rewritten, moved atomically, and guarded
against shrinking. Tested in HA's own busybox shell -- comment and pre-existing
keys preserved, idempotent across runs, no duplicate key.
Also records why an env var cannot be used at all: HA removed `!env_var` from
its YAML loader, so `!secret` is the only mechanism for recorder.db_url, and
config-sync deliberately does not manage secrets.yaml.
Back to busybox; HA's image is no longer needed for this.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2b is done: 12.77M statistics rows copied SQLite -> Postgres with HA
running and no downtime (read-only mount; SQLite WAL gives readers a consistent
snapshot).
statistics_meta 1449 OK
statistics 10799904 OK 153.9s
statistics_short_term 1969804 OK 27.0s
span in Postgres: 2024-03-13 -> 2026-08-19
1449 entities with long-term history, 2627859 rows carrying sum
database size 2252 MB against a 20Gi PVC
Records the high-water marks the cutover top-up must resume from, and two schema
details that cost a failed first attempt:
- has_mean/has_sum are INTEGER in SQLite, BOOLEAN in Postgres, and Postgres
will not coerce them.
- the legacy created/start/last_reset DATETIME columns are 100% NULL in both
bulk tables, which is what makes a tab-separated COPY safe.
Also documents why the schema was built from HA's own models with schema_changes
left EMPTY: HA's _initialize_database detects a new database via the
events.time_fired_ts index and writes SCHEMA_VERSION itself, so pre-stamping that
row would bypass HA's own initialisation.
ANALYZE was run on all three tables. After a bulk COPY the planner has no
statistics and HA seq-scans 10.8M rows on every history/energy view.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #524, and it was a genuine blocker.
The first version built "postgresql://${PGUSER}:${PGPASSWORD}@host/db" in
busybox sh with no encoding. Two defects:
1. The CNPG-generated password is random and may contain @ : / ? #, each of
which changes how the URL parses. This repo already hit this exact issue
with the same CNPG password-generation mechanism -- see
kubernetes/apps/woodpecker/server.yaml, which switched to a lib/pq
keyword DSN specifically to avoid escaping generated passwords. That
option does not exist here: HA's recorder.db_url must be a SQLAlchemy URL.
So the fix is percent-encoding.
2. Worse, and the reason this was worth blocking: a quote in the password
makes the YAML line malformed, so secrets.yaml stops parsing entirely and
HA loses EVERY secret, not just this one. The line-count guard could never
catch that -- the count is unchanged; the file is simply invalid.
Rewritten to run python3 from HA's own image (already pulled for the main
container, so no extra image) instead of busybox:
- urllib.parse.quote(..., safe="") on user, password and dbname.
- json.dumps for the YAML value: YAML is a JSON superset, so this emits a
correctly escaped double-quoted scalar even with quotes or backslashes.
- The file is still rewritten LINE-WISE rather than round-tripped through the
YAML parser, so hand-maintained comments and key order survive.
- The guard now validates the RESULT: parse the new content, assert no key
from the old file disappeared, and assert recorder_db_url round-trips to
exactly the URL that was built. That catches corruption the line count
cannot.
Verified in HA's image against a deliberately hostile password
(p@ss:w/o"rd#?&\x=1) before commit:
new version -> parses, comment preserved, idempotent, password decodes back
byte-exact, host and dbname parse correctly
naive version -> yaml.safe_load raises ParserError
i.e. the approach this replaces would have produced an unparseable
secrets.yaml.
Also updates MIGRATION-recorder-postgres.md for this step landing, including
why the ordering (init container before the configuration.yaml change) is
mandatory.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2a of the recorder move off SQLite (MIGRATION-recorder-postgres.md).
Adds an init container that writes `recorder_db_url` into /config/secrets.yaml
from the CNPG-generated ha-pg-app Secret.
INERT until the recorder is repointed. It only adds a key to secrets.yaml;
nothing reads it until `db_url: !secret recorder_db_url` lands in
configuration.yaml in the homeassistant repo. That ordering is REQUIRED in this
direction — HA refuses to start when a referenced secret is missing, so the
config change must not go first.
Why an init container rather than a hand-added line: the password is generated
by CNPG and can be rotated, so a hand-written entry goes stale silently and HA
then fails to start on a bad password. Re-rendering on every pod start is also
what makes a rotation take effect.
secrets.yaml is PVC runtime state that config-sync deliberately does not manage
(it holds influxdb_token and others), so this must EDIT the existing file, never
replace it. The script filters out any existing recorder_db_url line, appends
the current one, and guards the result: if the rewrite would drop any
non-recorder line it aborts rather than truncating the file holding every other
secret HA needs. Written to a temp path and moved atomically, chmod 600.
Verified in HA's own Alpine/busybox shell against a fixed sample before commit
-- pre-existing keys preserved, guard reports old=3 new=3, and a second run
produces a byte-identical 4-line file (idempotent, no duplicate key).
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.
CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.
CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".
DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.
The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.
Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.
Refs #522
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recorder's SQLite database is the largest client writer in the cluster.
Measured 2026-08-19 via cgroup io.stat on the home-assistant container,
corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD
volume, ~26% of all client writes, against only ~5 GB/day of actual row data.
The 8x gap is the write PATTERN, not the volume:
~30 state changes/sec (107k rows/hour, 2.57M/day)
x2 SQLite WAL + checkpoint -- every dirtied page written twice
x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit
(measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
= ~15 KB written per recorded state change
Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
This attacks the multiplier rather than trimming what gets recorded, which would
hide the problem instead of fixing it.
PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on
SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves.
Phase 2 migrates the statistics tables and repoints recorder.db_url in the
homeassistant repo; MIGRATION-recorder-postgres.md has the sequence.
What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT
migrate data between database engines. `statistics` holds 10,793,712 rows
spanning 2024-03-13 to today -- 2.4 years of energy history that is not in
InfluxDB and cannot be reconstructed. Repointing db_url at an empty database
silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7.
Postgres is tuned to minimise write volume, which is the point of the move:
synchronous_commit off -- the recorder is telemetry, not transactional
record-keeping. Postgres stays crash consistent;
only the last few hundred ms of commits are at risk
on unclean shutdown. In exchange, commits stop
forcing an fsync -- the exact cost that makes SQLite
expensive here.
wal_compression on -- recorder writes are small inserts scattered across
pages, so post-checkpoint full-page writes dominate
WAL volume.
checkpoint_timeout 15min / max_wal_size 2GB
-- fewer checkpoints means fewer repeat full-page
images for the hot rows the recorder rewrites.
full_page_writes -- deliberately LEFT ON. RBD can tear a page across a
4 MiB object boundary; trading that for write volume
risks an unrecoverable database.
Also included:
- ObjectStore on its own s3 prefix (two clusters sharing a destinationPath
interleave WAL streams and corrupt both).
- ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters.
- PodMonitor added WITH the cluster rather than after it, so the migration has
metrics from the start instead of discovering the gap later (#433).
- Cilium policies as DEDICATED policies with their own endpointSelector, not
rules on the namespace-wide default-deny -- that policy is endpointSelector:{}
so any rule there applies to every pod in the namespace, and :8000 is served
by more than the CNPG instance manager. Includes the ICMP allowance that
attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD.
The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same
coupling #397 tracks for woodpecker. Noted in the file with the exact bao command
to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed`
was a wrong-path error rather than missing permissions.
Refs #434, #397
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three Ceph monitors were writing 240 GB/day between them for a ~106 MB
database. That was ~62% of all OS-disk writes on the nodes and roughly a
quarter of everything the cluster writes -- more than every client workload
combined.
Measured 2026-08-19:
paxos transactions ~99,000/day (1.15/s, sampled over 60 s)
paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime)
ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three
(cgroup io.stat: 2.15 TB / 27 d)
=> ~98x write amplification
Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix
(including paxos) when we trim its old states". With paxos_trim_min at 250 and
~99k transactions/day, that is hundreds of forced range compactions per day,
each rewriting the store.
The churn driving it is not cluster activity. 99% of paxos versions are cluster
log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook
operator poll (mgr stat / versions / status / report / osd dump; 23 audit
entries per minute per mon) becomes a cluster-log entry, hence a paxos
transaction, replicated to all three mons. Reducing that polling is the
complementary fix and is tracked separately; this change removes the
amplification regardless of the rate.
This does NOT disable compaction. RocksDB's own levelled compaction still runs
and still reclaims trimmed keys; what stops is the extra forced compact_range
on every trim. Two backstops if that assumption is wrong:
- mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn
in monitoring-rules already alerts on.
- mon_compact_on_start is set to true here (upstream default false), so every
mon restart -- upgrade, node reboot, reschedule -- does a full compaction.
Cheap on a ~106 MB store.
Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z:
wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z.
Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly
double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212,
and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank
writers; every figure above comes from cgroup io.stat or kernel diskstats, which
reconcile with each other and with the drives' SMART counters.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With 3 OSDs on 3 hosts, size=3 meant every write landed on every drive in the
cluster. Per-drive write volume equalled TOTAL cluster write volume, so all
three drives wore at an identical rate and would have reached end-of-rated-life
together: measured 2026-08-19 from the drives' own SMART counters, 80/82/87%
endurance used, advancing ~0.13 points/day, exhausted between Nov 2026 and
Jan 2027 (#434). Replication gives no protection there — the wear-out is
correlated by construction, not independent.
size 2 cuts per-drive write volume by a third and desynchronises the wear.
min_size 1 is required, not incidental. Ceph does NOT lower min_size when size
is reduced; it was 2, and leaving it would make replicapool 2/2, where a single
OSD down blocks IO for its PGs. That would hang every RBD volume in the cluster
on every rolling node reboot and every OSD restart.
Accepted trade, stated explicitly: while degraded (one OSD down), a second
drive failure loses data, and at 80%+ endurance that window is not
hypothetical. What limits the blast radius is that CNPG barman and Velero both
ship offsite to Hetzner, so exposure is the RPO of those backups rather than
total loss. Node reboots must stay serialised with HEALTH_OK between them.
CephFS metadataPool deliberately stays at size 3: 467 MiB against data0's
275 GiB, so it contributes nothing measurable to wear, while losing it destroys
the entire filesystem rather than the objects on one OSD. The wear saving comes
from data0. The RGW pools in objectstore.yaml were already size 2.
One-time cost on apply: Ceph deletes the third replica of everything, which is
background IO, and raw used drops from ~1.4 TiB to ~0.95 TiB.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bluestore_prefer_deferred_size_ssd was set to 65536 and
bluestore_deferred_batch_ops_ssd to 64 as a QLC throughput tune. Ceph's SSD
defaults are 0 and 16. Reverted to the defaults, set explicitly so the stored
`ceph config` value is overwritten rather than merely un-declared.
prefer_deferred_size defers every write SMALLER than the threshold: it lands in
RocksDB's WAL first and is applied to the object store afterwards. That is a
double write, and at 64 KiB it caught essentially every write this cluster
makes -- SQLite pages are 4 KiB, Postgres pages 8 KiB. Ceph disables deferral on
SSD by default because solid state does not need the random-write batching that
helps spinning media.
Why the original tune was the wrong trade:
- The benchmark justifying it cannot have exercised it. The 2026-05-05
baseline used 4 MiB writes, 64x above the 64 KiB cutoff, so none of those
writes were ever deferred. The setting was validated by a workload it does
not affect.
- It doubles writes onto QLC, the lowest-endurance NAND type. Measured
2026-08-19 from the drives' own SMART counters: all three OSD devices at
80/82/87% endurance used, advancing ~0.13 points/day, i.e. rated life
exhausted between Nov 2026 and Jan 2027. Because size=3 on 3 OSDs, every
drive absorbs 100% of cluster writes, so they wear in lockstep and expire
together (see #434).
Measured amplification before this change: 640 GB/day written to the three NVMe
devices against 117 GB/day at the RBD client layer -- 5.5x, of which 3x is
replication and ~1.8x BlueStore. Removing the double write should take the
BlueStore factor toward ~1.1x.
The cluster is write-light, so throughput was never the binding constraint;
endurance is. Both parameters are runtime-updatable, so no OSD restart.
Also corrects the drive model in the comment: the OSD devices are
CT1000P3PSSD8 (Crucial P3 Plus), not the P2 named previously.
bdev_enable_discard / bdev_async_discard are unrelated and kept as-is.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the noise-log dedup, both of which silently mute real alerts.
1. Prose-mentioned fingerprints became suppressions.
The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons
routinely cite another alert's fingerprint in prose ("see the issue filed for
fingerprint <fp>"), so those citations were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints
for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert
that HAD been filed as a genuine issue, permanently muted purely because two
other entries mention it. Now only lines in the exact shape the write-back step
emits count:
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among
them.
2. Suppression never expired.
The fingerprint is sha256(alertname|scope) and is deliberately stable across
recurrences, so a single blip verdict muted that alertname+scope pair forever.
21 pairs were permanently muted, including openbao's KubePodNotReady,
KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real
openbao outage would have filed no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be
interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted
from the date already present in each line. Re-triaging a recurrence weeks later
is cheap; missing an outage because a July blip muted it is not. If it is still
noise, triage re-adds it automatically.
Rollout is gentle by construction: every current entry is dated 2026-07-21 or
later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge --
the oldest entries lapse a day later, and only re-triage if they actually fire
again.
Unparseable lines are dropped, i.e. they stop suppressing: this fails toward
more alerting, never less.
Verified end-to-end by running the shipped snippet inside the real
ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20,
21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox
date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch
arithmetic with `date -d @<epoch>`, which was confirmed working in that image.
Refs #326
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #516: the restart-triggering mosquitto_pub was unbounded,
unlike the sibling call in zigbee2mqtt-interview-watch.sh.
Neither client bounds a TCP connect that HANGS rather than being refused, and a
stalled connect is most likely precisely when this guard runs -- around a broker
restart. Unbounded, the loop would block forever and stop logging, which is the
silent-failure mode the guard exists to prevent.
Applied to the reads as well, not just the publish: mosquitto_sub -W bounds the
wait for a message once CONNECTED, so it does not cover the connect phase
either. The timeout wrapper is set above -W so the protocol-level wait still
wins in the normal case.
Not mosquitto_pub -W: that flag does not exist for mosquitto_pub and makes every
publish fail on a usage error.
The cooldown stamp was already written only on a successful publish, so a failed
restart request still retries on the next cycle rather than burning the cooldown.
`timeout` is confirmed present in eclipse-mosquitto:2.0.22 (already used by
interview-watch, and exercised directly in this image).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mosquitto runs `persistence false` with no PVC (deliberate and settled), so
every broker restart wipes the retained store. Zigbee2MQTT publishes its HA
discovery configs retained and has no way to restore them itself, so a broker
restart not followed by a Z2M restart leaves every Z2M entity unavailable in HA
indefinitely -- ~14 h on 2026-07-23.
Adds a discovery-guard sidecar to the Z2M pod that detects the wiped state and
asks Z2M to restart, which re-runs discover().
Detection. bridge/state and bridge/{info,devices,groups,definitions} are all
retained, but only bridge/state is republished on reconnect (mqtt.ts onConnect).
So "bridge/state present + bridge/groups absent" means Z2M is talking to this
broker but nothing has re-run start() since the store was wiped.
bridge/groups specifically, because the absent-half partner must not be
re-publishable by ordinary runtime events or it self-heals and the guard goes
blind while HA is still broken. bridge/devices, bridge/definitions and
bridge/info all reach inbound radio traffic via devicesChanged (receive.ts hands
deviceExposesChanged to every fromZigbee converter); bridge/health self-heals on
its own 10 min timer. bridge/groups is only republished by deliberate admin
actions, and publishGroups() always publishes, emitting `[]` when no groups
exist -- confirmed live: it currently holds a 2-byte payload, so presence is
testable.
Not AND-ed with bridge/info being absent. That would cut false positives at the
cost of false negatives, and the costs are asymmetric: a false positive is one
~60s Z2M restart, a false negative is every Zigbee entity dead until a human
notices. Tuned for sensitivity.
Spurious positives are handled by a 90s confirm re-check, which covers both the
startup race (bridge/state is published at mqtt.connect(), bridge/groups only at
the end of BridgeExtension.start()) and a dropped publish (these topics go out at
QoS 0). A 30 min cooldown keeps a Z2M that will not come up healthy from being
restarted in a loop.
Action is bridge/request/restart, which is a full in-process restart -- index.js
restart() -> stop(true) tears down every extension, then start() builds a new
Controller and re-runs HomeAssistant.start(). discover() then republishes only
missing or changed configs, which after a wipe is all of them.
Reuses the existing zigbee2mqtt broker credential (ACL `readwrite
zigbee2mqtt/#`), so no new broker user and no read grant on homeassistant/#,
which the ACL denies on purpose.
Verified in the eclipse-mosquitto:2.0.22 image the sidecar actually runs:
mosquitto_sub -C/-W, mosquitto_pub and `date -u +%s` all present; `sh -n` clean;
startup run logs correctly and enters the loop. Detection was exercised against
the live broker and correctly reports healthy (bridge/state online,
bridge/groups present) with no action.
NOTE: this rolls mosquitto, because the corrected ACL comment lives inside the
container command. That roll wipes the retained store, which the guard then
repairs within ~6 min if the Z2M roll does not already do so.
Closes#381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four comments across the broker manifests asserted that retained state is
re-populated by all clients on reconnect, and specifically that Zigbee2MQTT
re-publishes HA discovery when it sees the homeassistant/status birth message.
Both are false, and the second is the premise that kept issue #381 open.
Verified against Zigbee2MQTT 2.12.1 source:
- discover() runs only from HomeAssistant.start(); it is never called from
the reconnect path.
- lib/mqtt.ts onConnect() republishes bridge/state and resubscribes, nothing
more. republishRetainedTimer is armed only inside connect() and is never
re-armed on reconnect.
- lib/extension/homeassistant.ts onMQTTMessage(), on status_topic == "online",
republishes bridge/state plus cached device states after a hardcoded 30s
delay -- but never the discovery configs.
- Discovery is published retain:true, and the only knob is the global
mqtt.force_disable_retain, which upstream documents as breaking the HA
integration. Broker retention is the sole mechanism Z2M relies on.
- No bridge/request endpoint forces a rediscovery; upstream declined to add
one (Koenkk/zigbee2mqtt #27458, #22977, #16950, all closed NOT_PLANNED).
The mosquitto ACL was never the blocker: `read homeassistant/status` is already
granted. That grant is still correct -- the cached-state republish is what makes
HA show current values after an HA restart -- so it is kept and the comment now
says what it actually buys.
Also records that a RollingUpdate roll is equivalent to a broker restart for
retention: maxUnavailable:0/maxSurge:1 preserves connectivity, but the new pod
starts with an empty retained store.
Comment-only. The two deployment comments are plain YAML comments outside the
container command, so the rendered Deployment is byte-identical and the broker
does not roll; the ConfigMap carries no reloader annotation, so it updates in
place without restarting mosquitto.
Refs #381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stage 2 of the decommission the shim was written for. Deleting the file makes
the next apply destroy the three containers.
The containers were taken out of service months ago — they are stopped
(pve_up = 0 for all three) and rustfs-{1,2,3}.srv.renner.uno no longer resolve
— but the LXC objects and their disks still exist on saturn/jupiter/neptune.
Two earlier destroy attempts (#270) died instantly with "context deadline
exceeded", because the bpg provider reads timeout_delete from STATE and the
imported resources had stored 0.
That is now fixed: apply 5675 landed timeout_delete = 1800 and the rest of the
timeouts in state, in place, with 0 destroyed. The destroy has a real deadline
to work with for the first time.
Checked before removing, since the containers are tagged `state-backend` and
`terraform-backend`:
- The tofu backend is Hetzner S3 (renner-uno-tfstate at
fsn1.your-objectstorage.com), not rustfs. Destroying these cannot pull the
state out from under the apply. The tags are historical, from before that
migration.
- No Kubernetes or Ansible resource references rustfs.
- scripts/restore-authentik-db.sh mentions rustfs only as an `mc` alias name;
its endpoint is s3.renner.uno:9000, the Ceph RGW `migration` store.
- DNS for the three names is already gone.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Both of these were learned the expensive way today and existed only in commit
messages and PR descriptions, which is not where anyone looks.
providers.tf carried a warning that a UniFi PUT may require _id in the body,
which restapi does not send, and that a 400 should be answered by tainting the
object so CI recreates it. That is wrong on this controller. Tested against a
live rule with exactly the body tofu sends: HTTP 200, object returned intact,
rule byte-identical before and after, 60 rules before and after. Confirmed at
scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and
immich.renner.uno still 200.
That wrong warning had a cost. A note saying an apply might break firewall
rules is one nobody argues with, so the apply did not get run, and the
renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
while the WordPress backup failed nightly with "The specified bucket does not
exist". The warning is replaced with the evidence that disproves it.
unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and
cosmetic, and lists the four attempts to remove it with the plan number that
disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662),
restapi 3.x ignore_server_additions (5667), and create_returns_object (5677).
The first three cannot work in principle — `data` is a configured attribute, so
Terraform core diffs it against state before provider-side ignore logic runs.
The fourth was the better theory, that PUT responses re-polluted `data`, and it
is wrong too: the diff returned on the plan immediately after the apply. That
means the response is written into `data` on the read path, and settling it
needs the provider source rather than more guesses at its schema.
No behaviour change. Documentation only, so the next person inherits the
findings instead of the dead ends.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Root cause of the 60 permanently-dirty hairpin NAT rules.
The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.
No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:
ignore_changes_to = ["_id"] plan 5661 — 60 remained
ignore_all_server_changes = true plan 5662 — 60 remained
restapi 3.x + ignore_server_additions = true plan 5667 — 60 remained,
plus a new api_data diff
create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.
This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Plan 5657 rejected the previous commit:
Error: Missing required argument
on lxc_800_rustfs_decommission.tf line 98, in resource
"proxmox_virtual_environment_container" "rustfs_decommission":
The argument "template_file_id" is required, but no definition was found.
The plan's own state dump showed only `type = "debian"` inside
operating_system, so mirroring it looked complete. It is not:
template_file_id is create-time-only, PVE does not report the original
template for an existing container, and nothing else in opentofu/ declares
one to copy — these three are the only container resources in the repo.
Guessing a template would be a lie that stays invisible until something
recreates these containers, which is precisely the event this file exists to
prevent. So operating_system is left undeclared and ignored via lifecycle
instead, which stops it planning as a removal without asserting a value that
is not known.
The timeouts remain outside ignore_changes on purpose — they are the one
thing this shim needs to actually apply.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The v2 shim's own instructions say the plan MUST show an in-place update and
no replace before applying. It does not. Pipeline 5656 planned all three
containers as `must be replaced`, which would destroy and recreate LXCs
tagged critical / state-backend / terraform-backend instead of nudging a
timeout value into state.
Cause: v2 declared only node_name, vm_id, started, unprivileged and the
timeouts, on the stated assumption that "the other attributes mirror the real
containers so the plan stays in-place". They were not declared at all, so
every absent attribute planned as a removal — and two of those removals force
replacement:
- mount_point { ... } # forces replacement (size, volume)
- operating_system { ... } # forces replacement
Declaring the full attribute set, mirrored from the values the plan itself
reported out of state: cpu, memory, disk, mount_point, network_interface,
operating_system and initialization. hostname and mac_address differ per
container, so the local map carries them per key rather than being repeated
or guessed.
tags is declared for a separate reason: it was planning as
["critical","rustfs","state-backend","terraform-backend"] -> null, which does
not force replacement but would have stripped the tags off the live
containers on apply.
This does not unblock the WordPress backup on its own. That needs
renner-uno-k8s-mariadb, which the same plan wants to create — but the bucket
cannot be applied while the plan also carries three container replacements,
and a -target apply is not an acceptable way around it: it diverges state
from config and leaves the real problem in place.
Verification is the file's own gate — a plan showing in-place only.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Rebased onto master now that #511 landed the WOODPECKER_URL fix for both
rotators — that part is dropped from here rather than duplicated. What remains
is the half nothing else covered.
All four rules in prometheusrule-woodpecker-rotation.yaml were selecting
series that do not exist, so the whole group was silent:
- WoodpeckerServerNotReady queried kube_statefulset_status_replicas_ready.
The server became a Deployment when the sqlite->CNPG migration dropped its
PVC, so that metric has no series at all — a `critical` "all CI is down"
alert that could never fire, and had been dead well before the namespace
move. Now kube_deployment_status_replicas_ready in the woodpecker namespace.
- WoodpeckerAgentNotReady selected namespace="apps". Broken by the namespace
migration; mine.
- The two woodpecker-secret-rotate rules selected namespace="apps", but that
rotation runs from the openbao namespace. Broken since that move, which the
woodpecker kustomization documents.
Verified against live Prometheus instead of reasoned about: server ready=1,
agent available=1, cronjob 1 series. The originals returned nothing, which is
exactly why they never fired rather than firing wrongly.
Two stale claims in the same file corrected while here: it pointed at
kubernetes/apps/woodpecker/rotate-agent-secret.yaml, which no longer exists,
and asserted the CronJob/Job series "stay empty until the rotation feature is
merged" — that rotation has since landed in openbao.
Also finishes the netpol comments #511 left mid-update, which still credited
woodpecker-allow-rotators to `apps/*`. gitea.apps.svc:3000 in review-bot.yaml
is deliberately untouched: gitea really is still in apps.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
description:"The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
summary:"Ceph health check {{ $labels.name }} active for 6h"
description:"Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
- alert:CephPoolLostRedundancy
# Names the pool, which POOL_NO_REDUNDANCY does not.
#
# There is no pool-size metric, but bytes_used/stored IS the
# replication factor. The >1e6 floor drops pools too small to measure
# meaningfully, empty ones included.
#
# No pool is exempt.
expr:|
ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
for:30m
labels:
severity:critical
annotations:
summary:"Ceph pool {{ $labels.pool_id }} is running unreplicated"
description:"Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
- alert:CephHealthError
# HEALTH_ERR means data availability or redundancy is at risk.
summary:"CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
description:"{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
- alert:CNPGBackupFailed
# cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
# when no failure is recorded. Guard on > 0 so the epoch-zero default
summary:"CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
description:"A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
# DELIBERATELY ABSENT: a backup-age alert, which is the one you would
summary:"WordPress MariaDB backup has not succeeded in 26h+"
description:"The wordpress-mariadb-backup CronJob's last successful run is over 26 hours old. Check the newest Job's pod logs in the wordpress namespace: `kubectl -n wordpress get jobs`. The 2026-08-19 failure was `The specified bucket does not exist.` against renner-uno-k8s-mariadb while the endpoint itself probed healthy, so a green HetznerBackupEndpointDown does not clear this."
- alert:WordpressMariadbBackupNeverRan
# The staleness expression above goes blind when the metric is
# absent, which is what happens once the operator recreates the
# CronJob. Without this guard "never backed up" is quieter than
summary:"WordPress MariaDB backup has no recorded successful run"
description:"The wordpress-mariadb-backup CronJob exists but has no lastSuccessfulTime: either it has not completed once since the Backup CR was recreated, or every run since has failed. Check `kubectl -n wordpress get jobs` and that the hetzner-s3-creds ExternalSecret is synced."
# Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
# VLAN from this node (previously only k8s-edge had v30iot, so any
# IoT-pool LB lease landing here was unreachable from devices).
extra_network_bridges=["v30iot"]
disks=[
{
storage="local-zfs"
size="64G"
format="raw"
slot="virtio0"
file_id=local.talos_image
}
]
hostpci=[
{
mapping="osd-ssd"
},
{
mapping="x710-port1"
}
]
tags="kubernetes;talos;controlplane"
ssh_public_keys=local.common_ssh_keys
vm_state="running"
# Rebuild pipeline issues `-replace`; skip graceful shutdown so a
# non-responsive Talos guest can't hang the destroy for 30 minutes.
stop_on_destroy=true
}
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot
issue-fix-bot