The SHA tag was a workaround for not being able to compute a date in a
shell-less build step. That problem is gone — the build timestamp now
lives in the image at /etc/image-build-date — so the tag no longer has
to carry it.
A SHA reads as noise at the point of use and breaks the existing
convention every consumer pin follows (ci-k8s:2026-08-10,
ci-ansible:2026-04-29, alpine-jq:2026-07-30). Dated literals keep the
pins uniform and are bumped in the commit that changes an image, which
is the same trigger the `when:` path filters already use.
Completes the pattern across every Containerfile in the repo. ci-claude
was the last one without it.
It was skipped initially over the reproducibility work in a675bf6c, on a
misreading. That commit did not make the whole image reproducible — it
made the npm install layer byte-stable so a rebuild of the same version
dedups in the registry instead of orphaning ~260 MB (#336, #351).
A timestamp added as its own final layer leaves the npm layer untouched,
so that dedup still holds. The cost is a few dozen bytes per build, not
260 MB.
The date was previously computed by `date` in the pipeline step and used
as the image tag, which is why the build step needed a shell and hence
the kaniko debug image.
It does not need to cross between CI steps at all. The base images have
a shell, so a RUN in the Containerfile writes the timestamp during the
build itself — kaniko executes the layer and the value lands in the
image. Nothing has to be propagated, and the build step stays
shell-less.
This also separates two things the tag was conflating: the tag says
which source revision was built, /etc/image-build-date says when.
Placed last so it invalidates only its own layer. secret-rotator writes
it in the rust build stage and copies it across, since its final image
is FROM scratch and has no shell.
Verified against ci/images/alpine-jq:
$ docker run --rm localtest:jq cat /etc/image-build-date
2026-08-23T18:56:51Z
Note this makes builds non-reproducible byte-for-byte, which a675bf6c
deliberately arranged for ci-claude. ci-claude is built by
ci-claude-refresh.yml and is not touched here.
The `-debug` image ships a busybox shell so a failing build can be
inspected. It was being used as the build image because the steps run
shell to assemble registry auth and compute a tag, which is not what
that variant is for.
The plain executor has no shell, so the eight static builds now pass
flags via `entrypoint` and read credentials from DOCKER_CONFIG. A single
registry-auth step writes `.docker/config.json` into the workspace once,
in alpine, where a shell is the intended interface.
Images are tagged `${CI_COMMIT_SHA:0:8}` instead of a build date.
Woodpecker exposes no formatted-date variable — every timestamp built in
is UNIX — and kaniko takes `--destination` only as a literal argument,
with all its file-based flags being outputs. The commit SHA is available
through normal substitution and identifies the source revision the image
was built from, which a build date does not.
Existing dated tags stay in the registry, so the pins that reference
them keep resolving; only newly created tags change format.
Also moves off v1.23.2, picking up the go-git fix for CVE-2025-21613 and
containerd 1.7.18 -> 1.7.27.
ci-claude-refresh.yml keeps its shell step: its destination tag is a
runtime value read from a file written by an earlier step, and the build
is conditionally skipped, neither of which survives a static entrypoint.
Left for #552.
Needs a pipeline run to confirm that substitution applies to
`entrypoint` and `environment` values, and that the auth file written by
registry-auth is visible to the build steps.
Refs #426, #552
Comments should say what the setting does and what constrains it. The
blocks added in #542, #544, #545 and #547 had grown into incident
write-ups: measurement runs, prior values, how a problem was found,
what a check used to be. That belongs in commit messages and the wiki,
both of which already carry it.
Trimmed, no behaviour change:
cephcluster.yaml 22 lines -> 6 (cephx rotation)
cephfs-ci.yaml 26 lines -> 7 (why size 2)
prometheusrule-ceph.yaml (CephHealthWarn, 43 -> 13;
CephPoolLostRedundancy, 15 -> 7)
The operative facts are kept: which checks are excluded and why they
cannot clear, that POOL_NO_REDUNDANCY reports blocked maintenance and
not only data risk, and that size 1 needs dedicated OSDs.
The unreplicated CI pools introduced in #542 make every OSD on the
cluster permanently un-restartable.
`ceph osd ok-to-stop` refuses while stopping an OSD would take PGs
offline. A size-1 pool puts every PG on exactly one OSD, so the check
fails for all three:
Error EBUSY: unsafe to stop osd(s) at this time
(20 PGs are or would become offline)
bad_become_inactive: 13.6 13.9 ... 14.1c 14.1d
Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2
pools are not implicated — they run min_size 1 and appear under
ok_become_degraded, which is allowed.
Rook gates OSD restarts on that check, so this blocks CephX key
rotation, Ceph upgrades, node drains and reboots, and drive
replacement. It surfaced when #545 deadlocked: Rook retried
ok-to-stop against all three OSDs several times a second and could
never succeed, while `ceph -s` reported 217 pgs active+clean
throughout. Nothing is unhealthy; the restarts are simply refused.
The reasoning in #542 was that unreplicated CI data only risks
in-flight pipelines, since every byte is a fresh git clone. That part
holds. What it missed is that the cost is not scoped to the pool: one
unreplicated pool freezes maintenance for the entire cluster.
Most of #542's saving came from keeping CI churn off `cephfs`, whose
metadataPool is size 3 — that survives this change. Only the second-
order gain from dropping the replica is given up, and cephfs-ci-data0
stores 0 B with cephfs-ci-metadata at 182 MiB, so the backfill is
trivial.
size 1 becomes viable again once these pools sit on OSDs of their own,
where the permanently un-stoppable OSD holds nothing but throwaway
data. That is planned with the retired QLC drives.
Alerting follows the same change:
- POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was
excluded because cephfs-ci was expected to keep it permanently
active; no pool is unreplicated now, so the exclusion would only
hide a real one losing its replicas.
- CephPoolLostRedundancy drops its cephfs-ci exemption for the same
reason, and its description now states the maintenance consequence,
not just the data-loss one.
- The AUTH_INSECURE_* exclusion is reworded. Calling it
"known-accepted" reads as a decision that the risk is fine; it is a
kernel-blocked remediation (aes256k over the CSI mount path needs
kernel 7.0+, nodes run 6.18).
Refs #542, #545, #527
Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.
The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.
Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.
Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.
CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.
Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.
Refs #527, #528
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CephHealthWarn alerted on `ceph_health_status == 1`. It has been FIRING since
at least 2026-08-21T00:26Z and cannot clear: four AUTH_INSECURE_* checks are
permanently active from CVE-2025-30156 (#527) and stay that way until Rook
>=1.20.5 lands. A permanently-firing alert reports nothing -- every Ceph
warning since then has been invisible behind it.
This is a pre-existing gap, not a consequence of the cephfs-ci work, though
that work would have added a second permanent cause (POOL_NO_REDUNDANCY, since
the CI pools are size 1 on purpose).
ceph_health_detail exposes each check as its own series, so the alert now fires
on any check that is NOT known-accepted, and names it. Verified against live
data: 5 checks active, 0 unexpected, so it is quiet now and fires the moment
something genuinely new appears.
Excluding POOL_NO_REDUNDANCY wholesale would give something up: that check is
cluster-wide and does not say WHICH pool, so accepting it for cephfs-ci would
equally hide a production pool dropping to one replica. CephPoolLostRedundancy
restores that. There is no pool-size metric, but bytes_used/stored IS the
replication factor -- verified live as replicapool 2, cephfs-data0 2,
cephfs-metadata 3, cephfs-ci-metadata 1.
Both alerts were tested against live data before committing, in BOTH
directions. The positive control matters as much as the negative one: an early
draft used a >1e9 floor which silently excluded the CI pools entirely, so the
expression could not have matched anything. It looked correct precisely because
it returned no results. The control caught it; the floor is now 1e6.
The floor is not what suppresses the near-empty rgw pools. Their ratios are
9-36 because per-object overhead dominates, so they never match `< 1.5`.
6h on CephHealthWarn is unchanged -- Ceph warns for self-clearing conditions
and paging on those trains people to ignore the alert.
Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.
Measured 2026-08-21, polling both CephFS pools every 15s:
CI active 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB
CI idle 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB
CI share of writes: data0 100%, metadata 61%
data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.
Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.
RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.
Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.
The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.
cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.
Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.
one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
every interval with no CI pod: data0 ~0 metadata ~0
So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.
size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.
`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.
Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.
Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.
The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.
gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.
Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.
ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.
ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.
Also corrects two comments in ha-pg-cluster.yaml that this work falsified.
wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.
checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.
Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.
This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.
Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.
The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.
archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
`wal_compression: "on"` resolves to pglz, PostgreSQL's original 2016-era
algorithm. Every CNPG cluster here was set that way, so none of them were
getting the compression actually available to them: all seven builds accept
lz4 and zstd (verified via pg_settings.enumvals on each, including immich's
custom pgvector image and trmnl's PostgreSQL 18).
This matters because of what dominates recorder-style write volume. A page
is written to WAL in full -- all 8 KiB -- on its first modification after a
checkpoint, for torn-page protection. Measured on ha-pg, those full-page
images are 62.4% of WAL bytes. Compressing them is therefore the single
largest lever available on WAL volume, and pglz was the weakest option for
the job.
ha-pg additionally moves checkpoint_timeout 15min -> 30min. Full-page images
are emitted once per page per checkpoint interval, so doubling the interval
roughly halves how often each hot page pays the 8 KiB toll. max_wal_size
stays at 2GB deliberately: it must remain well above what one interval
generates, or checkpoints would trigger on size and the interval change
would do nothing. Recovery replays at most one interval of WAL -- under
200MB at the measured rate, i.e. seconds.
The other six clusters keep checkpoint_timeout at 15min. Their measured
write rates do not justify trading recovery time: gf-pg generates 4.5 MB/day
of WAL and wp-pg 62 MB/day, against ak-pg's 604 MB/day.
full_page_writes stays on everywhere. Disabling it would remove the
overhead entirely but RBD can tear a page across an object boundary, and
the trade is write volume against an unrecoverable database.
Both parameters are sighup-reloadable, so this does not restart any cluster.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
An image reference is not a secret. Encoding it as base64 `data` had two real
costs: no reviewer sees a version change in a diff, and no $imagepolicy marker
can attach to a base64 blob.
The result was a two-minor version skew in the component that performs every
base backup and WAL archive for seven Postgres clusters (#535):
controller plugin-barman-cloud:v0.14.0 <- auto-bumped by Flux
sidecar plugin-barman-cloud-sidecar:v0.12.0 <- invisible, never moved
Not stale pods: ha-pg-1 was created hours ago during the recorder migration and
still received v0.12.0.
Three changes:
1. The Secret now uses stringData with the tag in plaintext, carrying an
$imagepolicy marker. The Secret KIND is kept because the controller reads it
via secretKeyRef; switching to a ConfigMap would need a matching edit to that
reference and diverge further from upstream. stringData is accepted
identically by the API server.
2. ImageRepository + ImagePolicy added for the sidecar image, with the same
filter and 0.x range as the controller, so the two cannot drift apart again.
Only the controller was tracked before.
3. Tag aligned to v0.14.0 to match the controller.
The Secret name carries a kustomize hash suffix from upstream's secretGenerator
but is static here, so editing the value does not change the name and the
deployment's secretKeyRef stays valid — verified in the render, which still shows
the name twice (Secret + reference).
Verified the tag EXISTS before pinning it: pulling
plugin-barman-cloud-sidecar:v0.14.0 succeeds (the pod fails later on a missing
`true` binary in that minimal image, which is the container command, not the
pull). Pinning a non-existent tag would have put all seven Postgres clusters into
ImagePullBackOff.
EXPECTED IMPACT: bumping the sidecar changes every Postgres pod spec, so all
seven CNPG clusters roll. CNPG handles this as a normal rolling restart. Backups
should be confirmed working afterwards — 113 Backup CRs exist today, newest ~3h
old and completed.
Note for future re-imports: upstream ships this Secret as base64 `data` with no
marker, so a wholesale re-import silently reintroduces the skew. The manifest now
says so inline.
Refs #535
Co-Authored-By: Claude Bot <noreply@anthropic.com>
Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.
The copy is already done and verified byte-for-byte with paperless stopped:
src files: 75
dst files: 75
IDENTICAL: name, size and mtime match for every file
index segments: src 3, dst 3
mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)
Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.
cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.
paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.
It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:
paperless-data client request_load_avg 22 (all other sessions 0-1)
MDS over 180s 4,052 client requests = 22.5/s
2,718 journal events = 15/s
2 dir commits <- not file writes
Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.
cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.
PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).
Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten Postgres instances across seven CNPG clusters are ~48 GB/day of the
cluster's ~114 GB/day of client writes, and only ha-pg carried any
write-reduction tuning. The rest ran Postgres defaults, which favour throughput
over storage that wears out.
MEASURED, not assumed. ak-pg since 2026-05-17:
checkpoints_timed 27,624 = 291/day, i.e. one every 5 min (the default)
buffers_checkpoint 25,801,628 = ~206 GB flushed
database size 1,046 MB
That is the whole database rewritten roughly twice a day. The pathology is
specific to idle databases: sessions, tokens and job state re-dirty the same few
pages constantly, and checkpoint_timeout=5min forces each of them to disk 288
times a day instead of letting them coalesce. At 15min they are written a third
as often.
max_wal_size is raised alongside it. Without that, checkpoints trigger on size
instead and the timeout change does nothing. It also bounds crash recovery to at
most 2GB of WAL replay.
Applied to ak-pg, pprl-pg, tr-pg, wp-pg, im-pg, gf-pg. ha-pg already had it.
synchronous_commit is deliberately NOT changed here. It is off for ha-pg because
a state recorder is telemetry; these clusters hold credentials, documents and job
state, where losing the last few hundred ms of commits on an unclean shutdown is
not an acceptable trade for write volume.
Separately, ha-pg shared_buffers 256MB -> 512MB. After 15h on Postgres its
buffers_backend was 629,704 of 939,025 total page writes -- 67% written by
backends rather than the checkpointer or bgwriter, which means the working set
does not fit in the buffer pool and pages are written out repeatedly instead of
coalescing. Memory limit raised 1Gi -> 1536Mi to keep the same headroom (measured
383Mi in use, of which 256MB was shared_buffers).
Verified before commit by rendering every cluster and asserting the tuning landed
under spec.postgresql.parameters with the barman plugin block untouched. That
check exists because the first attempt at this matched the wrong `parameters:`
key -- the barman plugin's, at 6-space indent under plugins: -- and corrupted
wp-pg's barmanObjectName. Reverted and redone with an indent-aware match.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer was the single largest writer in the cluster -- more than every
Ceph OSD and every database combined -- and none of it needed to touch disk.
Measured 2026-08-20 from the container's own cgroup io.stat:
253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day
252:48 (its own PVC) 340 MB total = negligible
So the writes were not going to its volume at all; they were going to the
node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without
`medium: Memory` is node storage, which is easy to read as "ephemeral, so
free".
The cause is cadence rather than volume. schedules.template.json renders on
"* * * * *", so Chrome is launched 1440 times a day and each launch creates,
fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp
holds under 8 MB -- it is pure churn for data that never outlives the render.
tmpfs pages count against the container's memory limit, so the 512Mi cap is
kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample
headroom even if the cap were fully consumed.
Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a
4h window. This is a bigger cluster-wide reduction than the Home Assistant
recorder migration achieved, for a one-line change.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.
CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.
CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".
DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.
The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.
Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.
Refs #522
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 15:43:07 +02:00
33 changed files with 947 additions and 158 deletions
description:"The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
summary:"Ceph health check {{ $labels.name }} active for 6h"
description:"Ceph health check {{ $labels.name }} has been active for six hours, so it is not a self-clearing blip. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527), so this is a new condition. Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
- alert:CephPoolLostRedundancy
# Names the pool, which POOL_NO_REDUNDANCY does not.
#
# There is no pool-size metric, but bytes_used/stored IS the
# replication factor. The >1e6 floor drops pools too small to measure
# meaningfully, empty ones included.
#
# No pool is exempt.
expr:|
ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
for:30m
labels:
severity:critical
annotations:
summary:"Ceph pool {{ $labels.pool_id }} is running unreplicated"
description:"Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
- alert:CephHealthError
# HEALTH_ERR means data availability or redundancy is at risk.
summary:"CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
description:"{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
- alert:CNPGBackupFailed
# cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
# when no failure is recorded. Guard on > 0 so the epoch-zero default
summary:"CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
description:"A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
# DELIBERATELY ABSENT: a backup-age alert, which is the one you would
# Matches runAsUser on the rotation Jobs. There is no /etc/passwd in a scratch
# image, so this is a bare uid by necessity — which is also all Kubernetes needs.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-bot
k.renner
security-bot