perf(cnpg): roll archive_timeout out to the remaining six clusters

A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.

gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.

Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.

ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.

ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.

Also corrects two comments in ha-pg-cluster.yaml that this work falsified.

wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.

checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.

Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.

kubernetes/apps/authentik/cluster.yaml

@@ -67,6 +67,20 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 3.2 GB/day here, most of it padding.
+      #
+      # 15min, NOT the 30min used elsewhere in this rollout. This cluster holds
+      # credentials, sessions and enrolment state, so its RPO is worth more than
+      # the difference: 30min would save a further 0.3 GB/day, which does not
+      # justify doubling the worst-case loss window on this data.
+      #
+      # It does carry a synchronous standby (instances: 2), so the realistic
+      # exposure is losing both instances, not one -- but the conservative value
+      # is cheap here and the aggressive one buys almost nothing.
+      archive_timeout: "15min"
     # Async replication makes CNPG switchover slow because the standby has
     # to chase missing WAL via barman-cloud and replay queued records
     # before it can promote. With one synchronous standby, the standby is

kubernetes/apps/grafana-db/postgres.yaml

@@ -53,6 +53,28 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # THE dominant write source on this cluster, by three orders of magnitude.
+      #
+      # A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
+      # switch on a timer so the archive never lags reality by more than that
+      # interval -- and the switched-out segment is written and archived at its
+      # full 16 MiB regardless of how few bytes of real WAL it contains.
+      #
+      # CNPG defaults this to 5min, i.e. up to 288 segments/day = 4.6 GB/day.
+      # Measured here: this database is 21 MB and generates 4.5 MB/day of actual
+      # WAL, while its RBD volume takes 211 MB/hour. Essentially every byte
+      # written is padding -- roughly 1100x amplification.
+      #
+      # 30min cuts that to 48 segments/day. The cost is RPO: on total loss of
+      # the cluster, up to 30min of Grafana state is unrecoverable rather than
+      # 5min. Grafana's state here is dashboards, annotations and preferences,
+      # and the write rate says it changes ~4.5 MB/day -- so that window is
+      # nearly always empty anyway.
+      #
+      # Applied to gf-pg FIRST, alone, as a staged rollout: it is the lowest-risk
+      # cluster and has the cleanest signal, since its real WAL is small enough
+      # that any change in volume is unambiguously this parameter.
+      archive_timeout: "30min"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -86,8 +86,26 @@ spec:
       # inserts scattered across pages, so full-page writes after each
       # checkpoint dominate WAL volume; compression cuts that directly.
       #
-      # Measured on this cluster: full-page images are 62.4% of WAL bytes, so
-      # this is the single largest lever on recorder write volume.
+      # Full-page images really are ~62% of WAL here: 61.9% measured as a
+      # steady-state delta over one checkpoint cycle, against 62.4% lifetime.
+      #
+      # But switching pglz -> zstd changed NOTHING. Measured by
+      # pg_current_wal_lsn advance over 20min windows: 4911 and 4976 MB/day
+      # before, 5002 MB/day after.
+      #
+      # The reason is that `wal_compression: "on"` was never "no compression" --
+      # it is pglz, which was already compressing these page images. zstd only
+      # changes the ratio, and on recorder pages that margin is too small to
+      # measure. The original framing of this change (that the clusters "were
+      # not getting compression") was wrong; they were.
+      #
+      # Note the 62% figure is an upper bound: it is computed as
+      # wal_fpi * 8192 / wal_bytes, which assumes uncompressed 8 KiB pages. With
+      # compression active the true byte share is lower, which is consistent
+      # with there being little headroom left for a better algorithm.
+      #
+      # Kept because zstd is not worse than pglz and costs nothing. It is simply
+      # not a lever. The real one on this cluster was archive_timeout, below.
       #
       # zstd, not "on": the boolean spelling resolves to pglz, the original
       # 2016-era algorithm, which is both slower and weaker than the two
@@ -95,9 +113,35 @@ spec:
       # best ratio on 8 KiB page images, and the CPU cost lands on a cluster
       # that is IO-bound on QLC NVMe, not CPU-bound.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 4.6 GB/day that is largely padding.
+      #
+      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
+      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
+      # Two independent signals moving together is what confirms the volume was
+      # padding rather than data.
+      #
+      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
+      # its volume are lost outright. This is a single-instance cluster, so the
+      # archive is the only off-node copy.
+      archive_timeout: "30min"
       # Fewer checkpoints => fewer full-page writes, since a page is only
-      # fully-logged on its first modification after a checkpoint. Doubling the
-      # interval halves how often each hot page pays that 8 KiB toll.
+      # fully-logged on its first modification after a checkpoint.
+      #
+      # That reasoning did not show up in WAL volume: 15min -> 30min moved the
+      # WAL rate not at all (4911/4976 -> 5002 MB/day). The likely reason is
+      # that this workload is append-mostly -- a page is dirtied while it fills
+      # and then not touched again, so its full-page image is paid once
+      # regardless of how often checkpoints run. Interval only helps pages that
+      # are re-dirtied across many cycles, which is not what a recorder does.
+      #
+      # Kept, but NOT because WAL improved. checkpoint_timeout also governs
+      # data-file writes at checkpoint, and those are unmeasured here: this
+      # volume takes ~11 GB/day against 4.9 GB/day of WAL, so most of its
+      # traffic is outside what the WAL measurement covers. Revisit with a
+      # data-file measurement before treating this value as justified.
       #
       # Recovery cost is bounded by how much WAL a single interval produces,
       # not by the interval itself: at the measured rate 30min is well under

kubernetes/apps/immich/cluster.yaml

@@ -46,6 +46,20 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 4.6 GB/day that is largely padding.
+      #
+      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
+      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
+      # Two independent signals moving together is what confirms the volume was
+      # padding rather than data.
+      #
+      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
+      # its volume are lost outright. This is a single-instance cluster, so the
+      # archive is the only off-node copy.
+      archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.
       #

kubernetes/apps/paperless/cluster.yaml

@@ -58,6 +58,18 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 4.1 GB/day that is largely padding.
+      #
+      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
+      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
+      #
+      # The cost is RPO: up to 30min of WAL is unrecoverable if both instances
+      # are lost. The documents themselves live on the media PVC, not in
+      # Postgres -- this database holds their metadata.
+      archive_timeout: "30min"
     # See ak-pg cluster.yaml for the rationale on synchronous replication.
     synchronous:
       method: any

kubernetes/apps/trmnl/cluster.yaml

@@ -36,6 +36,20 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 4.6 GB/day that is largely padding.
+      #
+      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
+      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
+      # Two independent signals moving together is what confirms the volume was
+      # padding rather than data.
+      #
+      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
+      # its volume are lost outright. This is a single-instance cluster, so the
+      # archive is the only off-node copy.
+      archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.
       #

kubernetes/apps/woodpecker/cluster.yaml

@@ -80,6 +80,20 @@ spec:
       # and slower than the lz4/zstd this build also accepts). Compresses the
       # full-page images that dominate WAL volume after each checkpoint.
       wal_compression: "zstd"
+      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
+      # on a timer -- the segment is then written and archived at its full size
+      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
+      # to 288 segments/day = 4.6 GB/day that is largely padding.
+      #
+      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
+      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
+      # Two independent signals moving together is what confirms the volume was
+      # padding rather than data.
+      #
+      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
+      # its volume are lost outright. This is a single-instance cluster, so the
+      # archive is the only off-node copy.
+      archive_timeout: "30min"
   resources:
     requests:
       cpu: 100m