From 0a9d19a7c6f44c33d77d1048fd9f2b53f900508f Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 20 Aug 2026 14:21:03 +0200 Subject: [PATCH] perf(grafana-db): raise archive_timeout to 30min on gf-pg A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment switch on a timer so the archive never trails reality by more than that interval, and the switched-out segment is written and archived at its full 16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to 5min, which permits up to 288 segments/day = 4.6 GB/day per cluster. Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted from 12 switches of 16 MiB. Essentially every byte written to this volume is padding, roughly 1100x amplification. This resolves an anomaly that has been open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it. Segment padding does, exactly. Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster and gives the cleanest signal -- its real WAL is small enough that any change in volume is unambiguously attributable to this parameter. The same default affects all seven clusters and is worth an estimated ~24 GB/day of logical writes fleet-wide, but that only goes ahead once this one is verified. The cost is RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences, changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated separately rather than inheriting this value by default. archive_timeout is in CNPG's GlobalDefaultSettings, not its FixedConfigurationParameters, so it is user-overridable -- verified against pkg/postgres/configuration.go rather than assumed. Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear. --- kubernetes/apps/grafana-db/postgres.yaml | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/kubernetes/apps/grafana-db/postgres.yaml b/kubernetes/apps/grafana-db/postgres.yaml index c4026196..dc2e098e 100644 --- a/kubernetes/apps/grafana-db/postgres.yaml +++ b/kubernetes/apps/grafana-db/postgres.yaml @@ -53,6 +53,28 @@ spec: # and slower than the lz4/zstd this build also accepts). Compresses the # full-page images that dominate WAL volume after each checkpoint. wal_compression: "zstd" + # THE dominant write source on this cluster, by three orders of magnitude. + # + # A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment + # switch on a timer so the archive never lags reality by more than that + # interval -- and the switched-out segment is written and archived at its + # full 16 MiB regardless of how few bytes of real WAL it contains. + # + # CNPG defaults this to 5min, i.e. up to 288 segments/day = 4.6 GB/day. + # Measured here: this database is 21 MB and generates 4.5 MB/day of actual + # WAL, while its RBD volume takes 211 MB/hour. Essentially every byte + # written is padding -- roughly 1100x amplification. + # + # 30min cuts that to 48 segments/day. The cost is RPO: on total loss of + # the cluster, up to 30min of Grafana state is unrecoverable rather than + # 5min. Grafana's state here is dashboards, annotations and preferences, + # and the write rate says it changes ~4.5 MB/day -- so that window is + # nearly always empty anyway. + # + # Applied to gf-pg FIRST, alone, as a staged rollout: it is the lowest-risk + # cluster and has the cleanest signal, since its real WAL is small enough + # that any change in volume is unambiguously this parameter. + archive_timeout: "30min" resources: requests: cpu: 100m -- 2.54.0