#539 never reached master. I based it on the reduce-pg-wal-fpi feature branch instead of master, so merging it merged into a branch that had itself already been merged. The commit was stranded — git merge-base --is-ancestor confirms it is not an ancestor of master, and the live cluster still reports archive_timeout = 300.
This is the same commit, cherry-picked onto master. My error, not a review problem.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment switch on a timer so the archive never trails reality by more than that interval — and the switched-out segment is written and archived at its full 16 MiB regardless of how few bytes of real WAL it contains.
CNPG defaults this to 5min, permitting up to 288 segments/day = 4.6 GB/day, per cluster.
This resolves an anomaly open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it.
Measured, not inferred
gf-pg
database size
21 MB
actual WAL generated
4.5 MB/day
RBD volume writes
211 MB/hour
predicted from 12 switches x 16 MiB
192 MB/hour
Agreement within 10%. Essentially every byte written to this volume is padding — roughly 1100x amplification.
Why gf-pg alone
Staged rollout. Lowest-risk cluster, and the cleanest signal available: its real WAL is small enough that any change in volume is unambiguously this parameter and nothing else.
The same default affects all seven clusters, worth an estimated ~24 GB/day of logical writes fleet-wide — but that only proceeds once this one is measured. I will report actual before/after rather than assuming it generalises.
The cost
RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated individually rather than inheriting this.
Verification
archive_timeout is in CNPG GlobalDefaultSettings, not FixedConfigurationParameters, so it is user-overridable — checked against pkg/postgres/configuration.go, not assumed. (archive_mode and archive_commandare fixed; archive_timeout is not.)
Confirmed already-merged #538 did apply: gf-pg now reports wal_compression = zstd live.
## Why this exists
**#539 never reached master.** I based it on the `reduce-pg-wal-fpi` feature branch instead of `master`, so merging it merged into a branch that had itself already been merged. The commit was stranded — `git merge-base --is-ancestor` confirms it is not an ancestor of master, and the live cluster still reports `archive_timeout = 300`.
This is the same commit, cherry-picked onto master. My error, not a review problem.
## The finding (unchanged from #539)
A WAL segment is a fixed **16 MiB file**. `archive_timeout` forces a segment switch on a timer so the archive never trails reality by more than that interval — and the switched-out segment is written and archived **at its full 16 MiB regardless of how few bytes of real WAL it contains**.
CNPG defaults this to `5min`, permitting up to 288 segments/day = 4.6 GB/day, per cluster.
This resolves an anomaly open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it.
## Measured, not inferred
| | gf-pg |
|---|---|
| database size | 21 MB |
| actual WAL generated | 4.5 MB/day |
| RBD volume writes | **211 MB/hour** |
| predicted from 12 switches x 16 MiB | 192 MB/hour |
Agreement within 10%. Essentially every byte written to this volume is padding — roughly **1100x amplification**.
## Why gf-pg alone
Staged rollout. Lowest-risk cluster, and the cleanest signal available: its real WAL is small enough that any change in volume is unambiguously this parameter and nothing else.
The same default affects all seven clusters, worth an estimated **~24 GB/day of logical writes fleet-wide** — but that only proceeds once this one is measured. I will report actual before/after rather than assuming it generalises.
## The cost
RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated individually rather than inheriting this.
## Verification
`archive_timeout` is in CNPG `GlobalDefaultSettings`, not `FixedConfigurationParameters`, so it is user-overridable — checked against `pkg/postgres/configuration.go`, not assumed. (`archive_mode` and `archive_command` *are* fixed; `archive_timeout` is not.)
Confirmed already-merged #538 did apply: gf-pg now reports `wal_compression = zstd` live.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.
Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.
This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.
Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.
The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.
archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 0a9d19a7c6f44c33d77d1048fd9f2b53f900508f.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR re-targets a performance optimization for the Grafana database PostgreSQL cluster that was previously stranded in #539. The change raises archive_timeout from 5min (default) to 30min to reduce WAL segment padding, cutting write amplification by 6x (from ~4.6 GB/day to ~800 MB/day logical output, measured on gf-pg). The tradeoff is an increase in RPO from 5min to 30min, which is acceptable for this cluster since its data (dashboards, annotations) changes infrequently (~4.5 MB/day). No blocking issues.
🟢 Nits
Comment formatting consistency: Line 56 has an extra leading # compared to the comment block at lines 32-34, creating a subtle visual discontinuity. The multi-line comment at 56-77 uses proper alignment, so no actual issue — just a minor style note if you standardize.
Tests
This is a Kubernetes manifest change rather than application code. Testing will be observational: monitor RBD volume write rates and WAL metrics on gf-pg after deployment to confirm the predicted ~192 MB/hour reduction materializes. The PR description correctly identifies gf-pg as ideal for initial measurement due to its small real WAL volume (4.5 MB/day), making any change in storage writes unambiguous.
Recommendation
Looks reasonable — no blocking issues; the change is well-measured and conservatively scoped.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `0a9d19a7c6f44c33d77d1048fd9f2b53f900508f`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR re-targets a performance optimization for the Grafana database PostgreSQL cluster that was previously stranded in #539. The change raises `archive_timeout` from 5min (default) to 30min to reduce WAL segment padding, cutting write amplification by 6x (from ~4.6 GB/day to ~800 MB/day logical output, measured on gf-pg). The tradeoff is an increase in RPO from 5min to 30min, which is acceptable for this cluster since its data (dashboards, annotations) changes infrequently (~4.5 MB/day). No blocking issues.
## 🟢 Nits
**Comment formatting consistency:** Line 56 has an extra leading `#` compared to the comment block at lines 32-34, creating a subtle visual discontinuity. The multi-line comment at 56-77 uses proper alignment, so no actual issue — just a minor style note if you standardize.
## Tests
This is a Kubernetes manifest change rather than application code. Testing will be observational: monitor RBD volume write rates and WAL metrics on gf-pg after deployment to confirm the predicted ~192 MB/hour reduction materializes. The PR description correctly identifies gf-pg as ideal for initial measurement due to its small real WAL volume (4.5 MB/day), making any change in storage writes unambiguous.
## Recommendation
Looks reasonable — no blocking issues; the change is well-measured and conservatively scoped.
<!-- pr-review:sha=0a9d19a7c6f44c33d77d1048fd9f2b53f900508f -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
Why this exists
#539 never reached master. I based it on the
reduce-pg-wal-fpifeature branch instead ofmaster, so merging it merged into a branch that had itself already been merged. The commit was stranded —git merge-base --is-ancestorconfirms it is not an ancestor of master, and the live cluster still reportsarchive_timeout = 300.This is the same commit, cherry-picked onto master. My error, not a review problem.
The finding (unchanged from #539)
A WAL segment is a fixed 16 MiB file.
archive_timeoutforces a segment switch on a timer so the archive never trails reality by more than that interval — and the switched-out segment is written and archived at its full 16 MiB regardless of how few bytes of real WAL it contains.CNPG defaults this to
5min, permitting up to 288 segments/day = 4.6 GB/day, per cluster.This resolves an anomaly open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it.
Measured, not inferred
Agreement within 10%. Essentially every byte written to this volume is padding — roughly 1100x amplification.
Why gf-pg alone
Staged rollout. Lowest-risk cluster, and the cleanest signal available: its real WAL is small enough that any change in volume is unambiguously this parameter and nothing else.
The same default affects all seven clusters, worth an estimated ~24 GB/day of logical writes fleet-wide — but that only proceeds once this one is measured. I will report actual before/after rather than assuming it generalises.
The cost
RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated individually rather than inheriting this.
Verification
archive_timeoutis in CNPGGlobalDefaultSettings, notFixedConfigurationParameters, so it is user-overridable — checked againstpkg/postgres/configuration.go, not assumed. (archive_modeandarchive_commandare fixed;archive_timeoutis not.)Confirmed already-merged #538 did apply: gf-pg now reports
wal_compression = zstdlive.🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit0a9d19a7c6f44c33d77d1048fd9f2b53f900508f.Summary
This PR re-targets a performance optimization for the Grafana database PostgreSQL cluster that was previously stranded in #539. The change raises
archive_timeoutfrom 5min (default) to 30min to reduce WAL segment padding, cutting write amplification by 6x (from ~4.6 GB/day to ~800 MB/day logical output, measured on gf-pg). The tradeoff is an increase in RPO from 5min to 30min, which is acceptable for this cluster since its data (dashboards, annotations) changes infrequently (~4.5 MB/day). No blocking issues.🟢 Nits
Comment formatting consistency: Line 56 has an extra leading
#compared to the comment block at lines 32-34, creating a subtle visual discontinuity. The multi-line comment at 56-77 uses proper alignment, so no actual issue — just a minor style note if you standardize.Tests
This is a Kubernetes manifest change rather than application code. Testing will be observational: monitor RBD volume write rates and WAL metrics on gf-pg after deployment to confirm the predicted ~192 MB/hour reduction materializes. The PR description correctly identifies gf-pg as ideal for initial measurement due to its small real WAL volume (4.5 MB/day), making any change in storage writes unambiguous.
Recommendation
Looks reasonable — no blocking issues; the change is well-measured and conservatively scoped.