perf(velero): run the restore-test monthly instead of weekly #565

2026-08-25T13:41:57Z

claude-bot commented

2026-08-25 13:41:57 +00:00

The Velero restore-test sweep restores every discovered schedule, immich's ~180 GiB library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per week — ~47% of the endurance budget against a ~65 GB/day weekday baseline.

That baseline is already the product of the 2026-08-18..21 write-reduction work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs, paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive writes ~70% from ~195 GB/day. With the application tier down, the restore test is now the single largest consumer of remaining QLC endurance; the three OSD drives are at 81/82/87% used.

Monthly still exercises every restore path and cuts the sweep's share of drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.

Change (cadence only)

0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the 03:15-07:15 daily backup window.
ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still inspectable when the next one fires.
Comments and docs updated to match, including the BLUESTORE_SLOW_OP_ALERT runbook text that tells the reader to check whether a stall fell inside the restore-test window.

No day-of-week form is available: Kubernetes parses cron with robfig/cron's standard parser, whose day-of-week field allows only * / , - ? (no Quartz # nth-weekday), and restricting both day fields ORs them.

The Velero restore-test sweep restores every discovered schedule, immich's ~180 GiB library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per week — ~47% of the endurance budget against a ~65 GB/day weekday baseline. That baseline is already the product of the 2026-08-18..21 write-reduction work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs, paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive writes ~70% from ~195 GB/day. With the application tier down, the restore test is now the single largest consumer of remaining QLC endurance; the three OSD drives are at 81/82/87% used. Monthly still exercises every restore path and cuts the sweep's share of drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB. ## Change (cadence only) - `0 8 * * 0` -> `0 8 1 * *`, keeping the 08:00 UTC slot that clears the 03:15-07:15 daily backup window. - `ttlSecondsAfterFinished` 14 d -> 40 d, so the previous run is still inspectable when the next one fires. - Comments and docs updated to match, including the `BLUESTORE_SLOW_OP_ALERT` runbook text that tells the reader to check whether a stall fell inside the restore-test window. No day-of-week form is available: Kubernetes parses cron with robfig/cron's standard parser, whose day-of-week field allows only `* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields ORs them.

claude-bot added 1 commit 2026-08-25 13:41:58 +00:00

perf(velero): run the restore-test monthly instead of weekly

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

44aa0d6919

The sweep restores every discovered Velero schedule, immich's ~180 GiB
library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and
measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per
week — ~47% of the endurance budget, against a ~65 GB/day weekday
baseline.

That baseline is itself the result of the 2026-08-18..21 write-reduction
work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs,
paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive
writes ~70% from ~195 GB/day. With the application tier down, the restore
test is now the single largest consumer of remaining QLC endurance; the
three OSD drives are at 81/82/87% used.

Monthly still exercises every restore path and cuts the sweep's share of
drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.

Cadence-only change:
- 0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the
  03:15-07:15 daily backup window (see the comment in the manifest).
- ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still
  inspectable when the next one fires.
- Comments and docs that state the cadence updated to match, including
  the BLUESTORE_SLOW_OP_ALERT runbook text, which tells the reader to
  check whether a stall fell inside the restore-test window.

No day-of-week form is available: Kubernetes parses cron with
robfig/cron's standard parser, whose day-of-week field allows only
`* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields
ORs them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 47c1170d51 into master

2026-08-25 13:42:56 +00:00

k.renner referenced this issue from a commit

2026-08-25 13:42:56 +00:00

Merge pull request 'perf(velero): run the restore-test monthly instead of weekly' (#565) from velero-restore-test-monthly into master

review-bot reviewed

2026-08-25 13:45:05 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 44aa0d691914ee5fd08a52e394d89df15955baa9.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Cadence-only change: the Velero restore-test CronJob moves from weekly (0 8 * * 0, Sundays) to monthly (0 8 1 * *, the 1st), ttlSecondsAfterFinished is bumped from 14 d to 40 d to match, and all comments/docs/alert-runbook text referencing the old weekly/Sunday cadence are updated in lockstep. No application logic, RBAC, or script behavior changes. This is a low-risk, well-documented change and I found no blocking issues.

🟡 Suggestions

kubernetes/RECOVERY.md:245 — "Automating this as a weekly CronJob (mirroring the Velero one)…" is now stale: the Velero job it refers to mirroring is monthly as of this PR. The PR updated the other RECOVERY.md mention (line 170) but missed this one further down in the same file.

🟢 Nits

The arithmetic in the PR description (~740 GB/drive/week → ~530 GB/week after the ~4x cut to the sweep's share) doesn't quite reconcile with the other stated figures (~350 GB/week attributed to the sweep, ~390 GB/week baseline) — using those numbers, monthly-averaged weekly writes work out closer to ~480 GB than ~530 GB. This is PR-description text only, not part of any committed file, so it doesn't affect the change itself.

Verification performed

schedule: "0 8 1 * *" (restore-test-cronjob.yaml:43) is valid 5-field cron: minute 0, hour 8, day-of-month 1, any month, any weekday — fires once a month at 08:00 UTC on the 1st, as documented.
ttlSecondsAfterFinished: 3456000 (restore-test-cronjob.yaml:53) = 40 × 86400 s, matches the "40 d" comment.
The new in-file comment's claim that Kubernetes' cron parser (robfig/cron) has no # nth-weekday operator and ORs day-of-month/day-of-week when both are restricted is accurate standard cron semantics, and correctly explains why day-of-month (1) rather than a weekday-based expression was used.
Grepped the repo for lingering weekly/Sunday/old schedule string (0 8 * * 0) and old TTL (1209600) references outside this diff — none found except the one flagged above and one unrelated CI runbook line about "weekly CronJob" for CNPG (out of scope, tracked separately per #190).
prometheusrule-ceph.yaml's updated window text ("08:00-14:00Z on the 1st") is consistent with activeDeadlineSeconds: 21600 (6 h) on the CronJob, which is unchanged by this PR.
No RBAC, script logic, or resource permissions changed — restore-test-rbac.yaml and restore-test.sh only had comment text updated.

Tests

No tests apply — this is a cron schedule/comment/doc change to existing infrastructure YAML with no new logic. Nothing further to add; the change is inherently observable only by watching the next scheduled run.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `44aa0d691914ee5fd08a52e394d89df15955baa9`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Cadence-only change: the Velero restore-test CronJob moves from weekly (`0 8 * * 0`, Sundays) to monthly (`0 8 1 * *`, the 1st), `ttlSecondsAfterFinished` is bumped from 14 d to 40 d to match, and all comments/docs/alert-runbook text referencing the old weekly/Sunday cadence are updated in lockstep. No application logic, RBAC, or script behavior changes. This is a low-risk, well-documented change and I found no blocking issues. ## 🟡 Suggestions - `kubernetes/RECOVERY.md:245` — "Automating this as a **weekly** CronJob (mirroring the Velero one)…" is now stale: the Velero job it refers to mirroring is monthly as of this PR. The PR updated the other RECOVERY.md mention (line 170) but missed this one further down in the same file. ## 🟢 Nits - The arithmetic in the PR description (~740 GB/drive/week → ~530 GB/week after the ~4x cut to the sweep's share) doesn't quite reconcile with the other stated figures (~350 GB/week attributed to the sweep, ~390 GB/week baseline) — using those numbers, monthly-averaged weekly writes work out closer to ~480 GB than ~530 GB. This is PR-description text only, not part of any committed file, so it doesn't affect the change itself. ## Verification performed - `schedule: "0 8 1 * *"` (`restore-test-cronjob.yaml:43`) is valid 5-field cron: minute 0, hour 8, day-of-month 1, any month, any weekday — fires once a month at 08:00 UTC on the 1st, as documented. - `ttlSecondsAfterFinished: 3456000` (`restore-test-cronjob.yaml:53`) = 40 × 86400 s, matches the "40 d" comment. - The new in-file comment's claim that Kubernetes' cron parser (robfig/cron) has no `#` nth-weekday operator and ORs day-of-month/day-of-week when both are restricted is accurate standard cron semantics, and correctly explains why day-of-month (`1`) rather than a weekday-based expression was used. - Grepped the repo for lingering `weekly`/`Sunday`/old schedule string (`0 8 * * 0`) and old TTL (`1209600`) references outside this diff — none found except the one flagged above and one unrelated CI runbook line about "weekly CronJob" for CNPG (out of scope, tracked separately per #190). - `prometheusrule-ceph.yaml`'s updated window text ("08:00-14:00Z on the 1st") is consistent with `activeDeadlineSeconds: 21600` (6 h) on the CronJob, which is unchanged by this PR. - No RBAC, script logic, or resource permissions changed — `restore-test-rbac.yaml` and `restore-test.sh` only had comment text updated. ## Tests No tests apply — this is a cron schedule/comment/doc change to existing infrastructure YAML with no new logic. Nothing further to add; the change is inherently observable only by watching the next scheduled run. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#565