fix(velero): bump data-mover timeouts 4h→8h (immich-library hits 4h cap) #86

2026-06-28T09:16:06Z

k.renner commented

2026-06-28 09:16:06 +00:00

Problem

The weekly restore-test CronJob in velero intermittently fails with KubeJobFailed and then "fixes itself".

Root cause is not the restore-test itself but the immich daily backup hitting Velero's 4h defaultItemOperationTimeout:

The immich-library volume (~250 GiB, millions of small files) now takes ~3h50m–4h+ to data-move.
When it crosses 4h, the DataUpload is Canceled at exactly the cap → no kopia snapshot is written → the daily backup ends PartiallyFailed.
The weekly restore-test selects "most recent Completed or PartiallyFailed" backup, picks the broken one, and the restore fails with Unable to load snapshot: snapshot not found → job exits 1 → KubeJobFailed.
Whenever a later day's upload finishes under 4h, the snapshot exists again → "fixes itself".

Evidence (immich-library DataUploads)

Date	Phase	Elapsed
2026-06-26	Canceled (no snapshot)	4h 00m 09s
2026-06-27	Canceled (no snapshot)	4h 00m 12s
2026-06-28	Completed	3h 50m 18s

Cancellations land at exactly 4h; the success was 10 min under. The volume is still growing, so it'll trip more often over time.

Change

Raise both dataMoverPrepareTimeout and defaultItemOperationTimeout from 4h → 8h in the Velero HelmRelease, with an updated comment explaining the cancellation mechanism.

Follow-up (not in this PR)

The restore-test's RESTORE_PHASE_TIMEOUT_S is still 4h; if a legit immich-library restore ever exceeds 4h it would false-fail. Worth bumping for symmetry.
The restore-test selecting PartiallyFailed backups makes a backup problem surface as a restore failure; it could skip backups whose specific PVC DataUpload didn't Complete.

🤖 Generated with Claude Code

## Problem The weekly `restore-test` CronJob in `velero` intermittently fails with **KubeJobFailed** and then "fixes itself". Root cause is **not** the restore-test itself but the **immich daily backup** hitting Velero's 4h `defaultItemOperationTimeout`: 1. The `immich-library` volume (~250 GiB, millions of small files) now takes ~3h50m–4h+ to data-move. 2. When it crosses 4h, the DataUpload is **Canceled** at exactly the cap → **no kopia snapshot is written** → the daily backup ends `PartiallyFailed`. 3. The weekly restore-test selects "most recent Completed **or PartiallyFailed**" backup, picks the broken one, and the restore fails with `Unable to load snapshot: snapshot not found` → job exits 1 → **KubeJobFailed**. 4. Whenever a later day's upload finishes under 4h, the snapshot exists again → "fixes itself". ### Evidence (immich-library DataUploads) | Date | Phase | Elapsed | |------|-------|---------| | 2026-06-26 | **Canceled** (no snapshot) | 4h 00m 09s | | 2026-06-27 | **Canceled** (no snapshot) | 4h 00m 12s | | 2026-06-28 | Completed | 3h 50m 18s | Cancellations land at *exactly* 4h; the success was 10 min under. The volume is still growing, so it'll trip more often over time. ## Change Raise both `dataMoverPrepareTimeout` and `defaultItemOperationTimeout` from `4h` → `8h` in the Velero HelmRelease, with an updated comment explaining the cancellation mechanism. ## Follow-up (not in this PR) - The restore-test's `RESTORE_PHASE_TIMEOUT_S` is still 4h; if a legit immich-library *restore* ever exceeds 4h it would false-fail. Worth bumping for symmetry. - The restore-test selecting `PartiallyFailed` backups makes a *backup* problem surface as a *restore* failure; it could skip backups whose specific PVC DataUpload didn't `Complete`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-28 09:16:06 +00:00

fix(velero): bump data-mover timeouts 4h→8h (immich-library hits 4h cap) e50c622da3

The immich-library DataUpload (~250 GiB, millions of small files) now
runs ~3h50m–4h+ and was intermittently Canceled at exactly the 4h
`defaultItemOperationTimeout`. A canceled upload writes no kopia
snapshot, so the daily backup ends PartiallyFailed; the weekly
restore-test then selects that backup and fails restoring immich-library
with "Unable to load snapshot: snapshot not found", firing KubeJobFailed.

Observed cancellations:
  2026-06-26  immich-library DataUpload  Canceled  4h00m09s
  2026-06-27  immich-library DataUpload  Canceled  4h00m12s
  2026-06-28  immich-library DataUpload  Completed 3h50m18s

Raise both dataMoverPrepareTimeout and defaultItemOperationTimeout to 8h
for headroom as the library keeps growing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

k.renner merged commit d8eaa7d4a5 into master

2026-06-28 09:17:31 +00:00

k.renner referenced this issue from a commit

2026-06-28 09:17:32 +00:00

Merge pull request 'fix(velero): bump data-mover timeouts 4h→8h (immich-library hits 4h cap)' (#86) from velero-bump-item-operation-timeout-8h into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#86