claude-botandClaude Opus 5
2797f64bb0
The velero e2e check manufactured phantom PartiallyFailed Backups that then
polluted backup-health triage. Sequence, from Loki for e2e-1785031248:
02:01:05 Backup + PVC created (DataUpload -> phase Prepared)
... data path never starts (node-agent slot held elsewhere)
~02:06 the 60x5s = 300s wait expires -> check FAIL, but does NOT abort
-> script deletes ns velero-e2e and creates the Restore
02:07:03 restore error "no DataUpload result cm found" -> PartiallyFailed
02:10:08 script deletes Backup+Restore -> "Cancel du under phase Prepared
because it is being deleted" -> Backup lands PartiallyFailed
Both 2026-07-26 occurrences (02:00:55 and 12:04:05) fall inside a running
restore-test sweep. node-agent-config.yaml sets no loadConcurrency, so Velero's
default of one data path per node applies and the sweep's data-mover pods hold
the slot — a 1 GiB e2e DataUpload then sits in Prepared well past 5 minutes.
The backup was never actually faulty; the test cancelled it.
Three changes:
- Wait 120x15s (30 min) instead of 60x5s (5 min), so a queued DataUpload has
time to run once the slot frees.
- Guard the destructive half (namespace teardown + restore roundtrip) behind
whether the backup check actually passed. Tearing down while the upload is
in flight is what cancels it.
- Do not delete the Backup on the failure path either — that is the same race
relocated to cleanup. The Backup carries ttl: 2h0m0s, so Velero's GC reaps
it regardless and leaving it costs nothing.
Cleanup of the restore, namespace and released PVs stays unconditional, so
nothing leaks. Verified with `bash -n`.
Found while triaging #427; the restore-test failure there was unrelated and
is already fixed.
Refs #427
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>