One Velero Restore per Schedule (kubernetes/restore/<schedule>/), applied
by Flux through a restore-<schedule> Kustomization
(clusters/homelab/restore.yaml), restoring the schedule's newest completed
backup. Every app Kustomization with backed-up volumes depends on its own
restore Kustomization, so on a fresh cluster no app starts on an empty
volume, and no app waits for another app's data. This reverses the
decision on #200 (option B, 952e6150): that removed a restore Job that was
never wired in, while the seed-* rsync Kustomizations were still the
bootstrap path. Both are gone; restoring from Velero on a fresh cluster is
now wanted.
Gitea goes first: it is the registry for the in-house images. The gitea
Kustomization depends on restore-gitea alone; every other restore
Kustomization depends on restore-gitea, so Velero moves Gitea's data
first. Each app depends on its own: apps, audiobookshelf (libation through
it), authentik, home-assistant (matter-server, home-assistant-iot, otbr,
whisper too, same namespace and schedule), immich, influxdb, matrix,
music-assistant, navidrome, paperless, trmnl, wordpress. Timeouts follow the
data: immich 6h, audiobookshelf 3h, the rest 1h. One shared gate made
authentik wait for immich's photo library (45 minutes in restore rehearsal
run 28) and held a kustomize-controller worker all that time.
Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
for good (restore_controller.go). The restore Kustomizations therefore
depend on velero-backups-synced, which is Ready once the
BackupStorageLocation has completed a sync (status.lastSyncedTime), and
which depends on the velero Kustomization (the BackupStorageLocation and
its credentials).
- After that sync, FailedValidation with "No backups found for schedule"
means there is nothing to restore and counts as done; any other
FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
exists"), and the Restore completes with a warning. On the running
cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
a consuming pod.
The velero-schedules header points at the restores instead of the manual
runbook, and bootstrap.sh's wait-apps comment says the volumes come from
Velero. wait-apps waits for gitea, then apps, up to an hour each (the
restore-gitea timeout), since both now include restores.
RUNBOOK-cold-rebuild.md Step 5 describes the restores and how to retry a
failed one: a Restore never retries, and while it is failed every
Kustomization depending on its restore Kustomization stops applying.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-botandClaude Opus 5.5
93cb53130c