konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Restore test 7/9: the cluster restores its app volumes itself #1303

Closed
claude-bot wants to merge 1 commits from split/07-velero-self-restore into split/06-hardware-gating
pull from: split/07-velero-self-restore
merge into: :split/06-hardware-gating
Conversation 2 Commits 1 Files Changed 43
+568 -5
1 Commits
Author SHA1 Message Date
claude-botandClaude Opus 5.5 e82f55bbc7 feat(velero): the cluster restores its app volumes itself, Gitea first
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot reviewed: comment
Details 
One Velero Restore per Schedule (kubernetes/restore/<schedule>/), applied
by Flux through a restore-<schedule> Kustomization
(clusters/homelab/restore.yaml), restoring the schedule's newest completed
backup. Every app Kustomization with backed-up volumes depends on its own
restore Kustomization, so on a fresh cluster no app starts on an empty
volume, and no app waits for another app's data. Restores the capability
952e6150 removed without an owner decision.

Gitea goes first: it is the registry for the in-house images. The gitea
Kustomization depends on restore-gitea alone; every other restore
Kustomization depends on restore-gitea, so Velero moves Gitea's data
first. Each app depends on its own: apps, audiobookshelf (libation through
it), authentik, home-assistant (matter-server, home-assistant-iot, otbr,
whisper too, same namespace and schedule), immich, influxdb, matrix,
music-assistant, navidrome, paperless, trmnl, wordpress. Timeouts follow the
data: immich 6h, audiobookshelf 3h, the rest 1h. One shared gate made
authentik wait for immich's photo library (45 minutes in restore rehearsal
run 28) and held a kustomize-controller worker all that time.

Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
  for good (restore_controller.go). The restore Kustomizations therefore
  depend on velero-backups-synced, which is Ready once the
  BackupStorageLocation has completed a sync (status.lastSyncedTime), and
  which depends on the velero Kustomization (the BackupStorageLocation and
  its credentials).
- After that sync, FailedValidation with "No backups found for schedule"
  means there is nothing to restore and counts as done; any other
  FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
  exists"), and the Restore completes with a warning. On the running
  cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
  Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
  a consuming pod.

The velero-schedules header points at the restores instead of the manual
runbook, and bootstrap.sh's wait-apps comment says the volumes come from
Velero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 11:46:32 +02:00