svc-atlasandClaude Opus 5.5
8ef039f8a6
Owner decision on REN-240 (protection rule 5): - README.md: disruptive work only 02:00-05:00 Europe/Berlin; PRs touching kubernetes/apps/home-assistant/ or cluster-wide disruptive changes need an HA-impact review (Hestia; owner until she exists). - kubernetes/RUNBOOK-home-assistant-image-rollback.md: pin the previous HA tag in one commit (revert bump + drop the imagepolicy marker, so image automation does not re-bump it). Docs only. No manifest changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.0 KiB
Runbook — roll back a Home Assistant image bump
Use when an automatic Home Assistant update (a
chore(image-automation): bump ghcr.io/home-assistant/home-assistant:<new>
commit on master) breaks Home Assistant. The owner approves the rollback
and its timing. Atlas writes the PR; Argus merges it.
Why a plain git revert is not enough
Image automation (kubernetes/infrastructure/image-automation/) re-scans
the registry every hour and rewrites the tag every 30 minutes to the
highest release matching the home-assistant ImagePolicy. A plain revert
of the bump commit would be bumped forward again within about an hour.
So the rollback commit does two things at once: it reverts the tag and
drops the # {"$imagepolicy": ...} marker from that image line. Without
the marker the Setters strategy has nothing to rewrite, so the old tag
stays pinned. It is still one commit, and reverting that commit restores
automation.
Before you roll back
- Find the bump commit:
git log --oneline -5 origin/master -- kubernetes/apps/home-assistant/deployment.yaml - Check the release notes of the broken version for a database schema
change. Home Assistant migrates the recorder schema (CNPG cluster
ha-pg) on start and cannot downgrade it. An older version started on a newer schema runs without recorder history until the schema matches again. If the bump crossed a schema change, tell the owner before going on: the choice is roll back without history, or restore the recorder database from backup as a separate step.
Roll back (one commit, one PR)
git worktree add -b atlas/ha-rollback-<old> "$WT" origin/master && cd "$WT"
git revert --no-commit <bump-sha>
# In kubernetes/apps/home-assistant/deployment.yaml, delete the trailing
# # {"$imagepolicy": "flux-system:home-assistant"}
# from the home-assistant image line. Leave every other line alone.
git diff --cached --stat # expect: 1 file, 1 line changed
git commit -m "fix(home-assistant): pin <old>, pause image automation (bump to <new> broke HA)"
git push -u origin HEAD
The PR description states: broken version, pinned version, symptom, schema check result, and "revert this commit to resume automatic updates". HA-impact review and merge follow README.md.
On merge Flux applies the Deployment (strategy Recreate): Home Assistant
restarts once, about one to two minutes of downtime.
Verify
kubectl -n home-assistant get deploy home-assistant \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="home-assistant")].image}'
kubectl -n home-assistant rollout status deploy/home-assistant
Read-only commands only (see REN-240 protection rule 1). One hour after
merge, confirm no new chore(image-automation) commit touched
deployment.yaml.
Resume automatic updates
When a fixed Home Assistant release is out, revert the rollback commit. The marker comes back, and image automation bumps to the newest release on its next pass. That bump is itself a Home Assistant restart.