fix: [alert] KubePodCrashLooping (home-assistant) (#1082) #1086

2026-10-01T03:44:26Z

issue-fix-bot commented

2026-10-01 03:44:26 +00:00

What this PR does

zone-configurator crashlooped because the planned quarterly rotation of the Home Assistant LLATs revoked the token it was using. Nobody revoked it by hand, and the stored token is not bad. prometheus-token-secret-rotate ran at 2026-10-01T02:00:00Z (04:00 Europe/Berlin). Its order is mint → store in OpenBao → revoke the old token in HA. The home-assistant-prometheus-token Secret only picks up the new value on its next ESO refresh (1h interval, which came at ~02:35Z). zone-configurator reads that Secret as a startup env var and, unlike the other LLAT consumers, has no Reloader annotation. So for about 35 minutes it ran with a revoked token. The unhandled WebSocket auth rejection killed the process each time, which is the crashloop. The Prometheus scrape (401) failed for the same reason. The problem fixed itself once ESO synced the new token: the next container restart read it. This PR adds secret.reloader.stakater.com/reload: home-assistant-prometheus-token to the two unannotated consumers of that Secret: zone-configurator and speech-to-phrase. That matches what home-assistant and shelly-manager already do for their rotated tokens, so the next quarterly rotation (2027-01-01) restarts them right after the Secret syncs.

Changes

kubernetes/apps/zone-configurator/deployment.yaml: add a Reloader annotation for home-assistant-prometheus-token.
kubernetes/apps/speech-to-phrase/deployment.yaml: same annotation. This deployment also reads the token as an env var (HASS_TOKEN → --hass-token), and it is still running with the revoked token right now (see Verification).
kubernetes/infrastructure/openbao/rotation/home-assistant.yaml: comment only. The consumer list for prometheus-token now names these two env consumers.

Verification

Checked live this run (around 03:42Z):

kubectl -n openbao get jobs -l app.kubernetes.io/component=secret-rotator: prometheus-token-secret-rotate-29847000 started 02:00:00Z and succeeded. config-sync-token-… ran at 02:05Z and shelly-manager-llat-… at 02:10Z.
Loki {namespace="home-assistant",container="home-assistant"}: the first Login attempt or request with invalid authentication … '/api/websocket' appears at 04:00:07 local (02:00:07Z), seconds after the rotation. Before that there were no auth failures.
kubectl -n home-assistant get externalsecret …: all three HA token ExternalSecrets refreshed at :35 (latest 03:35Z), so the post-rotation sync was ~02:35Z.
kubectl -n home-assistant get rs: a new home-assistant ReplicaSet appeared at 02:35:39Z. shelly-manager is also 67m old. Both are Reloader-annotated and rolled on that sync, as designed.
zone-configurator-69c586bd7d-k4rtv is now 1/1 Running, 16 restarts (71m ago). The last failure was at 02:31:36Z, with --previous logs showing Error: Auth invalid: Invalid access token or password. The current container serves probes normally.
Prometheus up{job="home-assistant"} = 1 again.
speech-to-phrase-66bc576865-wtk88: age 7d9h, 0 restarts, so its HASS_TOKEN env still holds the token that was revoked at 02:00Z.

After merge and Flux reconcile: kubectl -n home-assistant get deploy zone-configurator speech-to-phrase -o yaml should show the annotation. On the next rotation, both pods should roll within seconds of the ESO refresh, and no KubePodCrashLooping should fire. Note that adding a metadata-level annotation does not restart the pods by itself.

Notes

The issue's diagnosis was wrong in two places. (1) The token was not revoked by hand and no new token needs to be created. The rotator replaced it automatically, and OpenBao/ESO already hold a valid token (scrape up=1, zone-configurator healthy). The issue's three manual steps are not needed. (2) "Restarts reuse the value it got at creation" is incorrect: secretKeyRef env vars are resolved at each container start, which is why zone-configurator recovered on its own after the 02:35Z sync.
Recommended manual action: run kubectl -n home-assistant rollout restart deploy/speech-to-phrase once now. It still holds the revoked token, so its next retrain or HA connection will fail. This PR's annotation only covers future rotations.
Even with Reloader, there is still a gap of up to 1h (ESO refreshInterval) between revoke and sync, during which consumers and the scrape get 401s. A zone-configurator crashloop that short would usually clear before the alert's for: window. The issue's alerts fired because there was no restart trigger and the gap ran ~35 min. Closing the gap completely would need the rotator to force-sync the ExternalSecret, or a shorter refresh interval. That is a larger design change, deliberately left out here.
The issue's suggestion to give zone-configurator a separate token (instead of reusing the Prometheus scrape token) is still a reasonable hardening step. It needs a new OpenBao path, ExternalSecret, rotation CronJob and policy entry, so it is out of scope for a surgical fix.

Closes #1082

## What this PR does `zone-configurator` crashlooped because the planned quarterly rotation of the Home Assistant LLATs revoked the token it was using. Nobody revoked it by hand, and the stored token is not bad. `prometheus-token-secret-rotate` ran at 2026-10-01T02:00:00Z (04:00 Europe/Berlin). Its order is mint → store in OpenBao → revoke the old token in HA. The `home-assistant-prometheus-token` Secret only picks up the new value on its next ESO refresh (1h interval, which came at ~02:35Z). `zone-configurator` reads that Secret as a startup env var and, unlike the other LLAT consumers, has no Reloader annotation. So for about 35 minutes it ran with a revoked token. The unhandled WebSocket auth rejection killed the process each time, which is the crashloop. The Prometheus scrape (401) failed for the same reason. The problem fixed itself once ESO synced the new token: the next container restart read it. This PR adds `secret.reloader.stakater.com/reload: home-assistant-prometheus-token` to the two unannotated consumers of that Secret: `zone-configurator` and `speech-to-phrase`. That matches what `home-assistant` and `shelly-manager` already do for their rotated tokens, so the next quarterly rotation (2027-01-01) restarts them right after the Secret syncs. ## Changes - `kubernetes/apps/zone-configurator/deployment.yaml`: add a Reloader annotation for `home-assistant-prometheus-token`. - `kubernetes/apps/speech-to-phrase/deployment.yaml`: same annotation. This deployment also reads the token as an env var (`HASS_TOKEN` → `--hass-token`), and it is **still running with the revoked token right now** (see Verification). - `kubernetes/infrastructure/openbao/rotation/home-assistant.yaml`: comment only. The consumer list for `prometheus-token` now names these two env consumers. ## Verification Checked live this run (around 03:42Z): - `kubectl -n openbao get jobs -l app.kubernetes.io/component=secret-rotator`: `prometheus-token-secret-rotate-29847000` started 02:00:00Z and succeeded. `config-sync-token-…` ran at 02:05Z and `shelly-manager-llat-…` at 02:10Z. - Loki `{namespace="home-assistant",container="home-assistant"}`: the first `Login attempt or request with invalid authentication … '/api/websocket'` appears at 04:00:07 local (02:00:07Z), seconds after the rotation. Before that there were no auth failures. - `kubectl -n home-assistant get externalsecret …`: all three HA token ExternalSecrets refreshed at :35 (latest 03:35Z), so the post-rotation sync was ~02:35Z. - `kubectl -n home-assistant get rs`: a new `home-assistant` ReplicaSet appeared at 02:35:39Z. `shelly-manager` is also 67m old. Both are Reloader-annotated and rolled on that sync, as designed. - `zone-configurator-69c586bd7d-k4rtv` is now `1/1 Running, 16 restarts (71m ago)`. The last failure was at 02:31:36Z, with `--previous` logs showing `Error: Auth invalid: Invalid access token or password`. The current container serves probes normally. - Prometheus `up{job="home-assistant"}` = `1` again. - `speech-to-phrase-66bc576865-wtk88`: age 7d9h, **0 restarts**, so its `HASS_TOKEN` env still holds the token that was revoked at 02:00Z. After merge and Flux reconcile: `kubectl -n home-assistant get deploy zone-configurator speech-to-phrase -o yaml` should show the annotation. On the next rotation, both pods should roll within seconds of the ESO refresh, and no `KubePodCrashLooping` should fire. Note that adding a metadata-level annotation does not restart the pods by itself. ## Notes - **The issue's diagnosis was wrong in two places.** (1) The token was not revoked by hand and no new token needs to be created. The rotator replaced it automatically, and OpenBao/ESO already hold a valid token (scrape `up=1`, zone-configurator healthy). The issue's three manual steps are **not needed**. (2) "Restarts reuse the value it got at creation" is incorrect: `secretKeyRef` env vars are resolved at each container start, which is why zone-configurator recovered on its own after the 02:35Z sync. - **Recommended manual action:** run `kubectl -n home-assistant rollout restart deploy/speech-to-phrase` once now. It still holds the revoked token, so its next retrain or HA connection will fail. This PR's annotation only covers future rotations. - Even with Reloader, there is still a gap of up to 1h (ESO `refreshInterval`) between revoke and sync, during which consumers and the scrape get 401s. A `zone-configurator` crashloop that short would usually clear before the alert's `for:` window. The issue's alerts fired because there was no restart trigger and the gap ran ~35 min. Closing the gap completely would need the rotator to force-sync the ExternalSecret, or a shorter refresh interval. That is a larger design change, deliberately left out here. - The issue's suggestion to give zone-configurator a separate token (instead of reusing the Prometheus scrape token) is still a reasonable hardening step. It needs a new OpenBao path, ExternalSecret, rotation CronJob and policy entry, so it is out of scope for a surgical fix. Closes #1082

issue-fix-bot added 1 commit 2026-10-01 03:44:27 +00:00

fix: [alert] KubePodCrashLooping (home-assistant) (#1082 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

747ebbb50c

issue-fix-bot referenced this pull request

2026-10-01 03:44:27 +00:00

[alert] KubePodCrashLooping (home-assistant) #1082

review-bot reviewed

2026-10-01 03:46:55 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: unknown), commit 747ebbb50c29f67cc2e94a020c9ba4903dca2b9f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds the secret.reloader.stakater.com/reload: home-assistant-prometheus-token annotation to zone-configurator and speech-to-phrase so both restart when the shared HA Prometheus LLAT rotates, plus a comment-only update to the rotation CronJob file documenting the new consumers. The change is small, correctly scoped, and matches the existing idiom used elsewhere in the repo (e.g. kubernetes/apps/home-assistant/deployment.yaml:16, kubernetes/apps/woodpecker/server.yaml:12, kubernetes/apps/uptime-monitor/deployment.yaml:28). No blocking issues found.

🟡 Suggestions

kubernetes/apps/speech-to-phrase/deployment.yaml — the PR body states this pod is currently running with the already-revoked token (0 restarts, 7d9h old) and recommends a manual kubectl rollout restart now, since the annotation only prevents the next rotation's gap. That's correct but worth flagging again here: merging this PR alone does not fix the live incident for speech-to-phrase; the manual restart is a separate, out-of-band step the merger needs to remember to run.
Both deployments inherit the pre-existing ~1h window between HA revoking the old token and the ExternalSecret's refreshInterval: 1h syncing the new one (confirmed in kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml:7). Reloader fixes the "never restarts" problem but not this residual sync-lag window, which the PR body acknowledges and explicitly defers — reasonable to leave out of a surgical fix, just noting it's still open.

Tests

No automated tests apply to this kind of change (GitOps manifest/annotation diff); the repo doesn't have unit tests for Kustomize manifests beyond whatever ci-manifests-test.yml builds/validates. The PR author instead did live verification against the cluster (CronJob run times, Loki auth-failure logs, ExternalSecret refresh timestamps, pod restart counts) documented in the PR body, which is appropriate rigor for this type of fix. Worth confirming ci-manifests-test.yml passes on this PR (kustomize build / schema validation) before merge, but I have no reason to expect it to fail — the YAML is well-formed and the annotation key/value follow the same pattern used successfully elsewhere in the repo.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: unknown), commit `747ebbb50c29f67cc2e94a020c9ba4903dca2b9f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds the `secret.reloader.stakater.com/reload: home-assistant-prometheus-token` annotation to `zone-configurator` and `speech-to-phrase` so both restart when the shared HA Prometheus LLAT rotates, plus a comment-only update to the rotation CronJob file documenting the new consumers. The change is small, correctly scoped, and matches the existing idiom used elsewhere in the repo (e.g. `kubernetes/apps/home-assistant/deployment.yaml:16`, `kubernetes/apps/woodpecker/server.yaml:12`, `kubernetes/apps/uptime-monitor/deployment.yaml:28`). No blocking issues found. ## 🟡 Suggestions - `kubernetes/apps/speech-to-phrase/deployment.yaml` — the PR body states this pod is *currently* running with the already-revoked token (0 restarts, 7d9h old) and recommends a manual `kubectl rollout restart` now, since the annotation only prevents the *next* rotation's gap. That's correct but worth flagging again here: merging this PR alone does not fix the live incident for `speech-to-phrase`; the manual restart is a separate, out-of-band step the merger needs to remember to run. - Both deployments inherit the pre-existing ~1h window between HA revoking the old token and the ExternalSecret's `refreshInterval: 1h` syncing the new one (confirmed in `kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml:7`). Reloader fixes the "never restarts" problem but not this residual sync-lag window, which the PR body acknowledges and explicitly defers — reasonable to leave out of a surgical fix, just noting it's still open. ## Tests No automated tests apply to this kind of change (GitOps manifest/annotation diff); the repo doesn't have unit tests for Kustomize manifests beyond whatever `ci-manifests-test.yml` builds/validates. The PR author instead did live verification against the cluster (CronJob run times, Loki auth-failure logs, ExternalSecret refresh timestamps, pod restart counts) documented in the PR body, which is appropriate rigor for this type of fix. Worth confirming `ci-manifests-test.yml` passes on this PR (kustomize build / schema validation) before merge, but I have no reason to expect it to fail — the YAML is well-formed and the annotation key/value follow the same pattern used successfully elsewhere in the repo. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

~~Ghost referenced this pull request 2026-10-01 05:23:44 +00:00~~

fix: [alert] TargetDown (home-assistant) (#1084) #1087

Ghost closed this pull request

2026-10-01 05:23:45 +00:00

~~Ghost referenced this pull request 2026-10-01 05:23:45 +00:00~~

fix: [alert] HomeAssistantDown (home-assistant) (#1081) #1083

hermes-bot referenced this pull request

2026-10-21 07:34:04 +00:00

fix(zone-configurator,speech-to-phrase): restart on HA prometheus-token rotation (#1082) #1409

hermes-bot referenced this pull request

2026-10-21 07:34:32 +00:00

[alert] KubePodCrashLooping (home-assistant) #1082