fix: [alert] HomeAssistantDown (home-assistant) (#1081) #1083

2026-10-01T02:44:11Z

issue-fix-bot commented

2026-10-01 02:44:11 +00:00

What this PR does

HomeAssistantDown fired at about 02:05Z on 2026-10-01 because Home Assistant was rejecting the Prometheus scrape with a 401 error. Home Assistant itself was not down. At 02:00Z the quarterly prometheus-token-secret-rotate job minted a new long-lived access token (LLAT), stored it in OpenBao, and then revoked the old one right away. The home-assistant-prometheus-token ExternalSecret only syncs every 1h, though, so until about 02:35Z the Kubernetes Secret still held the revoked token. The ServiceMonitor scrape failed with 401 during that window, and zone-configurator, which reads the same token as a startup env var, went into a crashloop. The rotator's revoke-immediately behaviour lives in secret-rotator/src/main.rs, which is outside this agent's edit scope. This PR instead shrinks the window on the consumer side:

All three rotated LLAT ExternalSecrets now refresh every 1m. That keeps the dead-token window under the alert's for: 5m.
The two Deployments that read the token at startup get a Reloader annotation, so they restart when the Secret changes.

Changes

kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml — refreshInterval: 1h → 1m, with a comment explaining why.
kubernetes/apps/home-assistant/config-sync-token-externalsecret.yaml — 1h → 1m. Its CronJob also revokes immediately (04:05 Berlin), so it has the same gap. The HA deployment already has a Reloader annotation, but that only fires once ESO actually syncs.
kubernetes/apps/home-assistant/shelly-manager-llat-externalsecret.yaml — 1h → 1m, for the same reason (04:10 Berlin rotation).
kubernetes/apps/zone-configurator/deployment.yaml — added secret.reloader.stakater.com/reload: home-assistant-prometheus-token. It reads the token once as HA_LONG_LIVED_TOKEN.
kubernetes/apps/speech-to-phrase/deployment.yaml — added the same annotation. It reads the token once via --hass-token=$(HASS_TOKEN).

Verification

Checked live during this run (about 02:43Z):

kubectl -n home-assistant get externalsecret home-assistant-prometheus-token -o jsonpath=... → 1h 2026-10-01T02:35:26Z SecretSynced. ESO picked up the rotated token at 02:35Z, as the issue predicted.
Prometheus query?query=up{namespace="home-assistant"} → job="home-assistant", pod home-assistant-7d97f854d6-8jtbs, value "1". The scrape has self-healed. The alert should resolve on its own whether or not this PR merges.
kubectl -n home-assistant get pods → home-assistant-7d97f854d6-8jtbs 4/4 Running 0 7m23s. The pod named in the issue (...-8zcfz) has been replaced since the issue was written. zone-configurator-69c586bd7d-k4rtv 1/1 Running 16 (11m ago), and its current log shows only successful probe requests. speech-to-phrase-66bc576865-wtk88 Running 0, 7d8h.
kubectl -n flux-system get kustomization zone-configurator speech-to-phrase → both READY True. The HealthCheckFailed mentioned in the issue has cleared.
kubectl get deploy -A -l app.kubernetes.io/name=reloader → kube-system/reloader-reloader 1/1. Reloader is running, so the new annotations will take effect.

After Flux reconciles, confirm spec.refreshInterval is 1m on the three ExternalSecrets and that the annotation is present on both Deployments. The real test is the next rotation (2027-01-01 04:00 Europe/Berlin). Each refreshTime should advance within about 1 minute of its job, zone-configurator and speech-to-phrase should restart once, and HomeAssistantDown should not fire.

Notes

Manual follow-up: speech-to-phrase has been running for 7d with no restart, so it was started with the now-revoked token. Its logs since 2026-09-23 show nothing that confirms or rules out an auth failure, because it apparently only talks to HA at (re)train. The new annotation won't fire for a Secret change that already happened. Run kubectl -n home-assistant rollout restart deploy/speech-to-phrase once to load the current token. --retrain-on-start makes startup slow, but the startupProbe allows 10 minutes for it.
This narrows the gap; it doesn't close it. For up to about 1m after each rotation the Secret still holds a revoked token. The proper fix is in secret-rotator/src/main.rs (outside the allowed edit paths): defer revocation, e.g. revoke the previous rotation's token on the next run or after a delay longer than the ESO interval, or trigger an ESO force-sync annotation before revoking. I recommend a follow-up in the rotator.
I chose 1m over the repo's usual 10m for rotated secrets because 10m is longer than HomeAssistantDown's for: 5m and could still let the alert fire. The cost is three small OpenBao reads per minute.
Not done (structural, out of scope): giving zone-configurator and speech-to-phrase their own LLATs instead of sharing the token named for the ServiceMonitor. The issue rightly points out that this sharing is why one rotation broke three consumers.
No alert rules were changed.

Closes #1081

## What this PR does `HomeAssistantDown` fired at about 02:05Z on 2026-10-01 because Home Assistant was rejecting the Prometheus scrape with a 401 error. Home Assistant itself was not down. At 02:00Z the quarterly `prometheus-token-secret-rotate` job minted a new long-lived access token (LLAT), stored it in OpenBao, and then revoked the old one right away. The `home-assistant-prometheus-token` ExternalSecret only syncs every `1h`, though, so until about 02:35Z the Kubernetes Secret still held the revoked token. The ServiceMonitor scrape failed with 401 during that window, and `zone-configurator`, which reads the same token as a startup env var, went into a crashloop. The rotator's revoke-immediately behaviour lives in `secret-rotator/src/main.rs`, which is outside this agent's edit scope. This PR instead shrinks the window on the consumer side: - All three rotated LLAT ExternalSecrets now refresh every `1m`. That keeps the dead-token window under the alert's `for: 5m`. - The two Deployments that read the token at startup get a Reloader annotation, so they restart when the Secret changes. ## Changes - `kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml` — `refreshInterval: 1h` → `1m`, with a comment explaining why. - `kubernetes/apps/home-assistant/config-sync-token-externalsecret.yaml` — `1h` → `1m`. Its CronJob also revokes immediately (04:05 Berlin), so it has the same gap. The HA deployment already has a Reloader annotation, but that only fires once ESO actually syncs. - `kubernetes/apps/home-assistant/shelly-manager-llat-externalsecret.yaml` — `1h` → `1m`, for the same reason (04:10 Berlin rotation). - `kubernetes/apps/zone-configurator/deployment.yaml` — added `secret.reloader.stakater.com/reload: home-assistant-prometheus-token`. It reads the token once as `HA_LONG_LIVED_TOKEN`. - `kubernetes/apps/speech-to-phrase/deployment.yaml` — added the same annotation. It reads the token once via `--hass-token=$(HASS_TOKEN)`. ## Verification Checked live during this run (about 02:43Z): - `kubectl -n home-assistant get externalsecret home-assistant-prometheus-token -o jsonpath=...` → `1h 2026-10-01T02:35:26Z SecretSynced`. ESO picked up the rotated token at 02:35Z, as the issue predicted. - Prometheus `query?query=up{namespace="home-assistant"}` → `job="home-assistant"`, pod `home-assistant-7d97f854d6-8jtbs`, value `"1"`. **The scrape has self-healed.** The alert should resolve on its own whether or not this PR merges. - `kubectl -n home-assistant get pods` → `home-assistant-7d97f854d6-8jtbs 4/4 Running 0 7m23s`. The pod named in the issue (`...-8zcfz`) has been replaced since the issue was written. `zone-configurator-69c586bd7d-k4rtv 1/1 Running 16 (11m ago)`, and its current log shows only successful probe requests. `speech-to-phrase-66bc576865-wtk88 Running 0, 7d8h`. - `kubectl -n flux-system get kustomization zone-configurator speech-to-phrase` → both `READY True`. The `HealthCheckFailed` mentioned in the issue has cleared. - `kubectl get deploy -A -l app.kubernetes.io/name=reloader` → `kube-system/reloader-reloader 1/1`. Reloader is running, so the new annotations will take effect. After Flux reconciles, confirm `spec.refreshInterval` is `1m` on the three ExternalSecrets and that the annotation is present on both Deployments. The real test is the next rotation (2027-01-01 04:00 Europe/Berlin). Each `refreshTime` should advance within about 1 minute of its job, `zone-configurator` and `speech-to-phrase` should restart once, and `HomeAssistantDown` should not fire. ## Notes - **Manual follow-up:** `speech-to-phrase` has been running for 7d with no restart, so it was started with the now-revoked token. Its logs since 2026-09-23 show nothing that confirms or rules out an auth failure, because it apparently only talks to HA at (re)train. The new annotation won't fire for a Secret change that already happened. Run `kubectl -n home-assistant rollout restart deploy/speech-to-phrase` once to load the current token. `--retrain-on-start` makes startup slow, but the startupProbe allows 10 minutes for it. - **This narrows the gap; it doesn't close it.** For up to about 1m after each rotation the Secret still holds a revoked token. The proper fix is in `secret-rotator/src/main.rs` (outside the allowed edit paths): defer revocation, e.g. revoke the *previous* rotation's token on the next run or after a delay longer than the ESO interval, or trigger an ESO `force-sync` annotation before revoking. I recommend a follow-up in the rotator. - I chose `1m` over the repo's usual `10m` for rotated secrets because `10m` is longer than `HomeAssistantDown`'s `for: 5m` and could still let the alert fire. The cost is three small OpenBao reads per minute. - Not done (structural, out of scope): giving `zone-configurator` and `speech-to-phrase` their own LLATs instead of sharing the token named for the ServiceMonitor. The issue rightly points out that this sharing is why one rotation broke three consumers. - No alert rules were changed. Closes #1081

issue-fix-bot added 1 commit 2026-10-01 02:44:11 +00:00

fix: [alert] HomeAssistantDown (home-assistant) (#1081 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

0bbd863049

issue-fix-bot referenced this pull request

2026-10-01 02:44:12 +00:00

[alert] HomeAssistantDown (home-assistant) #1081

review-bot reviewed

2026-10-01 02:47:22 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 0bbd8630499fe35d9bf2f29f739deb34c15e1602.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR shrinks the dead-token window that caused HomeAssistantDown (issue #1081): it drops refreshInterval from 1h to 1m on the three LLAT ExternalSecrets rotated by secret-rotator (prometheus-token, config-sync-token, shelly-manager-llat), and adds a secret.reloader.stakater.com/reload annotation to the two Deployments (zone-configurator, speech-to-phrase) that read home-assistant-prometheus-token once at startup. I verified the root-cause narrative against secret-rotator/src/main.rs::rotate_ha_token (mint → store in OpenBao → revoke old, in that order) and the CronJob schedules in kubernetes/infrastructure/openbao/rotation/home-assistant.yaml (04:00/04:05/04:10 Berlin quarterly) — both match the PR body. The new Reloader annotations target the exact secret name each Deployment's secretKeyRef uses, so they'll actually fire. No blocking issues.

🟡 Suggestions

Margin is tight, not provably safe. refreshInterval: 1m keeps the dead-token window under HomeAssistantDown's for: 5m only if ESO reconciles within roughly a minute of the rotation. kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml documents that a stuck ExternalSecret can sit in retry/backoff for a long time (the wiki-embed-basic-auth incident needed a manual force-sync). If that happens to coincide with a quarterly rotation, 1m doesn't actually bound the gap and the alert could still fire. Worth a one-line acknowledgment in the comment, or better, treating the rotator-side fix the PR author already identifies (defer revocation / force-sync annotation) as a real follow-up rather than a "nice to have."
Known stale-token deployment not restarted. The PR body notes speech-to-phrase has run 7 days on a now-revoked token and recommends kubectl rollout restart deploy/speech-to-phrase, but that's a manual step that lives only in the PR description. Since it only matters "at (re)train" per the author's own note, it's easy to forget; consider filing a follow-up issue so it isn't lost once the PR merges.

Tests

No automated tests accompany this change, consistent with the rest of the repo — other plain refreshInterval/annotation edits in this directory (e.g. kubernetes/apps/paperclip, kubernetes/apps/external-dns) aren't covered by promtool tests either; those exist only for PrometheusRule files, which aren't touched here. The PR body documents manual live verification (ESO refreshTime, Prometheus up{}, pod/Kustomization health) instead, which is reasonable for a manifest-only change like this.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `0bbd8630499fe35d9bf2f29f739deb34c15e1602`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR shrinks the dead-token window that caused `HomeAssistantDown` (issue #1081): it drops `refreshInterval` from `1h` to `1m` on the three LLAT `ExternalSecret`s rotated by `secret-rotator` (prometheus-token, config-sync-token, shelly-manager-llat), and adds a `secret.reloader.stakater.com/reload` annotation to the two Deployments (`zone-configurator`, `speech-to-phrase`) that read `home-assistant-prometheus-token` once at startup. I verified the root-cause narrative against `secret-rotator/src/main.rs::rotate_ha_token` (mint → store in OpenBao → revoke old, in that order) and the CronJob schedules in `kubernetes/infrastructure/openbao/rotation/home-assistant.yaml` (04:00/04:05/04:10 Berlin quarterly) — both match the PR body. The new Reloader annotations target the exact secret name each Deployment's `secretKeyRef` uses, so they'll actually fire. No blocking issues. ## 🟡 Suggestions - **Margin is tight, not provably safe.** `refreshInterval: 1m` keeps the dead-token window under `HomeAssistantDown`'s `for: 5m` only if ESO reconciles within roughly a minute of the rotation. `kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml` documents that a stuck ExternalSecret can sit in retry/backoff for a long time (the `wiki-embed-basic-auth` incident needed a manual force-sync). If that happens to coincide with a quarterly rotation, 1m doesn't actually bound the gap and the alert could still fire. Worth a one-line acknowledgment in the comment, or better, treating the rotator-side fix the PR author already identifies (defer revocation / force-sync annotation) as a real follow-up rather than a "nice to have." - **Known stale-token deployment not restarted.** The PR body notes `speech-to-phrase` has run 7 days on a now-revoked token and recommends `kubectl rollout restart deploy/speech-to-phrase`, but that's a manual step that lives only in the PR description. Since it only matters "at (re)train" per the author's own note, it's easy to forget; consider filing a follow-up issue so it isn't lost once the PR merges. ## Tests No automated tests accompany this change, consistent with the rest of the repo — other plain `refreshInterval`/annotation edits in this directory (e.g. `kubernetes/apps/paperclip`, `kubernetes/apps/external-dns`) aren't covered by `promtool` tests either; those exist only for `PrometheusRule` files, which aren't touched here. The PR body documents manual live verification (ESO `refreshTime`, Prometheus `up{}`, pod/Kustomization health) instead, which is reasonable for a manifest-only change like this. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

~~Ghost referenced this pull request 2026-10-01 05:23:44 +00:00~~

fix: [alert] TargetDown (home-assistant) (#1084) #1087

~~Ghost referenced this pull request 2026-10-01 05:23:45 +00:00~~

fix: [alert] KubePodCrashLooping (home-assistant) (#1082) #1086

Ghost closed this pull request

2026-10-01 05:23:45 +00:00

~~hermes-bot referenced this pull request 2026-10-21 07:41:47 +00:00~~

fix(secret-rotator): keep the replaced HA token until the next rotation (#1081, #1084) #1414

hermes-bot referenced this pull request

2026-10-21 07:42:23 +00:00

[alert] HomeAssistantDown (home-assistant) #1081