fix: [alert] TargetDown (home-assistant) (#1084) #1087

2026-10-01T04:43:14Z

issue-fix-bot commented

2026-10-01 04:43:14 +00:00

What this PR does

The quarterly Home Assistant LLAT rotation (kubernetes/infrastructure/openbao/rotation/home-assistant.yaml, secret-rotator ha-token) stores the new token in OpenBao and then revokes the old one straight away. The k8s Secrets that consumers read are synced from OpenBao by ExternalSecrets with refreshInterval: 1h. So until ESO next re-read OpenBao, every consumer held a revoked token. On 2026-10-01 that lasted ~35 min. The Prometheus scrape got 401s, which fired TargetDown and HomeAssistantDown. zone-configurator reuses home-assistant-prometheus-token, so it crashlooped on Auth invalid. This PR shortens refreshInterval to 1m on the three rotated-token ExternalSecrets. That cuts the stale-token window from up to 1 h to about 1 min, well under the 5m for: on HomeAssistantDown. It is a stopgap (option 3 in the issue). The real fix is in the rotator binary, which this pipeline is not allowed to edit (see Notes).

Changes

kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml — refreshInterval: 1h → 1m, plus a comment explaining why.
kubernetes/apps/home-assistant/config-sync-token-externalsecret.yaml — 1h → 1m.
kubernetes/apps/home-assistant/shelly-manager-llat-externalsecret.yaml — 1h → 1m.

Verification

Checked live this run:

./.ci-diag/kubectl get externalsecret -n home-assistant showed home-assistant-prometheus-token, home-assistant-config-sync-token and shelly-manager-llat at REFRESH INTERVAL 1h, all SecretSynced True. This matches the repo.
./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zone-configurator showed zone-configurator-69c586bd7d-k4rtv with 16 restarts.
./.ci-diag/kubectl logs -n home-assistant zone-configurator-69c586bd7d-k4rtv --previous --tail=25 showed "WsReadTransport: Auth invalid" and then an uncaught Error: Auth invalid: Invalid access token or password. The timestamp is 1790821896152, which is 2026-10-01 02:31:36Z, inside the window between revoke and ESO refresh.
In the repo: zone-configurator/deployment.yaml takes HA_LONG_LIVED_TOKEN from home-assistant-prometheus-token. HomeAssistantDown has for: 5m (monitoring-rules/prometheusrule-home-assistant.yaml, and the mutation test pins it). The rotator logs "revoked old ..." right after the store (secret-rotator/src/main.rs ~L493).

Not checked by me: the Loki lines for HA's invalid authentication and the 02:35Z ESO refreshTime values come from the issue.

After Flux reconciles, kubectl get externalsecret -n home-assistant should show 1m for the three tokens. To test end to end, run an off-cycle job, e.g. kubectl create job -n openbao --from=cronjob/prometheus-token-secret-rotate test-rotate (needs a human with write access). Then check that up{job="home-assistant"} recovers within ~1–2 min and that HomeAssistantDown does not fire.

Notes

Stopgap, not the full fix. The right fix is to stop the break-before-make: either delay the revoke in the rotator (grace period, or a separate revoke pass), or have the rotator force an ESO sync and wait for it before revoking. Both need changes to secret-rotator/src/main.rs, and the force-sync option also needs new RBAC. That is outside this pipeline's edit scope (secret-rotator/ is not allowlisted), so it needs a human follow-up.
Remaining window. Consumers can still see a revoked token for up to ~1 min, plus pod restart time where Reloader applies (home-assistant, shelly-manager). zone-configurator may crash once or twice and then recover on its own: a container restart re-reads the env var. That should stay below the for: durations of the alerts, but this has not been tested.
Cost. Three ExternalSecrets now read OpenBao once a minute instead of once an hour. This is negligible, and the namespace already runs others at 10m.
Not done: decoupling zone-configurator from the ServiceMonitor token (option 4). It would need a new LLAT minted in HA, a new OpenBao path and policy, and a new rotation CronJob, which needs manual bootstrapping and is beyond a surgical change. It is a reasonable follow-up. Adding secret.reloader.stakater.com/reload: home-assistant-prometheus-token to the zone-configurator deployment would also be harmless, but it would not prevent the crash.
The issue's hypothesis that config-sync and shelly-manager were also failing auth during their windows was not checked this run.

Closes #1084

## What this PR does The quarterly Home Assistant LLAT rotation (`kubernetes/infrastructure/openbao/rotation/home-assistant.yaml`, `secret-rotator ha-token`) stores the new token in OpenBao and then revokes the old one straight away. The k8s Secrets that consumers read are synced from OpenBao by ExternalSecrets with `refreshInterval: 1h`. So until ESO next re-read OpenBao, every consumer held a revoked token. On 2026-10-01 that lasted ~35 min. The Prometheus scrape got 401s, which fired `TargetDown` and `HomeAssistantDown`. `zone-configurator` reuses `home-assistant-prometheus-token`, so it crashlooped on `Auth invalid`. This PR shortens `refreshInterval` to `1m` on the three rotated-token ExternalSecrets. That cuts the stale-token window from up to 1 h to about 1 min, well under the 5m `for:` on `HomeAssistantDown`. It is a stopgap (option 3 in the issue). The real fix is in the rotator binary, which this pipeline is not allowed to edit (see Notes). ## Changes - `kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml` — `refreshInterval: 1h` → `1m`, plus a comment explaining why. - `kubernetes/apps/home-assistant/config-sync-token-externalsecret.yaml` — `1h` → `1m`. - `kubernetes/apps/home-assistant/shelly-manager-llat-externalsecret.yaml` — `1h` → `1m`. ## Verification Checked live this run: - `./.ci-diag/kubectl get externalsecret -n home-assistant` showed `home-assistant-prometheus-token`, `home-assistant-config-sync-token` and `shelly-manager-llat` at `REFRESH INTERVAL 1h`, all `SecretSynced True`. This matches the repo. - `./.ci-diag/kubectl get pods -n home-assistant -l app.kubernetes.io/name=zone-configurator` showed `zone-configurator-69c586bd7d-k4rtv` with 16 restarts. - `./.ci-diag/kubectl logs -n home-assistant zone-configurator-69c586bd7d-k4rtv --previous --tail=25` showed `"WsReadTransport: Auth invalid"` and then an uncaught `Error: Auth invalid: Invalid access token or password`. The timestamp is `1790821896152`, which is 2026-10-01 02:31:36Z, inside the window between revoke and ESO refresh. - In the repo: `zone-configurator/deployment.yaml` takes `HA_LONG_LIVED_TOKEN` from `home-assistant-prometheus-token`. `HomeAssistantDown` has `for: 5m` (`monitoring-rules/prometheusrule-home-assistant.yaml`, and the mutation test pins it). The rotator logs "revoked old ..." right after the store (`secret-rotator/src/main.rs` ~L493). Not checked by me: the Loki lines for HA's `invalid authentication` and the 02:35Z ESO `refreshTime` values come from the issue. After Flux reconciles, `kubectl get externalsecret -n home-assistant` should show `1m` for the three tokens. To test end to end, run an off-cycle job, e.g. `kubectl create job -n openbao --from=cronjob/prometheus-token-secret-rotate test-rotate` (needs a human with write access). Then check that `up{job="home-assistant"}` recovers within ~1–2 min and that `HomeAssistantDown` does not fire. ## Notes - **Stopgap, not the full fix.** The right fix is to stop the break-before-make: either delay the revoke in the rotator (grace period, or a separate revoke pass), or have the rotator force an ESO sync and wait for it before revoking. Both need changes to `secret-rotator/src/main.rs`, and the force-sync option also needs new RBAC. That is outside this pipeline's edit scope (`secret-rotator/` is not allowlisted), so it needs a human follow-up. - **Remaining window.** Consumers can still see a revoked token for up to ~1 min, plus pod restart time where Reloader applies (`home-assistant`, `shelly-manager`). `zone-configurator` may crash once or twice and then recover on its own: a container restart re-reads the env var. That should stay below the `for:` durations of the alerts, but this has not been tested. - **Cost.** Three ExternalSecrets now read OpenBao once a minute instead of once an hour. This is negligible, and the namespace already runs others at 10m. - **Not done: decoupling zone-configurator** from the ServiceMonitor token (option 4). It would need a new LLAT minted in HA, a new OpenBao path and policy, and a new rotation CronJob, which needs manual bootstrapping and is beyond a surgical change. It is a reasonable follow-up. Adding `secret.reloader.stakater.com/reload: home-assistant-prometheus-token` to the zone-configurator deployment would also be harmless, but it would not prevent the crash. - The issue's hypothesis that config-sync and shelly-manager were also failing auth during their windows was not checked this run. Closes #1084

issue-fix-bot added 1 commit 2026-10-01 04:43:15 +00:00

fix: [alert] TargetDown (home-assistant) (#1084 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

44556132f4

issue-fix-bot referenced this pull request

2026-10-01 04:43:15 +00:00

[alert] TargetDown (home-assistant) #1084

review-bot reviewed

2026-10-01 04:45:47 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 44556132f4a6726fb9646db2af06f09268f115f4.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Lowers refreshInterval from 1h to 1m on the three home-assistant ExternalSecrets fed by the ha-token rotator (prometheus-token-externalsecret.yaml, config-sync-token-externalsecret.yaml, shelly-manager-llat-externalsecret.yaml), shrinking the stale-token window after each quarterly rotation from up to 1h to about 1m. This is a minimal, well-targeted stopgap for issue #1084 (TargetDown/HomeAssistantDown/zone-configurator crashloop). I verified the claims the PR body makes and they hold up; no blocking issues.

Verification performed

secret-rotator/src/main.rs:493 confirms make-before-break with an unconditional revoke of the old token right after the store succeeds — matches the PR's root-cause claim.
kubernetes/apps/monitoring-rules/prometheusrule-home-assistant.yaml:43 confirms HomeAssistantDown has for: 5m; at a 1m refresh the new stale-token window (≈1m + ESO reconcile jitter) stays comfortably under that threshold, unlike the old 1h window.
kubernetes/apps/zone-configurator/deployment.yaml:28-32 confirms HA_LONG_LIVED_TOKEN is sourced via secretKeyRef from home-assistant-prometheus-token with no Reloader annotation on the Deployment, so a Secret update alone won't restart the pod — recovery here genuinely depends on the container itself crash-looping (as the PR body notes was observed: 16 restarts) until a restart happens to land after ESO has refreshed. That's an accepted, explicitly-documented limitation of this stopgap rather than something the PR claims to fix.
Other ExternalSecrets in the same namespace already run at 10m (mosquitto-health-credentials-externalsecret.yaml:28, shelly-manager-auth-externalsecret.yaml:9), supporting the "cost is negligible, namespace already does this" claim.
No kustomize/kyverno/admission policy in kubernetes/infrastructure constrains refreshInterval, and the promtool test (tests/home-assistant_test.yaml) doesn't touch ExternalSecret timing, so neither CI gate is affected by this change.
All three YAML files remain structurally valid (indentation, keys unchanged aside from the edited line and added comments).

🟡 Suggestions

kubernetes/apps/zone-configurator/deployment.yaml has no secret.reloader.stakater.com/reload annotation for home-assistant-prometheus-token. The PR body already flags this as a known follow-up ("harmless... would not prevent the crash"), but adding it here would actually help: a Reloader-triggered restart, timed by ESO's refresh rather than by the app's own crash, is a cleaner recovery path than waiting for Auth invalid to kill the pod. Worth a fast follow-up PR even though it's out of scope for this stopgap.
The real fix (delay revoke / force-sync-then-revoke in secret-rotator) is correctly identified as out of this pipeline's edit scope — worth tracking as a separate issue so this 1m workaround doesn't become permanent.

Tests

No automated test covers refreshInterval values (ESO reconciliation timing isn't something the repo's promtool-based alert tests exercise, and there's no other test harness for ExternalSecret specs in this repo). That's consistent with how the existing 10m/1h values elsewhere in the namespace are handled — untested config values, not a gap introduced by this PR. The PR body documents manual verification steps (live kubectl get externalsecret, pod restart logs) and a suggested end-to-end test via an off-cycle rotation job; that's reasonable given there's no CI path that could exercise live OpenBao/ESO/HA timing.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `44556132f4a6726fb9646db2af06f09268f115f4`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Lowers `refreshInterval` from `1h` to `1m` on the three home-assistant ExternalSecrets fed by the `ha-token` rotator (`prometheus-token-externalsecret.yaml`, `config-sync-token-externalsecret.yaml`, `shelly-manager-llat-externalsecret.yaml`), shrinking the stale-token window after each quarterly rotation from up to 1h to about 1m. This is a minimal, well-targeted stopgap for issue #1084 (`TargetDown`/`HomeAssistantDown`/zone-configurator crashloop). I verified the claims the PR body makes and they hold up; no blocking issues. ## Verification performed - `secret-rotator/src/main.rs:493` confirms make-before-break with an unconditional revoke of the old token right after the store succeeds — matches the PR's root-cause claim. - `kubernetes/apps/monitoring-rules/prometheusrule-home-assistant.yaml:43` confirms `HomeAssistantDown` has `for: 5m`; at a 1m refresh the new stale-token window (≈1m + ESO reconcile jitter) stays comfortably under that threshold, unlike the old 1h window. - `kubernetes/apps/zone-configurator/deployment.yaml:28-32` confirms `HA_LONG_LIVED_TOKEN` is sourced via `secretKeyRef` from `home-assistant-prometheus-token` with no Reloader annotation on the Deployment, so a Secret update alone won't restart the pod — recovery here genuinely depends on the container itself crash-looping (as the PR body notes was observed: 16 restarts) until a restart happens to land after ESO has refreshed. That's an accepted, explicitly-documented limitation of this stopgap rather than something the PR claims to fix. - Other ExternalSecrets in the same namespace already run at `10m` (`mosquitto-health-credentials-externalsecret.yaml:28`, `shelly-manager-auth-externalsecret.yaml:9`), supporting the "cost is negligible, namespace already does this" claim. - No kustomize/kyverno/admission policy in `kubernetes/infrastructure` constrains `refreshInterval`, and the promtool test (`tests/home-assistant_test.yaml`) doesn't touch ExternalSecret timing, so neither CI gate is affected by this change. - All three YAML files remain structurally valid (indentation, keys unchanged aside from the edited line and added comments). ## 🟡 Suggestions - `kubernetes/apps/zone-configurator/deployment.yaml` has no `secret.reloader.stakater.com/reload` annotation for `home-assistant-prometheus-token`. The PR body already flags this as a known follow-up ("harmless... would not prevent the crash"), but adding it here would actually help: a Reloader-triggered restart, timed by ESO's refresh rather than by the app's own crash, is a cleaner recovery path than waiting for `Auth invalid` to kill the pod. Worth a fast follow-up PR even though it's out of scope for this stopgap. - The real fix (delay revoke / force-sync-then-revoke in `secret-rotator`) is correctly identified as out of this pipeline's edit scope — worth tracking as a separate issue so this `1m` workaround doesn't become permanent. ## Tests No automated test covers `refreshInterval` values (ESO reconciliation timing isn't something the repo's promtool-based alert tests exercise, and there's no other test harness for ExternalSecret specs in this repo). That's consistent with how the existing `10m`/`1h` values elsewhere in the namespace are handled — untested config values, not a gap introduced by this PR. The PR body documents manual verification steps (live `kubectl get externalsecret`, pod restart logs) and a suggested end-to-end test via an off-cycle rotation job; that's reasonable given there's no CI path that could exercise live OpenBao/ESO/HA timing. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Ghost closed this pull request

2026-10-01 05:23:44 +00:00

~~Ghost referenced this pull request 2026-10-01 05:23:45 +00:00~~

fix: [alert] KubePodCrashLooping (home-assistant) (#1082) #1086

~~Ghost referenced this pull request 2026-10-01 05:23:45 +00:00~~

fix: [alert] HomeAssistantDown (home-assistant) (#1081) #1083

hermes-bot referenced this pull request

2026-10-21 07:38:44 +00:00

fix(secret-rotator): keep the replaced HA token valid until the next rotation (#1084, #1081) #1412

hermes-bot referenced this pull request

2026-10-21 07:39:18 +00:00

[alert] TargetDown (home-assistant) #1084