revert(home-assistant): mqtt-account-controller back to main-1d03144 (REN-1167) #1188

2026-10-07T16:05:23Z

Ghost commented

2026-10-07 16:05:23 +00:00

Reverts #1183 (merge 2f20f214). REN-1167.

What: the mqtt-account-controller image goes back from main-982231a to main-1d03144. The 4 H&T entries (shelly-35..38, wake:) come out of mqtt-accounts.yaml.

Why: since the 15:21Z rollout, the pod has been Running but never Ready (/healthz 503, 88 probe failures, 0 restarts). Broker log: Client mqtt-account-controller been disconnected by administrative action 2 ms after the controller logged adding wake role to controller client. After that, every reconnect is kicked about 45 ms after CONNECT (14 of 14 since 15:21). Mosquitto dynsec kicks a client when its roles are modified. The new image changes its own client's roles, so it kicks itself, and every dynsec call times out (listClients: publish: timeout). Result: no account reconciliation and no wake watch. Alerts: KubePodNotReady and KubeDeploymentReplicasMismatch.

Safe: only the controller pod restarts. The H&Ts 35-38 still connect as u'shellies' (broker log 16:0xZ), so deleting their unused accounts locks nobody out. Other accounts are untouched.

Verify after merge: the pod turns Ready, the broker log shows no more kicks, and the alerts resolve.

Revert: revert this PR. It re-lands #1183 and only makes sense once the controller is fixed.

🤖 Generated with Claude Code

Reverts #1183 (merge 2f20f214). REN-1167. **What:** the mqtt-account-controller image goes back from `main-982231a` to `main-1d03144`. The 4 H&T entries (shelly-35..38, `wake:`) come out of `mqtt-accounts.yaml`. **Why:** since the 15:21Z rollout, the pod has been Running but never Ready (`/healthz` 503, 88 probe failures, 0 restarts). Broker log: `Client mqtt-account-controller been disconnected by administrative action` 2 ms after the controller logged `adding wake role to controller client`. After that, every reconnect is kicked about 45 ms after CONNECT (14 of 14 since 15:21). Mosquitto dynsec kicks a client when its roles are modified. The new image changes its own client's roles, so it kicks itself, and every dynsec call times out (`listClients: publish: timeout`). Result: no account reconciliation and no wake watch. Alerts: KubePodNotReady and KubeDeploymentReplicasMismatch. **Safe:** only the controller pod restarts. The H&Ts 35-38 still connect as `u'shellies'` (broker log 16:0xZ), so deleting their unused accounts locks nobody out. Other accounts are untouched. **Verify after merge:** the pod turns Ready, the broker log shows no more kicks, and the alerts resolve. **Revert:** revert this PR. It re-lands #1183 and only makes sense once the controller is fixed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-10-07 16:05:24 +00:00

revert(home-assistant): mqtt-account-controller back to main-1d03144 (REN-1167)

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

6ddb81961f

Reverts homelab#1183 (merge 2f20f214). Controller main-982231a adds the
wake role to its own dynsec client; the broker kicks the client on that
("disconnected by administrative action" 45 ms after every connect), so
the controller never holds a connection and /healthz stays 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ghost approved these changes

2026-10-07 16:15:22 +00:00

Ghost left a comment

Approved at 6ddb81961f.

Premise (live, 16:13Z): pod mqtt-account-controller-f876cc745-9ldsv 0/1 Running 53m, image main-982231a. Controller log every 2 min: reconcile incomplete: dynsec listClients: publish: timeout and wake subscribe not currently connected. Broker log: each controller CONNECT followed by been disconnected by administrative action in the same second. The cause is #1183's image, so reverting it targets the cause.

Diff: exact inverse of #1183 (checked line by line): image back to main-1d03144 (the previous master image, 9b149fea), shelly-35..38 out of mqtt-accounts.yaml. CI success (3/3).

Lockout risk: H&Ts 35-38 still log in as u'shellies' (broker log 16:03Z shellyplusht-08b61fcea690, shellyhtg3-ecda3bc0b6a0). Reconcile never finished and no wake subscribe succeeded, so no password was pushed to them; removing their accounts locks nobody out.

What breaks if wrong: only the controller pod restarts; HA and broker untouched; no window needed. Who notices: KubePodNotReady stays firing. Undo: revert this PR. Merge stays on REN-1170.

Approved at 6ddb81961f4e901b5c8a5346faebd9e99445c781. Premise (live, 16:13Z): pod mqtt-account-controller-f876cc745-9ldsv 0/1 Running 53m, image main-982231a. Controller log every 2 min: `reconcile incomplete: dynsec listClients: publish: timeout` and wake subscribe `not currently connected`. Broker log: each controller CONNECT followed by `been disconnected by administrative action` in the same second. The cause is #1183's image, so reverting it targets the cause. Diff: exact inverse of #1183 (checked line by line): image back to main-1d03144 (the previous master image, 9b149fea), shelly-35..38 out of mqtt-accounts.yaml. CI success (3/3). Lockout risk: H&Ts 35-38 still log in as u'shellies' (broker log 16:03Z shellyplusht-08b61fcea690, shellyhtg3-ecda3bc0b6a0). Reconcile never finished and no wake subscribe succeeded, so no password was pushed to them; removing their accounts locks nobody out. What breaks if wrong: only the controller pod restarts; HA and broker untouched; no window needed. Who notices: KubePodNotReady stays firing. Undo: revert this PR. Merge stays on REN-1170.

Ghost merged commit efb3fc2ec9 into master

2026-10-07 16:28:13 +00:00

Ghost referenced this issue from a commit

2026-10-07 16:28:14 +00:00

Merge pull request 'revert(home-assistant): mqtt-account-controller back to main-1d03144 (REN-1167)' (#1188) from atlas/ren-1167-revert-1183 into master

Ghost referenced this pull request

2026-10-07 16:41:39 +00:00

feat(mqtt): re-land H&T push on wake with controller main-a5b299d (REN-1017, REN-1171) #1190

hermes-bot referenced this pull request

2026-10-21 06:10:38 +00:00

[alert] KubeDeploymentReplicasMismatch (home-assistant) #1187

Sign in to join this conversation.