feat(mosquitto): retire legacy password users (admin stays, generated Secret) #1433

2026-10-24T18:46:30Z

claude-bot commented

2026-10-24 18:46:30 +00:00

What

Removes the Mosquitto password_file users that the mqtt-account-controller register (Dynamic Security, mqtt-accounts.yaml) has replaced. Only admin stays in the files.

Deleted mosquitto-bootstrap.yaml: 5 Jobs (they apk add at runtime) plus SA/Role/RoleBinding.
mosquitto-deployment.yaml: the shim hashes only admin. The ACL has only user admin / topic readwrite #. Env is down to ADMIN_PASSWORD. The Dynamic Security seed logic is unchanged.
New mosquitto-admin-password.yaml: ESO Password generator + ExternalSecret with refreshPolicy: CreatedOnce. This is the house pattern from apps/hermes/api-key.yaml. It creates Secret mosquitto-admin-password, key password.
mosquitto-config.yaml: the comment now says Dynamic Security is the account source for every client and the files hold admin only. password_file/acl_file stay because of admin. defaultACLAccess still denies.
Deleted mosquitto-evcc-credentials-externalsecret.yaml and mosquitto-health-credentials-externalsecret.yaml. Also deleted the health rotator (openbao/rotation/mosquitto-health.yaml + kustomization entry, openbao/policies/rotator-mosquitto-health.hcl). Nothing reads them: evcc uses cert account evcc-client, and the probes use mosquitto-probe.
Deleted PROPOSAL-mosquitto-credential-rotation.md. It proposed rotating the 8 legacy users and described the bootstrap Jobs as current.
MosquittoDown: the comment and the test now use the mosquitto-dynsec-test pod as the masking pod. The bootstrap Job pod no longer exists. The ha-mosquitto-loose-regex mutation is still caught.
Docs/comments: mqtt-accounts.yaml header and per-entry shadowing notes, mqtt-account-controller.yaml, zigbee2mqtt-migration-runbook.md step 0, kustomization.yaml, ansible/docs/edge-host.md (secrets + bridge section).

Why `admin` is not a register account

The owner wants to keep admin (readwrite #), used from the LAN through the LoadBalancer. The controller README (konairius/mqtt-account-controller) has no delivery mode for that:

auth: password → deliver names exactly one of shelly-manager or esp32, and the plaintext is stored nowhere.
auth: cert → Secret mqtt-<name>-tls. Cert logins only work on the mTLS listener :8884, which is not on mosquitto-iot (in-cluster only).

So admin stays a password_file user. A future register entry named admin would replace it, because Dynamic Security is checked first.

Evidence

Loki (logcli, ns home-assistant, container mosquitto, 30 days):

Last logins by legacy users homeassistant, sml2mqtt, open3e, zigbee2mqtt were around 2026-10-08. admin and evcc have none at all.
Since the broker restart around 2026-10-22, only register accounts connect: home-assistant, edge-1, evcc-client, zigbee2mqtt-bridge, mosquitto-probe, shelly-*, mqtt-account-controller.
Re-checked 2026-10-24 with |~ "u.(homeassistant|admin|sml2mqtt|open3e|evcc|zigbee2mqtt).\\)". The newest hit is 2026-10-08T10:09:38Z (edge1-bridge-sml2mqtt / edge1-bridge-open3e) and nothing is later.

Listeners 1883/8883 without legacy users (docs)

mosquitto.conf(5): with per_listener_settings false (the default, not set here), "this option applies to all listeners". Also: "If password_file, or acl_file are used in the config file alongside plugin, the plugin checks will run before the built in checks." https://mosquitto.org/man/mosquitto-conf-5.html
Dynamic Security docs: "It is recommended to use per_listener_settings false with this plugin, so all listeners use the same authentication and access control." The client username/password "maps to the username/password provided in the CONNECT packet". https://mosquitto.org/documentation/dynamic-security/
Live: shelly-* (1883) and edge-1 (8883) already log in through Dynamic Security today. Neither is in the password_file.

Merge notes

Merging restarts the broker (pod template change). Retained state is lost. Zigbee2MQTT needs kubectl rollout restart deploy/zigbee2mqtt -n home-assistant afterwards (#381).
admin gets a new password. It now lives in mosquitto-admin-password: kubectl -n home-assistant get secret mosquitto-admin-password -o jsonpath='{.data.password}' | base64 -d. The rolled pod waits (CreateContainerConfigError) until ESO has created the Secret. maxUnavailable: 0 keeps the old broker serving until then.
Orphaned Secrets: mosquitto-credentials, mosquitto-admin-credentials, mosquitto-sml2mqtt-credentials, mosquitto-open3e-credentials, mosquitto-zigbee2mqtt-credentials were created by the Jobs and are not Flux-managed. They stay until deleted by hand or via adopt-then-prune. mosquitto-evcc-credentials and mosquitto-health-credentials are ESO-owned and go with their ExternalSecrets.
Manual OpenBao leftovers (not GitOps-managed): policy + kubernetes role rotator-mosquitto-health, and the KV paths homelab/apps/home-assistant/mosquitto-health-credentials and …/mosquitto-evcc-credentials.
Must merge before #1432 (egress lockdown), which removes the Jobs' internet access.

Not in this PR

ansible/roles/mqtt-bridge still renders bridge.d/legacy.conf (the open3e vault-password bridge) as a fallback until the controller's first push. The cluster broker no longer accepts it. Removing it and the vault key changes the edge-host play, so it is left for a separate PR.

Verification

kubectl kustomize kubernetes/apps/home-assistant and kubernetes/infrastructure/openbao/rotation: render clean.
kubectl apply --dry-run=server of the rendered home-assistant output: all objects accepted, including the new Password generator + ExternalSecret.
scripts/ci/promtool-rules.sh in prom/prometheus:v3.15.0: 24 manifests, 9 suites green, 42 mutations caught.

🤖 Generated with Claude Code

## What Removes the Mosquitto password_file users that the mqtt-account-controller register (Dynamic Security, `mqtt-accounts.yaml`) has replaced. Only `admin` stays in the files. - **Deleted** `mosquitto-bootstrap.yaml`: 5 Jobs (they `apk add` at runtime) plus SA/Role/RoleBinding. - **`mosquitto-deployment.yaml`**: the shim hashes only `admin`. The ACL has only `user admin / topic readwrite #`. Env is down to `ADMIN_PASSWORD`. The Dynamic Security seed logic is unchanged. - **New `mosquitto-admin-password.yaml`**: ESO `Password` generator + ExternalSecret with `refreshPolicy: CreatedOnce`. This is the house pattern from `apps/hermes/api-key.yaml`. It creates Secret `mosquitto-admin-password`, key `password`. - **`mosquitto-config.yaml`**: the comment now says Dynamic Security is the account source for every client and the files hold `admin` only. `password_file`/`acl_file` stay because of `admin`. `defaultACLAccess` still denies. - **Deleted** `mosquitto-evcc-credentials-externalsecret.yaml` and `mosquitto-health-credentials-externalsecret.yaml`. Also deleted the health rotator (`openbao/rotation/mosquitto-health.yaml` + kustomization entry, `openbao/policies/rotator-mosquitto-health.hcl`). Nothing reads them: evcc uses cert account `evcc-client`, and the probes use `mosquitto-probe`. - **Deleted** `PROPOSAL-mosquitto-credential-rotation.md`. It proposed rotating the 8 legacy users and described the bootstrap Jobs as current. - **MosquittoDown**: the comment and the test now use the `mosquitto-dynsec-test` pod as the masking pod. The bootstrap Job pod no longer exists. The `ha-mosquitto-loose-regex` mutation is still caught. - **Docs/comments**: `mqtt-accounts.yaml` header and per-entry shadowing notes, `mqtt-account-controller.yaml`, `zigbee2mqtt-migration-runbook.md` step 0, `kustomization.yaml`, `ansible/docs/edge-host.md` (secrets + bridge section). ## Why `admin` is not a register account The owner wants to keep `admin` (`readwrite #`), used from the LAN through the LoadBalancer. The controller README (konairius/mqtt-account-controller) has no delivery mode for that: - `auth: password` → `deliver` names exactly one of `shelly-manager` or `esp32`, and the plaintext is stored nowhere. - `auth: cert` → Secret `mqtt-<name>-tls`. Cert logins only work on the mTLS listener :8884, which is not on `mosquitto-iot` (in-cluster only). So `admin` stays a password_file user. A future register entry named `admin` would replace it, because Dynamic Security is checked first. ## Evidence Loki (`logcli`, ns home-assistant, container mosquitto, 30 days): - Last logins by legacy users `homeassistant`, `sml2mqtt`, `open3e`, `zigbee2mqtt` were around 2026-10-08. `admin` and `evcc` have none at all. - Since the broker restart around 2026-10-22, only register accounts connect: `home-assistant`, `edge-1`, `evcc-client`, `zigbee2mqtt-bridge`, `mosquitto-probe`, `shelly-*`, `mqtt-account-controller`. - Re-checked 2026-10-24 with `|~ "u.(homeassistant|admin|sml2mqtt|open3e|evcc|zigbee2mqtt).\\)"`. The newest hit is 2026-10-08T10:09:38Z (`edge1-bridge-sml2mqtt` / `edge1-bridge-open3e`) and nothing is later. ## Listeners 1883/8883 without legacy users (docs) - mosquitto.conf(5): with `per_listener_settings` false (the default, not set here), "this option applies to all listeners". Also: "If password_file, or acl_file are used in the config file alongside plugin, the plugin checks will run before the built in checks." https://mosquitto.org/man/mosquitto-conf-5.html - Dynamic Security docs: "It is recommended to use `per_listener_settings false` with this plugin, so all listeners use the same authentication and access control." The client username/password "maps to the username/password provided in the CONNECT packet". https://mosquitto.org/documentation/dynamic-security/ - Live: `shelly-*` (1883) and `edge-1` (8883) already log in through Dynamic Security today. Neither is in the password_file. ## Merge notes - **Merging restarts the broker** (pod template change). Retained state is lost. Zigbee2MQTT needs `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant` afterwards (#381). - **`admin` gets a new password.** It now lives in `mosquitto-admin-password`: `kubectl -n home-assistant get secret mosquitto-admin-password -o jsonpath='{.data.password}' | base64 -d`. The rolled pod waits (CreateContainerConfigError) until ESO has created the Secret. `maxUnavailable: 0` keeps the old broker serving until then. - **Orphaned Secrets**: `mosquitto-credentials`, `mosquitto-admin-credentials`, `mosquitto-sml2mqtt-credentials`, `mosquitto-open3e-credentials`, `mosquitto-zigbee2mqtt-credentials` were created by the Jobs and are not Flux-managed. They stay until deleted by hand or via adopt-then-prune. `mosquitto-evcc-credentials` and `mosquitto-health-credentials` are ESO-owned and go with their ExternalSecrets. - **Manual OpenBao leftovers** (not GitOps-managed): policy + kubernetes role `rotator-mosquitto-health`, and the KV paths `homelab/apps/home-assistant/mosquitto-health-credentials` and `…/mosquitto-evcc-credentials`. - **Must merge before #1432** (egress lockdown), which removes the Jobs' internet access. ## Not in this PR `ansible/roles/mqtt-bridge` still renders `bridge.d/legacy.conf` (the `open3e` vault-password bridge) as a fallback until the controller's first push. The cluster broker no longer accepts it. Removing it and the vault key changes the edge-host play, so it is left for a separate PR. ## Verification - `kubectl kustomize kubernetes/apps/home-assistant` and `kubernetes/infrastructure/openbao/rotation`: render clean. - `kubectl apply --dry-run=server` of the rendered home-assistant output: all objects accepted, including the new Password generator + ExternalSecret. - `scripts/ci/promtool-rules.sh` in prom/prometheus:v3.15.0: 24 manifests, 9 suites green, 42 mutations caught. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-10-24 18:46:30 +00:00

feat(mosquitto): retire legacy password users; admin stays on a generated Secret

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

bb090e6809

Since the broker restart on 2026-10-22 only register accounts connect; the
last legacy logins (homeassistant, sml2mqtt, open3e, zigbee2mqtt) were on
2026-10-08, and admin/evcc never logged in within the 30-day Loki window.

- Delete mosquitto-bootstrap.yaml (5 Jobs that apk add at runtime + RBAC).
- The broker's password_file/acl_file now hold only admin, the owner's
  workstation login over the LoadBalancer. The controller has no delivery
  mode that puts a password account's password into a Secret, and the cert
  listener :8884 is in-cluster only, so admin cannot be a register account.
  Its password comes from an ESO Password generator (CreatedOnce), the
  pattern of apps/hermes/api-key.yaml, in Secret mosquitto-admin-password.
- Delete the evcc and health ExternalSecrets and the health rotator
  CronJob/policy; nothing reads them.
- Delete PROPOSAL-mosquitto-credential-rotation.md; its subject is gone.
- MosquittoDown test masks with the dynsec-test pod instead of a
  bootstrap Job pod; comments and docs follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review-bot reviewed

2026-10-24 18:50:15 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-5-5, commit bb090e68091f65162c8109d7b4c563a9ce2222b8.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This PR retires the Mosquitto password_file users that the Dynamic Security register has replaced. It deletes the five apk add bootstrap Jobs and their RBAC, the evcc/health ExternalSecrets, the health rotator with its policy, and the old proposal doc. admin is now the only file login, and its password comes from an ESO Password generator with CreatedOnce, the same setup as apps/hermes/api-key.yaml. I found no blocking issues. The PR is mergeable, its base 55cd083 is the current main tip, and no recent main commit touches these files.

What I checked:

Remaining consumers of the deleted Secrets: none. Z2M (zigbee2mqtt-deployment.yaml), evcc (apps/evcc/deployment.yaml), HA (the ghostunnel mqtt-tls sidecar in deployment.yaml) and the broker probes all log in by certificate on :8884.
Remaining references to the deleted resources: none. I read the e2e test, the Flux Kustomization, RBAC, network policies, the rotation kustomization, the cold-rebuild runbook and the Job/rotation alert rules, and none of them name the bootstrap Jobs, mosquitto-health*, rotator-mosquitto-health or the deleted Secrets.
MosquittoDown test still works. The new masking pod name mosquitto-dynsec-test-6d9f8b7c5-r4t8w does not match the anchored regex mosquitto-[a-z0-9]{6,10}-[a-z0-9]{5}, but it does match the mutated mosquitto-.*. So the test still catches the ha-mosquitto-loose-regex mutation.
Merge order with #1432. #1432 states it must merge after this PR; I found no conflict between the two.

🟡 Suggestions

Short outage risk during the rollout. At the base commit, mosquitto-deployment.yaml reads EVCC_PASSWORD from mosquitto-evcc-credentials, and that reference is not optional. The Secret is owned by ESO, so pruning the ExternalSecret deletes it (the deleted health ExternalSecret's header warned about exactly this). Until the new pod is Ready, the old pod keeps serving, but its container now depends on a Secret that is gone. If that container restarts in this window (liveness failure, OOM), kubelet resolves the env again and fails with CreateContainerConfigError. The new pod may meanwhile still be waiting for mosquitto-admin-password. The window is short, so this is optional. To close it, give the evcc ExternalSecret creationPolicy: Orphan, or remove it one reconcile later.
evcc port 1883 now has no use (kubernetes/infrastructure/network-policies/home-assistant-ns.yaml, evcc ingress rule in default-deny-ingress). The rule still opens 1883 to the evcc namespace, with a comment about "the legacy password". evcc now uses only :8884, and no evcc password login exists any more. Dropping 1883 and fixing the comment would close an unused path. #1432 also edits this file, so a follow-up PR after both merge is the simpler route.

🟢 Nits

kubernetes/apps/monitoring-rules/tests/mutations.txt: the comment above ha-mosquitto-loose-regex still says "lets a Ready bootstrap Job pod mask a down broker".
ansible/inventory/group_vars/edge_host/vars.yml: the "Local MQTT contract" comment still says the bridge logs in to the cluster broker "AS the open3e user". This fits with the ansible cleanup the PR leaves for later, but the comment now contradicts the updated edge-host.md.

Tests

For a manifest-only change, the coverage is adequate. The MosquittoDown promtool case now uses the dynsec-test pod as its masking series, and the mutation check still catches the loose regex. Beyond that, the PR relies on render and dry-run checks, which is normal for this repo. No new test is needed.

Recommendation

Looks reasonable: no blocking issues; the comments above are optional.

🤖 **Automated review** — model `claude-opus-5-5`, commit `bb090e68091f65162c8109d7b4c563a9ce2222b8`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR retires the Mosquitto `password_file` users that the Dynamic Security register has replaced. It deletes the five `apk add` bootstrap Jobs and their RBAC, the evcc/health ExternalSecrets, the health rotator with its policy, and the old proposal doc. `admin` is now the only file login, and its password comes from an ESO `Password` generator with `CreatedOnce`, the same setup as `apps/hermes/api-key.yaml`. I found no blocking issues. The PR is `mergeable`, its base `55cd083` is the current `main` tip, and no recent `main` commit touches these files. What I checked: - **Remaining consumers of the deleted Secrets: none.** Z2M (`zigbee2mqtt-deployment.yaml`), evcc (`apps/evcc/deployment.yaml`), HA (the ghostunnel `mqtt-tls` sidecar in `deployment.yaml`) and the broker probes all log in by certificate on `:8884`. - **Remaining references to the deleted resources: none.** I read the e2e test, the Flux Kustomization, RBAC, network policies, the rotation kustomization, the cold-rebuild runbook and the Job/rotation alert rules, and none of them name the bootstrap Jobs, `mosquitto-health*`, `rotator-mosquitto-health` or the deleted Secrets. - **MosquittoDown test still works.** The new masking pod name `mosquitto-dynsec-test-6d9f8b7c5-r4t8w` does not match the anchored regex `mosquitto-[a-z0-9]{6,10}-[a-z0-9]{5}`, but it does match the mutated `mosquitto-.*`. So the test still catches the `ha-mosquitto-loose-regex` mutation. - **Merge order with #1432.** #1432 states it must merge after this PR; I found no conflict between the two. ## 🟡 Suggestions - **Short outage risk during the rollout.** At the base commit, `mosquitto-deployment.yaml` reads `EVCC_PASSWORD` from `mosquitto-evcc-credentials`, and that reference is not `optional`. The Secret is owned by ESO, so pruning the ExternalSecret deletes it (the deleted health ExternalSecret's header warned about exactly this). Until the new pod is Ready, the old pod keeps serving, but its container now depends on a Secret that is gone. If that container restarts in this window (liveness failure, OOM), kubelet resolves the env again and fails with `CreateContainerConfigError`. The new pod may meanwhile still be waiting for `mosquitto-admin-password`. The window is short, so this is optional. To close it, give the evcc ExternalSecret `creationPolicy: Orphan`, or remove it one reconcile later. - **evcc port 1883 now has no use** (`kubernetes/infrastructure/network-policies/home-assistant-ns.yaml`, evcc ingress rule in `default-deny-ingress`). The rule still opens `1883` to the evcc namespace, with a comment about "the legacy password". evcc now uses only `:8884`, and no evcc password login exists any more. Dropping `1883` and fixing the comment would close an unused path. #1432 also edits this file, so a follow-up PR after both merge is the simpler route. ## 🟢 Nits - `kubernetes/apps/monitoring-rules/tests/mutations.txt`: the comment above `ha-mosquitto-loose-regex` still says "lets a Ready bootstrap Job pod mask a down broker". - `ansible/inventory/group_vars/edge_host/vars.yml`: the "Local MQTT contract" comment still says the bridge logs in to the cluster broker "AS the open3e user". This fits with the ansible cleanup the PR leaves for later, but the comment now contradicts the updated `edge-host.md`. ## Tests For a manifest-only change, the coverage is adequate. The MosquittoDown promtool case now uses the dynsec-test pod as its masking series, and the mutation check still catches the loose regex. Beyond that, the PR relies on render and dry-run checks, which is normal for this repo. No new test is needed. ## Recommendation Looks reasonable: no blocking issues; the comments above are optional.

k.renner merged commit aad24d4186 into main

2026-10-25 14:50:12 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1433