mosquitto: availability hardening — stateless broker + priority + fast failover #129

2026-07-08T08:30:31Z

k.renner commented

2026-07-08 08:30:31 +00:00

The broker is now load-bearing (heat pump, energy meters, EV charging, HA all ride on it) but was a slow-to-recover SPOF: single replica on a Ceph RWO PVC, no priority, default 5-min fault tolerations. mosquitto can't cluster, so rather than fake HA this minimizes recovery time and removes the storage coupling.

Changes

persistence false + drop the Ceph PVC. The db was only ~178 KB of retained messages (evcc state, HA discovery configs, Gen1 Shelly announce/state) — and I verified all of it is re-published by clients on reconnect (evcc republishes, Shellies re-send announce/online, HA re-publishes discovery). So the broker no longer needs Ceph to start, and reschedules with no RWO volume to fence/detach/reattach (the slowest part of failover today). Trade-off: a broker restart briefly empties retained state until clients reconnect — self-healing.
PriorityClass home-critical (1e6, below system-cluster-critical) — evicted last under node pressure, reschedules ahead of ordinary workloads. (New cluster-scoped resource in infrastructure/cluster-settings, which apps dependsOn, so it exists before the deployment references it.)
not-ready/unreachable tolerations → 30s (was default 300s): node-failure reschedule in ~30s instead of ~5min.
terminationGracePeriodSeconds 10 for faster Recreate rollovers.

Single replica + Recreate retained (two brokers would split clients/state). Net effect: Ceph-coupled, slow RWO failover → stateless, reschedules anywhere in seconds. Both kustomizations build clean.

Apply

Recreates the pod (brief MQTT reconnect; retained state re-populates as clients reconnect).
The old mosquitto-data PVC has prune: disabled, so Flux won't delete it — reclaim the Ceph space manually with kubectl -n home-assistant delete pvc mosquitto-data once you're happy.

This is the last big item from the review. Only path to true active-active HA beyond this would be re-platforming to a clustered broker (EMQX/VerneMQ) — out of scope.

The broker is now load-bearing (heat pump, energy meters, EV charging, HA all ride on it) but was a **slow-to-recover SPOF**: single replica on a **Ceph RWO PVC**, no priority, default **5-min** fault tolerations. mosquitto can't cluster, so rather than fake HA this **minimizes recovery time and removes the storage coupling**. ## Changes - **`persistence false` + drop the Ceph PVC.** The db was only **~178 KB** of retained messages (evcc state, HA discovery configs, Gen1 Shelly announce/state) — and I verified all of it is **re-published by clients on reconnect** (evcc republishes, Shellies re-send `announce`/`online`, HA re-publishes discovery). So the broker no longer needs Ceph to start, and reschedules with **no RWO volume to fence/detach/reattach** (the slowest part of failover today). *Trade-off:* a broker restart briefly empties retained state until clients reconnect — self-healing. - **PriorityClass `home-critical`** (1e6, below `system-cluster-critical`) — evicted last under node pressure, reschedules ahead of ordinary workloads. (New cluster-scoped resource in `infrastructure/cluster-settings`, which `apps` dependsOn, so it exists before the deployment references it.) - **not-ready/unreachable tolerations → 30s** (was default 300s): node-failure reschedule in ~30s instead of ~5min. - **terminationGracePeriodSeconds 10** for faster `Recreate` rollovers. Single replica + `Recreate` retained (two brokers would split clients/state). Net effect: **Ceph-coupled, slow RWO failover → stateless, reschedules anywhere in seconds.** Both kustomizations build clean. ## Apply - Recreates the pod (brief MQTT reconnect; retained state re-populates as clients reconnect). - The old `mosquitto-data` PVC has `prune: disabled`, so Flux won't delete it — reclaim the Ceph space manually with `kubectl -n home-assistant delete pvc mosquitto-data` once you're happy. This is the last big item from the review. Only path to *true* active-active HA beyond this would be re-platforming to a clustered broker (EMQX/VerneMQ) — out of scope.

k.renner added 1 commit 2026-07-08 08:30:31 +00:00

mosquitto: availability hardening — stateless broker + priority + fast failover 9738f6fd6e

The broker is now load-bearing (heat pump, energy meters, EV charging, HA all
depend on it) but was a slow-to-recover SPOF: single replica on a Ceph RWO PVC
with no priority and default 5-min fault tolerations. mosquitto can't cluster,
so instead of faking HA this minimizes recovery time and removes the storage
coupling.

- persistence false + drop the Ceph PVC. The db was only ~178 KB of retained
  messages (evcc state, HA discovery, Gen1 Shelly announce/state) — all of
  which clients re-publish on reconnect. Removing it means the broker no
  longer depends on Ceph to start, and reschedules with no RWO volume to
  fence/detach/reattach. Trade-off: a restart briefly empties retained state
  until clients reconnect (self-healing).
- PriorityClass `home-critical` (1e6, below system-cluster-critical) so the
  broker is evicted last and reschedules ahead of ordinary workloads.
- not-ready/unreachable tolerations at 30s (was default 300s) → node-failure
  reschedule in ~30s instead of ~5min.
- terminationGracePeriodSeconds 10 for faster Recreate rollovers.

Single replica + Recreate retained (two brokers would split clients/state).

Apply notes:
- Recreates the pod (brief MQTT reconnect; retained state re-populates).
- The old PVC has prune:disabled, so Flux won't delete it — reclaim manually
  with `kubectl -n home-assistant delete pvc mosquitto-data` once happy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit de3ff25c19 into master

2026-07-08 08:31:12 +00:00

k.renner referenced this issue from a commit

2026-07-08 08:31:14 +00:00

Merge pull request 'mosquitto: availability hardening — stateless broker + priority + fast failover' (#129) from feat/mosquitto-availability into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#129