The broker is now load-bearing (heat pump, energy meters, EV charging, HA all
depend on it) but was a slow-to-recover SPOF: single replica on a Ceph RWO PVC
with no priority and default 5-min fault tolerations. mosquitto can't cluster,
so instead of faking HA this minimizes recovery time and removes the storage
coupling.
- persistence false + drop the Ceph PVC. The db was only ~178 KB of retained
messages (evcc state, HA discovery, Gen1 Shelly announce/state) — all of
which clients re-publish on reconnect. Removing it means the broker no
longer depends on Ceph to start, and reschedules with no RWO volume to
fence/detach/reattach. Trade-off: a restart briefly empties retained state
until clients reconnect (self-healing).
- PriorityClass `home-critical` (1e6, below system-cluster-critical) so the
broker is evicted last and reschedules ahead of ordinary workloads.
- not-ready/unreachable tolerations at 30s (was default 300s) → node-failure
reschedule in ~30s instead of ~5min.
- terminationGracePeriodSeconds 10 for faster Recreate rollovers.
Single replica + Recreate retained (two brokers would split clients/state).
Apply notes:
- Recreates the pod (brief MQTT reconnect; retained state re-populates).
- The old PVC has prune:disabled, so Flux won't delete it — reclaim manually
with `kubectl -n home-assistant delete pvc mosquitto-data` once happy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8