The broker's liveness+readiness were tcpSocket:1883 — a bare TCP accept
proves a listener exists, not that the broker routes. Even a QoS-1 PUBACK
proves only that the broker acknowledged the publish at the protocol level;
a broker wedged so it ACKs writes but drops them (a write blackhole) would
still pass.
So each probe publishes a fresh RANDOM value retained and reads it back,
succeeding only if the exact bytes round-trip. That proves the broker
authenticated, stored the message, and delivered it to a subscriber:
auth + write + storage + read + delivery. Retained (not a live
subscribe-then-publish) so the read never races the connect and can't
false-fail; liveness and readiness use separate topics (probe/live vs
probe/ready) so their rotating retained values never collide.
Runs as a dedicated least-privilege `health` user, not admin. The probes
are exec probes running INSIDE the mosquitto container; admin's password is
only reachable there because the startup shim hashes every user into the
passwd file, but admin is readwrite # and meant for an external workstation
client, so it's the wrong credential for a probe.
- new mosquitto-health-credentials-bootstrap Job -> Secret
mosquitto-health-credentials (same per-user pattern as the other six).
- ACL: readwrite health/# and NOTHING else (read needed for the
read-back). A leak can only touch the throwaway probe subtree that
nothing subscribes to.
- HEALTH_PASSWORD env (the shim needs it to build passwd); the
in-container exec probes read it directly.
startupProbe tcpSocket :1883, 5s x 12 = 60s (passwd/acl build + bind)
readinessProbe round-trip, 15s x 3, timeout 10s
livenessProbe round-trip, 30s x 6 = 3 min sustained before restart, timeout 10s
Verified live before merge (with the admin cred, which already exists —
identical command shape):
- exact retained round-trip script -> exit 0, value read back matches.
- negative control (subscribe to an empty topic) -> times out, mismatch,
non-zero exit. Confirms the probe fails on a blackhole, not just on TCP.
- round-trip latency ~0 ms on loopback, far under the 10s timeout.
SPOF safety (single replica; HA, Shelly, sml2mqtt, open3e, evcc):
- liveness fires slowly (3 min sustained) so a transient hiccup never
bounces the broker.
- Deployment is maxUnavailable:0 / maxSurge:1: the new pod must pass
readiness before the old is torn down, so even if the health wiring were
wrong the old broker keeps serving — a stalled rollout, not an outage.
kubectl kustomize builds. Post-merge: bootstrap Job creates the secret, pod
rolls (old serves until new Ready), health user round-trips green, 0
restarts, clients stay connected; negative-control a wrong cred/blackhole.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8