mosquitto: real MQTT round-trip probes instead of tcpSocket #150

2026-07-10T18:12:22Z

k.renner commented

2026-07-10 18:12:22 +00:00

Why

mosquitto's liveness+readiness were tcpSocket:1883 — proves a listener exists, not that the broker routes. Even a QoS-1 PUBACK isn't enough: it only confirms the broker acknowledged the publish at the protocol level. A broker wedged so it ACKs writes but drops them (a write blackhole) would still pass.

What — a real pub/sub round-trip

Each probe publishes a fresh random value retained and reads it back, succeeding only if the exact bytes round-trip:

V="probe-$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' ')"
mosquitto_pub … -q 1 -r -t health/probe/live -m "$V" &&
[ "$(mosquitto_sub … -t health/probe/live -C 1 -W 5)" = "$V" ]

That proves the broker authenticated, stored the message, and delivered it to a subscriber — auth + write + storage + read + delivery. A blackhole → read-back empty/mismatch → fail.

Retained, not a live subscribe-then-publish, so the read never races the connect and can't false-fail.
Separate topics for liveness (probe/live) vs readiness (probe/ready) so their rotating retained values never collide.

Dedicated least-privilege `health` user (not admin)

The probes are exec probes running inside the mosquitto container. admin's password is reachable there only because the startup shim hashes every user into the passwd file — but admin is readwrite #, meant for an external workstation client, so it's the wrong credential for a probe.

New mosquitto-health-credentials-bootstrap Job → Secret mosquitto-health-credentials (same per-user pattern as the other six).
ACL: readwrite health/# and nothing else (read is needed for the read-back). A leak can only touch the throwaway probe subtree nothing subscribes to.
HEALTH_PASSWORD env (the shim needs it to build passwd); the in-container probes read it directly.

probe	check	budget
startupProbe	tcpSocket :1883	5s × 12 = 60s
readinessProbe	round-trip	15s × 3, timeout 10s
livenessProbe	round-trip	30s × 6 = 3 min sustained before restart, timeout 10s

Verified live before merge

With the admin cred (already exists — identical command shape):

exact retained round-trip     -> exit 0, value read back matches
negative control (empty topic)-> times out, mismatch, non-zero exit
round-trip latency            -> ~0 ms on loopback (timeout is 10s)

The negative control is the point: the probe fails on a blackhole, not just on a dead TCP port.

SPOF safety (single replica: HA, Shelly, sml2mqtt, open3e, evcc)

Liveness fires slowly — 3 min sustained — so a transient hiccup never bounces the broker.
maxUnavailable: 0 / maxSurge: 1 — the new pod must pass readiness before the old is torn down. Even if the health wiring were wrong, the old broker keeps serving: a stalled rollout, not an outage.

Post-merge

Bootstrap Job creates the secret → pod rolls (old serves until new Ready) → health user round-trips green → 0 restarts, clients stay connected. I'll negative-control a wrong cred to confirm it genuinely fails.

## Why mosquitto's liveness+readiness were `tcpSocket:1883` — proves a listener exists, not that the broker routes. **Even a QoS-1 PUBACK isn't enough**: it only confirms the broker acknowledged the publish at the protocol level. A broker wedged so it ACKs writes but drops them (a write blackhole) would still pass. ## What — a real pub/sub round-trip Each probe publishes a fresh **random value retained** and **reads it back**, succeeding only if the exact bytes round-trip: ```sh V="probe-$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' ')" mosquitto_pub … -q 1 -r -t health/probe/live -m "$V" && [ "$(mosquitto_sub … -t health/probe/live -C 1 -W 5)" = "$V" ] ``` That proves the broker **authenticated, stored the message, and delivered it to a subscriber** — auth + write + storage + read + delivery. A blackhole → read-back empty/mismatch → fail. - **Retained**, not a live subscribe-then-publish, so the read never races the connect and can't false-fail. - **Separate topics** for liveness (`probe/live`) vs readiness (`probe/ready`) so their rotating retained values never collide. ## Dedicated least-privilege `health` user (not admin) The probes are `exec` probes running **inside the mosquitto container**. admin's password is reachable there only because the startup shim hashes every user into the passwd file — but admin is `readwrite #`, meant for an external workstation client, so it's the wrong credential for a probe. - New `mosquitto-health-credentials-bootstrap` Job → Secret `mosquitto-health-credentials` (same per-user pattern as the other six). - ACL: **`readwrite health/#` and nothing else** (read is needed for the read-back). A leak can only touch the throwaway probe subtree nothing subscribes to. - `HEALTH_PASSWORD` env (the shim needs it to build passwd); the in-container probes read it directly. | probe | check | budget | |---|---|---| | startupProbe | tcpSocket :1883 | 5s × 12 = 60s | | readinessProbe | round-trip | 15s × 3, timeout 10s | | livenessProbe | round-trip | 30s × **6 = 3 min** sustained before restart, timeout 10s | ## Verified live before merge With the admin cred (already exists — identical command shape): ``` exact retained round-trip -> exit 0, value read back matches negative control (empty topic)-> times out, mismatch, non-zero exit round-trip latency -> ~0 ms on loopback (timeout is 10s) ``` The negative control is the point: the probe fails on a **blackhole**, not just on a dead TCP port. ## SPOF safety (single replica: HA, Shelly, sml2mqtt, open3e, evcc) 1. **Liveness fires slowly** — 3 min sustained — so a transient hiccup never bounces the broker. 2. **`maxUnavailable: 0` / `maxSurge: 1`** — the new pod must pass readiness before the old is torn down. Even if the health wiring were wrong, the old broker keeps serving: a stalled rollout, not an outage. ## Post-merge Bootstrap Job creates the secret → pod rolls (old serves until new Ready) → health user round-trips green → 0 restarts, clients stay connected. I'll negative-control a wrong cred to confirm it genuinely fails.

k.renner added 1 commit 2026-07-10 18:12:22 +00:00

mosquitto: real MQTT round-trip probes instead of tcpSocket 8ce3343037

The broker's liveness+readiness were tcpSocket:1883 — a bare TCP accept
proves a listener exists, not that the broker routes. A wedged broker
(accepts connections, doesn't process pub/sub) showed healthy.

Replace with an authenticated QoS-1 publish: `mosquitto_pub -q 1` exits 0
only if the broker accepted an authenticated CONNECT and returned a PUBACK
for the PUBLISH — i.e. auth and the routing path both work. A wedged broker
blocks with no PUBACK and the probe times out.

Reuses the existing `admin` credential (readwrite #, already in the
container env as ADMIN_PASSWORD): no new user, ACL, Secret or bootstrap Job,
so nothing new can misfire and restart-loop the broker. The target topic
health/mosquitto-probe has no subscriber, so the message is PUBACK'd and
dropped.

mosquitto is a single-replica SPOF (HA, Shelly, sml2mqtt, open3e, evcc), so
liveness is deliberately slow: failureThreshold 6 x 30s = 3 min of
sustained failure before a restart, timeout 8s. A transient hiccup never
bounces the broker. The startupProbe (tcpSocket) suspends the pub/sub
probes while the container builds passwd/acl and mosquitto binds :1883.

  startupProbe   tcpSocket :1883, 5s x 12 = 60s
  readinessProbe mosquitto_pub, 15s x 3
  livenessProbe  mosquitto_pub, 30s x 6 = 3 min sustained before restart

Verified against the LIVE broker before merge (essential for a SPOF
liveness probe):
  - exact probe command with the real admin cred -> exit 0.
  - negative control (wrong password) -> exit 5 (rejected), proving the
    probe actually validates auth+routing, not just TCP.
  - ADMIN_PASSWORD present in the container env (32 chars).
kubectl kustomize builds.

Owner chose full pub/sub liveness+readiness (over keeping tcpSocket) with
the SPOF risk understood; reusing the proven admin credential and the
3-minute liveness threshold are the mitigations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner force-pushed fix/mosquitto-mqtt-probes from 8ce3343037 to 7f39d73250

2026-07-10 18:20:27 +00:00

Compare

k.renner force-pushed fix/mosquitto-mqtt-probes from 7f39d73250 to 50f5f7d5ee

2026-07-10 18:26:24 +00:00

Compare

k.renner merged commit 0ce6b209a2 into master

2026-07-10 18:27:25 +00:00

k.renner referenced this issue from a commit

2026-07-10 18:27:27 +00:00

Merge pull request 'mosquitto: real MQTT round-trip probes instead of tcpSocket' (#150) from fix/mosquitto-mqtt-probes into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#150