From 50f5f7d5ee8f2863214f7f7eedc6bfd9826ae2f5 Mon Sep 17 00:00:00 2001 From: Konstantin Renner Date: Fri, 10 Jul 2026 20:11:23 +0200 Subject: [PATCH] mosquitto: pub/sub round-trip probes via a dedicated health user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The broker's liveness+readiness were tcpSocket:1883 — a bare TCP accept proves a listener exists, not that the broker routes. Even a QoS-1 PUBACK proves only that the broker acknowledged the publish at the protocol level; a broker wedged so it ACKs writes but drops them (a write blackhole) would still pass. So each probe publishes a fresh RANDOM value retained and reads it back, succeeding only if the exact bytes round-trip. That proves the broker authenticated, stored the message, and delivered it to a subscriber: auth + write + storage + read + delivery. Retained (not a live subscribe-then-publish) so the read never races the connect and can't false-fail; liveness and readiness use separate topics (probe/live vs probe/ready) so their rotating retained values never collide. Runs as a dedicated least-privilege `health` user, not admin. The probes are exec probes running INSIDE the mosquitto container; admin's password is only reachable there because the startup shim hashes every user into the passwd file, but admin is readwrite # and meant for an external workstation client, so it's the wrong credential for a probe. - new mosquitto-health-credentials-bootstrap Job -> Secret mosquitto-health-credentials (same per-user pattern as the other six). - ACL: readwrite health/# and NOTHING else (read needed for the read-back). A leak can only touch the throwaway probe subtree that nothing subscribes to. - HEALTH_PASSWORD env (the shim needs it to build passwd); the in-container exec probes read it directly. startupProbe tcpSocket :1883, 5s x 12 = 60s (passwd/acl build + bind) readinessProbe round-trip, 15s x 3, timeout 10s livenessProbe round-trip, 30s x 6 = 3 min sustained before restart, timeout 10s Verified live before merge (with the admin cred, which already exists — identical command shape): - exact retained round-trip script -> exit 0, value read back matches. - negative control (subscribe to an empty topic) -> times out, mismatch, non-zero exit. Confirms the probe fails on a blackhole, not just on TCP. - round-trip latency ~0 ms on loopback, far under the 10s timeout. SPOF safety (single replica; HA, Shelly, sml2mqtt, open3e, evcc): - liveness fires slowly (3 min sustained) so a transient hiccup never bounces the broker. - Deployment is maxUnavailable:0 / maxSurge:1: the new pod must pass readiness before the old is torn down, so even if the health wiring were wrong the old broker keeps serving — a stalled rollout, not an outage. kubectl kustomize builds. Post-merge: bootstrap Job creates the secret, pod rolls (old serves until new Ready), health user round-trips green, 0 restarts, clients stay connected; negative-control a wrong cred/blackhole. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../home-assistant/mosquitto-bootstrap.yaml | 68 +++++++++++++++ .../home-assistant/mosquitto-deployment.yaml | 85 +++++++++++++++++-- 2 files changed, 147 insertions(+), 6 deletions(-) diff --git a/kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml b/kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml index 0008942c..7286194b 100644 --- a/kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml +++ b/kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml @@ -369,3 +369,71 @@ spec: memory: 32Mi limits: memory: 64Mi +--- +# Broker user `health`: a dedicated, least-privilege credential used ONLY by +# mosquitto's own liveness/readiness probes (mosquitto-deployment.yaml). Its +# ACL is `write health/#` and nothing else — it cannot read any topic or +# write outside the throwaway probe subtree, so a leak is worthless (nothing +# subscribes to health/#). Kept on its own Secret, like every other user, so +# the probe never has to borrow the admin credential. +apiVersion: batch/v1 +kind: Job +metadata: + name: mosquitto-health-credentials-bootstrap + namespace: home-assistant +spec: + backoffLimit: 4 + template: + metadata: + labels: + app.kubernetes.io/name: mosquitto-bootstrap + spec: + serviceAccountName: mosquitto-bootstrap + restartPolicy: OnFailure + containers: + - name: bootstrap + image: alpine:3.21 + command: + - sh + - -c + - | + set -eu + apk add --no-cache curl jq >/dev/null + + NS=home-assistant + NAME=mosquitto-health-credentials + TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) + CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + API=https://kubernetes.default.svc + + if curl -fsSL --cacert "$CA" -H "Authorization: Bearer $TOKEN" \ + "$API/api/v1/namespaces/$NS/secrets/$NAME" >/dev/null 2>&1; then + echo "$NAME already exists; nothing to do." + exit 0 + fi + + PW=$(tr -dc 'A-Za-z0-9' /dev/null + + echo "generated $NAME" + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi diff --git a/kubernetes/apps/home-assistant/mosquitto-deployment.yaml b/kubernetes/apps/home-assistant/mosquitto-deployment.yaml index 7a667cae..05151967 100644 --- a/kubernetes/apps/home-assistant/mosquitto-deployment.yaml +++ b/kubernetes/apps/home-assistant/mosquitto-deployment.yaml @@ -72,6 +72,7 @@ spec: mosquitto_passwd -b /mosquitto/auth/passwd sml2mqtt "$SML2MQTT_PASSWORD" mosquitto_passwd -b /mosquitto/auth/passwd open3e "$OPEN3E_PASSWORD" mosquitto_passwd -b /mosquitto/auth/passwd evcc "$EVCC_PASSWORD" + mosquitto_passwd -b /mosquitto/auth/passwd health "$HEALTH_PASSWORD" # ACL: HA keeps full access (subscribes to everything); the # shellies user is `readwrite shellies/#` (devices publish state # and subscribe to their command topics) + `write homeassistant/#` @@ -119,6 +120,16 @@ spec: topic read open3e/# topic readwrite evcc/# topic read energy/# + + # `health` is the broker's own probe credential (see the + # liveness/readiness probes below). Scoped to health/# and + # nothing else: the probes publish a random value retained and + # read it back to prove the broker actually stores+delivers + # (not just PUBACKs) — so the user needs read AND write, but + # only under health/#. A leak can only touch the throwaway + # probe subtree that nothing else subscribes to. + user health + topic readwrite health/# EOF chown 1883:1883 /mosquitto/auth/passwd /mosquitto/auth/acl chmod 0600 /mosquitto/auth/passwd /mosquitto/auth/acl @@ -154,6 +165,15 @@ spec: secretKeyRef: name: mosquitto-evcc-credentials key: password + # Probe-only credential (mosquitto-health-credentials-bootstrap). + # Present here for the same reason as the others — the startup + # shim hashes it into the passwd file — and reused by the exec + # probes below, which run inside this container. + - name: HEALTH_PASSWORD + valueFrom: + secretKeyRef: + name: mosquitto-health-credentials + key: password ports: - name: mqtt containerPort: 1883 @@ -170,15 +190,68 @@ spec: - name: tls mountPath: /mosquitto/tls readOnly: true - livenessProbe: + # Real MQTT round-trip probes, replacing the old tcpSocket checks. + # A bare TCP accept proved only that a listener existed. Even a + # QoS-1 PUBACK proves only that the broker acknowledged the publish + # at the protocol level — a broker wedged so it ACKs writes but + # drops them (a write blackhole) would still pass. So each probe + # publishes a fresh RANDOM value retained and reads it back, and + # succeeds only if the exact bytes round-trip. That proves the + # broker authenticated, stored the message, and delivered it to a + # subscriber — auth + write + storage + read + delivery. Verified + # live before merge (retained round-trip returns the value; + # subscribing to an empty topic times out and fails). + # + # Retained (not a live subscribe-then-publish) on purpose: the + # subscriber always gets the stored value regardless of connect + # timing, so there is no race that could false-fail the probe. + # Liveness and readiness use SEPARATE topics (probe/live vs + # probe/ready) so their rotating retained values never collide. + # + # Uses the dedicated least-privilege `health` user (ACL: readwrite + # health/# only — see the passwd/acl build above and the + # mosquitto-health-credentials-bootstrap Job). Its password is in + # this container's env (HEALTH_PASSWORD) because the startup shim + # hashes every user into the passwd file; the exec probes run + # inside this container and read it directly. A leak of this + # credential can only touch the health/# probe subtree, which + # nothing else subscribes to. + # + # mosquitto is a single-replica SPOF (HA, Shelly, sml2mqtt, open3e, + # evcc). Liveness is deliberately slow to fire — failureThreshold 6 + # x 30s = 3 min of sustained failure before a restart — so a + # transient hiccup never bounces the broker every client depends + # on. The startupProbe (tcpSocket) suspends both while the + # container builds the passwd/acl files and mosquitto binds :1883. + startupProbe: tcpSocket: port: mqtt - initialDelaySeconds: 10 - periodSeconds: 30 + periodSeconds: 5 + failureThreshold: 12 readinessProbe: - tcpSocket: - port: mqtt - periodSeconds: 10 + exec: + command: + - sh + - -c + - | + V="probe-$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' ')" + mosquitto_pub -h 127.0.0.1 -p 1883 -u health -P "$HEALTH_PASSWORD" -q 1 -r -t health/probe/ready -m "$V" && + [ "$(mosquitto_sub -h 127.0.0.1 -p 1883 -u health -P "$HEALTH_PASSWORD" -t health/probe/ready -C 1 -W 5)" = "$V" ] + periodSeconds: 15 + timeoutSeconds: 10 + failureThreshold: 3 + livenessProbe: + exec: + command: + - sh + - -c + - | + V="probe-$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' ')" + mosquitto_pub -h 127.0.0.1 -p 1883 -u health -P "$HEALTH_PASSWORD" -q 1 -r -t health/probe/live -m "$V" && + [ "$(mosquitto_sub -h 127.0.0.1 -p 1883 -u health -P "$HEALTH_PASSWORD" -t health/probe/live -C 1 -W 5)" = "$V" ] + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 6 resources: requests: cpu: 20m -- 2.54.0