From debf76f54873be3d8698a13863572b0c810d3685 Mon Sep 17 00:00:00 2001 From: Konstantin Renner Date: Fri, 10 Jul 2026 19:46:26 +0200 Subject: [PATCH] matter-server: add startup/readiness/liveness probes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit matter-server had NO probes (2026-07-10 audit, flagged HIGH — it's the otbr sibling and had a real incident). A hung or crashed server kept the pod Running while HA's WebSocket to it was dead. `GET /` on :5580 returns 200 from the aiohttp control server, so it's a real "server up and answering HTTP" check, stronger than a bare tcpSocket. httpGet targets the pod IP, which is exactly where the server binds (--listen-address=$(POD_IP), PR #144) — verified GET / -> 200 on the pod IP, and the /ws upgrade returns 101. startupProbe GET / :5580, 10s x 30 = 5 min. Suspends the other two while the SDK loads the fabric and binds the socket. readinessProbe GET / :5580, 15s x 3. Gates HA's ClusterIP path. livenessProbe GET / :5580, 30s x 5 = 2.5 min sustained failure before a restart. Deliberately conservative: a restart churns the whole Matter fabric (~13 nodes re-subscribe over minutes, observed in #144), so a transient blip must never trigger it. The check is a loopback-reliable aiohttp handler, very low false-positive. SCOPE, stated honestly: this catches server-down / hung / not-listening — the current gap. It does NOT catch "fabric lost / 0 nodes": that failure leaves the :5580 server (on eth0) answering while net1/SDK is broken, so no :5580 probe can see it. Closing that would need an app-level node-count signal matter-server doesn't expose over HTTP; out of scope. Verified: kubectl kustomize builds; rendered probes as intended. Post-merge: pod rolls, startupProbe passes once :5580 answers, Ready, and the 13-node fabric re-subscribes (expected churn on any matter-server restart). Co-Authored-By: Claude Opus 4.8 (1M context) --- kubernetes/apps/matter-server/deployment.yaml | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/kubernetes/apps/matter-server/deployment.yaml b/kubernetes/apps/matter-server/deployment.yaml index 794559e7..524a91df 100644 --- a/kubernetes/apps/matter-server/deployment.yaml +++ b/kubernetes/apps/matter-server/deployment.yaml @@ -85,6 +85,48 @@ spec: - name: ws containerPort: 5580 protocol: TCP + # matter-server had NO probes — a hung or crashed server kept the + # pod Running while HA's WebSocket to it was dead. These probes fix + # that. `GET /` on :5580 returns 200 from the aiohttp server, so it + # is a real "the control server is up and answering HTTP" check — + # stronger than a bare tcpSocket. httpGet targets the pod IP, which + # is exactly where the server binds (--listen-address=$(POD_IP), + # PR #144), so this reaches the right socket. + # + # SCOPE, honestly: this catches server-down / hung / not-listening. + # It does NOT catch "fabric lost / 0 nodes" — that failure leaves + # the :5580 server (on eth0) answering while the net1 macvlan / SDK + # is broken, so no :5580 probe can see it. Closing that would need + # an app-level node-count signal matter-server doesn't expose over + # HTTP; out of scope here. + # + # Liveness is deliberately conservative: a restart churns the whole + # Matter fabric (~13 nodes re-subscribe over minutes, observed in + # #144). failureThreshold 5 x 30s = 2.5 min of sustained failure + # before a restart, so a transient blip never triggers fabric + # churn. The startupProbe (5 min budget) suspends liveness/readiness + # while the SDK loads the fabric and binds the socket. + startupProbe: + httpGet: + path: / + port: ws + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 30 + readinessProbe: + httpGet: + path: / + port: ws + periodSeconds: 15 + timeoutSeconds: 5 + failureThreshold: 3 + livenessProbe: + httpGet: + path: / + port: ws + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 5 volumeMounts: - name: data mountPath: /data -- 2.54.0