matter-server: add startup/readiness/liveness probes #149

2026-07-10T17:46:52Z

k.renner commented

2026-07-10 17:46:52 +00:00

Why

matter-server had zero probes (2026-07-10 audit, flagged HIGH — it's the otbr sibling and had a real incident). A hung or crashed server kept the pod Running while HA's WebSocket to it was dead.

What

GET / on :5580 returns 200 from the aiohttp control server — a real "server up and answering HTTP" check, stronger than a bare tcpSocket. httpGet targets the pod IP, exactly where the server binds (--listen-address=$(POD_IP), #144). Verified on the pod IP: GET / → 200, /ws upgrade → 101.

probe	check	budget
startupProbe	`GET /:5580`	10s × 30 = 5 min (SDK loads fabric, binds socket)
readinessProbe	`GET /:5580`	15s × 3
livenessProbe	`GET /:5580`	30s × 5 = 2.5 min sustained before restart

Liveness is deliberately conservative. A restart churns the whole Matter fabric (~13 nodes re-subscribe over minutes, observed in #144), so a transient blip must never trigger it. The check is a loopback-reliable aiohttp handler — very low false-positive. startupProbe suspends the other two during the slow fabric load.

Scope, stated honestly

This catches server-down / hung / not-listening — the current gap. It does not catch "fabric lost / 0 nodes": that failure leaves the :5580 server (on eth0) answering while net1/SDK is broken, so no :5580 probe can see it. Closing that would need an app-level node-count signal matter-server doesn't expose over HTTP — out of scope, not an oversight.

Post-merge

Pod rolls, startupProbe passes once :5580 answers, Ready, and the 13-node fabric re-subscribes (expected churn on any matter-server restart — same as #144).

## Why matter-server had **zero probes** (2026-07-10 audit, flagged HIGH — it's the otbr sibling and had a real incident). A hung or crashed server kept the pod `Running` while HA's WebSocket to it was dead. ## What `GET /` on `:5580` returns `200` from the aiohttp control server — a real "server up and answering HTTP" check, stronger than a bare tcpSocket. httpGet targets the pod IP, exactly where the server binds (`--listen-address=$(POD_IP)`, #144). Verified on the pod IP: `GET / → 200`, `/ws` upgrade → `101`. | probe | check | budget | |---|---|---| | startupProbe | `GET /:5580` | 10s × 30 = 5 min (SDK loads fabric, binds socket) | | readinessProbe | `GET /:5580` | 15s × 3 | | livenessProbe | `GET /:5580` | 30s × **5 = 2.5 min** sustained before restart | **Liveness is deliberately conservative.** A restart churns the whole Matter fabric (~13 nodes re-subscribe over minutes, observed in #144), so a transient blip must never trigger it. The check is a loopback-reliable aiohttp handler — very low false-positive. startupProbe suspends the other two during the slow fabric load. ## Scope, stated honestly This catches **server-down / hung / not-listening** — the current gap. It does **not** catch "fabric lost / 0 nodes": that failure leaves the `:5580` server (on eth0) answering while `net1`/SDK is broken, so no `:5580` probe can see it. Closing that would need an app-level node-count signal matter-server doesn't expose over HTTP — out of scope, not an oversight. ## Post-merge Pod rolls, startupProbe passes once `:5580` answers, Ready, and the 13-node fabric re-subscribes (expected churn on any matter-server restart — same as #144).

k.renner added 1 commit 2026-07-10 17:46:52 +00:00

matter-server: add startup/readiness/liveness probes debf76f548

matter-server had NO probes (2026-07-10 audit, flagged HIGH — it's the
otbr sibling and had a real incident). A hung or crashed server kept the
pod Running while HA's WebSocket to it was dead.

`GET /` on :5580 returns 200 from the aiohttp control server, so it's a
real "server up and answering HTTP" check, stronger than a bare tcpSocket.
httpGet targets the pod IP, which is exactly where the server binds
(--listen-address=$(POD_IP), PR #144) — verified GET / -> 200 on the pod
IP, and the /ws upgrade returns 101.

  startupProbe   GET / :5580, 10s x 30 = 5 min. Suspends the other two
                 while the SDK loads the fabric and binds the socket.
  readinessProbe GET / :5580, 15s x 3. Gates HA's ClusterIP path.
  livenessProbe  GET / :5580, 30s x 5 = 2.5 min sustained failure before a
                 restart. Deliberately conservative: a restart churns the
                 whole Matter fabric (~13 nodes re-subscribe over minutes,
                 observed in #144), so a transient blip must never trigger
                 it. The check is a loopback-reliable aiohttp handler, very
                 low false-positive.

SCOPE, stated honestly: this catches server-down / hung / not-listening —
the current gap. It does NOT catch "fabric lost / 0 nodes": that failure
leaves the :5580 server (on eth0) answering while net1/SDK is broken, so
no :5580 probe can see it. Closing that would need an app-level node-count
signal matter-server doesn't expose over HTTP; out of scope.

Verified: kubectl kustomize builds; rendered probes as intended. Post-merge:
pod rolls, startupProbe passes once :5580 answers, Ready, and the 13-node
fabric re-subscribes (expected churn on any matter-server restart).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 0da51b0e8f into master

2026-07-10 18:27:35 +00:00

k.renner referenced this issue from a commit

2026-07-10 18:27:36 +00:00

Merge pull request 'matter-server: add startup/readiness/liveness probes' (#149) from fix/matter-server-probes into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#149