From e174e1d227a7779a4f71fb6fdf8aaef973eda0b0 Mon Sep 17 00:00:00 2001 From: Konstantin Renner Date: Thu, 9 Jul 2026 20:23:26 +0200 Subject: [PATCH] otbr: make the probes watch Thread, not just the process The only probe on this container was `pgrep -x otbr-agent`. It caught the upstream entrypoint's habit of keeping PID 1 alive via `tail -f` after the agent dies. It could not catch what actually happened on 2026-06-27: otbr-agent ran for twelve days with `ot-ctl state` = disabled and the border router stopped, while the pod reported Running 1/1 the whole time. A live process is not a live Thread mesh. Add a startupProbe and a readinessProbe, and teach liveness the one protocol state a restart repairs. startupProbe ot-ctl's control socket answers. 10s x 60 = 10 minutes, covering the apt install, the socat TCP<->PTY bridge to the SLZB radio, and the spinel handshake. While it runs the other probes are suspended, so the entrypoint gets to finish activating Thread before liveness may act. readinessProbe state is leader|router|child|detached. `detached` counts as ready: the stack is up and attaching, REST still serves, and treating normal mesh churn as NotReady would flap HA's OTBR integration for nothing. `disabled` does not count -- it is the twelve-day failure and the one state that never resolves on its own. livenessProbe pgrep, plus: restart when state is `disabled` AND a dataset is persisted. That combination means activation never ran, and since #139 a restart re-runs a path that polls for the dataset and starts Thread. Self-healing. The dataset guard on liveness is load-bearing. On a clean install there is no dataset and `disabled` is correct until otbr-dataset-bootstrap seeds one over `kubectl exec`. Restarting there would fight the Job, which has already been seen to die with BackoffLimitExceeded against a CrashLoopBackOff. Likewise, if ot-ctl does not answer, the state check falls through and liveness rests on pgrep alone -- a wedged control socket must not trigger a restart storm. All three probes strip CR. ot-ctl speaks a serial-console protocol and terminates lines with CRLF, so `ot-ctl state` yields `leader\r`, which matches no shell glob (#140). curl and wget are absent from this image, so the probes go through ot-ctl rather than REST. Removing endpoints affects only HA's OTBR REST integration on :8081. matter-server does not consume this Service -- it learns the OMR prefix by RA on its own macvlan interface. Verified. The three scripts were run against the live pod (state=leader): startup/readiness/liveness all exit 0. The states that cannot be produced on a healthy border router were driven with stubbed ot-ctl/pgrep: state dataset agent | startup readiness liveness leader yes alive | 0 0 0 router yes alive | 0 0 0 detached yes alive | 0 0 0 disabled yes alive | 0 1 1 <- the bug disabled no alive | 0 1 0 <- clean install yes alive | 1 1 0 leader yes dead | 0 0 1 `sh -n` clean on both probe scripts as rendered by kustomize; kustomize builds. Entrypoint untouched. Caveat: an operator running `ot-ctl thread stop` by hand on a provisioned node gets the container restarted ~3 minutes later. Deliberate -- `disabled` with a dataset is a fault on this deployment. Co-Authored-By: Claude Opus 4.8 (1M context) --- kubernetes/apps/otbr/deployment.yaml | 86 +++++++++++++++++++++++++--- 1 file changed, 79 insertions(+), 7 deletions(-) diff --git a/kubernetes/apps/otbr/deployment.yaml b/kubernetes/apps/otbr/deployment.yaml index f8cd9dd1..6792573d 100644 --- a/kubernetes/apps/otbr/deployment.yaml +++ b/kubernetes/apps/otbr/deployment.yaml @@ -236,20 +236,92 @@ spec: # and re-election on every restart). - name: data mountPath: /var/lib/thread + # Probes must watch the *protocol*, not just the process. + # # The upstream entrypoint backgrounds otbr-agent and ends in # `tail -f /var/log/syslog`, so PID 1 stays alive even when - # otbr-agent has died — the pod stays Ready while the radio - # is broken. A liveness probe restores honest signaling: if - # otbr-agent isn't running, the container is restarted and - # CrashLoopBackOff surfaces the failure. + # otbr-agent has died. `pgrep -x otbr-agent` fixed that half. + # It does not fix the other half: on 2026-06-27 otbr-agent ran + # happily for twelve days with `ot-ctl state` = `disabled` and + # the border router `stopped`, and the pod reported Running 1/1 + # the entire time. A live process is not a live Thread mesh. + # + # All three probes must strip CR: ot-ctl speaks a serial-console + # protocol and terminates lines with CRLF, so `ot-ctl state` + # yields `leader\r`, which matches no shell glob (see the + # entrypoint's attach check and PR #140). curl/wget are absent + # from this image, so probes go through ot-ctl, not REST. + # + # startupProbe: the agent's control socket answers. Covers the + # apt install, the socat TCP↔PTY bridge to the SLZB radio, and + # the spinel handshake. Generous: 10s × 60 = 10 minutes. While + # it runs, the other two probes are suspended, so the entrypoint + # gets to finish activating Thread before liveness may act. + startupProbe: + exec: + command: + - sh + - -c + - ot-ctl state >/dev/null 2>&1 + periodSeconds: 10 + timeoutSeconds: 10 + failureThreshold: 60 + # readinessProbe: Thread is up. `detached` counts as ready — + # it means the stack is running and trying to attach, REST + # still serves, and treating normal mesh churn as NotReady + # would flap HA's OTBR integration for nothing. `disabled` does + # NOT count: that is the twelve-day failure, and it is the one + # state that never resolves on its own. + # + # Removing endpoints only affects HA's OTBR REST integration on + # :8081. matter-server does not consume this Service — it learns + # the OMR prefix by RA on its own macvlan interface. + readinessProbe: + exec: + command: + - sh + - -c + - | + S=$(ot-ctl state 2>/dev/null | head -1 | tr -d '\r') + case "$S" in + leader|router|child|detached) exit 0 ;; + esac + exit 1 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 2 + # livenessProbe: the agent process, plus the one protocol state + # a restart actually repairs. `disabled` WITH a persisted dataset + # means the entrypoint's activation never ran (or was undone) — + # restarting re-runs it, and since #139 that path polls for the + # dataset and starts Thread. Self-healing. + # + # The dataset guard is load-bearing. On a clean install there is + # no dataset and `disabled` is the correct state until the + # otbr-dataset-bootstrap Job seeds one over `kubectl exec`. + # Restarting there would fight the Job — which has already been + # seen to die with BackoffLimitExceeded against a CrashLoopBackOff. + # + # If ot-ctl does not answer, S is empty, the state check falls + # through, and liveness rests on pgrep alone. A busy or wedged + # control socket must not trigger a restart storm. + # + # Caveat: an operator running `ot-ctl thread stop` by hand on a + # provisioned node gets the container restarted ~3 minutes later. + # That is deliberate — `disabled` with a dataset is a fault here. livenessProbe: exec: command: - sh - -c - - pgrep -x otbr-agent >/dev/null - initialDelaySeconds: 60 - periodSeconds: 15 + - | + pgrep -x otbr-agent >/dev/null || exit 1 + S=$(ot-ctl state 2>/dev/null | head -1 | tr -d '\r') + [ "$S" = "disabled" ] || exit 0 + ot-ctl dataset active 2>&1 | grep -qE "Active Timestamp|Network Name" || exit 0 + exit 1 + periodSeconds: 60 + timeoutSeconds: 15 failureThreshold: 3 volumes: - name: data -- 2.54.0