otbr: make the probes watch Thread, not just the process #141
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  k.rennerandClaude Opus 4.8
|
e174e1d227 |
otbr: make the probes watch Thread, not just the process
The only probe on this container was `pgrep -x otbr-agent`. It caught the
upstream entrypoint's habit of keeping PID 1 alive via `tail -f` after the
agent dies. It could not catch what actually happened on 2026-06-27:
otbr-agent ran for twelve days with `ot-ctl state` = disabled and the
border router stopped, while the pod reported Running 1/1 the whole time.
A live process is not a live Thread mesh.
Add a startupProbe and a readinessProbe, and teach liveness the one
protocol state a restart repairs.
startupProbe ot-ctl's control socket answers. 10s x 60 = 10 minutes,
covering the apt install, the socat TCP<->PTY bridge to
the SLZB radio, and the spinel handshake. While it runs
the other probes are suspended, so the entrypoint gets to
finish activating Thread before liveness may act.
readinessProbe state is leader|router|child|detached. `detached` counts
as ready: the stack is up and attaching, REST still
serves, and treating normal mesh churn as NotReady would
flap HA's OTBR integration for nothing. `disabled` does
not count -- it is the twelve-day failure and the one
state that never resolves on its own.
livenessProbe pgrep, plus: restart when state is `disabled` AND a
dataset is persisted. That combination means activation
never ran, and since #139 a restart re-runs a path that
polls for the dataset and starts Thread. Self-healing.
The dataset guard on liveness is load-bearing. On a clean install there is
no dataset and `disabled` is correct until otbr-dataset-bootstrap seeds one
over `kubectl exec`. Restarting there would fight the Job, which has
already been seen to die with BackoffLimitExceeded against a
CrashLoopBackOff. Likewise, if ot-ctl does not answer, the state check
falls through and liveness rests on pgrep alone -- a wedged control socket
must not trigger a restart storm.
All three probes strip CR. ot-ctl speaks a serial-console protocol and
terminates lines with CRLF, so `ot-ctl state` yields `leader\r`, which
matches no shell glob (#140). curl and wget are absent from this image, so
the probes go through ot-ctl rather than REST.
Removing endpoints affects only HA's OTBR REST integration on :8081.
matter-server does not consume this Service -- it learns the OMR prefix by
RA on its own macvlan interface.
Verified. The three scripts were run against the live pod (state=leader):
startup/readiness/liveness all exit 0. The states that cannot be produced
on a healthy border router were driven with stubbed ot-ctl/pgrep:
state dataset agent | startup readiness liveness
leader yes alive | 0 0 0
router yes alive | 0 0 0
detached yes alive | 0 0 0
disabled yes alive | 0 1 1 <- the bug
disabled no alive | 0 1 0 <- clean install
<wedged> yes alive | 1 1 0
leader yes dead | 0 0 1
`sh -n` clean on both probe scripts as rendered by kustomize; kustomize
builds. Entrypoint untouched.
Caveat: an operator running `ot-ctl thread stop` by hand on a provisioned
node gets the container restarted ~3 minutes later. Deliberate --
`disabled` with a dataset is a fault on this deployment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|