konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

otbr: make the probes watch Thread, not just the process

Browse Source 
The only probe on this container was `pgrep -x otbr-agent`. It caught the
upstream entrypoint's habit of keeping PID 1 alive via `tail -f` after the
agent dies. It could not catch what actually happened on 2026-06-27:
otbr-agent ran for twelve days with `ot-ctl state` = disabled and the
border router stopped, while the pod reported Running 1/1 the whole time.
A live process is not a live Thread mesh.

Add a startupProbe and a readinessProbe, and teach liveness the one
protocol state a restart repairs.

  startupProbe   ot-ctl's control socket answers. 10s x 60 = 10 minutes,
                 covering the apt install, the socat TCP<->PTY bridge to
                 the SLZB radio, and the spinel handshake. While it runs
                 the other probes are suspended, so the entrypoint gets to
                 finish activating Thread before liveness may act.

  readinessProbe state is leader|router|child|detached. `detached` counts
                 as ready: the stack is up and attaching, REST still
                 serves, and treating normal mesh churn as NotReady would
                 flap HA's OTBR integration for nothing. `disabled` does
                 not count -- it is the twelve-day failure and the one
                 state that never resolves on its own.

  livenessProbe  pgrep, plus: restart when state is `disabled` AND a
                 dataset is persisted. That combination means activation
                 never ran, and since #139 a restart re-runs a path that
                 polls for the dataset and starts Thread. Self-healing.

The dataset guard on liveness is load-bearing. On a clean install there is
no dataset and `disabled` is correct until otbr-dataset-bootstrap seeds one
over `kubectl exec`. Restarting there would fight the Job, which has
already been seen to die with BackoffLimitExceeded against a
CrashLoopBackOff. Likewise, if ot-ctl does not answer, the state check
falls through and liveness rests on pgrep alone -- a wedged control socket
must not trigger a restart storm.

All three probes strip CR. ot-ctl speaks a serial-console protocol and
terminates lines with CRLF, so `ot-ctl state` yields `leader\r`, which
matches no shell glob (#140). curl and wget are absent from this image, so
the probes go through ot-ctl rather than REST.

Removing endpoints affects only HA's OTBR REST integration on :8081.
matter-server does not consume this Service -- it learns the OMR prefix by
RA on its own macvlan interface.

Verified. The three scripts were run against the live pod (state=leader):
startup/readiness/liveness all exit 0. The states that cannot be produced
on a healthy border router were driven with stubbed ot-ctl/pgrep:

  state       dataset  agent  | startup readiness liveness
  leader      yes      alive  |    0        0         0
  router      yes      alive  |    0        0         0
  detached    yes      alive  |    0        0         0
  disabled    yes      alive  |    0        1         1   <- the bug
  disabled    no       alive  |    0        1         0   <- clean install
  <wedged>    yes      alive  |    1        1         0
  leader      yes      dead   |    0        0         1

`sh -n` clean on both probe scripts as rendered by kustomize; kustomize
builds. Entrypoint untouched.

Caveat: an operator running `ot-ctl thread stop` by hand on a provisioned
node gets the container restarted ~3 minutes later. Deliberate --
`disabled` with a dataset is a fault on this deployment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
k.renner
2026-07-09 20:23:26 +02:00
co-authored by Claude Opus 4.8
parent 3ec270fa89
commit e174e1d227
1 changed files with 79 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/apps/otbr/deployment.yaml
+79 -7
View File
@@ -236,20 +236,92 @@ spec:
# and re-election on every restart).
- name: data
mountPath: /var/lib/thread
# Probes must watch the *protocol*, not just the process.
#
# The upstream entrypoint backgrounds otbr-agent and ends in
# `tail -f /var/log/syslog`, so PID 1 stays alive even when
# otbr-agent has died — the pod stays Ready while the radio
# is broken. A liveness probe restores honest signaling: if
# otbr-agent isn't running, the container is restarted and
# CrashLoopBackOff surfaces the failure.
# otbr-agent has died. `pgrep -x otbr-agent` fixed that half.
# It does not fix the other half: on 2026-06-27 otbr-agent ran
# happily for twelve days with `ot-ctl state` = `disabled` and
# the border router `stopped`, and the pod reported Running 1/1
# the entire time. A live process is not a live Thread mesh.
#
# All three probes must strip CR: ot-ctl speaks a serial-console
# protocol and terminates lines with CRLF, so `ot-ctl state`
# yields `leader\r`, which matches no shell glob (see the
# entrypoint's attach check and PR #140). curl/wget are absent
# from this image, so probes go through ot-ctl, not REST.
#
# startupProbe: the agent's control socket answers. Covers the
# apt install, the socat TCP↔PTY bridge to the SLZB radio, and
# the spinel handshake. Generous: 10s × 60 = 10 minutes. While
# it runs, the other two probes are suspended, so the entrypoint
# gets to finish activating Thread before liveness may act.
startupProbe:
exec:
command:
- sh
- -c
- ot-ctl state >/dev/null 2>&1
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 60
# readinessProbe: Thread is up. `detached` counts as ready —
# it means the stack is running and trying to attach, REST
# still serves, and treating normal mesh churn as NotReady
# would flap HA's OTBR integration for nothing. `disabled` does
# NOT count: that is the twelve-day failure, and it is the one
# state that never resolves on its own.
#
# Removing endpoints only affects HA's OTBR REST integration on
# :8081. matter-server does not consume this Service — it learns
# the OMR prefix by RA on its own macvlan interface.
readinessProbe:
exec:
command:
- sh
- -c
- |
S=$(ot-ctl state 2>/dev/null | head -1 | tr -d '\r')
case "$S" in
leader|router|child|detached) exit 0 ;;
esac
exit 1
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 2
# livenessProbe: the agent process, plus the one protocol state
# a restart actually repairs. `disabled` WITH a persisted dataset
# means the entrypoint's activation never ran (or was undone) —
# restarting re-runs it, and since #139 that path polls for the
# dataset and starts Thread. Self-healing.
#
# The dataset guard is load-bearing. On a clean install there is
# no dataset and `disabled` is the correct state until the
# otbr-dataset-bootstrap Job seeds one over `kubectl exec`.
# Restarting there would fight the Job — which has already been
# seen to die with BackoffLimitExceeded against a CrashLoopBackOff.
#
# If ot-ctl does not answer, S is empty, the state check falls
# through, and liveness rests on pgrep alone. A busy or wedged
# control socket must not trigger a restart storm.
#
# Caveat: an operator running `ot-ctl thread stop` by hand on a
# provisioned node gets the container restarted ~3 minutes later.
# That is deliberate — `disabled` with a dataset is a fault here.
livenessProbe:
exec:
command:
- sh
- -c
- pgrep -x otbr-agent >/dev/null
initialDelaySeconds: 60
periodSeconds: 15
- |
pgrep -x otbr-agent >/dev/null || exit 1
S=$(ot-ctl state 2>/dev/null | head -1 | tr -d '\r')
[ "$S" = "disabled" ] || exit 0
ot-ctl dataset active 2>&1 | grep -qE "Active Timestamp|Network Name" || exit 0
exit 1
periodSeconds: 60
timeoutSeconds: 15
failureThreshold: 3
volumes:
- name: data