claude-botandClaude Opus 5
601290dcd9
The NTP health check asked "is chronyd alive", never "is this host safe to
take time from". Those came apart on 2026-08-04.
edge-1's CMOS battery is dead. After a power loss its clock came up ~16
days in the past and, with no makestep directive, chronyd never corrected
it. All three health checks passed continuously for ~18h:
- chronyd was active
- `chronyc tracking` responded
- udp/123 was listening
and `local stratum 10` meant chronyd advertised itself to clients as a
valid reference the whole time. keepalived would have kept the VIP on a
host ready to serve a clock from July.
The sharpest part: check 2 ran `chronyc tracking >/dev/null 2>&1` and
discarded the output. The one command that knew the answer was executed
and its answer thrown away. During the incident that output read:
System time : 1376235.375000000 seconds slow of NTP time
This captures the output and adds two assertions:
- offset above keepalived_ntp_max_offset (default 1.0s) fails, so a host
that knows its clock is wrong drops the VIP
- no upstream source selected in `chronyc sources` fails, because with
`local stratum N` chronyd advertises itself as a usable reference even
when it has never synchronised to anything -- upstream chrony docs are
explicit that the local reference activates immediately in that case
Both predicates were validated against live output from edge-1 rather than
assumed: the offset comparison against the real incident value, and the
`chronyc sources` selected-source column against a real run.
Failing the check costs the configured weight (-20). When the WAN is down
every host fails together and loses the same weight, so relative priority
is unchanged and the VIP stays put -- the local-reference fallback still
works. What this prevents is one host with a broken clock winning.
Scope: this hardens the keepalived path, which is legacy. It does not
address the documented end state in plays/install-ntp-ha.yml, where edge-1
becomes the SOLE NTP server -- there, a failing check means no host serves
time at all, converting "serves the wrong time" into "serves no time".
That needs an architectural answer, not a better health check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>