Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 2c5c416787 fix(chrony): step the clock at boot so a bad RTC self-corrects
chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

edge-1 hit this on 2026-08-04. Its CMOS battery is dead, so after a power
loss the RTC came up reading 2018-08-20. The kernel set the system clock
from it:

  PM: RTC time: 09:18:56, date: 2018-08-20
  rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC

systemd floored that forward to its own build date:

  systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

and it ended up at ~2026-07-19, roughly 16 days behind. chronyd then
synced to ptbtime1 at stratum 2 with a sub-millisecond RMS offset, and
still reported "System time: 1376235.375 seconds slow of NTP time" while
correcting none of it.

With host time before the cluster certificate's NotBefore, every outbound
TLS handshake failed x509 validation:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

The 16 days is incidental - it is just where the software flooring landed.
The fault this commit addresses is that nothing steps. Recovery required a
manual `chronyc makestep`; the directive lets it self-correct at boot.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Note this does not fix edge-1's dead RTC battery, which is hardware.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 17:28:41 +02:00
2 changed files with 25 additions and 0 deletions
@@ -18,6 +18,26 @@ chrony_upstream_pools:
# Local stratum if disconnected from upstream
chrony_local_stratum: 10
# Step (rather than slew) the clock when the offset exceeds the threshold, but
# only for the first N updates after chronyd starts. Without a makestep
# directive chronyd only ever slews, so a large offset is never closed in
# practice.
#
# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
# that forward to its own built-in epoch (2026-06-26), and it ended up at
# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
# With host time before the cluster certificate's NotBefore, every outbound TLS
# handshake failed x509 validation, taking out the MQTT bridge (heating
# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
# is just where the software flooring landed; the fault is that nothing steps.
#
# Limiting to the first few updates keeps steps confined to boot, so a
# long-running server is never jumped out from under its workloads.
chrony_makestep_threshold: 1.0
chrony_makestep_limit: 3
# Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
# (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
# error per interface when asked for it. Overridden to false in
@@ -14,6 +14,11 @@ pool {{ pool }}
# Record the rate at which the system clock gains/losses time
driftfile /var/lib/chrony/chrony.drift
# Step the clock instead of slewing it when the offset is large, limited to the
# first few updates after start so only boot-time corrections jump. Rationale
# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
# Allow NTP client access from local networks
{% for network in chrony_allowed_networks %}
allow {{ network }}