fix(chrony): step the clock at boot so a bad RTC self-corrects
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful

chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-05 16:35:18 +02:00
co-authored by Claude Opus 5
parent 11bf13883f
commit 7fcaa4b035
2 changed files with 17 additions and 0 deletions
@@ -18,6 +18,18 @@ chrony_upstream_pools:
# Local stratum if disconnected from upstream
chrony_local_stratum: 10
# Step (rather than slew) the clock when the offset exceeds the threshold, but
# only for the first N updates after chronyd starts. Without this, chronyd only
# ever slews, and a large offset is never closed in practice: on 2026-08-04
# edge-1 rebooted with an RTC ~16 days in the past and chronyd sat synchronised
# to ptbtime1 reporting "System time: 1376235s slow" while correcting nothing.
# Every outbound TLS handshake then failed x509 NotBefore validation, taking out
# the MQTT bridge (heating telemetry) and the Loki log shipper at once.
# Limiting to the first few updates keeps steps confined to boot, so a
# long-running server is never jumped out from under its workloads.
chrony_makestep_threshold: 1.0
chrony_makestep_limit: 3
# Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
# (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
# error per interface when asked for it. Overridden to false in
@@ -14,6 +14,11 @@ pool {{ pool }}
# Record the rate at which the system clock gains/losses time
driftfile /var/lib/chrony/chrony.drift
# Step the clock instead of slewing it when the offset is large, limited to the
# first few updates after start so only boot-time corrections jump. Without a
# makestep directive chronyd slews exclusively and never closes a large gap.
makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
# Allow NTP client access from local networks
{% for network in chrony_allowed_networks %}
allow {{ network }}