homelab

konairius/homelab

Fork 0

Author	SHA1	Message	Date
claude-botandClaude Opus 5	2c5c416787	fix(chrony): step the clock at boot so a bad RTC self-corrects chrony.conf.j2 had no makestep directive, so chronyd only ever slewed. A large offset is therefore never closed in practice. edge-1 hit this on 2026-08-04. Its CMOS battery is dead, so after a power loss the RTC came up reading 2018-08-20. The kernel set the system clock from it: PM: RTC time: 09:18:56, date: 2018-08-20 rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC systemd floored that forward to its own build date: systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC and it ended up at ~2026-07-19, roughly 16 days behind. chronyd then synced to ptbtime1 at stratum 2 with a sub-millisecond RMS offset, and still reported "System time: 1376235.375 seconds slow of NTP time" while correcting none of it. With host time before the cluster certificate's NotBefore, every outbound TLS handshake failed x509 validation: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-07-20T16:15:36+02:00 is before 2026-07-23T06:15:01Z That took out the local mosquitto -> cluster MQTT bridge (all Vitocal 250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously. Because the same skew broke log shipping, the outage was invisible to log-based alerting. The 16 days is incidental - it is just where the software flooring landed. The fault this commit addresses is that nothing steps. Recovery required a manual `chronyc makestep`; the directive lets it self-correct at boot. The limit keeps steps confined to the first few updates after chronyd starts, so a long-running NTP server is never jumped out from under its workloads mid-flight. Note this does not fix edge-1's dead RTC battery, which is hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 17:28:41 +02:00

Author

SHA1

Message

Date

claude-botandClaude Opus 5

2c5c416787

fix(chrony): step the clock at boot so a bad RTC self-corrects

chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

edge-1 hit this on 2026-08-04. Its CMOS battery is dead, so after a power
loss the RTC came up reading 2018-08-20. The kernel set the system clock
from it:

  PM: RTC time: 09:18:56, date: 2018-08-20
  rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC

systemd floored that forward to its own build date:

  systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

and it ended up at ~2026-07-19, roughly 16 days behind. chronyd then
synced to ptbtime1 at stratum 2 with a sub-millisecond RMS offset, and
still reported "System time: 1376235.375 seconds slow of NTP time" while
correcting none of it.

With host time before the cluster certificate's NotBefore, every outbound
TLS handshake failed x509 validation:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

The 16 days is incidental - it is just where the software flooring landed.
The fault this commit addresses is that nothing steps. Recovery required a
manual `chronyc makestep`; the directive lets it self-correct at boot.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Note this does not fix edge-1's dead RTC battery, which is hardware.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-05 17:28:41 +02:00

Compare commits

Diff Content Not Available