Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 32d12f93d8 docs(chrony): correct the root cause recorded for the makestep fix
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#465 landed the makestep directive with the wrong root cause in its
comment. It claimed edge-1 "rebooted with an RTC ~16 days in the past".
That is false, and the real cause matters for what to do next.

edge-1's CMOS battery is dead. After a power loss the RTC came up reading
2018-08-20 -- roughly eight years stale, not 16 days:

  PM: RTC time: 09:18:56, date: 2018-08-20
  rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC

The clock was then reconstructed entirely in software. systemd floored it
forward to its own build date:

  systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

and it settled at ~2026-07-19, about 16 days behind. So the 16 days is
incidental -- it is just where the flooring landed. There is no 16-day
mechanism, and nothing "drifted". The two real faults are a dead RTC
battery and a chrony config that never steps. #465 fixed the second.

Also collapses the duplicated rationale in chrony.conf.j2 to a pointer at
defaults/main.yml, as suggested in review on #465, so the explanation
lives in exactly one place.

No behaviour change: the makestep directive and its defaults are
untouched. Comments only.

The commit message on 7fcaa4b0 still carries the wrong explanation and is
left alone rather than rewriting merged history.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 17:35:51 +02:00
k.renner d41f4870ad Merge pull request 'fix(chrony): step the clock at boot so a bad RTC self-corrects' (#465) from worktree-chrony-makestep into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#465
2026-08-05 15:14:21 +00:00
claude-botandClaude Opus 5 7fcaa4b035 fix(chrony): step the clock at boot so a bad RTC self-corrects
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:35:18 +02:00
k.renner 11bf13883f Merge pull request 'ci: validate the GitOps tree and pipeline configs on every PR' (#460) from ci-k8s-validate into master
ci/woodpecker/cron/update-all Pipeline failed
ci/woodpecker/cron/update-audit Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#460
2026-08-01 10:20:58 +00:00
security-bot 4b502daa98 chore(image-automation): bump woodpeckerci/woodpecker-server:v3.17.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-31 14:33:28 +00:00
security-bot 83a20bfa80 chore(image-automation): bump woodpeckerci/woodpecker-agent:v3.17.0
Updates managed by image-automation-controller.
2026-07-31 14:33:17 +00:00
security-bot c1a1a59ecc chore(image-automation): bump redis:8.10.0-alpine
ci/woodpecker/cron/e2e Pipeline was successful
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-30 21:32:22 +00:00
5 changed files with 28 additions and 3 deletions
@@ -18,6 +18,26 @@ chrony_upstream_pools:
# Local stratum if disconnected from upstream
chrony_local_stratum: 10
# Step (rather than slew) the clock when the offset exceeds the threshold, but
# only for the first N updates after chronyd starts. Without a makestep
# directive chronyd only ever slews, so a large offset is never closed in
# practice.
#
# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
# that forward to its own built-in epoch (2026-06-26), and it ended up at
# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
# With host time before the cluster certificate's NotBefore, every outbound TLS
# handshake failed x509 validation, taking out the MQTT bridge (heating
# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
# is just where the software flooring landed; the fault is that nothing steps.
#
# Limiting to the first few updates keeps steps confined to boot, so a
# long-running server is never jumped out from under its workloads.
chrony_makestep_threshold: 1.0
chrony_makestep_limit: 3
# Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
# (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
# error per interface when asked for it. Overridden to false in
@@ -14,6 +14,11 @@ pool {{ pool }}
# Record the rate at which the system clock gains/losses time
driftfile /var/lib/chrony/chrony.drift
# Step the clock instead of slewing it when the offset is large, limited to the
# first few updates after start so only boot-time corrections jump. Rationale
# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
# Allow NTP client access from local networks
{% for network in chrony_allowed_networks %}
allow {{ network }}
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
type: RuntimeDefault
containers:
- name: redis
image: redis:8.8.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
args:
- redis-server
- --save
+1 -1
View File
@@ -34,7 +34,7 @@ spec:
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
# track the same range/filter so they bump together on each release.
image: woodpeckerci/woodpecker-agent:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
env:
- name: WOODPECKER_SERVER
value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+1 -1
View File
@@ -40,7 +40,7 @@ spec:
# minor + patch auto-update within 3.x; only the 4.0 major is held for
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
# -bump hazard is gone). Stays in lockstep with the agent.
image: woodpeckerci/woodpecker-server:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
env:
- name: WOODPECKER_HOST
value: https://ci.renner.uno