jupiter, neptune and saturn were the last entries in pve_hosts.yml still
reached via ansible_host=<IPv4 literal>. Every other host here --
authentik, netbird, proxy, nas-1, grafana, edge-1 -- has always used its
FQDN, so these three were the outlier rather than the convention.
An address in an inventory is allocator state standing in for identity, and
it pins access to the legacy family: the names resolve IPv6-first, which is
the primary path on this network. DNS is authoritative for where a host is;
the inventory should say which host.
The short inventory_hostname is kept deliberately -- plays key on it
(ntp_servers[inventory_hostname] in install-ntp-ha.yml, node names in
opentofu, the prometheus targets) -- so only ansible_host changes.
No claim that this fixes the connection stalls seen while applying the NTP
work on 2026-08-07. Those were the IPS reacting to the connection pattern I
was generating (a port sweep, then rapid repeated SSH across hosts
interleaved with Ansible's own churn), not an address-family problem. This
change stands on convention and on not encoding a lease as identity.
claude-ws is left alone: its comment documents that obg.renner.uno is
split-horizon and resolves to the public proxy from home. That explains why
that particular name is unusable, not why a literal is right -- a name that
resolves to the wrong target calls for one that resolves correctly. Out of
scope here, and I do not know whether such a record exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chrony_allowed_networks contained only 10.2.0.0/16, so the rendered
chrony.conf permitted IPv4 clients and nothing else. IPv6 is the primary
address family here: ntp.srv.renner.uno resolves to fd40::123, so an
IPv4-only allow list refuses every client that uses the documented name.
This was live but invisible. The hosts carried `allow fd00::/8` in
chrony.conf for a long time without it ever being in this repository.
Applying the role on 2026-08-07 rendered the template over the live config
and dropped the line on all four NTP servers at once, taking internal NTP
down over IPv6 until it was added here.
Same class of untracked host state as the anycast service address, and the
same failure: config that exists only on the box survives right up until
something re-renders it. Reading the drifted file is not enough -- the rest
of it has to be diffed against what the template would produce.
The outage was not caught by the play, which reported the v6 master as
"serving ntp.srv.renner.uno" throughout. It was caught by querying the
service:
chronyd -Q "server ntp.srv.renner.uno iburst"
-> No suitable source for synchronisation
and after the fix:
chronyd -Q "server ntp.srv.renner.uno iburst"
-> System clock wrong by 0.001999 seconds (ignored)
with a control query against ptbtime1.ptb.de to confirm the method rather
than the target was sound.
fe80::/10 is included as well: a client on the same segment may source its
query from a link-local address, and without it chrony drops the request,
which presents as the server being unreachable rather than as a policy
refusal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applying #469 to edge-1 exposed two defects that only show up on real
hardware.
1. The VRRP address was unusable.
keepalived took MASTER and assigned fd40::123, but the kernel left it:
inet6 fd40::123/128 scope global dadfailed deprecated tentative
proto keepalived
valid_lft forever preferred_lft 0sec
`-nodad` did not take effect -- no nodad flag on the address, and
net.ipv6.conf.enp1s0/40.accept_dad = 1 -- so DAD ran and found a real
duplicate: jupiter, saturn and neptune still held the address statically.
edge-1 believed it was the v6 master while serving nothing.
The flag is dropped rather than fixed. It was inherited from the previous
design, where every NTP server carried the address at once and duplicate
detection had to be suppressed for that to work. Under single-master VRRP
the opposite holds: exactly one host should have it, so a duplicate is a
genuine fault and detecting it is the point.
What that costs is that the migration can no longer be done host by host.
A static copy anywhere and a VRRP copy elsewhere cannot coexist, and the
dadfailed state is sticky -- the kernel does not retry. install-ntp-ha.yml
is therefore split in two: a first play withdraws every static assignment
across the whole group, then the second hands the address to keepalived.
Deliberately not serial, since it must finish everywhere before any host
claims the address. There is a gap between the phases where nobody holds
it, bounded by one play's run time; clients retry.
The withdrawal tasks move to roles/keepalived-ntp/tasks/withdraw-static.yml
for the first play to include. main.yml keeps the detect-and-assert guards,
which still run every time and fail loudly if any static assignment
survives.
2. keepalived warned:
(NTP_VIP6) the first IPv6 VIP address should be link local
RFC 5798 gives an IPv6 virtual router a link-local primary. keepalived
carried on regardless, but the instance was sourcing VRRPv3 from something
the protocol does not expect. fe80::123 is now the first
virtual_ipaddress: it mirrors the service address and cannot collide with
a NIC's EUI-64 link-local.
ntp_vip6 and ntp_vip6_linklocal move to group_vars/ntp_cluster.yml so both
plays read one definition rather than duplicating a service address.
Verified: `keepalived -t` on edge-1 (keepalived 2.4.1) parses the two-address
form clean, and the playbook passes --syntax-check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #469 found the Debian path still fails open, and was right that
my previous commit message was wrong to claim otherwise.
The whole Debian chain -- detect, withdraw, confirm, assert -- was gated on
`keepalived_ntp_static6_removed.changed`, which is only true when the
lineinfile regex matches exactly, prefix length included. The hook it
targets is hand-written and was never in version control, so any deviation
(a different mask, unusual whitespace, a second assignment elsewhere)
leaves .changed false and silently skips every following task. The static
assignment then coexists with the VRRP-managed one, which is the
double-assignment failure this PR exists to prevent -- just relocated from
edge-1 to a PVE host.
Same fail-open class as the Arch grep fixed in 5e01e2f1. The argument in
that commit message -- that the Debian path was safe because
`ip -6 addr show ... to <prefix>` matches by range -- is true of the runtime
checks but says nothing about the lineinfile text match, which is a
different mechanism and was exactly as fragile as the pre-fix Arch grep.
Fixed by checking the CONFIG FILE unconditionally, prefix-agnostic and
\b-anchored, mirroring the networkd check.
Note this is deliberately not the fix the review proposed. It suggested
running the runtime `ip -6 addr show ... to <addr>` check and its assert
unconditionally. That would break the PR: after migration the address is
legitimately present on whichever host is the elected v6 master, so an
unconditional "must be absent" assert fails on the master on every run. The
config file is the right thing to check every time precisely because
keepalived never writes there -- anything found is by definition static,
whereas the live address is expected on the master. The runtime withdrawal
stays gated on .changed, which remains correct for it.
Verified on jupiter:
grep -rlE 'fd40::123\b' /etc/network/interfaces /etc/network/interfaces.d/
-> /etc/network/interfaces
Sequencing holds: on the migration run the lineinfile removes the hook
first, so the grep finds nothing and the assert passes; if the regex does
not match, the grep finds it and the assert fails loudly instead of
skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Arch guard added in the previous commit grepped for the literal
"fd40::123/128" -- the full address/prefixlen from ntp_vip6. A hand-written
[Address] using any other mask (/64, say) would not have matched, the
assert would have passed, and keepalived would have been handed an address
networkd was still assigning.
That fails open, which is the wrong direction for this particular check.
It is detect-only: there is no delete afterwards whose result gets
re-verified, so a miss is silent, and the outcome is the dual-ownership
state the guard exists to prevent.
The Debian path does not share the problem. `ip -6 addr show ... to
<prefix>` matches by range regardless of the prefix length the address was
actually assigned with, and its `ip addr del` -- which IS prefix-sensitive
-- is re-verified by a following assert, so a mismatch there fails loudly.
Now matches the bare address, anchored with \b so it cannot instead hit a
longer address sharing the prefix. Verified both directions:
grep -rlE 'fd40::123\b' /etc/systemd/network/ on edge-1
-> /etc/systemd/network/10-enp1s0.40.network (still detected)
grep -cE 'fd40::123\b' against 'Address=fd40::1234/64'
-> 0 (no false positive)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second review round on #469 found two ways the migration could silently
leave the address outside VRRP control -- both of which would defeat the
health gating while the play reported success.
1. The one-shot migration had no verification.
Gating the withdrawal on `keepalived_ntp_static6_removed.changed` is what
makes it idempotent, but it also means it gets exactly one attempt and
never retries. Combined with `failed_when: false` on the delete, a
mismatch would pass silently: `ip addr del` matches on address AND prefix
length, so a host whose static assignment used a different mask or
interface than ntp_vip6 / keepalived_ntp_interface would see the delete
no-op, keep the address permanently outside VRRP control, and still
report success.
The idempotency fix from the previous commit created this: making the
task run once removed the retry that would otherwise have masked it.
Now asserted explicitly after the delete, with a message naming the
likely cause and the commands to inspect it.
2. edge-1 -- the host the incident actually happened on -- was not covered.
The removal tasks are gated to Debian, so they never touch edge-1 (Arch).
Its static [Address] block belongs to the edge-host role, and
install-edge-host.yml is a separate play with no ordering relationship to
this one. Nothing guaranteed it had been re-applied.
edge_host already sets ntp_keepalived_enabled: true, so the next run of
install-ntp-ha.yml gives edge-1 the NTP_VIP6 instance immediately. With
the [Address] block still present it would hold fd40::123 statically AND
run VRRP for it: keepalived would believe it had withdrawn the address
while networkd kept it assigned, and the host would go on answering NTP
with a clock check_ntp had already rejected. That is the failure of
2026-08-04, on the same host.
This is not hypothetical -- edge-1 still has the block right now:
grep -rl fd40::123/128 /etc/systemd/network/
-> /etc/systemd/network/10-enp1s0.40.network
The role now detects this and fails fast, naming the play to run first,
rather than proceeding into a split-ownership state.
Also documents why the post-deploy presence check is meaningful: presence
alone cannot distinguish a VRRP-held address from a static leftover, and
only means what it claims because the role asserts no static assignment
survives on either distro.
Not taken: narrowing serial: 1 so routine runs are not serialised through
the chrony settle pause. Real cost, but correctness of the address handover
comes first at four hosts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #469 caught a real defect. The withdrawal task used
ansible.builtin.command with no changed_when, so it reported changed and
fired `notify: restart keepalived` on every run where the address was
present.
That is fine exactly once. After this lands, fd40::123 is legitimately
present on whichever host is the elected v6 master, put there by keepalived
-- and an ungated delete cannot tell that apart from a static assignment,
because the address looks identical either way. So every later run of this
play (adding a host, changing a priority, plain re-convergence) would have
deleted the master's address and restarted keepalived, dropping the service
address and forcing a re-election each time. That is precisely the flapping
this change exists to prevent.
Both the check and the delete are now gated on the ifupdown hook having just
been removed, so they run once during the migration off static config and
never again. Steady-state runs leave the master's address alone.
Also from review:
- failed_when: false on the delete. The desired end state is "address
absent". If check_ntp fails in the window between the check and the
delete, keepalived withdraws the address first and `ip addr del` errors
on an address that is already gone -- the outcome the task wants, and no
reason to abort the play for that host.
- The play's post-deploy verification only ever checked the IPv4 VIP. The
v6 address is the path clients actually use, so a broken v6 rollout --
address never claimed, or -nodad not behaving as expected on a given
kernel -- would not have surfaced in the play at all. It now gets the
same check, and the message points at check_ntp and the health log for
the case where no host holds it.
Not taken: the suggestion to narrow serial: 1 so it does not also serialise
chrony. With four hosts it costs nothing, and scoping it per-role would mean
splitting the play. Worth revisiting if the cluster grows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this the health gating is defeated on three of the four NTP
servers. fd40::123 was assigned statically on every one of them, by hand
and in two different ways, none of it in version control:
PVE nodes `up ip -6 addr replace fd40::123/128 dev vmbr1.40 nodad`
in /etc/network/interfaces
edge-1 an [Address] block in /etc/systemd/network/
A statically assigned copy does not disappear when keepalived decides the
host is unfit to serve time. Leaving it in place means the host keeps
answering NTP on an address VRRP believes it has withdrawn -- exactly the
failure this whole change exists to prevent, with the added confusion of
keepalived reporting the address as released.
The ifupdown hook is removed here. edge-1 needs nothing: the edge-host
networkd template has never carried the address, so applying that role
drops it.
The live address is deleted as well, because removing the config line alone
does not take effect until the next network reload, and a reload on a PVE
node interrupts guest connectivity including the Talos VM and its Ceph OSD.
Deleting it is safe where keepalived already holds it: the deletion
notifies a keepalived restart, and the elected master re-adds it on the way
back up. Doing it unconditionally avoids having to tell "assigned
statically" from "assigned by keepalived", which the address alone does not
reveal.
The play now runs serial: 1. Withdrawing the address and restarting
keepalived on every NTP server at once would leave the fleet with nobody
holding fd40::123 until the elections settle; serialised, the remaining
members answer throughout.
Verified against the live hosts rather than assumed:
- all three PVE nodes carry the identical hook (jupiter:54, saturn:55,
neptune:56), so one task covers them
- the removal regexp matches exactly one line per host
- `ip -6 addr show dev <if> to <addr>` prints the matching address when
present and nothing at all when absent, so the guard condition is sound
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit set keepalived_ntp_vip6 in group_vars/edge_host, which
would have given the v6 VRRP instance to edge-1 alone. The other NTP
servers would have rendered no NTP_VIP6 instance at all and stayed silently
outside the election -- so the address could never fail over to them, and
the health gating would have amounted to "edge-1 withdraws and nobody takes
over".
Moves it alongside ntp_vip in plays/install-ntp-ha.yml, which is already
where the service-wide address lives, and passes it to the role for every
host in ntp_cluster.
Verified that edge-1 is currently BACKUP and does not hold the IPv4 VIP: a
VRRP BACKUP receiving no advertisements promotes itself to MASTER, so the
other members are alive and one of them holds it. They are real
participants, not retired hosts.
Not verified: whether the other members also carry fd40::123 statically, as
edge-1 did. They have no DNS records -- jupiter/saturn/neptune resolve only
through the *.renner.uno wildcard to the proxy -- so there is no hostname
path to them to check. If they do carry it statically, that address must be
removed at the same time this lands, or two hosts will hold it outside VRRP
control.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ntp.srv.renner.uno resolves to fd40::123 (IPv6) and to the keepalived VIP
(IPv4, legacy). IPv6 is the primary family here, so the v6 address is the
path clients actually use -- and it was the one with no health gating at
all.
It was assigned statically in /etc/systemd/network/ on edge-1, so networkd
carried it unconditionally from boot. That is how the host answered NTP
queries on 2026-08-04 while its clock was ~16 days in the past: chronyd was
running, udp/123 was listening, the address was up, and nothing anywhere
was checking whether the clock was right. Fixing the keepalived health
check does not help, because keepalived was not holding that address.
This moves it into keepalived as a second VRRP instance tracking the same
check_ntp script, so the address exists only while the host is fit to serve
time.
Three details that are load-bearing rather than stylistic:
- A separate vrrp_instance is mandatory. keepalived requires that "all
addresses in virtual_ipaddress must be of the same family", so the v6
address cannot join NTP_VIP.
- No authentication block. VRRPv3 removed authentication (RFC 5798) and
keepalived always uses version 3 for IPv6 instances.
- track_script uses `weight 0`, overriding the script's -20. A non-zero
weight only subtracts from priority on failure, so the last remaining
host -- having nobody to lose the election to -- would stay MASTER and
keep serving a clock it had already reported as wrong. Weight 0
transitions the instance to FAULT instead, releasing the address
unconditionally. That is the required behaviour: a host unfit to serve
time must stop answering whether or not a replacement exists.
Semantic change, deliberate and worth stating: VRRP is a single-master
unicast VIP, not anycast. Previously every NTP server carried fd40::123 and
clients reached whichever answered neighbour discovery first. Now exactly
one host holds it. For NTP this is the better trade -- query volume makes
load spreading pointless, and on failover keepalived sends unsolicited
Neighbour Advertisements so client neighbour caches update in seconds,
where the static address had no withdrawal signal at all and clients kept
using a dead holder until their own neighbour unreachability detection
expired. True routed anycast with health withdrawal would mean announcing
the /128 via an IGP, which the gateway would have to speak.
The edge-host role is deliberately untouched: its template already has no
[Address] block, so applying it removes the hand-placed address, and
keepalived then adds it back under health control. The service address is
NTP-role configuration and now lives there.
Validated with `keepalived -t` against keepalived 2.4.1 on edge-1 using the
config as it renders for that host; parses clean apart from the pre-existing
auth_pass truncation warning on the IPv4 instance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.
On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:
tls: failed to verify certificate: x509: certificate has expired or is
not yet valid: current time 2026-07-20T16:15:36+02:00 is before
2026-07-23T06:15:01Z
That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.
Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.
The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot