Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 24ed2485c9 fix(chrony): allow IPv6 clients -- the allow list was IPv4-only
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
chrony_allowed_networks contained only 10.2.0.0/16, so the rendered
chrony.conf permitted IPv4 clients and nothing else. IPv6 is the primary
address family here: ntp.srv.renner.uno resolves to fd40::123, so an
IPv4-only allow list refuses every client that uses the documented name.

This was live but invisible. The hosts carried `allow fd00::/8` in
chrony.conf for a long time without it ever being in this repository.
Applying the role on 2026-08-07 rendered the template over the live config
and dropped the line on all four NTP servers at once, taking internal NTP
down over IPv6 until it was added here.

Same class of untracked host state as the anycast service address, and the
same failure: config that exists only on the box survives right up until
something re-renders it. Reading the drifted file is not enough -- the rest
of it has to be diffed against what the template would produce.

The outage was not caught by the play, which reported the v6 master as
"serving ntp.srv.renner.uno" throughout. It was caught by querying the
service:

  chronyd -Q "server ntp.srv.renner.uno iburst"
    -> No suitable source for synchronisation

and after the fix:

  chronyd -Q "server ntp.srv.renner.uno iburst"
    -> System clock wrong by 0.001999 seconds (ignored)

with a control query against ptbtime1.ptb.de to confirm the method rather
than the target was sound.

fe80::/10 is included as well: a client on the same segment may source its
query from a link-local address, and without it chrony drops the request,
which presents as the server being unreachable rather than as a policy
refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:33:22 +02:00
claude-botandClaude Opus 5 ea439b110d fix(ntp): link-local VRRP primary, and sequence the migration around DAD
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Applying #469 to edge-1 exposed two defects that only show up on real
hardware.

1. The VRRP address was unusable.

   keepalived took MASTER and assigned fd40::123, but the kernel left it:

     inet6 fd40::123/128 scope global dadfailed deprecated tentative
           proto keepalived
           valid_lft forever preferred_lft 0sec

   `-nodad` did not take effect -- no nodad flag on the address, and
   net.ipv6.conf.enp1s0/40.accept_dad = 1 -- so DAD ran and found a real
   duplicate: jupiter, saturn and neptune still held the address statically.
   edge-1 believed it was the v6 master while serving nothing.

   The flag is dropped rather than fixed. It was inherited from the previous
   design, where every NTP server carried the address at once and duplicate
   detection had to be suppressed for that to work. Under single-master VRRP
   the opposite holds: exactly one host should have it, so a duplicate is a
   genuine fault and detecting it is the point.

   What that costs is that the migration can no longer be done host by host.
   A static copy anywhere and a VRRP copy elsewhere cannot coexist, and the
   dadfailed state is sticky -- the kernel does not retry. install-ntp-ha.yml
   is therefore split in two: a first play withdraws every static assignment
   across the whole group, then the second hands the address to keepalived.
   Deliberately not serial, since it must finish everywhere before any host
   claims the address. There is a gap between the phases where nobody holds
   it, bounded by one play's run time; clients retry.

   The withdrawal tasks move to roles/keepalived-ntp/tasks/withdraw-static.yml
   for the first play to include. main.yml keeps the detect-and-assert guards,
   which still run every time and fail loudly if any static assignment
   survives.

2. keepalived warned:

     (NTP_VIP6) the first IPv6 VIP address should be link local

   RFC 5798 gives an IPv6 virtual router a link-local primary. keepalived
   carried on regardless, but the instance was sourcing VRRPv3 from something
   the protocol does not expect. fe80::123 is now the first
   virtual_ipaddress: it mirrors the service address and cannot collide with
   a NIC's EUI-64 link-local.

ntp_vip6 and ntp_vip6_linklocal move to group_vars/ntp_cluster.yml so both
plays read one definition rather than duplicating a service address.

Verified: `keepalived -t` on edge-1 (keepalived 2.4.1) parses the two-address
form clean, and the playbook passes --syntax-check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:07:22 +02:00
claude-botandClaude Opus 5 a1e3b65cbf fix(ntp): verify the ifupdown side unconditionally, not only after a match
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #469 found the Debian path still fails open, and was right that
my previous commit message was wrong to claim otherwise.

The whole Debian chain -- detect, withdraw, confirm, assert -- was gated on
`keepalived_ntp_static6_removed.changed`, which is only true when the
lineinfile regex matches exactly, prefix length included. The hook it
targets is hand-written and was never in version control, so any deviation
(a different mask, unusual whitespace, a second assignment elsewhere)
leaves .changed false and silently skips every following task. The static
assignment then coexists with the VRRP-managed one, which is the
double-assignment failure this PR exists to prevent -- just relocated from
edge-1 to a PVE host.

Same fail-open class as the Arch grep fixed in 5e01e2f1. The argument in
that commit message -- that the Debian path was safe because
`ip -6 addr show ... to <prefix>` matches by range -- is true of the runtime
checks but says nothing about the lineinfile text match, which is a
different mechanism and was exactly as fragile as the pre-fix Arch grep.

Fixed by checking the CONFIG FILE unconditionally, prefix-agnostic and
\b-anchored, mirroring the networkd check.

Note this is deliberately not the fix the review proposed. It suggested
running the runtime `ip -6 addr show ... to <addr>` check and its assert
unconditionally. That would break the PR: after migration the address is
legitimately present on whichever host is the elected v6 master, so an
unconditional "must be absent" assert fails on the master on every run. The
config file is the right thing to check every time precisely because
keepalived never writes there -- anything found is by definition static,
whereas the live address is expected on the master. The runtime withdrawal
stays gated on .changed, which remains correct for it.

Verified on jupiter:
  grep -rlE 'fd40::123\b' /etc/network/interfaces /etc/network/interfaces.d/
    -> /etc/network/interfaces

Sequencing holds: on the migration run the lineinfile removes the hook
first, so the grep finds nothing and the assert passes; if the regex does
not match, the grep finds it and the assert fails loudly instead of
skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:29:53 +02:00
claude-botandClaude Opus 5 5e01e2f1b5 fix(ntp): detect a static networkd assignment at any prefix length
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Arch guard added in the previous commit grepped for the literal
"fd40::123/128" -- the full address/prefixlen from ntp_vip6. A hand-written
[Address] using any other mask (/64, say) would not have matched, the
assert would have passed, and keepalived would have been handed an address
networkd was still assigning.

That fails open, which is the wrong direction for this particular check.
It is detect-only: there is no delete afterwards whose result gets
re-verified, so a miss is silent, and the outcome is the dual-ownership
state the guard exists to prevent.

The Debian path does not share the problem. `ip -6 addr show ... to
<prefix>` matches by range regardless of the prefix length the address was
actually assigned with, and its `ip addr del` -- which IS prefix-sensitive
-- is re-verified by a following assert, so a mismatch there fails loudly.

Now matches the bare address, anchored with \b so it cannot instead hit a
longer address sharing the prefix. Verified both directions:

  grep -rlE 'fd40::123\b' /etc/systemd/network/   on edge-1
    -> /etc/systemd/network/10-enp1s0.40.network  (still detected)
  grep -cE 'fd40::123\b' against 'Address=fd40::1234/64'
    -> 0                                          (no false positive)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:11:36 +02:00
claude-botandClaude Opus 5 40ef53fa67 fix(ntp): assert the static service address is really gone before handing over
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Second review round on #469 found two ways the migration could silently
leave the address outside VRRP control -- both of which would defeat the
health gating while the play reported success.

1. The one-shot migration had no verification.

   Gating the withdrawal on `keepalived_ntp_static6_removed.changed` is what
   makes it idempotent, but it also means it gets exactly one attempt and
   never retries. Combined with `failed_when: false` on the delete, a
   mismatch would pass silently: `ip addr del` matches on address AND prefix
   length, so a host whose static assignment used a different mask or
   interface than ntp_vip6 / keepalived_ntp_interface would see the delete
   no-op, keep the address permanently outside VRRP control, and still
   report success.

   The idempotency fix from the previous commit created this: making the
   task run once removed the retry that would otherwise have masked it.

   Now asserted explicitly after the delete, with a message naming the
   likely cause and the commands to inspect it.

2. edge-1 -- the host the incident actually happened on -- was not covered.

   The removal tasks are gated to Debian, so they never touch edge-1 (Arch).
   Its static [Address] block belongs to the edge-host role, and
   install-edge-host.yml is a separate play with no ordering relationship to
   this one. Nothing guaranteed it had been re-applied.

   edge_host already sets ntp_keepalived_enabled: true, so the next run of
   install-ntp-ha.yml gives edge-1 the NTP_VIP6 instance immediately. With
   the [Address] block still present it would hold fd40::123 statically AND
   run VRRP for it: keepalived would believe it had withdrawn the address
   while networkd kept it assigned, and the host would go on answering NTP
   with a clock check_ntp had already rejected. That is the failure of
   2026-08-04, on the same host.

   This is not hypothetical -- edge-1 still has the block right now:
     grep -rl fd40::123/128 /etc/systemd/network/
     -> /etc/systemd/network/10-enp1s0.40.network

   The role now detects this and fails fast, naming the play to run first,
   rather than proceeding into a split-ownership state.

Also documents why the post-deploy presence check is meaningful: presence
alone cannot distinguish a VRRP-held address from a static leftover, and
only means what it claims because the role asserts no static assignment
survives on either distro.

Not taken: narrowing serial: 1 so routine runs are not serialised through
the chrony settle pause. Real cost, but correctness of the address handover
comes first at four hosts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:02:26 +02:00
claude-botandClaude Opus 5 fbf5c4dcb8 fix(ntp): make the static-address withdrawal run once, not every play
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #469 caught a real defect. The withdrawal task used
ansible.builtin.command with no changed_when, so it reported changed and
fired `notify: restart keepalived` on every run where the address was
present.

That is fine exactly once. After this lands, fd40::123 is legitimately
present on whichever host is the elected v6 master, put there by keepalived
-- and an ungated delete cannot tell that apart from a static assignment,
because the address looks identical either way. So every later run of this
play (adding a host, changing a priority, plain re-convergence) would have
deleted the master's address and restarted keepalived, dropping the service
address and forcing a re-election each time. That is precisely the flapping
this change exists to prevent.

Both the check and the delete are now gated on the ifupdown hook having just
been removed, so they run once during the migration off static config and
never again. Steady-state runs leave the master's address alone.

Also from review:

  - failed_when: false on the delete. The desired end state is "address
    absent". If check_ntp fails in the window between the check and the
    delete, keepalived withdraws the address first and `ip addr del` errors
    on an address that is already gone -- the outcome the task wants, and no
    reason to abort the play for that host.

  - The play's post-deploy verification only ever checked the IPv4 VIP. The
    v6 address is the path clients actually use, so a broken v6 rollout --
    address never claimed, or -nodad not behaving as expected on a given
    kernel -- would not have surfaced in the play at all. It now gets the
    same check, and the message points at check_ntp and the health log for
    the case where no host holds it.

Not taken: the suggestion to narrow serial: 1 so it does not also serialise
chrony. With four hosts it costs nothing, and scoping it per-role would mean
splitting the play. Worth revisiting if the cluster grows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:28:21 +02:00
claude-botandClaude Opus 5 112299da49 fix(ntp): remove the static service address so keepalived can own it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Without this the health gating is defeated on three of the four NTP
servers. fd40::123 was assigned statically on every one of them, by hand
and in two different ways, none of it in version control:

  PVE nodes  `up ip -6 addr replace fd40::123/128 dev vmbr1.40 nodad`
             in /etc/network/interfaces
  edge-1     an [Address] block in /etc/systemd/network/

A statically assigned copy does not disappear when keepalived decides the
host is unfit to serve time. Leaving it in place means the host keeps
answering NTP on an address VRRP believes it has withdrawn -- exactly the
failure this whole change exists to prevent, with the added confusion of
keepalived reporting the address as released.

The ifupdown hook is removed here. edge-1 needs nothing: the edge-host
networkd template has never carried the address, so applying that role
drops it.

The live address is deleted as well, because removing the config line alone
does not take effect until the next network reload, and a reload on a PVE
node interrupts guest connectivity including the Talos VM and its Ceph OSD.
Deleting it is safe where keepalived already holds it: the deletion
notifies a keepalived restart, and the elected master re-adds it on the way
back up. Doing it unconditionally avoids having to tell "assigned
statically" from "assigned by keepalived", which the address alone does not
reveal.

The play now runs serial: 1. Withdrawing the address and restarting
keepalived on every NTP server at once would leave the fleet with nobody
holding fd40::123 until the elections settle; serialised, the remaining
members answer throughout.

Verified against the live hosts rather than assumed:
  - all three PVE nodes carry the identical hook (jupiter:54, saturn:55,
    neptune:56), so one task covers them
  - the removal regexp matches exactly one line per host
  - `ip -6 addr show dev <if> to <addr>` prints the matching address when
    present and nothing at all when absent, so the guard condition is sound

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:06:43 +02:00
claude-botandClaude Opus 5 01e843beef fix(ntp): apply the v6 service address to every ntp_cluster member
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The previous commit set keepalived_ntp_vip6 in group_vars/edge_host, which
would have given the v6 VRRP instance to edge-1 alone. The other NTP
servers would have rendered no NTP_VIP6 instance at all and stayed silently
outside the election -- so the address could never fail over to them, and
the health gating would have amounted to "edge-1 withdraws and nobody takes
over".

Moves it alongside ntp_vip in plays/install-ntp-ha.yml, which is already
where the service-wide address lives, and passes it to the role for every
host in ntp_cluster.

Verified that edge-1 is currently BACKUP and does not hold the IPv4 VIP: a
VRRP BACKUP receiving no advertisements promotes itself to MASTER, so the
other members are alive and one of them holds it. They are real
participants, not retired hosts.

Not verified: whether the other members also carry fd40::123 statically, as
edge-1 did. They have no DNS records -- jupiter/saturn/neptune resolve only
through the *.renner.uno wildcard to the proxy -- so there is no hostname
path to them to check. If they do carry it statically, that address must be
removed at the same time this lands, or two hosts will hold it outside VRRP
control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 20:07:05 +02:00
claude-botandClaude Opus 5 2fbce12b0b feat(ntp): hold the IPv6 service address in keepalived, gated on clock health
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ntp.srv.renner.uno resolves to fd40::123 (IPv6) and to the keepalived VIP
(IPv4, legacy). IPv6 is the primary family here, so the v6 address is the
path clients actually use -- and it was the one with no health gating at
all.

It was assigned statically in /etc/systemd/network/ on edge-1, so networkd
carried it unconditionally from boot. That is how the host answered NTP
queries on 2026-08-04 while its clock was ~16 days in the past: chronyd was
running, udp/123 was listening, the address was up, and nothing anywhere
was checking whether the clock was right. Fixing the keepalived health
check does not help, because keepalived was not holding that address.

This moves it into keepalived as a second VRRP instance tracking the same
check_ntp script, so the address exists only while the host is fit to serve
time.

Three details that are load-bearing rather than stylistic:

  - A separate vrrp_instance is mandatory. keepalived requires that "all
    addresses in virtual_ipaddress must be of the same family", so the v6
    address cannot join NTP_VIP.

  - No authentication block. VRRPv3 removed authentication (RFC 5798) and
    keepalived always uses version 3 for IPv6 instances.

  - track_script uses `weight 0`, overriding the script's -20. A non-zero
    weight only subtracts from priority on failure, so the last remaining
    host -- having nobody to lose the election to -- would stay MASTER and
    keep serving a clock it had already reported as wrong. Weight 0
    transitions the instance to FAULT instead, releasing the address
    unconditionally. That is the required behaviour: a host unfit to serve
    time must stop answering whether or not a replacement exists.

Semantic change, deliberate and worth stating: VRRP is a single-master
unicast VIP, not anycast. Previously every NTP server carried fd40::123 and
clients reached whichever answered neighbour discovery first. Now exactly
one host holds it. For NTP this is the better trade -- query volume makes
load spreading pointless, and on failover keepalived sends unsolicited
Neighbour Advertisements so client neighbour caches update in seconds,
where the static address had no withdrawal signal at all and clients kept
using a dead holder until their own neighbour unreachability detection
expired. True routed anycast with health withdrawal would mean announcing
the /128 via an IGP, which the gateway would have to speak.

The edge-host role is deliberately untouched: its template already has no
[Address] block, so applying it removes the hand-placed address, and
keepalived then adds it back under health control. The service address is
NTP-role configuration and now lives there.

Validated with `keepalived -t` against keepalived 2.4.1 on edge-1 using the
config as it renders for that host; parses clean apart from the pre-existing
auth_pass truncation warning on the IPv4 instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 19:59:40 +02:00
8 changed files with 487 additions and 8 deletions
@@ -0,0 +1,18 @@
---
# Shared configuration for every NTP server (see plays/install-ntp-ha.yml).
# IPv6 service address -- what ntp.srv.renner.uno resolves to, and the path
# clients actually use. The IPv4 VIP is legacy.
#
# Defined at group level rather than in the play because install-ntp-ha.yml runs
# two plays over this group -- withdraw the static assignments fleet-wide, then
# hand the address to keepalived -- and both need the same value. Duplicating a
# service address across plays is exactly the sort of thing that drifts.
ntp_vip6: "fd40::123/128"
# Link-local address carried as the FIRST virtual_ipaddress of the IPv6 VRRP
# instance. RFC 5798 gives an IPv6 virtual router a link-local primary, and
# keepalived warns "the first IPv6 VIP address should be link local" without
# one. fe80::123 mirrors the service address and cannot collide with a real
# NIC's EUI-64 link-local.
ntp_vip6_linklocal: "fe80::123/64"
+88 -1
View File
@@ -22,12 +22,54 @@
# so it stays BACKUP while the PVE nodes exist and wins the VIP by default once
# they are gone -- no config change required at cutover.
- name: Install NTP Server (HA)
# Phase 1. Every static assignment of the IPv6 service address must be gone
# fleet-wide before ANY host claims it via VRRP.
#
# DAD is enabled on the VRRP address (deliberately -- under single-master VRRP a
# duplicate is a real fault worth detecting). So a static copy on one host and a
# VRRP-managed copy on another cannot coexist: the kernel marks the newcomer
# `dadfailed deprecated`, and its master then believes it is serving while
# carrying an unusable address. That is not hypothetical -- it is what edge-1
# did on 2026-08-07 when it took the address while three PVE nodes still held it
# statically.
#
# Deliberately NOT serial: this play must complete everywhere before phase 2
# starts anywhere. Doing the withdrawal per host inside phase 2 cannot avoid the
# overlap, because with serial: 1 the first host claims the address while the
# rest are still static.
#
# There is a gap between the phases where no host holds the address. It is
# bounded by one play's run time and clients simply retry.
- name: Withdraw static assignments of the NTP service address
hosts: ntp_cluster
become: true
vars:
keepalived_ntp_vip6: "{{ ntp_vip6 }}"
keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
tasks:
- name: Withdraw any static assignment of the service address
ansible.builtin.include_role:
name: keepalived-ntp
tasks_from: withdraw-static.yml
when: ntp_keepalived_enabled | default(false)
# Phase 2. Configure chrony and hand the address to keepalived.
- name: Install NTP Server (HA)
hosts: ntp_cluster
become: true
# One node at a time. The keepalived role withdraws the statically assigned
# IPv6 service address and restarts keepalived so the address is re-taken
# under VRRP control; doing that across every NTP server at once would leave
# the whole fleet with nobody holding fd40::123 until the elections settle.
# Serialised, the remaining members keep answering throughout.
serial: 1
vars:
ntp_vip: "10.2.40.4"
# ntp_vip6 / ntp_vip6_linklocal come from group_vars/ntp_cluster.yml, so
# both plays in this file see one definition.
ntp_servers:
jupiter:
priority: 100
@@ -49,6 +91,8 @@
when: ntp_keepalived_enabled | default(false)
vars:
keepalived_ntp_vip: "{{ ntp_vip }}"
keepalived_ntp_vip6: "{{ ntp_vip6 }}"
keepalived_ntp_vip6_linklocal: "{{ ntp_vip6_linklocal }}"
keepalived_ntp_state: "{{ ntp_servers[inventory_hostname].state }}"
keepalived_ntp_priority: "{{ ntp_servers[inventory_hostname].priority }}"
keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
@@ -90,6 +134,49 @@
{% endif %}
when: ntp_keepalived_enabled | default(false)
# The v6 address is the path clients actually use, so it gets the same
# check as the v4 VIP. Without this a broken v6 rollout -- the address
# never claimed, or -nodad not behaving as expected on a given kernel --
# would not surface here at all, only later by reading
# /var/log/keepalived-state.log by hand.
#
# `ip -6 addr show ... to <addr>` prints the address when present and
# nothing when absent, so emptiness is the test.
#
# Presence cannot itself distinguish a VRRP-held address from a static
# leftover -- they look identical. It is only meaningful because the
# keepalived-ntp role asserts, on both distros, that no static assignment
# survives: the play fails before reaching here if one does.
- name: Check if this server holds the IPv6 service address
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ ntp_keepalived_interface }}"
- to
- "{{ ntp_vip6 }}"
register: vip6_check
failed_when: false
changed_when: false
when: ntp_keepalived_enabled | default(false)
- name: Display IPv6 service address status
ansible.builtin.debug:
msg: >
{% if vip6_check.stdout | default('') | length > 0 %}
This server ({{ inventory_hostname }}) holds the IPv6 service address
{{ ntp_vip6 }} -- it is the v6 MASTER and is serving ntp.srv.renner.uno
{% else %}
This server ({{ inventory_hostname }}) does not hold {{ ntp_vip6 }}.
Expected on a v6 BACKUP. If NO host holds it, check_ntp is failing
everywhere -- inspect `chronyc tracking` and
/var/log/keepalived-ntp-health.log
{% endif %}
when: ntp_keepalived_enabled | default(false)
- name: Report hosts serving NTP without participating in the VIP
ansible.builtin.debug:
msg: >
@@ -1,9 +1,25 @@
---
# Chrony NTP Server Role Defaults
# Networks allowed to use this server as NTP source
# Networks allowed to use this server as NTP source.
#
# BOTH families are required. IPv6 is the primary address family here -- the
# service address ntp.srv.renner.uno resolves to fd40::123 -- so an IPv4-only
# allow list refuses every client that uses the documented name.
#
# The hosts carried `allow fd00::/8` for a long time without it ever being in
# this file. Applying the role on 2026-08-07 rendered the template over it and
# dropped the line on all four NTP servers at once, which took internal NTP
# down over IPv6 until it was added here. Same class of untracked host state as
# the anycast service address, and the same failure: config that exists only on
# the box survives right up until something re-renders it.
chrony_allowed_networks:
- "10.2.0.0/16"
- "fd00::/8"
# Link-local. A client on the same segment may source its query from fe80::,
# and without this chrony refuses it -- the request arrives and is dropped,
# which looks like the server being unreachable rather than a policy refusal.
- "fe80::/10"
# Primary NTP servers (PTB - Physikalisch-Technische Bundesanstalt, Stratum 1)
chrony_upstream_servers:
@@ -4,6 +4,29 @@
keepalived_ntp_router_id: 4
keepalived_ntp_vip: "10.2.40.4"
keepalived_ntp_interface: "vmbr1.40"
# IPv6 service address for NTP (ntp.srv.renner.uno). Empty disables the v6
# instance entirely. IPv6 is the primary address family on this network, so
# this is the path that actually matters -- the v4 VIP above is legacy.
#
# Held by keepalived rather than assigned statically by networkd so that it is
# gated on check_ntp: present only while this host is fit to serve time, and
# withdrawn the moment it is not. A statically configured address is carried
# unconditionally from boot, which is how edge-1 came to answer NTP queries on
# 2026-08-04 while its clock was ~16 days in the past.
#
# Needs its own virtual_router_id: it is a distinct VRRP instance, and a VRID
# collision with the IPv4 instance on the same interface would have the two
# fighting over a single election.
keepalived_ntp_vip6: ""
keepalived_ntp_router_id6: 6
# Link-local address for the IPv6 VRRP instance, carried as its FIRST
# virtual_ipaddress. RFC 5798 gives an IPv6 virtual router a link-local primary,
# and keepalived warns "the first IPv6 VIP address should be link local" without
# one. Chosen to mirror the service address rather than derived from the virtual
# MAC, since no real NIC generates fe80::123 so there is nothing to collide with.
keepalived_ntp_vip6_linklocal: "fe80::123/64"
# NOTE: plaintext VRRP auth, unvaulted. Carried over as-is from the original
# role; VRRP PASS auth is weak regardless (sent in the clear on the segment),
# so this is a documented weakness rather than a secret worth vaulting.
@@ -1,22 +1,39 @@
#!/bin/bash
# Keepalived State Change Notification Script
#
# Two VRRP instances call this: NTP_VIP (IPv4, legacy) passes MASTER/BACKUP/
# FAULT, and NTP_VIP6 (IPv6, the primary path) passes the same suffixed with
# '6'. They are logged distinctly because the two can legitimately diverge --
# a host can hold one family and not the other -- and a log line that does not
# say which family it means is useless during an incident.
TYPE=$1
NAME=$(hostname)
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
log() {
echo "$TIMESTAMP: $NAME $1" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME $1"
}
case $TYPE in
MASTER)
echo "$TIMESTAMP: $NAME transitioned to MASTER state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now NTP MASTER"
log "transitioned to MASTER state (NTP, IPv4)"
;;
BACKUP)
echo "$TIMESTAMP: $NAME transitioned to BACKUP state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now NTP BACKUP"
log "transitioned to BACKUP state (NTP, IPv4)"
;;
FAULT)
echo "$TIMESTAMP: $NAME transitioned to FAULT state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now in NTP FAULT state"
log "transitioned to FAULT state (NTP, IPv4)"
;;
MASTER6)
log "transitioned to MASTER state (NTP, IPv6) - now serving ntp.srv.renner.uno"
;;
BACKUP6)
log "transitioned to BACKUP state (NTP, IPv6) - released ntp.srv.renner.uno"
;;
FAULT6)
log "transitioned to FAULT state (NTP, IPv6) - withdrew ntp.srv.renner.uno, unfit to serve time"
;;
*)
echo "$TIMESTAMP: $NAME unknown state: $TYPE" >> /var/log/keepalived-state.log
+115
View File
@@ -35,6 +35,121 @@
state: present
when: ansible_facts['os_family'] == 'Archlinux'
# --- Verify nothing else is assigning the IPv6 service address ---------------
#
# The removal itself lives in withdraw-static.yml, run as a separate fleet-wide
# play before any host reaches this role (see install-ntp-ha.yml). It has to
# complete everywhere first: DAD is enabled on the VRRP address, so a static
# copy on one host and a VRRP-managed copy on another cannot coexist -- the
# kernel marks the newcomer `dadfailed` and its master serves nothing.
#
# What remains here are the guards. They run on every execution, not just the
# migration, and fail the play if any static assignment survives -- which would
# both defeat the health gating (a static copy does not disappear when
# keepalived decides the host is unfit) and break the VRRP copy via DAD.
#
# The config file is what gets checked, not the running address, and that is
# what makes it safe to run every time: keepalived never writes to these files,
# so anything found is by definition static, whereas the live address is
# legitimately present on whichever host is master.
#
# Prefix-agnostic and \b-anchored on purpose. Matching the full
# address/prefixlen would fail open against a hand-written assignment using any
# other mask.
- name: Detect any remaining static assignment of the service address (ifupdown)
ansible.builtin.command:
argv:
- grep
- -rlE
- "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
- /etc/network/interfaces
- /etc/network/interfaces.d/
register: keepalived_ntp_static6_ifupdown
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
- name: Assert ifupdown is not also assigning the service address
ansible.builtin.assert:
that:
- keepalived_ntp_static6_ifupdown.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned statically by ifupdown on
this host
({{ keepalived_ntp_static6_ifupdown.stdout_lines | default([]) | join(', ') }}).
The automatic removal only matches the exact form
`up ip -6 addr replace {{ keepalived_ntp_vip6 }} dev <if> nodad`, so a
different prefix length or layout is left in place. Handing the address
to keepalived while ifupdown also assigns it means it would NOT be
withdrawn when check_ntp fails -- the host would keep serving time on an
address VRRP believes it has released. Remove the assignment from the
file(s) above, then re-run.
success_msg: >-
No static ifupdown assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
# On Arch (edge-1) the static assignment lives in a systemd-networkd file owned
# by the edge-host role, so this role must not rewrite it -- but it must not
# quietly proceed either. install-edge-host.yml is a separate play with no
# ordering relationship to this one, so nothing guarantees it has been re-run
# to drop the [Address] block. If it has not, edge-1 would end up holding
# fd40::123 statically AND running a VRRP instance for the same address:
# keepalived would believe it had withdrawn the address while networkd kept it
# assigned, and the host would go on answering NTP with a clock the health
# check had already rejected.
#
# That is the precise failure of 2026-08-04, on the host it happened to, so it
# fails fast rather than being left to inspection.
# Matched on the bare address, not the full address/prefixlen, and anchored
# with \b so it cannot also hit a longer address sharing the prefix (fd40::1234).
#
# Searching for the literal "fd40::123/128" would fail open: a hand-written
# [Address] using any other mask -- /64, say -- would not match, the assert
# below would pass, and keepalived would be handed an address networkd is
# still assigning. Since this check is detect-only, with no delete afterwards
# to catch the mistake, a miss here is silent. The Debian path does not share
# the problem: `ip -6 addr show ... to <prefix>` matches by range regardless of
# the assigned prefix length, and its delete is re-verified by assert.
- name: Detect a static assignment of the service address (systemd-networkd)
ansible.builtin.command:
argv:
- grep
- -rlE
- "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
- /etc/systemd/network/
register: keepalived_ntp_static6_networkd
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Archlinux'
- name: Assert networkd is not also assigning the service address
ansible.builtin.assert:
that:
- keepalived_ntp_static6_networkd.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned statically by
systemd-networkd on this host
({{ keepalived_ntp_static6_networkd.stdout_lines | default([]) | join(', ') }}).
Handing the address to keepalived while networkd also assigns it means
it would NOT be withdrawn when check_ntp fails -- the host would keep
serving time on an address VRRP believes it has released.
Re-apply the edge-host role first, whose template no longer carries the
[Address] block: ansible-playbook -i inventory/ plays/install-edge-host.yml
--limit {{ inventory_hostname }} --tags vlan40
success_msg: >-
No static networkd assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Archlinux'
- name: Create keepalived configuration
ansible.builtin.template:
src: keepalived.conf.j2
@@ -0,0 +1,134 @@
---
# Withdraw a static assignment of the IPv6 NTP service address.
#
# Included by install-ntp-ha.yml as a FIRST play across the whole ntp_cluster,
# before any host is given the NTP_VIP6 instance. That ordering is the point of
# splitting it out.
#
# DAD is enabled on the VRRP address (see keepalived.conf.j2), so a static copy
# on any host and a VRRP-managed copy on another cannot coexist: the kernel sees
# the duplicate and marks the newcomer `dadfailed`, leaving a master that
# believes it is serving while carrying an unusable address. Observed on edge-1
# on 2026-08-07, where keepalived reported MASTER while the address sat
# `dadfailed deprecated tentative` because three PVE nodes still held it
# statically.
#
# Doing the withdrawal per host inside the main role could not avoid that: with
# serial: 1 the first host would claim the address via VRRP while the rest were
# still static. Fleet-wide first, then claim.
#
# fd40::123 was assigned by hand on all four servers, in two different ways,
# none of it in version control:
#
# PVE nodes an `up ip -6 addr replace ... nodad` hook in
# /etc/network/interfaces
# edge-1 an [Address] block in /etc/systemd/network/, owned by the
# edge-host role and removed by re-applying it -- not touched
# here, since this role must not rewrite another role's file
#
# There is a service gap between this play and the next, while no host holds
# the address. It is bounded by the run time of one play and clients simply
# retry; the alternative is the dadfailed state above, which does not resolve
# on its own.
- name: Remove static assignment of the NTP service address (ifupdown)
ansible.builtin.lineinfile:
path: /etc/network/interfaces
regexp: '^\s*up\s+ip\s+-6\s+addr\s+(replace|add)\s+{{ keepalived_ntp_vip6 | regex_escape }}\b'
state: absent
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
register: keepalived_ntp_static6_removed
# Everything below is gated on the config line having just been removed, so the
# runtime delete happens once -- on the migration -- and never again.
#
# The gate is load-bearing. Once this has landed, the address is legitimately
# present on whichever host is the elected v6 master, placed there by
# keepalived, and a delete cannot tell that apart from a static assignment (the
# address looks identical either way). Ungated, every later run would delete the
# master's address and restart keepalived, dropping the service address and
# forcing a re-election each time -- the flapping this whole change exists to
# prevent.
- name: Check whether the service address is still assigned statically
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ keepalived_ntp_interface }}"
- to
- "{{ keepalived_ntp_vip6 }}"
register: keepalived_ntp_vip6_present
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
# failed_when: false because the desired end state is "address absent". An
# `ip addr del` against an address that has already gone is the outcome this
# task wants, not a reason to abort the play for that host.
- name: Withdraw the statically assigned service address
ansible.builtin.command:
argv:
- ip
- -6
- addr
- del
- "{{ keepalived_ntp_vip6 }}"
- dev
- "{{ keepalived_ntp_interface }}"
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
- keepalived_ntp_vip6_present.stdout | default('') | length > 0
failed_when: false
# The withdrawal gets exactly one attempt -- the gate that makes it idempotent
# also means it never retries -- and the delete swallows its own errors. So the
# result is asserted rather than assumed.
#
# `ip addr del` matches on address AND prefix length: if a host's static
# assignment used a different mask or interface than ntp_vip6 /
# keepalived_ntp_interface, the delete is a silent no-op and the address stays
# outside VRRP control permanently, with the play reporting success.
- name: Confirm the static service address was actually withdrawn
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ keepalived_ntp_interface }}"
- to
- "{{ keepalived_ntp_vip6 }}"
register: keepalived_ntp_vip6_after
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
- name: Assert the service address is no longer assigned outside keepalived
ansible.builtin.assert:
that:
- keepalived_ntp_vip6_after.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned on
{{ keepalived_ntp_interface }} after the static assignment was removed.
`ip -6 addr del` matches address AND prefix length, so this usually means
the on-host assignment differs from ntp_vip6 or uses another interface.
Leaving it would both keep the address outside VRRP control and make the
VRRP-managed copy fail DAD elsewhere. Inspect with
`ip -6 addr show dev {{ keepalived_ntp_interface }}` and
`grep -rn {{ keepalived_ntp_vip6 }} /etc/network/ /etc/systemd/network/`.
success_msg: >-
{{ keepalived_ntp_vip6 }} withdrawn; safe for keepalived to claim it.
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
@@ -39,3 +39,72 @@ vrrp_instance NTP_VIP {
notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP"
notify_fault "/usr/local/bin/keepalived-notify.sh FAULT"
}
{% if keepalived_ntp_vip6 %}
# IPv6 service address for NTP (ntp.srv.renner.uno). IPv6 is the primary
# address family here, so this instance -- not the IPv4 one above -- is the
# path that matters.
#
# A separate instance is mandatory, not stylistic: keepalived requires that
# "all addresses in virtual_ipaddress must be of the same family", so the v6
# address cannot be added to NTP_VIP.
#
# No authentication block: VRRPv3 removed authentication (RFC 5798), and
# keepalived always uses version 3 for IPv6 instances. Configuring auth here
# is rejected rather than ignored.
#
# The first address is link-local, and must be. RFC 5798 gives an IPv6 virtual
# router a link-local address as its primary, and keepalived warns
# "the first IPv6 VIP address should be link local" otherwise. It carries on,
# but the instance is then sourcing VRRPv3 from something the protocol does not
# expect.
#
# DAD is deliberately left ON (no -nodad). The flag was inherited from the
# previous design, where every NTP server carried fd40::123 simultaneously and
# duplicate detection had to be suppressed for that to work at all. Under
# single-master VRRP the opposite is true: exactly one host should hold the
# address, so a duplicate is a genuine fault and detecting it is the point.
#
# This does mean the migration cannot leave a static copy anywhere while a
# VRRP-managed one exists -- DAD sees the two and marks the newcomer
# `dadfailed`, leaving a master that believes it is serving while carrying an
# unusable address. Observed on edge-1 on 2026-08-07. install-ntp-ha.yml
# therefore withdraws every static assignment fleet-wide in a first play
# before any host configures this instance.
#
# track_script is what makes this health-gated: the address is only held while
# check_ntp passes, and is withdrawn as soon as it fails. On withdrawal
# keepalived sends unsolicited Neighbour Advertisements so client neighbour
# caches are updated immediately -- a statically assigned address cannot do
# this, and clients would keep sending to the dead holder until their own
# neighbour unreachability detection expired.
vrrp_instance NTP_VIP6 {
state {{ keepalived_ntp_state }}
interface {{ keepalived_ntp_interface }}
virtual_router_id {{ keepalived_ntp_router_id6 }}
priority {{ keepalived_ntp_priority }}
advert_int 1
virtual_ipaddress {
{{ keepalived_ntp_vip6_linklocal }} dev {{ keepalived_ntp_interface }}
{{ keepalived_ntp_vip6 }} dev {{ keepalived_ntp_interface }}
}
# weight 0 overrides the script's own weight, and the distinction decides
# whether this works at all. A non-zero weight only SUBTRACTS from priority
# on failure -- so the last remaining host, having no one to lose the
# election to, stays MASTER and keeps serving time with a clock it has
# already reported as wrong. Weight 0 instead transitions the instance to
# FAULT after <fall> failures, which releases the address unconditionally.
#
# That is the required behaviour here: a host unfit to serve time must stop
# answering, whether or not a replacement exists.
track_script {
check_ntp weight 0
}
notify_master "/usr/local/bin/keepalived-notify.sh MASTER6"
notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP6"
notify_fault "/usr/local/bin/keepalived-notify.sh FAULT6"
}
{% endif %}