Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 6cfa63f6f3 fix(edge): keep the docker log-driver config out of the shared role
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #474 caught that the previous commit put the daemon.json task in
docker-install, which is shared by ten plays: install-netbird-server,
install-netbird-proxy, install-n8n, install-authentik-ha, install-owncloud,
install-gatus, install-ical-proxy, install-grist and
install-music-assistant.

None of those hosts has an /etc/docker/daemon.json today -- the template and
the default were both new here -- so the next routine run of any of those
plays would have created one, reported changed, fired the Restart docker
handler, and bounced every container on that host. Unannounced restarts
across authentik, n8n, owncloud, grist, netbird, gatus, music-assistant and
ical-proxy, from a change about edge-host bootstrapping.

The --check verification could not have caught it: it only ever exercised
install-edge-host.yml, and the damage was in the other nine plays.

The comment I wrote there was the tell. It said "the services that use this
host run with network_mode: host" -- true of edge-1, false of most consumers
of that role. A justification that only held for the one host I was looking
at, written into a shared file.

Moved to edge-flash-wear, which is edge-host-only and already owns this
rationale: routing container stdout to the journal is flash-wear handling,
not generic docker setup. The networking aside is dropped -- in a flash-wear
role the only thing worth saying about daemon.json is why logging goes to
the journal.

docker-install is now byte-identical to master (git diff origin/master --
roles/docker-install/ is empty).

Two details the move needed: /etc/docker is created explicitly, because
edge-flash-wear runs before docker-install on a first bootstrap and the
directory will not exist yet; and the Restart docker handler fires at
end-of-play, by which time docker is installed.

Verified: ok=67 changed=25. The changed count is unchanged from before the
move, and no docker task appears in it -- daemon.json renders byte-identical
to what the old role produced, so it reports ok and the handler never fires
on edge-1.

Also from review: noted on the play that heating-can, zwave-gateway and
power-meter-sml each declare edge_base_dir in their own defaults so they
stay usable standalone, that play vars win, and that the two must be kept in
step.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 12:27:13 +02:00
claude-botandClaude Opus 5 9cc20251fd fix(edge): retire the legacy compose project before any role starts a container
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #474 found two blocking defects, both in exactly the place the
--check verification is blind: command and shell tasks are skipped, not
simulated, under check mode. So "verified with --check --diff" covered
everything except the two things that would actually have broken.

1. Ordering: a real run would have aborted.

   The legacy-project teardown lived in power-meter-sml, which the play runs
   last. zwave-gateway runs before it and does `docker compose up -d` for a
   container named zwave-js -- while the old shared project still owns that
   name. Docker enforces container names globally, not per project, so the
   command fails with "Conflict: container name already in use". That is not
   the missing-USB-device string the task tolerates, so the task fails and
   the play aborts before ever reaching the cleanup.

   Moved to pre_tasks, which is its correct home anyway: it is a migration
   step for both services, and putting it inside either one meant whichever
   ran first hit the conflict.

   The play now also declares edge_base_dir and edge_legacy_compose in its
   own vars rather than borrowing them from a role's defaults, since
   pre_tasks must not depend on role default resolution. And it removes the
   now-orphaned zwavejs.env left at the old shared project root.

2. Dangling handler.

   The libeatmydata task still notified "Restart edge compose services", a
   handler that was deleted with the old role. Masked today only because
   `creates:` makes the task report ok -- but the role's own comment says to
   delete that .so when the Alpine base bumps, and the next time it does,
   Ansible would fatally error with "handler not found". Now notifies
   "Restart zwave-js".

   Swept every notify in the new roles against every defined handler; that
   was the only unresolved one.

Also from review: kubernetes/bootstrap.sh referenced the deleted role.
Repointed to ansible/docs/edge-host.md.

Re-verified: ok=66 changed=25. The teardown moved from the power-meter-sml
group into pre_tasks and gained the orphaned-env removal, so the count goes
5 + 5 + 14 -> 5 pre-existing + 6 migration + 14 header repointing. Nothing
else moved.

The limit is worth stating plainly: this re-run still cannot prove the
teardown works, for the same reason it missed the bug. The first real apply
is the test.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 11:47:39 +02:00
claude-botandClaude Opus 5 2d9c3ba213 refactor(edge): split the edge-host role into one role per concern
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
edge-host was a single role doing eight unrelated jobs: flash-wear
mitigation, log shipping, VLAN networking, an MQTT bridge, a CAN interface,
the heating CAN services, a Z-Wave gateway and a power-meter reader.

That is not just untidy. It is what blocked a second thin client from
joining as an NTP peer: the host-generic work (journald to RAM, Alloy, the
VLAN sub-interface) could not be applied without also handing the box a
Z-Wave radio and a CAN bus it does not have. The role bound its services to
specific USB device paths, so a second host would have tried to start
containers for hardware that is not there.

Now:

  edge-flash-wear   journald -> RAM, noatime/lazytime, TRIM, eMMC health
  edge-log-shipper  Alloy: journal -> Loki
  edge-vlan         tagged VLAN sub-interface
  mqtt-bridge       loopback mosquitto -> cluster broker over TLS
  socketcan         can0 only
  heating-can       open3e (Vitocal/VX3) + e3oncan (E380 meter)
  zwave-gateway     zwave-js-ui, own compose project
  power-meter-sml   sml2mqtt, own compose project

The E380 reader lives with open3e rather than in its own role: it is part
of the same heating installation -- it feeds the heat pump's own energy
management -- not a separate system that happens to share a bus. can0 does
stay separate, because an interface is not the same concern as the services
consuming it.

Each role installs its own packages, so it stands alone. An edge host with
no peripherals takes the first five and none of the rest.

docker-daemon.json moves to docker-install with its own docker_log_driver
variable and Restart docker handler, since configuring the docker daemon was
never an edge-host concern. Handlers split ten ways; defaults split per
role; all sixteen templates and files moved with git mv.

Two deliberate behaviour changes on the node, both accepted:

1. Compose split. zwave-js and sml2mqtt shared a compose file, so touching
   either recreated both. They are now separate projects in their own
   directories. container_name is fixed, so the old project must release the
   names first -- a guarded one-time task downs it and removes the file.
   First apply costs a brief down/up on both. Data paths are unchanged; the
   zwave store is not migrated.

2. Header repointing. Templates and scripts carried "Managed by
   roles/edge-host" headers, which render onto the host -- they would have
   sent an operator to a role that no longer exists. Updating them changes
   file content, so it restarts the services that consume those files.

Verified by --check --diff against the live host. 24 changed items, and
every one is accounted for:

    5  pre-existing, unchanged from the baseline before this work:
       common timesyncd; open3e venv; E3onCAN venv; and the open3e /
       e3oncan restart handlers. The venv tasks install from a git URL,
       which pip cannot verify, so they report changed on every run --
       a separate defect, not introduced here.
    5  the compose split: zwave env + compose and sml compose to their
       new paths, plus the two legacy-project retirement tasks.
   14  the header repointing: journald conf, eMMC health script, alloy
       config, mosquitto bridge config, 80-can0.network, open3e overlay /
       env / unit, e3oncan unit, zwave backup script + env, sml2mqtt
       config -- and the resulting restarts of journald, alloy, mosquitto,
       open3e and e3oncan.

Nothing changed outside those three groups.

One consequence check mode cannot show: 80-can0.network.j2 changed, so the
"Reload networkd" handler fires on a real run and reconfigures can0,
briefly dropping the bus under open3e and e3oncan. It is absent from the
check-mode output only because shell handlers are skipped there. The VLAN
templates did NOT change, so the riskier reconfigure of the parent uplink
on this headless host is not triggered.

Also fixed in passing: the keepalived-ntp guard told operators to recover
with `--limit <host> --tags vlan40`, and referred to the edge-host role.
Both corrected. The role README becomes ansible/docs/edge-host.md -- it
documents the host, not any single role now.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 11:33:57 +02:00
claude-botandClaude Opus 5 07faf0ae69 fix(edge-host-iso): carry the bootstrap fixes into the ISO side too
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #474 caught that the previous commit fixed the procedure in
ansible/roles/edge-host/README.md and left the same two defects in the two
places an operator actually reads them first.

That happened because my grep for the stale patterns ran with the working
directory set to ansible/, so `images/edge-host-iso/README.md` was a path
that did not exist. grep returned nothing and I read that as clean. A false
negative from a bad path, not a clean sweep.

Fixed now, verified from the repository root:

  airootfs/usr/local/bin/edge-install
    Printed the superseded two-command sequence at the console immediately
    after imaging -- the very first instructions anyone sees, telling them
    to run common.yml with -l edge_host. Now prints the single unrestricted
    run, plus the two steps that precede it (confirm the host resolves by
    name, add it to [edge_host]).

  README.md
    Described the image as being for `edge-1.srv.renner.uno`, a name that
    does not resolve, and repeated the same two-command bootstrap. Both
    replaced. This file is the link target of step 2 in the ansible-side
    README, so following the procedure end to end landed straight back on
    the defect #474 claims to close.

Also corrected while there: the ISO README called the target "the bare-metal
N100". edge-1 is a Wyse 3040 (Atom x5-Z8350), and the installer text further
down already says it "covers the Wyse 3040". Rewritten to describe the image
as not tied to one box or model, since the installer autodetects the target
disk -- which is what makes it usable for the second thin client.

Also from review: a comment on why `become: true` is set when the connection
is already root. Not taken -- ignore_unreachable, which common.yml sets. The
review's own reasoning applies: Ansible already skips an unreachable host and
continues with the others, so it adds nothing here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:21:27 +02:00
claude-botandClaude Opus 5 e288ebe39d fix(edge-host): make the bootstrap self-contained and the runbook followable
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The documented base-install procedure had three defects, two of which are
standing house rules and one of which simply does not work.

1. It required a DHCP reservation.

   "Add the DHCP reservation/DNS entry for <host> on the UDM" was step 3 --
   a reservation as a documented precondition. Reservations are off the
   table here. The UDM registers DHCP leases in DNS on its own, so the
   record appears without one, and the host is identified by name
   everywhere: inventory, plays, README. Its address is never recorded.

2. It required --limit, because the plays were split wrong.

   Step 4 was:

     ansible-playbook plays/common.yml -l edge_host
     ansible-playbook plays/install-edge-host.yml

   The --limit was load-bearing, not sloppiness: common.yml is
   `hosts: all:!...`, so running it unrestricted while bootstrapping a
   single new box would reconfigure the entire fleet. The split left no
   other option. But --limit narrows the target list rather than staging
   anything, and leaves every other host outside convergence with no report
   that they were skipped.

   Fixed structurally rather than by deleting the flag: install-edge-host.yml
   now carries `certificates` and `common` itself -- the parts of common.yml
   that apply to an edge host -- so the bootstrap is one unrestricted run
   over the edge_host group. No second command, no limit.

   Not carried over: qemu-guest-agent (these are bare metal, not VMs) and
   monitoring-agent (Debian-oriented; edge hosts ship logs via edge-alloy).
   common.yml's loki/prometheus vars go with monitoring-agent and are not
   needed -- verified that the common role does not reference them.

3. It named a host that does not exist.

   The README said `edge-1.srv.renner.uno` in both the base-install steps
   and the Z-Wave cutover checklist. The host is `edge-1.iot.renner.uno`;
   the .srv name does not resolve, so anyone following the runbook hits a
   dead name at step 3.

Verified idempotent against the live host before committing:
`ansible-playbook plays/install-edge-host.yml --check --diff` over the whole
edge_host group -> ok=53 changed=5 failed=0, with all five changes inside
the edge-host role (open3e venv, E3onCAN checkout/venv), which was already
in this play. certificates and common contributed no changes at all.

Also checked the common role branches only on Debian/RedHat os_family, so
its OS-specific tasks skip cleanly on Arch.

Prompted by preparing a second Wyse 3040 as an NTP peer -- this is the
procedure that would have been followed to build it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 08:07:15 +02:00
claude-botandClaude Opus 5 17c12fe13f refactor(prometheus): address the PVE nodes by name in scrape targets
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Follow-up to #472, which changed the inventory but left the same three IPv4
literals in the monitoring config and a runbook. Raised in review there.

prometheus.yml.j2:
  - pve job targets -> FQDNs
  - chrony job targets -> FQDNs, and restructured to one static_config per
    host so each carries its own hostname label

The chrony restructure is the substantive part. The previous form listed all
three targets together and recovered the hostname in relabel_configs by
regex-matching __address__ against each node's IP:

    - source_labels: [__address__]
      regex: '10\.2\.40\.201:.*'
      target_label: hostname
      replacement: 'jupiter'

That silently coupled the labels to the addressing. Swapping a target for a
name would have left every regex unmatched and the hostname label simply
absent, with the scrape still succeeding and nothing failing -- a
find-and-replace on the addresses alone would have broken the labels
quietly. Per-host static_configs remove the coupling entirely.

nodes.yml.j2: targets -> FQDNs. Already one entry per host with an explicit
hostname label, so nothing else to change.

claude-workstation/RUNBOOK.md: the kubeconfig command pointed --server at
https://10.2.40.201:6443. That is jupiter, a Proxmox host, not a control
plane node -- stale from when k8s ran as VMs on it. The cluster's endpoint
is https://k8s.renner.uno:6443 (kubernetes/talos/patches/cluster.yaml), and
the name is also what the API server certificate carries as a SAN, so an
address there fails TLS verification even when reachable. The documented
command was very likely broken as written.

Deliberately NOT changed: kubernetes/apps/pve/endpointslice.yaml. An
EndpointSlice takes literal addresses; hostnames are not valid in that
field, so those three are correct as they are.

Note this changes the `instance` label on the pve job from an address to a
name, since the relabel chain feeds __address__ into __param_target and on
to instance. Anything selecting on the old value needs updating.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:52:49 +02:00
k.renner 332950b564 Merge pull request 'refactor(inventory): address the PVE nodes by name, not by IPv4 literal' (#472) from inventory-pve-hostnames into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#472
2026-08-07 17:49:05 +00:00
claude-botandClaude Opus 5 ba369a37b0 refactor(inventory): address the PVE nodes by name, not by IPv4 literal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
jupiter, neptune and saturn were the last entries in pve_hosts.yml still
reached via ansible_host=<IPv4 literal>. Every other host here --
authentik, netbird, proxy, nas-1, grafana, edge-1 -- has always used its
FQDN, so these three were the outlier rather than the convention.

An address in an inventory is allocator state standing in for identity, and
it pins access to the legacy family: the names resolve IPv6-first, which is
the primary path on this network. DNS is authoritative for where a host is;
the inventory should say which host.

The short inventory_hostname is kept deliberately -- plays key on it
(ntp_servers[inventory_hostname] in install-ntp-ha.yml, node names in
opentofu, the prometheus targets) -- so only ansible_host changes.

No claim that this fixes the connection stalls seen while applying the NTP
work on 2026-08-07. Those were the IPS reacting to the connection pattern I
was generating (a port sweep, then rapid repeated SSH across hosts
interleaved with Ansible's own churn), not an address-family problem. This
change stands on convention and on not encoding a lease as identity.

claude-ws is left alone: its comment documents that obg.renner.uno is
split-horizon and resolves to the public proxy from home. That explains why
that particular name is unusable, not why a literal is right -- a name that
resolves to the wrong target calls for one that resolves correctly. Out of
scope here, and I do not know whether such a record exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:44:54 +02:00
k.renner eacc821222 Merge pull request 'fix(chrony): allow IPv6 clients -- the allow list was IPv4-only' (#471) from chrony-allow-ipv6 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#471
2026-08-07 17:35:07 +00:00
claude-botandClaude Opus 5 24ed2485c9 fix(chrony): allow IPv6 clients -- the allow list was IPv4-only
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
chrony_allowed_networks contained only 10.2.0.0/16, so the rendered
chrony.conf permitted IPv4 clients and nothing else. IPv6 is the primary
address family here: ntp.srv.renner.uno resolves to fd40::123, so an
IPv4-only allow list refuses every client that uses the documented name.

This was live but invisible. The hosts carried `allow fd00::/8` in
chrony.conf for a long time without it ever being in this repository.
Applying the role on 2026-08-07 rendered the template over the live config
and dropped the line on all four NTP servers at once, taking internal NTP
down over IPv6 until it was added here.

Same class of untracked host state as the anycast service address, and the
same failure: config that exists only on the box survives right up until
something re-renders it. Reading the drifted file is not enough -- the rest
of it has to be diffed against what the template would produce.

The outage was not caught by the play, which reported the v6 master as
"serving ntp.srv.renner.uno" throughout. It was caught by querying the
service:

  chronyd -Q "server ntp.srv.renner.uno iburst"
    -> No suitable source for synchronisation

and after the fix:

  chronyd -Q "server ntp.srv.renner.uno iburst"
    -> System clock wrong by 0.001999 seconds (ignored)

with a control query against ptbtime1.ptb.de to confirm the method rather
than the target was sound.

fe80::/10 is included as well: a client on the same segment may source its
query from a link-local address, and without it chrony drops the request,
which presents as the server being unreachable rather than as a policy
refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:33:22 +02:00
k.renner 6628f6eaef Merge pull request 'fix(ntp): link-local VRRP primary, and sequence the migration around DAD' (#470) from ntp-vrrp-linklocal-and-dad-sequencing into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#470
2026-08-07 17:18:11 +00:00
claude-botandClaude Opus 5 ea439b110d fix(ntp): link-local VRRP primary, and sequence the migration around DAD
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Applying #469 to edge-1 exposed two defects that only show up on real
hardware.

1. The VRRP address was unusable.

   keepalived took MASTER and assigned fd40::123, but the kernel left it:

     inet6 fd40::123/128 scope global dadfailed deprecated tentative
           proto keepalived
           valid_lft forever preferred_lft 0sec

   `-nodad` did not take effect -- no nodad flag on the address, and
   net.ipv6.conf.enp1s0/40.accept_dad = 1 -- so DAD ran and found a real
   duplicate: jupiter, saturn and neptune still held the address statically.
   edge-1 believed it was the v6 master while serving nothing.

   The flag is dropped rather than fixed. It was inherited from the previous
   design, where every NTP server carried the address at once and duplicate
   detection had to be suppressed for that to work. Under single-master VRRP
   the opposite holds: exactly one host should have it, so a duplicate is a
   genuine fault and detecting it is the point.

   What that costs is that the migration can no longer be done host by host.
   A static copy anywhere and a VRRP copy elsewhere cannot coexist, and the
   dadfailed state is sticky -- the kernel does not retry. install-ntp-ha.yml
   is therefore split in two: a first play withdraws every static assignment
   across the whole group, then the second hands the address to keepalived.
   Deliberately not serial, since it must finish everywhere before any host
   claims the address. There is a gap between the phases where nobody holds
   it, bounded by one play's run time; clients retry.

   The withdrawal tasks move to roles/keepalived-ntp/tasks/withdraw-static.yml
   for the first play to include. main.yml keeps the detect-and-assert guards,
   which still run every time and fail loudly if any static assignment
   survives.

2. keepalived warned:

     (NTP_VIP6) the first IPv6 VIP address should be link local

   RFC 5798 gives an IPv6 virtual router a link-local primary. keepalived
   carried on regardless, but the instance was sourcing VRRPv3 from something
   the protocol does not expect. fe80::123 is now the first
   virtual_ipaddress: it mirrors the service address and cannot collide with
   a NIC's EUI-64 link-local.

ntp_vip6 and ntp_vip6_linklocal move to group_vars/ntp_cluster.yml so both
plays read one definition rather than duplicating a service address.

Verified: `keepalived -t` on edge-1 (keepalived 2.4.1) parses the two-address
form clean, and the playbook passes --syntax-check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 19:07:22 +02:00
k.renner b1c06b9d43 Merge pull request 'feat(ntp): hold the IPv6 service address in keepalived, gated on clock health' (#469) from ntp-service-address-health-gated into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#469
2026-08-07 11:27:30 +00:00
security-bot 436c24a394 chore(image-automation): bump ghcr.io/music-assistant/server:2.9.11
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-07 02:34:14 +00:00
claude-botandClaude Opus 5 a1e3b65cbf fix(ntp): verify the ifupdown side unconditionally, not only after a match
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #469 found the Debian path still fails open, and was right that
my previous commit message was wrong to claim otherwise.

The whole Debian chain -- detect, withdraw, confirm, assert -- was gated on
`keepalived_ntp_static6_removed.changed`, which is only true when the
lineinfile regex matches exactly, prefix length included. The hook it
targets is hand-written and was never in version control, so any deviation
(a different mask, unusual whitespace, a second assignment elsewhere)
leaves .changed false and silently skips every following task. The static
assignment then coexists with the VRRP-managed one, which is the
double-assignment failure this PR exists to prevent -- just relocated from
edge-1 to a PVE host.

Same fail-open class as the Arch grep fixed in 5e01e2f1. The argument in
that commit message -- that the Debian path was safe because
`ip -6 addr show ... to <prefix>` matches by range -- is true of the runtime
checks but says nothing about the lineinfile text match, which is a
different mechanism and was exactly as fragile as the pre-fix Arch grep.

Fixed by checking the CONFIG FILE unconditionally, prefix-agnostic and
\b-anchored, mirroring the networkd check.

Note this is deliberately not the fix the review proposed. It suggested
running the runtime `ip -6 addr show ... to <addr>` check and its assert
unconditionally. That would break the PR: after migration the address is
legitimately present on whichever host is the elected v6 master, so an
unconditional "must be absent" assert fails on the master on every run. The
config file is the right thing to check every time precisely because
keepalived never writes there -- anything found is by definition static,
whereas the live address is expected on the master. The runtime withdrawal
stays gated on .changed, which remains correct for it.

Verified on jupiter:
  grep -rlE 'fd40::123\b' /etc/network/interfaces /etc/network/interfaces.d/
    -> /etc/network/interfaces

Sequencing holds: on the migration run the lineinfile removes the hook
first, so the grep finds nothing and the assert passes; if the regex does
not match, the grep finds it and the assert fails loudly instead of
skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:29:53 +02:00
claude-botandClaude Opus 5 5e01e2f1b5 fix(ntp): detect a static networkd assignment at any prefix length
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Arch guard added in the previous commit grepped for the literal
"fd40::123/128" -- the full address/prefixlen from ntp_vip6. A hand-written
[Address] using any other mask (/64, say) would not have matched, the
assert would have passed, and keepalived would have been handed an address
networkd was still assigning.

That fails open, which is the wrong direction for this particular check.
It is detect-only: there is no delete afterwards whose result gets
re-verified, so a miss is silent, and the outcome is the dual-ownership
state the guard exists to prevent.

The Debian path does not share the problem. `ip -6 addr show ... to
<prefix>` matches by range regardless of the prefix length the address was
actually assigned with, and its `ip addr del` -- which IS prefix-sensitive
-- is re-verified by a following assert, so a mismatch there fails loudly.

Now matches the bare address, anchored with \b so it cannot instead hit a
longer address sharing the prefix. Verified both directions:

  grep -rlE 'fd40::123\b' /etc/systemd/network/   on edge-1
    -> /etc/systemd/network/10-enp1s0.40.network  (still detected)
  grep -cE 'fd40::123\b' against 'Address=fd40::1234/64'
    -> 0                                          (no false positive)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:11:36 +02:00
claude-botandClaude Opus 5 40ef53fa67 fix(ntp): assert the static service address is really gone before handing over
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Second review round on #469 found two ways the migration could silently
leave the address outside VRRP control -- both of which would defeat the
health gating while the play reported success.

1. The one-shot migration had no verification.

   Gating the withdrawal on `keepalived_ntp_static6_removed.changed` is what
   makes it idempotent, but it also means it gets exactly one attempt and
   never retries. Combined with `failed_when: false` on the delete, a
   mismatch would pass silently: `ip addr del` matches on address AND prefix
   length, so a host whose static assignment used a different mask or
   interface than ntp_vip6 / keepalived_ntp_interface would see the delete
   no-op, keep the address permanently outside VRRP control, and still
   report success.

   The idempotency fix from the previous commit created this: making the
   task run once removed the retry that would otherwise have masked it.

   Now asserted explicitly after the delete, with a message naming the
   likely cause and the commands to inspect it.

2. edge-1 -- the host the incident actually happened on -- was not covered.

   The removal tasks are gated to Debian, so they never touch edge-1 (Arch).
   Its static [Address] block belongs to the edge-host role, and
   install-edge-host.yml is a separate play with no ordering relationship to
   this one. Nothing guaranteed it had been re-applied.

   edge_host already sets ntp_keepalived_enabled: true, so the next run of
   install-ntp-ha.yml gives edge-1 the NTP_VIP6 instance immediately. With
   the [Address] block still present it would hold fd40::123 statically AND
   run VRRP for it: keepalived would believe it had withdrawn the address
   while networkd kept it assigned, and the host would go on answering NTP
   with a clock check_ntp had already rejected. That is the failure of
   2026-08-04, on the same host.

   This is not hypothetical -- edge-1 still has the block right now:
     grep -rl fd40::123/128 /etc/systemd/network/
     -> /etc/systemd/network/10-enp1s0.40.network

   The role now detects this and fails fast, naming the play to run first,
   rather than proceeding into a split-ownership state.

Also documents why the post-deploy presence check is meaningful: presence
alone cannot distinguish a VRRP-held address from a static leftover, and
only means what it claims because the role asserts no static assignment
survives on either distro.

Not taken: narrowing serial: 1 so routine runs are not serialised through
the chrony settle pause. Real cost, but correctness of the address handover
comes first at four hosts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 22:02:26 +02:00
claude-botandClaude Opus 5 fbf5c4dcb8 fix(ntp): make the static-address withdrawal run once, not every play
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review on #469 caught a real defect. The withdrawal task used
ansible.builtin.command with no changed_when, so it reported changed and
fired `notify: restart keepalived` on every run where the address was
present.

That is fine exactly once. After this lands, fd40::123 is legitimately
present on whichever host is the elected v6 master, put there by keepalived
-- and an ungated delete cannot tell that apart from a static assignment,
because the address looks identical either way. So every later run of this
play (adding a host, changing a priority, plain re-convergence) would have
deleted the master's address and restarted keepalived, dropping the service
address and forcing a re-election each time. That is precisely the flapping
this change exists to prevent.

Both the check and the delete are now gated on the ifupdown hook having just
been removed, so they run once during the migration off static config and
never again. Steady-state runs leave the master's address alone.

Also from review:

  - failed_when: false on the delete. The desired end state is "address
    absent". If check_ntp fails in the window between the check and the
    delete, keepalived withdraws the address first and `ip addr del` errors
    on an address that is already gone -- the outcome the task wants, and no
    reason to abort the play for that host.

  - The play's post-deploy verification only ever checked the IPv4 VIP. The
    v6 address is the path clients actually use, so a broken v6 rollout --
    address never claimed, or -nodad not behaving as expected on a given
    kernel -- would not have surfaced in the play at all. It now gets the
    same check, and the message points at check_ntp and the health log for
    the case where no host holds it.

Not taken: the suggestion to narrow serial: 1 so it does not also serialise
chrony. With four hosts it costs nothing, and scoping it per-role would mean
splitting the play. Worth revisiting if the cluster grows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:28:21 +02:00
claude-botandClaude Opus 5 112299da49 fix(ntp): remove the static service address so keepalived can own it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Without this the health gating is defeated on three of the four NTP
servers. fd40::123 was assigned statically on every one of them, by hand
and in two different ways, none of it in version control:

  PVE nodes  `up ip -6 addr replace fd40::123/128 dev vmbr1.40 nodad`
             in /etc/network/interfaces
  edge-1     an [Address] block in /etc/systemd/network/

A statically assigned copy does not disappear when keepalived decides the
host is unfit to serve time. Leaving it in place means the host keeps
answering NTP on an address VRRP believes it has withdrawn -- exactly the
failure this whole change exists to prevent, with the added confusion of
keepalived reporting the address as released.

The ifupdown hook is removed here. edge-1 needs nothing: the edge-host
networkd template has never carried the address, so applying that role
drops it.

The live address is deleted as well, because removing the config line alone
does not take effect until the next network reload, and a reload on a PVE
node interrupts guest connectivity including the Talos VM and its Ceph OSD.
Deleting it is safe where keepalived already holds it: the deletion
notifies a keepalived restart, and the elected master re-adds it on the way
back up. Doing it unconditionally avoids having to tell "assigned
statically" from "assigned by keepalived", which the address alone does not
reveal.

The play now runs serial: 1. Withdrawing the address and restarting
keepalived on every NTP server at once would leave the fleet with nobody
holding fd40::123 until the elections settle; serialised, the remaining
members answer throughout.

Verified against the live hosts rather than assumed:
  - all three PVE nodes carry the identical hook (jupiter:54, saturn:55,
    neptune:56), so one task covers them
  - the removal regexp matches exactly one line per host
  - `ip -6 addr show dev <if> to <addr>` prints the matching address when
    present and nothing at all when absent, so the guard condition is sound

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 21:06:43 +02:00
claude-botandClaude Opus 5 01e843beef fix(ntp): apply the v6 service address to every ntp_cluster member
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The previous commit set keepalived_ntp_vip6 in group_vars/edge_host, which
would have given the v6 VRRP instance to edge-1 alone. The other NTP
servers would have rendered no NTP_VIP6 instance at all and stayed silently
outside the election -- so the address could never fail over to them, and
the health gating would have amounted to "edge-1 withdraws and nobody takes
over".

Moves it alongside ntp_vip in plays/install-ntp-ha.yml, which is already
where the service-wide address lives, and passes it to the role for every
host in ntp_cluster.

Verified that edge-1 is currently BACKUP and does not hold the IPv4 VIP: a
VRRP BACKUP receiving no advertisements promotes itself to MASTER, so the
other members are alive and one of them holds it. They are real
participants, not retired hosts.

Not verified: whether the other members also carry fd40::123 statically, as
edge-1 did. They have no DNS records -- jupiter/saturn/neptune resolve only
through the *.renner.uno wildcard to the proxy -- so there is no hostname
path to them to check. If they do carry it statically, that address must be
removed at the same time this lands, or two hosts will hold it outside VRRP
control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 20:07:05 +02:00
claude-botandClaude Opus 5 2fbce12b0b feat(ntp): hold the IPv6 service address in keepalived, gated on clock health
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ntp.srv.renner.uno resolves to fd40::123 (IPv6) and to the keepalived VIP
(IPv4, legacy). IPv6 is the primary family here, so the v6 address is the
path clients actually use -- and it was the one with no health gating at
all.

It was assigned statically in /etc/systemd/network/ on edge-1, so networkd
carried it unconditionally from boot. That is how the host answered NTP
queries on 2026-08-04 while its clock was ~16 days in the past: chronyd was
running, udp/123 was listening, the address was up, and nothing anywhere
was checking whether the clock was right. Fixing the keepalived health
check does not help, because keepalived was not holding that address.

This moves it into keepalived as a second VRRP instance tracking the same
check_ntp script, so the address exists only while the host is fit to serve
time.

Three details that are load-bearing rather than stylistic:

  - A separate vrrp_instance is mandatory. keepalived requires that "all
    addresses in virtual_ipaddress must be of the same family", so the v6
    address cannot join NTP_VIP.

  - No authentication block. VRRPv3 removed authentication (RFC 5798) and
    keepalived always uses version 3 for IPv6 instances.

  - track_script uses `weight 0`, overriding the script's -20. A non-zero
    weight only subtracts from priority on failure, so the last remaining
    host -- having nobody to lose the election to -- would stay MASTER and
    keep serving a clock it had already reported as wrong. Weight 0
    transitions the instance to FAULT instead, releasing the address
    unconditionally. That is the required behaviour: a host unfit to serve
    time must stop answering whether or not a replacement exists.

Semantic change, deliberate and worth stating: VRRP is a single-master
unicast VIP, not anycast. Previously every NTP server carried fd40::123 and
clients reached whichever answered neighbour discovery first. Now exactly
one host holds it. For NTP this is the better trade -- query volume makes
load spreading pointless, and on failover keepalived sends unsolicited
Neighbour Advertisements so client neighbour caches update in seconds,
where the static address had no withdrawal signal at all and clients kept
using a dead holder until their own neighbour unreachability detection
expired. True routed anycast with health withdrawal would mean announcing
the /128 via an IGP, which the gateway would have to speak.

The edge-host role is deliberately untouched: its template already has no
[Address] block, so applying it removes the hand-placed address, and
keepalived then adds it back under health control. The service address is
NTP-role configuration and now lives there.

Validated with `keepalived -t` against keepalived 2.4.1 on edge-1 using the
config as it renders for that host; parses clean apart from the pre-existing
auth_pass truncation warning on the IPv4 instance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 19:59:40 +02:00
k.renner ea54e558fd Merge pull request 'fix(ntp): make the keepalived health check verify the clock, not liveness' (#467) from ntp-health-clock-correctness into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#467
2026-08-05 17:49:35 +00:00
k.renner d39115a55d Merge pull request 'docs(chrony): correct the root cause recorded for the makestep fix' (#466) from chrony-comment-correction into master
Reviewed-on: k.renner/homelab#466
2026-08-05 17:49:08 +00:00
claude-botandClaude Opus 5 601290dcd9 fix(ntp): make the keepalived health check verify the clock, not liveness
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The NTP health check asked "is chronyd alive", never "is this host safe to
take time from". Those came apart on 2026-08-04.

edge-1's CMOS battery is dead. After a power loss its clock came up ~16
days in the past and, with no makestep directive, chronyd never corrected
it. All three health checks passed continuously for ~18h:

  - chronyd was active
  - `chronyc tracking` responded
  - udp/123 was listening

and `local stratum 10` meant chronyd advertised itself to clients as a
valid reference the whole time. keepalived would have kept the VIP on a
host ready to serve a clock from July.

The sharpest part: check 2 ran `chronyc tracking >/dev/null 2>&1` and
discarded the output. The one command that knew the answer was executed
and its answer thrown away. During the incident that output read:

  System time     : 1376235.375000000 seconds slow of NTP time

This captures the output and adds two assertions:

  - offset above keepalived_ntp_max_offset (default 1.0s) fails, so a host
    that knows its clock is wrong drops the VIP
  - no upstream source selected in `chronyc sources` fails, because with
    `local stratum N` chronyd advertises itself as a usable reference even
    when it has never synchronised to anything -- upstream chrony docs are
    explicit that the local reference activates immediately in that case

Both predicates were validated against live output from edge-1 rather than
assumed: the offset comparison against the real incident value, and the
`chronyc sources` selected-source column against a real run.

Failing the check costs the configured weight (-20). When the WAN is down
every host fails together and loses the same weight, so relative priority
is unchanged and the VIP stays put -- the local-reference fallback still
works. What this prevents is one host with a broken clock winning.

Scope: this hardens the keepalived path, which is legacy. It does not
address the documented end state in plays/install-ntp-ha.yml, where edge-1
becomes the SOLE NTP server -- there, a failing check means no host serves
time at all, converting "serves the wrong time" into "serves no time".
That needs an architectural answer, not a better health check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 19:27:59 +02:00
claude-botandClaude Opus 5 32d12f93d8 docs(chrony): correct the root cause recorded for the makestep fix
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#465 landed the makestep directive with the wrong root cause in its
comment. It claimed edge-1 "rebooted with an RTC ~16 days in the past".
That is false, and the real cause matters for what to do next.

edge-1's CMOS battery is dead. After a power loss the RTC came up reading
2018-08-20 -- roughly eight years stale, not 16 days:

  PM: RTC time: 09:18:56, date: 2018-08-20
  rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC

The clock was then reconstructed entirely in software. systemd floored it
forward to its own build date:

  systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

and it settled at ~2026-07-19, about 16 days behind. So the 16 days is
incidental -- it is just where the flooring landed. There is no 16-day
mechanism, and nothing "drifted". The two real faults are a dead RTC
battery and a chrony config that never steps. #465 fixed the second.

Also collapses the duplicated rationale in chrony.conf.j2 to a pointer at
defaults/main.yml, as suggested in review on #465, so the explanation
lives in exactly one place.

No behaviour change: the makestep directive and its defaults are
untouched. Comments only.

The commit message on 7fcaa4b0 still carries the wrong explanation and is
left alone rather than rewriting merged history.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 17:35:51 +02:00
security-bot 0fd92f2fa2 chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.8.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-05 15:35:09 +00:00
k.renner d41f4870ad Merge pull request 'fix(chrony): step the clock at boot so a bad RTC self-corrects' (#465) from worktree-chrony-makestep into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#465
2026-08-05 15:14:21 +00:00
claude-botandClaude Opus 5 7fcaa4b035 fix(chrony): step the clock at boot so a bad RTC self-corrects
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
chrony.conf.j2 had no makestep directive, so chronyd only ever slewed.
A large offset is therefore never closed in practice.

On 2026-08-04 edge-1 rebooted with its RTC ~16 days in the past. chronyd
was synchronised to ptbtime1 at stratum 2 with a sub-millisecond RMS
offset, yet reported "System time: 1376235.375 seconds slow of NTP time"
and corrected none of it. With host time at 2026-07-20, every outbound
TLS handshake failed x509 NotBefore validation against a cluster
certificate issued 2026-07-23:

  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-07-20T16:15:36+02:00 is before
  2026-07-23T06:15:01Z

That took out the local mosquitto -> cluster MQTT bridge (all Vitocal
250-A / Vitocharge VX3 / E380 heating telemetry went unavailable in Home
Assistant for ~18h) and the Alloy -> Loki log shipper simultaneously.
Because the same skew broke log shipping, the outage was invisible to
log-based alerting.

Recovery required a manual `chronyc makestep`. Adding the directive lets
this correct itself at boot instead.

The limit keeps steps confined to the first few updates after chronyd
starts, so a long-running NTP server is never jumped out from under its
workloads mid-flight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:35:18 +02:00
k.renner 11bf13883f Merge pull request 'ci: validate the GitOps tree and pipeline configs on every PR' (#460) from ci-k8s-validate into master
ci/woodpecker/cron/update-all Pipeline failed
ci/woodpecker/cron/update-audit Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#460
2026-08-01 10:20:58 +00:00
security-bot 4b502daa98 chore(image-automation): bump woodpeckerci/woodpecker-server:v3.17.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-31 14:33:28 +00:00
security-bot 83a20bfa80 chore(image-automation): bump woodpeckerci/woodpecker-agent:v3.17.0
Updates managed by image-automation-controller.
2026-07-31 14:33:17 +00:00
security-bot c1a1a59ecc chore(image-automation): bump redis:8.10.0-alpine
ci/woodpecker/cron/e2e Pipeline was successful
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-30 21:32:22 +00:00
77 changed files with 1586 additions and 770 deletions
@@ -33,13 +33,21 @@ Build and boot the dedicated installer image — see
(partitions the internal disk — NVMe if present, else the eMMC; the
installer autodetects — installs the ansible-ready base system with SSH
keys, DHCP, docker — asks for confirmation before wiping).
3. Add the DHCP reservation/DNS entry for `edge-1.srv.renner.uno` on the
UDM, confirm `ssh root@edge-1.srv.renner.uno` works.
4. From the workstation:
3. Confirm the box resolves and is reachable by name — `ssh root@<host>` must
work before anything else. The UDM registers DHCP leases in DNS, so the
record appears on its own; the host is identified by name everywhere
(inventory, plays, this file) and its address is never recorded. Do not
pin it to a reservation.
4. Add the host to `[edge_host]` in `inventory/pve_hosts.yml`, by name.
5. From the workstation:
```
ansible-playbook -i inventory/pve_hosts.yml plays/common.yml -l edge_host
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
```
One unrestricted run over the whole group. The play is self-contained --
it carries `certificates` and `common` itself, so there is no second
command and no `--limit`. Running it converges every edge host, which is
the point: a run that touches only the new box tells you nothing about
the others.
## Secrets (before first play run)
@@ -85,7 +93,7 @@ zwave-js (order matters — never two servers on one radio):
`kubectl -n home-assistant exec deploy/zwave-js -- tar cf - -C /usr/src/app store | tar xf - -C /opt/edge-host/zwave-js/`
- [ ] Scale down in-cluster: `kubectl -n home-assistant scale deploy/zwave-js --replicas 0`
- [ ] Move the ZWA-2 stick to edge-1, `docker compose up -d zwave-js`
- [ ] Point HA `zwave_js` integration at `ws://edge-1.srv.renner.uno:3000`
- [ ] Point HA `zwave_js` integration at `ws://edge-1.iot.renner.uno:3000`
- [ ] Nodes alive in the UI on :8091 → remove the flux kustomization
`home-assistant-zwavejs` + PVC in a follow-up PR
@@ -1,6 +1,6 @@
---
# Secret indirection — real values live in vault.yml (ansible-vault) next
# to this file; see roles/edge-host/README.md for how to populate it from
# to this file; see docs/edge-host.md for how to populate it from
# the cluster secrets.
edge_sml2mqtt_mqtt_password: "{{ vault_edge_sml2mqtt_mqtt_password }}"
edge_open3e_mqtt_password: "{{ vault_edge_open3e_mqtt_password }}"
@@ -0,0 +1,18 @@
---
# Shared configuration for every NTP server (see plays/install-ntp-ha.yml).
# IPv6 service address -- what ntp.srv.renner.uno resolves to, and the path
# clients actually use. The IPv4 VIP is legacy.
#
# Defined at group level rather than in the play because install-ntp-ha.yml runs
# two plays over this group -- withdraw the static assignments fleet-wide, then
# hand the address to keepalived -- and both need the same value. Duplicating a
# service address across plays is exactly the sort of thing that drifts.
ntp_vip6: "fd40::123/128"
# Link-local address carried as the FIRST virtual_ipaddress of the IPv6 VRRP
# instance. RFC 5798 gives an IPv6 virtual router a link-local primary, and
# keepalived warns "the first IPv6 VIP address should be link local" without
# one. fe80::123 mirrors the service address and cannot collide with a real
# NIC's EUI-64 link-local.
ntp_vip6_linklocal: "fe80::123/64"
+15 -3
View File
@@ -1,7 +1,19 @@
# Reached by name, like every other host in this file. These three were the last
# entries still addressed by IPv4 literal; authentik, netbird, proxy, nas-1,
# grafana and edge-1 have always used their FQDN.
#
# An address in an inventory is allocator state standing in for identity, and it
# pins access to the legacy family -- the names resolve IPv6-first, which is the
# primary path on this network. DNS is the authoritative source for where a host
# is; this file should say which host, not where it currently sits.
#
# The short inventory_hostname is kept deliberately: plays key on it
# (ntp_servers[inventory_hostname] in install-ntp-ha.yml, node names in opentofu,
# the prometheus targets), so only ansible_host changes.
[pve_cluster]
jupiter ansible_host=10.2.40.201 ansible_user=root
neptune ansible_host=10.2.40.202 ansible_user=root
saturn ansible_host=10.2.40.203 ansible_user=root
jupiter ansible_host=jupiter.renner.uno ansible_user=root
neptune ansible_host=neptune.renner.uno ansible_user=root
saturn ansible_host=saturn.renner.uno ansible_user=root
[authentik_db]
authentik-db-1.srv.renner.uno ansible_user=root
+84 -8
View File
@@ -1,13 +1,89 @@
---
# Bare-metal Arch Linux edge host (Dell Wyse 3040, internal eMMC) — replaces
# the k8s-edge Talos VM
# (opentofu vm_153) as the home for hardware-bound services:
# - zwave-js-ui (ZWA-2 Z-Wave USB radio, docker compose)
# - sml2mqtt (FTDI power-meter reader, docker compose)
# - open3e (Innomaker USB2CAN -> Vitocal/VX3, native venv + systemd)
# Base install procedure and cutover checklist: ansible/roles/edge-host/README.md
# Bare-metal Arch edge hosts (Dell Wyse 3040 and similar) — the home for
# hardware-bound services, because Talos supports neither SocketCAN nor USB
# passthrough.
#
# One role per concern. Previously all of this lived in a single `edge-host`
# role, which meant a host could not take the flash-wear handling or the log
# shipper without also being handed the Z-Wave radio and the CAN bus. That is
# what blocked a second thin client from joining as an NTP peer.
#
# Self-contained: `certificates` and `common` are the parts of plays/common.yml
# that apply to an edge host, carried here so bootstrapping one is a single
# unrestricted run over the group. Not carried over: qemu-guest-agent (bare
# metal, not VMs) and monitoring-agent (Debian-oriented; these hosts ship logs
# via edge-log-shipper).
#
# Order is dependency order, not alphabetical:
# mqtt-bridge before its clients, so they have somewhere to publish
# edge-flash-wear before docker, so the journal is volatile before
# containers start logging into it
# socketcan before heating-can, which consumes can0
- name: Install Arch edge host
hosts: edge_host
# Connection is already root (ansible_user=root), so this is a root-to-root
# no-op today. Kept explicit, matching plays/common.yml, so the roles below
# do not silently depend on the connection user staying privileged.
become: true
# Defined on the play, not taken from a role's defaults: these are used by
# pre_tasks, which must not depend on role default resolution.
#
# heating-can, zwave-gateway and power-meter-sml each declare edge_base_dir
# in their own defaults too, so they stay usable standalone. Play vars win,
# so these are the effective values -- keep them in step if the path moves.
vars:
edge_base_dir: /opt/edge-host
edge_legacy_compose: "{{ edge_base_dir }}/docker-compose.yml"
# One-time migration off the shared compose project, BEFORE any role tries
# to start a container.
#
# zwave-js and sml2mqtt used to live in one project at edge_legacy_compose,
# both with a fixed container_name. Docker enforces container names globally,
# not per project, so while the old project still owns those names the new
# per-service projects cannot create them -- `docker compose up -d` fails with
# "Conflict: container name already in use", which is not the missing-USB-device
# error the roles tolerate, so the play would abort.
#
# It belongs here rather than in either role: it is a migration step for both
# of them, and putting it in one meant whichever role ran first hit the
# conflict. Guarded on the legacy file, so it runs once and then never again.
#
# check mode cannot verify this: command tasks are skipped, not simulated, so
# a --check run neither hits the conflict nor proves the teardown works.
pre_tasks:
- name: Retire the legacy shared compose project
ansible.builtin.command:
cmd: docker compose -f "{{ edge_legacy_compose }}" down
args:
removes: "{{ edge_legacy_compose }}"
failed_when: false
- name: Remove the legacy shared compose file
ansible.builtin.file:
path: "{{ edge_legacy_compose }}"
state: absent
- name: Remove the legacy zwave-js env from the shared project root
ansible.builtin.file:
path: "{{ edge_base_dir }}/zwavejs.env"
state: absent
roles:
- certificates
- common
# Host-generic. An edge host with no attached peripherals takes these and
# nothing below them.
- edge-flash-wear
- docker-install
- edge-host
- edge-log-shipper
- mqtt-bridge
- edge-vlan
# Hardware-bound. Each needs its own device present.
- socketcan
- heating-can
- zwave-gateway
- power-meter-sml
+88 -1
View File
@@ -22,12 +22,54 @@
# so it stays BACKUP while the PVE nodes exist and wins the VIP by default once
# they are gone -- no config change required at cutover.
- name: Install NTP Server (HA)
# Phase 1. Every static assignment of the IPv6 service address must be gone
# fleet-wide before ANY host claims it via VRRP.
#
# DAD is enabled on the VRRP address (deliberately -- under single-master VRRP a
# duplicate is a real fault worth detecting). So a static copy on one host and a
# VRRP-managed copy on another cannot coexist: the kernel marks the newcomer
# `dadfailed deprecated`, and its master then believes it is serving while
# carrying an unusable address. That is not hypothetical -- it is what edge-1
# did on 2026-08-07 when it took the address while three PVE nodes still held it
# statically.
#
# Deliberately NOT serial: this play must complete everywhere before phase 2
# starts anywhere. Doing the withdrawal per host inside phase 2 cannot avoid the
# overlap, because with serial: 1 the first host claims the address while the
# rest are still static.
#
# There is a gap between the phases where no host holds the address. It is
# bounded by one play's run time and clients simply retry.
- name: Withdraw static assignments of the NTP service address
hosts: ntp_cluster
become: true
vars:
keepalived_ntp_vip6: "{{ ntp_vip6 }}"
keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
tasks:
- name: Withdraw any static assignment of the service address
ansible.builtin.include_role:
name: keepalived-ntp
tasks_from: withdraw-static.yml
when: ntp_keepalived_enabled | default(false)
# Phase 2. Configure chrony and hand the address to keepalived.
- name: Install NTP Server (HA)
hosts: ntp_cluster
become: true
# One node at a time. The keepalived role withdraws the statically assigned
# IPv6 service address and restarts keepalived so the address is re-taken
# under VRRP control; doing that across every NTP server at once would leave
# the whole fleet with nobody holding fd40::123 until the elections settle.
# Serialised, the remaining members keep answering throughout.
serial: 1
vars:
ntp_vip: "10.2.40.4"
# ntp_vip6 / ntp_vip6_linklocal come from group_vars/ntp_cluster.yml, so
# both plays in this file see one definition.
ntp_servers:
jupiter:
priority: 100
@@ -49,6 +91,8 @@
when: ntp_keepalived_enabled | default(false)
vars:
keepalived_ntp_vip: "{{ ntp_vip }}"
keepalived_ntp_vip6: "{{ ntp_vip6 }}"
keepalived_ntp_vip6_linklocal: "{{ ntp_vip6_linklocal }}"
keepalived_ntp_state: "{{ ntp_servers[inventory_hostname].state }}"
keepalived_ntp_priority: "{{ ntp_servers[inventory_hostname].priority }}"
keepalived_ntp_interface: "{{ ntp_keepalived_interface }}"
@@ -90,6 +134,49 @@
{% endif %}
when: ntp_keepalived_enabled | default(false)
# The v6 address is the path clients actually use, so it gets the same
# check as the v4 VIP. Without this a broken v6 rollout -- the address
# never claimed, or -nodad not behaving as expected on a given kernel --
# would not surface here at all, only later by reading
# /var/log/keepalived-state.log by hand.
#
# `ip -6 addr show ... to <addr>` prints the address when present and
# nothing when absent, so emptiness is the test.
#
# Presence cannot itself distinguish a VRRP-held address from a static
# leftover -- they look identical. It is only meaningful because the
# keepalived-ntp role asserts, on both distros, that no static assignment
# survives: the play fails before reaching here if one does.
- name: Check if this server holds the IPv6 service address
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ ntp_keepalived_interface }}"
- to
- "{{ ntp_vip6 }}"
register: vip6_check
failed_when: false
changed_when: false
when: ntp_keepalived_enabled | default(false)
- name: Display IPv6 service address status
ansible.builtin.debug:
msg: >
{% if vip6_check.stdout | default('') | length > 0 %}
This server ({{ inventory_hostname }}) holds the IPv6 service address
{{ ntp_vip6 }} -- it is the v6 MASTER and is serving ntp.srv.renner.uno
{% else %}
This server ({{ inventory_hostname }}) does not hold {{ ntp_vip6 }}.
Expected on a v6 BACKUP. If NO host holds it, check_ntp is failing
everywhere -- inspect `chronyc tracking` and
/var/log/keepalived-ntp-health.log
{% endif %}
when: ntp_keepalived_enabled | default(false)
- name: Report hosts serving NTP without participating in the VIP
ansible.builtin.debug:
msg: >
@@ -1,9 +1,25 @@
---
# Chrony NTP Server Role Defaults
# Networks allowed to use this server as NTP source
# Networks allowed to use this server as NTP source.
#
# BOTH families are required. IPv6 is the primary address family here -- the
# service address ntp.srv.renner.uno resolves to fd40::123 -- so an IPv4-only
# allow list refuses every client that uses the documented name.
#
# The hosts carried `allow fd00::/8` for a long time without it ever being in
# this file. Applying the role on 2026-08-07 rendered the template over it and
# dropped the line on all four NTP servers at once, which took internal NTP
# down over IPv6 until it was added here. Same class of untracked host state as
# the anycast service address, and the same failure: config that exists only on
# the box survives right up until something re-renders it.
chrony_allowed_networks:
- "10.2.0.0/16"
- "fd00::/8"
# Link-local. A client on the same segment may source its query from fe80::,
# and without this chrony refuses it -- the request arrives and is dropped,
# which looks like the server being unreachable rather than a policy refusal.
- "fe80::/10"
# Primary NTP servers (PTB - Physikalisch-Technische Bundesanstalt, Stratum 1)
chrony_upstream_servers:
@@ -18,6 +34,26 @@ chrony_upstream_pools:
# Local stratum if disconnected from upstream
chrony_local_stratum: 10
# Step (rather than slew) the clock when the offset exceeds the threshold, but
# only for the first N updates after chronyd starts. Without a makestep
# directive chronyd only ever slews, so a large offset is never closed in
# practice.
#
# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
# that forward to its own built-in epoch (2026-06-26), and it ended up at
# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
# With host time before the cluster certificate's NotBefore, every outbound TLS
# handshake failed x509 validation, taking out the MQTT bridge (heating
# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
# is just where the software flooring landed; the fault is that nothing steps.
#
# Limiting to the first few updates keeps steps confined to boot, so a
# long-running server is never jumped out from under its workloads.
chrony_makestep_threshold: 1.0
chrony_makestep_limit: 3
# Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
# (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
# error per interface when asked for it. Overridden to false in
@@ -14,6 +14,11 @@ pool {{ pool }}
# Record the rate at which the system clock gains/losses time
driftfile /var/lib/chrony/chrony.drift
# Step the clock instead of slewing it when the offset is large, limited to the
# first few updates after start so only boot-time corrections jump. Rationale
# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
# Allow NTP client access from local networks
{% for network in chrony_allowed_networks %}
allow {{ network }}
+5 -2
View File
@@ -118,8 +118,11 @@ After the `kubernetes/apps/claude-workstation/` PR merges and Flux reconciles:
NS=apps
TOKEN=$(kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.token}' | base64 -d)
kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt
# server = the API endpoint reachable over the site-to-site route (a CP node or VIP):
kubectl config set-cluster homelab --server=https://10.2.40.201:6443 \
# server = the cluster's stable API endpoint (kubernetes/talos/patches/cluster.yaml).
# Use the name, not an address: it is what the API server certificate carries as a
# SAN, so an IP here fails TLS verification even when it is reachable. It must
# resolve from the workstation over the site-to-site route.
kubectl config set-cluster homelab --server=https://k8s.renner.uno:6443 \
--certificate-authority=/tmp/ca.crt --embed-certs=true --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-credentials claude-workstation --token="$TOKEN" --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-context homelab --cluster=homelab --user=claude-workstation --kubeconfig=/tmp/claude-ws.kubeconfig
@@ -0,0 +1,15 @@
---
# Flash-wear mitigation.
# --- Logging & eMMC wear mitigation ------------------------------------
# edge-1 runs on a small internal eMMC (Dell Wyse 3040, 7.3 GB), so logs are
# kept off local flash: containers + native services log to a RAM-only
# journal and Grafana Alloy ships it to Loki. TRIM, noatime and a weekly
# wear check round out the flash-longevity measures. See README "Logging &
# eMMC".
edge_journald_runtime_max: "128M" # cap the volatile (RAM) journal
# Container stdout -> journal, no json log files on the flash. The journal
# itself is volatile (see the tasks above), so container logs land in RAM and
# are shipped off the box by edge-log-shipper.
docker_log_driver: journald
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# edge-1 eMMC wear/health check. Managed by roles/edge-host.
# edge-1 eMMC wear/health check. Managed by roles/edge-flash-wear.
#
# Reads the JEDEC eMMC 5.x life-time and pre-EOL estimators exposed via sysfs
# and emits a warning-priority journal message when the flash leaves the
@@ -0,0 +1,17 @@
---
- name: Restart journald
ansible.builtin.service:
name: systemd-journald
state: restarted
# Handlers run in definition order, so this fires after 'Restart journald':
# once the journal is volatile, the on-disk store is dead weight on the flash.
- name: Purge persistent journal store
ansible.builtin.file:
path: /var/log/journal
state: absent
- name: Restart docker
ansible.builtin.service:
name: docker
state: restarted
@@ -0,0 +1,100 @@
---
# Flash-wear mitigation for hosts on small eMMC (Dell Wyse 3040 and similar).
#
# journald -> volatile (RAM) and size-capped, root mounted noatime+lazytime,
# periodic TRIM, and a weekly wear check. Shipping the volatile journal off the
# box is a separate concern -- see the edge-log-shipper role.
- name: Install mmc-utils
ansible.builtin.package:
name: mmc-utils # eMMC lifetime/EOL introspection (mmc extcsd read)
state: present
- name: Ensure journald drop-in directory exists
ansible.builtin.file:
path: /etc/systemd/journald.conf.d
state: directory
mode: "0755"
- name: Switch the systemd journal to volatile (RAM) storage
ansible.builtin.template:
src: journald-edge.conf.j2
dest: /etc/systemd/journald.conf.d/00-edge-emmc.conf
mode: "0644"
notify:
- Restart journald
- Purge persistent journal store
- name: Resolve root filesystem facts (for noatime)
ansible.builtin.set_fact:
edge_root_fs: "{{ ansible_facts.mounts | selectattr('mount', 'equalto', '/') | first }}"
- name: Refuse to touch fstab without a stable root UUID
ansible.builtin.assert:
that:
- edge_root_fs.uuid | default('') | length > 0
fail_msg: "Could not determine root UUID from facts; not rewriting fstab."
# lazytime batches inode-timestamp writeback (e.g. the zwave-js lockfile
# heartbeat that otherwise dirties the journal every couple seconds); noatime
# drops access-time writes. Neither weakens data durability.
- name: Mount root with noatime + lazytime (persisted in fstab)
ansible.posix.mount:
path: /
src: "UUID={{ edge_root_fs.uuid }}"
fstype: "{{ edge_root_fs.fstype }}"
opts: "{{ ((edge_root_fs.options.split(',') | reject('in', ['atime', 'noatime', 'relatime', 'diratime', 'lazytime', 'nolazytime']) | list) + ['noatime', 'lazytime']) | unique | join(',') }}"
dump: "0"
passno: "1" # root fs — keep boot-time fsck ordering
backup: true
state: mounted
- name: Enable periodic TRIM (helps eMMC wear-leveling)
ansible.builtin.systemd:
name: fstrim.timer
enabled: true
state: started
- name: Install eMMC health-check script
ansible.builtin.copy:
src: edge-emmc-health
dest: /usr/local/bin/edge-emmc-health
mode: "0755"
- name: Install eMMC health-check service + timer
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- edge-emmc-health.service
- edge-emmc-health.timer
- name: Enable eMMC health-check timer
ansible.builtin.systemd:
name: edge-emmc-health.timer
enabled: true
state: started
daemon_reload: true
# Container stdout into the journal instead of json files on disk. This lives
# here, with the rest of the flash-wear handling, rather than in docker-install:
# that role is shared by ten other plays (authentik, n8n, owncloud, grist,
# netbird, gatus, music-assistant, ical-proxy), none of which have a
# daemon.json today. Adding it there would have created one on the next run of
# any of those plays and restarted every container on those hosts.
#
# /etc/docker is created explicitly because this role runs before
# docker-install on a first bootstrap, so the directory may not exist yet. The
# handler runs at the end of the play, by which time docker is installed.
- name: Ensure the docker config directory exists
ansible.builtin.file:
path: /etc/docker
state: directory
mode: "0755"
- name: Route container logs to the journal (keeps json logs off the flash)
ansible.builtin.template:
src: docker-daemon.json.j2
dest: /etc/docker/daemon.json
mode: "0644"
notify: Restart docker
@@ -0,0 +1,3 @@
{
"log-driver": "{{ docker_log_driver }}"
}
@@ -1,4 +1,4 @@
# Managed by roles/edge-host. edge-1 boots off a small internal eMMC, so the
# Managed by roles/edge-flash-wear. edge-1 boots off a small internal eMMC, so the
# journal must not persist to flash: keep it in RAM (/run/log/journal) and cap
# its size. Grafana Alloy (edge-alloy.service) ships this journal to Loki
# before it rotates out of RAM — nothing durable lives here.
-118
View File
@@ -1,118 +0,0 @@
---
edge_base_dir: /opt/edge-host
edge_tz: Europe/Berlin
# --- Logging & eMMC wear mitigation ------------------------------------
# edge-1 runs on a small internal eMMC (Dell Wyse 3040, 7.3 GB), so logs are
# kept off local flash: containers + native services log to a RAM-only
# journal and Grafana Alloy ships it to Loki. TRIM, noatime and a weekly
# wear check round out the flash-longevity measures. See README "Logging &
# eMMC".
edge_journald_runtime_max: "128M" # cap the volatile (RAM) journal
edge_docker_log_driver: journald # container stdout -> journal, no json logs on eMMC
# Grafana Alloy log shipper. The srv-network Loki (loki.srv.renner.uno:3100,
# no auth) is firewalled off the IoT VLAN; the only endpoint edge-1 can reach
# is the authenticated cluster ingress (Traefik basic-auth).
edge_alloy_config_file: /etc/alloy/edge.alloy
edge_alloy_http_port: 12345
edge_alloy_bin: /usr/bin/grafana-alloy # Arch `grafana-alloy` pkg names the binary this, not `alloy`
edge_loki_push_url: "https://loki.renner.uno/loki/api/v1/push"
edge_loki_username: edge-1
edge_alloy_labels:
environment: production
datacenter: homelab
# --- MQTT: edge clients (open3e, e3oncan, sml2mqtt) publish to a LOCAL
# mosquitto on loopback, which bridges to the central cluster broker over TLS
# (open3e/e3oncan have no native MQTTS). This keeps plaintext on loopback only
# — off the untrusted IoT VLAN — and buffers telemetry in RAM during cluster
# blips. The local broker writes nothing to the eMMC (persistence off, logs to
# the volatile journal). See edge-mqtt-bridge.conf.j2 and README "MQTT bridge".
edge_mqtt_host: 127.0.0.1
edge_mqtt_port: 1883
# Upstream cluster broker the local bridge connects to (TLS; the public
# Let's Encrypt cert is validated against the system CA bundle).
edge_mqtt_cluster_host: mqtt.renner.uno
edge_mqtt_cluster_tls_port: 8883
edge_mqtt_bridge_cafile: /etc/ssl/certs/ca-certificates.crt
# --- USB devices (stable by-id symlinks). After physically moving the
# sticks to edge-1, verify with `ls -l /dev/serial/by-id/` — the by-id
# names travel with the device, not the host, so these should not change.
edge_zwave_device: /dev/serial/by-id/usb-Nabu_Casa_ZWA-2_80B54EE15720-if00
edge_stromzaehler_device: /dev/serial/by-id/usb-FTDI_FT230X_Basic_UART_DK0E1GP5-if00-port0
# --- Images. Versions mirror the k8s manifests at migration time.
# sml2mqtt stays hand-pinned (critical meter bridge — was deliberately
# excluded from image-automation in k8s, keep that property here).
edge_zwavejs_image: ghcr.io/zwave-js/zwave-js-ui:11.21.1
edge_sml2mqtt_image: spacemanspiff2007/sml2mqtt:3.6.0
# Alpine version the zwave-js-ui image is built on — the source for the musl
# libeatmydata.so preloaded into that container (see tasks "Extract musl
# libeatmydata.so"). Keep in sync with the base if edge_zwavejs_image bumps
# its Alpine major (check: docker exec zwave-js cat /etc/os-release).
edge_zwave_alpine_base: alpine:3.22
# --- CAN / open3e (Viessmann E3 external bus) ---
edge_can_interface: can0
edge_can_bitrate: 250000
edge_open3e_pkg: git+https://github.com/open3e/open3e.git
edge_open3e_mqtt_user: open3e
edge_open3e_mqtt_topic: open3e
edge_open3e_mqtt_format: "{device}_{didNumber:04d}_{didName}"
edge_open3e_poll_seconds: 30
# Read-only datapoint set (Phase 1 extended set, established 2026-07-02 on
# the commissioning workstation; see vican repo CLAUDE.md for the rationale).
# NO -w / write DIDs here — live heating + battery, writes are Phase 3.
# vx3.266 = ErrorDtcHistory (O3EList) — the VX3 fault log; polled to catch
# recurrences of FailureCommunicationEnergyMeter (the E380 energy-meter link
# the internal PV-surplus EMS depends on). Decodes via the master O3EList
# codec inherited through the vx3plus overlay.
# vitocal.543/2350/2543/2560: SmartGrid-/EMS-Status-Beobachtung (2026-07-06)
# — klärt, ob die Eigenstromoptimierung (EMS) bei PV-Überschuss von selbst
# den WW-Boost auslöst (2543.2 steht laut Scan bereits auf +10 K), bevor
# SG-Ready-Verdrahtung oder andere Trigger gebaut werden. Alle vier ro.
# vitocal.259/260 = InfoDtcList/InfoDtcHistory (O3EList) — the HPMU event log,
# the Vitocal-side counterpart to vx3.266. Both are capped lists (2 and 9
# entries when added), so the multi-frame read is cheap at a 30 s poll.
# Records faults/info events only (NormalOperation, LagDeviceDtcReported,
# LegionellaProtectionTemperatureReached) — NOT setpoint or program changes.
edge_open3e_datapoints: >-
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791
# ECU addresses; -dev selects codec set, tx selects the UDS address.
# vcmu (refrigerant-circuit controller) has no dedicated codec set —
# omitting dpList makes open3e fall back to the generic datapoint list,
# which decodes all polled VCMU DIDs.
edge_open3e_devices:
vitocal: { tx: "0x680", dpList: "vcal" }
# dpList als Pfad: Overlay-Datei mit den Batteriemodul-DIDs (siehe
# files/open3e-vx3plus.py) — die Stock-vx3-Liste filtert sie sonst weg.
vx3: { tx: "0x6a1", dpList: "/opt/edge-host/open3e/vx3plus.py" }
vcmu: { tx: "0x68c" }
# --- E3onCAN: passive E380 energy-meter listener (no bus writes, so it
# coexists with the single open3e instance). The meter broadcasts at
# ~5 Hz; the gap throttles MQTT to one update per DID per interval.
edge_e3oncan_repo: https://github.com/MyHomeMyData/E3onCAN.git
edge_e3oncan_version: 461dcbdf97e8d72d2edb9929c63b39382af6c3a1 # V0.5.2
edge_e3oncan_gap_seconds: 10
edge_e3oncan_mqtt_format: "{device}_{didNumber:04d}_{didName}"
# --- Secrets: define these in inventory/group_vars/edge_host/vault.yml
# (ansible-vault). See README.md for the kubectl one-liners that read the
# current values out of the cluster during migration.
# edge_sml2mqtt_mqtt_password: "{{ vault_edge_sml2mqtt_mqtt_password }}"
# edge_open3e_mqtt_password: "{{ vault_edge_open3e_mqtt_password }}"
# edge_zwavejs_session_secret: "{{ vault_edge_zwavejs_session_secret }}"
# --- Tagged VLAN 40 sub-interface. Off by default; enabled in
# inventory/group_vars/edge_host. Exists so edge-1 can join the NTP keepalived
# VRRP group, which needs layer-2 adjacency with the other members on the
# service VLAN. The untagged VLAN 30 lease stays this host's only default
# route -- see templates/10-vlan40.network.j2.
edge_vlan40_enabled: false
edge_vlan40_parent: enp1s0
edge_vlan40_id: 40
edge_vlan40_name: enp1s0.40
edge_vlan40_parent_network: 20-wired.network
-66
View File
@@ -1,66 +0,0 @@
---
# reload alone only re-reads config files; existing links keep their old
# state until reconfigured.
- name: Reload networkd
ansible.builtin.shell:
cmd: networkctl reload && networkctl reconfigure {{ edge_can_interface }}
changed_when: true
# Reconfiguring the parent is what actually materialises the VLAN child -- a
# bare reload leaves the existing link as-is (see above). This re-runs DHCP on
# enp1s0, which is this headless host's only path to the network, so it is the
# one genuinely risky step in the VLAN change.
- name: Reload networkd for VLAN
ansible.builtin.shell:
cmd: networkctl reload && networkctl reconfigure {{ edge_vlan40_parent }}
changed_when: true
# Ordered before 'Restart edge compose services': the daemon must pick up the
# new default log driver before the containers are recreated onto it.
- name: Restart docker
ansible.builtin.service:
name: docker
state: restarted
- name: Restart journald
ansible.builtin.service:
name: systemd-journald
state: restarted
# Handlers run in definition order, so this fires after 'Restart journald':
# once the journal is volatile, the on-disk store is dead weight on the eMMC.
- name: Purge persistent journal store
ansible.builtin.file:
path: /var/log/journal
state: absent
- name: Restart alloy
ansible.builtin.systemd:
name: edge-alloy
state: restarted
daemon_reload: true
# Ordered before the MQTT clients (compose/open3e/e3oncan) so they reconnect
# to a freshly-configured local broker.
- name: Restart mosquitto
ansible.builtin.service:
name: mosquitto
state: restarted
- name: Restart edge compose services
ansible.builtin.command:
cmd: docker compose up -d --force-recreate
chdir: "{{ edge_base_dir }}"
changed_when: true
- name: Restart open3e
ansible.builtin.systemd:
name: open3e
state: restarted
daemon_reload: true
- name: Restart e3oncan
ansible.builtin.systemd:
name: e3oncan
state: restarted
daemon_reload: true
-440
View File
@@ -1,440 +0,0 @@
---
# Arch edge host: zwave-js-ui + sml2mqtt as docker compose services,
# open3e natively (venv + systemd) because it needs a SocketCAN interface,
# which is host network state, not something to hide in a container.
# Docker itself comes from the docker-install role (Arch-aware).
- name: Install base packages
ansible.builtin.package:
name:
- python
- python-packaging # ansible's pip module imports it on the target
- git
- mmc-utils # eMMC lifetime/EOL introspection (mmc extcsd read)
- grafana-alloy # log shipper -> Loki (see Alloy section below)
- mosquitto # local MQTT bridge broker -> cluster over TLS
state: present
# --- Local MQTT bridge broker. Edge clients publish to this over loopback;
# it bridges to the cluster broker over TLS (open3e/e3oncan can't do MQTTS).
# Configured with persistence off + stdout logging so it writes nothing to the
# eMMC. Deployed before the clients so they have somewhere to connect.
# See edge-mqtt-bridge.conf.j2 / README "MQTT bridge".
- name: Deploy local mosquitto bridge config
ansible.builtin.template:
src: edge-mqtt-bridge.conf.j2
dest: /etc/mosquitto/mosquitto.conf
owner: root
group: mosquitto
mode: "0640" # carries the cluster bridge credentials
notify: Restart mosquitto
- name: Enable local mosquitto bridge
ansible.builtin.systemd:
name: mosquitto
enabled: true
state: started
# ======================================================================
# eMMC wear mitigation + log shipping. edge-1 (Dell Wyse 3040) runs on a
# small internal eMMC, so logs must not churn local flash. Strategy:
# - journald -> volatile (RAM), size-capped (journald-edge.conf)
# - docker -> journald log driver, no json files (docker-daemon.json)
# - Grafana Alloy tails the RAM journal -> Loki, WAL on tmpfs
# plus TRIM, noatime and a weekly wear check for flash longevity.
# See roles/edge-host/README.md "Logging & eMMC".
# ======================================================================
- name: Ensure journald drop-in directory exists
ansible.builtin.file:
path: /etc/systemd/journald.conf.d
state: directory
mode: "0755"
- name: Switch the systemd journal to volatile (RAM) storage
ansible.builtin.template:
src: journald-edge.conf.j2
dest: /etc/systemd/journald.conf.d/00-edge-emmc.conf
mode: "0644"
notify:
- Restart journald
- Purge persistent journal store
- name: Resolve root filesystem facts (for noatime)
ansible.builtin.set_fact:
edge_root_fs: "{{ ansible_facts.mounts | selectattr('mount', 'equalto', '/') | first }}"
- name: Refuse to touch fstab without a stable root UUID
ansible.builtin.assert:
that:
- edge_root_fs.uuid | default('') | length > 0
fail_msg: "Could not determine root UUID from facts; not rewriting fstab."
# lazytime batches inode-timestamp writeback (e.g. the zwave-js lockfile
# heartbeat that otherwise dirties the journal every couple seconds); noatime
# drops access-time writes. Neither weakens data durability.
- name: Mount root with noatime + lazytime (persisted in fstab)
ansible.posix.mount:
path: /
src: "UUID={{ edge_root_fs.uuid }}"
fstype: "{{ edge_root_fs.fstype }}"
opts: "{{ ((edge_root_fs.options.split(',') | reject('in', ['atime', 'noatime', 'relatime', 'diratime', 'lazytime', 'nolazytime']) | list) + ['noatime', 'lazytime']) | unique | join(',') }}"
dump: "0"
passno: "1" # root fs — keep boot-time fsck ordering
backup: true
state: mounted
- name: Enable periodic TRIM (helps eMMC wear-leveling)
ansible.builtin.systemd:
name: fstrim.timer
enabled: true
state: started
- name: Install eMMC health-check script
ansible.builtin.copy:
src: edge-emmc-health
dest: /usr/local/bin/edge-emmc-health
mode: "0755"
- name: Install eMMC health-check service + timer
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- edge-emmc-health.service
- edge-emmc-health.timer
- name: Enable eMMC health-check timer
ansible.builtin.systemd:
name: edge-emmc-health.timer
enabled: true
state: started
daemon_reload: true
# --- Grafana Alloy: ship the volatile journal to Loki. Container logs reach
# the journal via docker's journald driver (configured below), native
# services log to the journal directly, so a single journal source covers
# everything. Storage/WAL sits on /run (tmpfs) — see edge-alloy.service.
- name: Create alloy system group
ansible.builtin.group:
name: alloy
system: true
- name: Create alloy system user
ansible.builtin.user:
name: alloy
group: alloy
groups: systemd-journal
system: true
create_home: false
shell: /usr/bin/nologin
- name: Create alloy config directory
ansible.builtin.file:
path: /etc/alloy
state: directory
owner: alloy
group: alloy
mode: "0750"
- name: Deploy alloy config (journal -> Loki)
ansible.builtin.template:
src: edge-alloy.alloy.j2
dest: "{{ edge_alloy_config_file }}"
owner: alloy
group: alloy
mode: "0600" # carries the Loki basic-auth password
notify: Restart alloy
- name: Deploy alloy systemd unit
ansible.builtin.template:
src: edge-alloy.service.j2
dest: /etc/systemd/system/edge-alloy.service
mode: "0644"
notify: Restart alloy
- name: Enable edge-alloy
ansible.builtin.systemd:
name: edge-alloy
enabled: true
state: started
daemon_reload: true
# --- SocketCAN: systemd-networkd owns can0. gs_usb is an in-tree module
# and autoloads when the Innomaker adapter enumerates; networkd then
# matches the interface and applies bitrate + up. RestartSec recovers the
# link after bus-off without manual intervention.
- name: Configure can0 via systemd-networkd
ansible.builtin.template:
src: 80-can0.network.j2
dest: /etc/systemd/network/80-can0.network
mode: "0644"
notify: Reload networkd
# --- Tagged VLAN 40. Additive by construction: the parent link keeps its own
# .network file untouched and gains the VLAN via a drop-in, so reverting is a
# matter of deleting these three paths. This host is headless and enp1s0 is its
# only route to the network, hence the caution.
- name: Create VLAN sub-interface netdev
ansible.builtin.template:
src: 10-vlan40.netdev.j2
dest: "/etc/systemd/network/10-{{ edge_vlan40_name }}.netdev"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Configure VLAN sub-interface addressing
ansible.builtin.template:
src: 10-vlan40.network.j2
dest: "/etc/systemd/network/10-{{ edge_vlan40_name }}.network"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Ensure parent network drop-in directory exists
ansible.builtin.file:
path: "/etc/systemd/network/{{ edge_vlan40_parent_network }}.d"
state: directory
mode: "0755"
when: edge_vlan40_enabled
tags: [vlan40]
- name: Attach VLAN to parent link via drop-in
ansible.builtin.template:
src: vlan40-parent-dropin.conf.j2
dest: "/etc/systemd/network/{{ edge_vlan40_parent_network }}.d/10-vlan40.conf"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Enable systemd-networkd
ansible.builtin.service:
name: systemd-networkd
state: started
enabled: true
# Docker networking needs no daemon.json (both services use network_mode:
# host — real addresses, v4+v6, no bridge/NAT). The daemon.json below exists
# only to route container stdout into the journald log driver so json logs
# never touch the eMMC; the journal itself is volatile (RAM) and shipped to
# Loki by edge-alloy. Both compose services inherit this default driver.
- name: Configure docker daemon (journald log driver, eMMC-friendly)
ansible.builtin.template:
src: docker-daemon.json.j2
dest: /etc/docker/daemon.json
mode: "0644"
notify:
- Restart docker
- Restart edge compose services
# --- Directory layout ---
- name: Create service directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- "{{ edge_base_dir }}"
- "{{ edge_base_dir }}/zwave-js/store"
- "{{ edge_base_dir }}/sml2mqtt"
# --- zwave-js eMMC write reduction: batch fsync via libeatmydata ---------
# zwave-js persists its value DB with an fsync on *every* value change
# (jsonl-db, throttleFS unset by zwave-js-ui). On a chatty network that
# fsync storm churns the eMMC (~2.6 GiB/day for ~0.1 KiB/s of real data).
# Preloading libeatmydata makes fsync a no-op so the kernel batches the
# writes via normal writeback — full HA data resolution and cache
# persistence are untouched (see docker-compose.yml.j2 for the mount + env).
# The .so must match the zwave-js-ui image's musl/Alpine base, so extract it
# from that exact Alpine version (bridge networking is broken on this host,
# hence --network=host). Refresh by deleting the .so if the base ever bumps
# its Alpine major version.
- name: Extract musl libeatmydata.so (matches the zwave-js-ui Alpine base)
ansible.builtin.shell:
cmd: |
set -euo pipefail
tmp="$(mktemp)"
trap 'rm -f "$tmp"' EXIT
docker run --rm --network=host {{ edge_zwave_alpine_base }} \
sh -c 'apk add --no-cache libeatmydata >/dev/null 2>&1 && cat /usr/lib/libeatmydata.so' > "$tmp"
test "$(stat -c%s "$tmp")" -ge 8000 # sanity: real .so is ~14 KB, guards against a failed/empty pull
install -m 0644 "$tmp" {{ edge_base_dir }}/zwave-js/libeatmydata.so
creates: "{{ edge_base_dir }}/zwave-js/libeatmydata.so"
notify: Restart edge compose services
# Because fsync is neutralized, a hard power-loss during the (rare) settings
# write could truncate settings.json — which holds the Z-Wave network keys.
# Keep a validated rolling backup on the eMMC so the keys stay recoverable.
- name: Install zwave-js settings.json backup script
ansible.builtin.template:
src: edge-zwave-settings-backup.j2
dest: /usr/local/bin/edge-zwave-settings-backup
mode: "0755"
- name: Install zwave-js settings backup service + timer
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- edge-zwave-settings-backup.service
- edge-zwave-settings-backup.timer
- name: Enable zwave-js settings backup timer
ansible.builtin.systemd:
name: edge-zwave-settings-backup.timer
enabled: true
state: started
daemon_reload: true
# --- Compose services: zwave-js-ui + sml2mqtt ---
- name: Deploy docker-compose.yml
ansible.builtin.template:
src: docker-compose.yml.j2
dest: "{{ edge_base_dir }}/docker-compose.yml"
mode: "0644"
notify: Restart edge compose services
- name: Deploy zwave-js env (session secret)
ansible.builtin.template:
src: zwavejs.env.j2
dest: "{{ edge_base_dir }}/zwavejs.env"
mode: "0600"
notify: Restart edge compose services
# sml2mqtt reads only a config file and has no env override for the
# password, so the rendered config carries it — 0600, root-owned, same
# trust boundary as the k8s init-container approach it replaces.
- name: Deploy sml2mqtt config
ansible.builtin.template:
src: sml2mqtt-config.yml.j2
dest: "{{ edge_base_dir }}/sml2mqtt/config.yml"
mode: "0600"
notify: Restart edge compose services
- name: Pull compose images
ansible.builtin.command:
cmd: docker compose pull
chdir: "{{ edge_base_dir }}"
changed_when: false
# Tolerates missing USB devices: before the physical cutover the ZWA-2 /
# FTDI sticks are still on the old k8s-edge VM, and a container whose
# device is absent must not block the rest of the role (open3e/CAN moves
# first). Re-run the play — or `docker compose up -d` — after each stick
# arrives.
- name: Start compose services
ansible.builtin.command:
cmd: docker compose up -d
chdir: "{{ edge_base_dir }}"
register: compose_up
failed_when: >-
compose_up.rc != 0
and 'error gathering device information' not in compose_up.stderr
changed_when: >-
'Started' in compose_up.stderr or 'Created' in compose_up.stderr
or 'Started' in compose_up.stdout or 'Creating' in compose_up.stdout
- name: Warn about containers waiting for their USB device
ansible.builtin.debug:
msg: >-
docker compose could not start every service (USB device not present
yet — expected before cutover): {{ compose_up.stderr_lines | select('search', 'device') | list }}
when: compose_up.rc != 0
# --- open3e: native install. CAN_RAW sockets need no capabilities, so
# the service runs as an unprivileged system user.
- name: Create open3e system user
ansible.builtin.user:
name: open3e
system: true
create_home: false
shell: /usr/bin/nologin
# Owned by the service user: udsoncan writes its log file into the
# working directory.
- name: Create open3e directory
ansible.builtin.file:
path: "{{ edge_base_dir }}/open3e"
state: directory
owner: open3e
group: open3e
mode: "0755"
- name: Install open3e into a venv
ansible.builtin.pip:
name: "{{ edge_open3e_pkg }}"
virtualenv: "{{ edge_base_dir }}/open3e/venv"
virtualenv_command: python -m venv
notify: Restart open3e
- name: Deploy open3e vx3 datapoint overlay (battery-module DIDs)
ansible.builtin.copy:
src: open3e-vx3plus.py
dest: "{{ edge_base_dir }}/open3e/vx3plus.py"
mode: "0644"
notify: Restart open3e
- name: Deploy open3e devices.json
ansible.builtin.copy:
content: "{{ edge_open3e_devices | to_nice_json }}\n"
dest: "{{ edge_base_dir }}/open3e/devices.json"
mode: "0644"
notify: Restart open3e
- name: Deploy open3e environment (broker credential)
ansible.builtin.template:
src: open3e.env.j2
dest: "{{ edge_base_dir }}/open3e/open3e.env"
owner: open3e
mode: "0600"
notify: Restart open3e
- name: Deploy open3e systemd unit
ansible.builtin.template:
src: open3e.service.j2
dest: /etc/systemd/system/open3e.service
mode: "0644"
notify: Restart open3e
- name: Enable open3e
ansible.builtin.systemd:
name: open3e
enabled: true
state: started
daemon_reload: true
# --- E3onCAN: direct E380 meter reads (passive; per-review decision not
# to trust the VX3's relayed meter values). Same broker user as open3e;
# topics land under the same open3e/ prefix as e380_<did>_<name>.
- name: Check out E3onCAN (pinned)
ansible.builtin.git:
repo: "{{ edge_e3oncan_repo }}"
dest: "{{ edge_base_dir }}/e3oncan/src"
version: "{{ edge_e3oncan_version }}"
notify: Restart e3oncan
- name: Create E3onCAN venv with requirements
ansible.builtin.pip:
requirements: "{{ edge_base_dir }}/e3oncan/src/requirements.txt"
virtualenv: "{{ edge_base_dir }}/e3oncan/venv"
virtualenv_command: python -m venv
notify: Restart e3oncan
- name: Deploy e3oncan systemd unit
ansible.builtin.template:
src: e3oncan.service.j2
dest: /etc/systemd/system/e3oncan.service
mode: "0644"
notify: Restart e3oncan
- name: Enable e3oncan
ansible.builtin.systemd:
name: e3oncan
enabled: true
state: started
daemon_reload: true
@@ -1,56 +0,0 @@
# Managed by ansible (roles/edge-host). Mirrors the former k8s manifests:
# kubernetes/apps/home-assistant-zwavejs/deployment.yaml and
# kubernetes/apps/home-assistant/sml2mqtt-deployment.yaml.
#
# No `privileged` needed here: docker's --device handles the device-cgroup
# grant that hostPath in k8s could not (that was the only reason the pods
# ran privileged).
# Both services use host networking: real host addresses (v4+v6), no
# bridge, no NAT. The broker LB is IPv6-first and docker bridge NAT66 is
# a tarpit (unmasqueraded SYNs die silently); the host stack just works.
services:
zwave-js:
image: {{ edge_zwavejs_image }}
container_name: zwave-js
restart: unless-stopped
stop_signal: SIGINT
tty: true
environment:
- TZ={{ edge_tz }}
# Batch the value-DB fsync storm to spare the eMMC (see role tasks /
# README "Logging & eMMC"). Neutralizes fsync only for this container;
# the WS/MQTT value stream to HA and cache persistence are unchanged.
- LD_PRELOAD=/usr/lib/libeatmydata.so
env_file:
- zwavejs.env
devices:
- "{{ edge_zwave_device }}:/dev/zwave"
volumes:
# musl libeatmydata.so extracted from the image's Alpine base (ro)
- ./zwave-js/libeatmydata.so:/usr/lib/libeatmydata.so:ro
# Persistent store: settings.json, device DB, network keys. Seeded
# from the k8s PVC `zwavejs-store` at cutover (see README). Since
# this host is stateful there is no init-container merge step —
# settings.json is edited via the UI and persists here; zwave.port
# stays /dev/zwave.
- ./zwave-js/store:/usr/src/app/store
# Host networking: binds 3000 (Z-Wave JS websocket for HA) and 8091
# (web UI) directly on the host.
network_mode: host
sml2mqtt:
image: {{ edge_sml2mqtt_image }}
container_name: sml2mqtt
restart: unless-stopped
# Outbound-only (serial in, MQTT out) — host networking sidesteps the
# docker NAT66 gap: the broker LB is IPv6-first and bridge-network v6
# SYNs leave unmasqueraded and die. The host stack reaches fd99::81
# fine (open3e proves it continuously).
network_mode: host
# Serial exclusivity: this is the ONLY reader of the meter port —
# HA's edl21 integration stays disabled (see sml2mqtt-config comment
# history in the k8s repo).
devices:
- "{{ edge_stromzaehler_device }}:/dev/stromzaehler"
volumes:
- ./sml2mqtt/config.yml:/sml2mqtt/config.yml:ro
@@ -1,3 +0,0 @@
{
"log-driver": "{{ edge_docker_log_driver }}"
}
@@ -0,0 +1,15 @@
---
# Journal -> Loki shipping.
#
# The srv-network Loki (loki.srv.renner.uno:3100, no auth) is firewalled off
# the IoT VLAN; the only endpoint these hosts can reach is the authenticated
# cluster ingress. edge_loki_password comes from the vault.
edge_alloy_config_file: /etc/alloy/edge.alloy
edge_alloy_http_port: 12345
edge_alloy_bin: /usr/bin/grafana-alloy # Arch `grafana-alloy` pkg names the binary this, not `alloy`
edge_loki_push_url: "https://loki.renner.uno/loki/api/v1/push"
edge_loki_username: edge-1
edge_alloy_labels:
environment: production
datacenter: homelab
@@ -0,0 +1,6 @@
---
- name: Restart alloy
ansible.builtin.systemd:
name: edge-alloy
state: restarted
daemon_reload: true
@@ -0,0 +1,57 @@
---
# Ship the systemd journal to Loki with Grafana Alloy.
#
# Pairs with edge-flash-wear, which makes the journal volatile: container logs
# reach the journal via docker s journald driver and native services log to it
# directly, so one journal source covers everything. Alloy s storage/WAL sits
# on /run (tmpfs) so it does not write to flash either.
- name: Install grafana-alloy
ansible.builtin.package:
name: grafana-alloy
state: present
- name: Create alloy system group
ansible.builtin.group:
name: alloy
system: true
- name: Create alloy system user
ansible.builtin.user:
name: alloy
group: alloy
groups: systemd-journal
system: true
create_home: false
shell: /usr/bin/nologin
- name: Create alloy config directory
ansible.builtin.file:
path: /etc/alloy
state: directory
owner: alloy
group: alloy
mode: "0750"
- name: Deploy alloy config (journal -> Loki)
ansible.builtin.template:
src: edge-alloy.alloy.j2
dest: "{{ edge_alloy_config_file }}"
owner: alloy
group: alloy
mode: "0600" # carries the Loki basic-auth password
notify: Restart alloy
- name: Deploy alloy systemd unit
ansible.builtin.template:
src: edge-alloy.service.j2
dest: /etc/systemd/system/edge-alloy.service
mode: "0644"
notify: Restart alloy
- name: Enable edge-alloy
ansible.builtin.systemd:
name: edge-alloy
enabled: true
state: started
daemon_reload: true
@@ -1,4 +1,4 @@
// Grafana Alloy — edge-1 log shipper. Managed by roles/edge-host.
// Grafana Alloy — edge-1 log shipper. Managed by roles/edge-log-shipper.
//
// edge-1 runs on internal eMMC, so logs are kept out of local flash: the
// systemd journal is volatile (RAM, see journald-edge.conf) and container
+11
View File
@@ -0,0 +1,11 @@
---
# --- Tagged VLAN 40 sub-interface. Off by default; enabled in
# inventory/group_vars/edge_host. Exists so edge-1 can join the NTP keepalived
# VRRP group, which needs layer-2 adjacency with the other members on the
# service VLAN. The untagged VLAN 30 lease stays this host's only default
# route -- see templates/10-vlan40.network.j2.
edge_vlan40_enabled: false
edge_vlan40_parent: enp1s0
edge_vlan40_id: 40
edge_vlan40_name: enp1s0.40
edge_vlan40_parent_network: 20-wired.network
@@ -0,0 +1,9 @@
---
# Reconfiguring the parent is what actually materialises the VLAN child -- a
# bare reload leaves the existing link as-is. This re-runs DHCP on the parent,
# which is a headless host's only path to the network, so it is the one
# genuinely risky step in the VLAN change.
- name: Reload networkd for VLAN
ansible.builtin.shell:
cmd: networkctl reload && networkctl reconfigure {{ edge_vlan40_parent }}
changed_when: true
+48
View File
@@ -0,0 +1,48 @@
---
# Tagged VLAN sub-interface via systemd-networkd.
#
# Additive by construction: the parent link keeps its own .network file and
# gains the VLAN through a drop-in, so reverting is deleting three paths. These
# hosts are headless and the parent is their only route to the network, hence
# the caution.
- name: Create VLAN sub-interface netdev
ansible.builtin.template:
src: 10-vlan40.netdev.j2
dest: "/etc/systemd/network/10-{{ edge_vlan40_name }}.netdev"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Configure VLAN sub-interface addressing
ansible.builtin.template:
src: 10-vlan40.network.j2
dest: "/etc/systemd/network/10-{{ edge_vlan40_name }}.network"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Ensure parent network drop-in directory exists
ansible.builtin.file:
path: "/etc/systemd/network/{{ edge_vlan40_parent_network }}.d"
state: directory
mode: "0755"
when: edge_vlan40_enabled
tags: [vlan40]
- name: Attach VLAN to parent link via drop-in
ansible.builtin.template:
src: vlan40-parent-dropin.conf.j2
dest: "/etc/systemd/network/{{ edge_vlan40_parent_network }}.d/10-vlan40.conf"
mode: "0644"
when: edge_vlan40_enabled
notify: Reload networkd for VLAN
tags: [vlan40]
- name: Enable systemd-networkd
ansible.builtin.service:
name: systemd-networkd
state: started
enabled: true
@@ -0,0 +1,52 @@
---
# Heating system over the Viessmann E3 CAN bus: open3e (heat pump) and
# e3oncan (E380 energy meter). The can0 interface itself is the socketcan
# role.
# Service root. Kept per-role so the role is usable on its own.
edge_base_dir: /opt/edge-host
edge_open3e_pkg: git+https://github.com/open3e/open3e.git
edge_open3e_mqtt_user: open3e
edge_open3e_mqtt_topic: open3e
edge_open3e_mqtt_format: "{device}_{didNumber:04d}_{didName}"
edge_open3e_poll_seconds: 30
# Read-only datapoint set (Phase 1 extended set, established 2026-07-02 on
# the commissioning workstation; see vican repo CLAUDE.md for the rationale).
# NO -w / write DIDs here — live heating + battery, writes are Phase 3.
# vx3.266 = ErrorDtcHistory (O3EList) — the VX3 fault log; polled to catch
# recurrences of FailureCommunicationEnergyMeter (the E380 energy-meter link
# the internal PV-surplus EMS depends on). Decodes via the master O3EList
# codec inherited through the vx3plus overlay.
# vitocal.543/2350/2543/2560: SmartGrid-/EMS-Status-Beobachtung (2026-07-06)
# — klärt, ob die Eigenstromoptimierung (EMS) bei PV-Überschuss von selbst
# den WW-Boost auslöst (2543.2 steht laut Scan bereits auf +10 K), bevor
# SG-Ready-Verdrahtung oder andere Trigger gebaut werden. Alle vier ro.
# vitocal.259/260 = InfoDtcList/InfoDtcHistory (O3EList) — the HPMU event log,
# the Vitocal-side counterpart to vx3.266. Both are capped lists (2 and 9
# entries when added), so the multi-frame read is cheap at a 30 s poll.
# Records faults/info events only (NormalOperation, LagDeviceDtcReported,
# LegionellaProtectionTemperatureReached) — NOT setpoint or program changes.
edge_open3e_datapoints: >-
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791
# ECU addresses; -dev selects codec set, tx selects the UDS address.
# vcmu (refrigerant-circuit controller) has no dedicated codec set —
# omitting dpList makes open3e fall back to the generic datapoint list,
# which decodes all polled VCMU DIDs.
edge_open3e_devices:
vitocal: { tx: "0x680", dpList: "vcal" }
# dpList als Pfad: Overlay-Datei mit den Batteriemodul-DIDs (siehe
# files/open3e-vx3plus.py) — die Stock-vx3-Liste filtert sie sonst weg.
vx3: { tx: "0x6a1", dpList: "/opt/edge-host/open3e/vx3plus.py" }
vcmu: { tx: "0x68c" }
# --- E3onCAN: passive E380 energy-meter listener (no bus writes, so it
# coexists with the single open3e instance). The meter broadcasts at
# ~5 Hz; the gap throttles MQTT to one update per DID per interval.
edge_e3oncan_repo: https://github.com/MyHomeMyData/E3onCAN.git
edge_e3oncan_version: 461dcbdf97e8d72d2edb9929c63b39382af6c3a1 # V0.5.2
edge_e3oncan_gap_seconds: 10
edge_e3oncan_mqtt_format: "{device}_{didNumber:04d}_{didName}"
# Secrets come from inventory/group_vars/edge_host/vault.yml:
# edge_open3e_mqtt_password
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host). Datapoint overlay for the VX3:
# Managed by ansible (roles/heating-can). Datapoint overlay for the VX3:
# the stock vx3 list acts as a FILTER (DIDs missing from it are popped
# from the master table, Open3Eclass.py), so the per-battery-module DIDs
# were never pollable with dpList "vx3". This overlay re-adds them with
@@ -0,0 +1,12 @@
---
- name: Restart open3e
ansible.builtin.systemd:
name: open3e
state: restarted
daemon_reload: true
- name: Restart e3oncan
ansible.builtin.systemd:
name: e3oncan
state: restarted
daemon_reload: true
+116
View File
@@ -0,0 +1,116 @@
---
# Heating system over the Viessmann E3 CAN bus.
#
# Two readers of the same heating installation:
# open3e UDS request/response to the Vitocal heat pump and Vitocharge VX3
# e3oncan passive CANopen listener for the E380 grid energy meter, which
# feeds the heat pump s own energy management
#
# They coexist because e3oncan issues no bus writes. Both are native installs
# (venv + systemd) rather than containers: they need a SocketCAN interface,
# which is host network state. CAN_RAW sockets need no capabilities, so they
# run as unprivileged system users.
#
# The can0 interface itself belongs to the socketcan role -- an interface is
# not the same concern as the services that consume it -- so this role must
# run after it.
- name: Install build/runtime packages for the CAN services
ansible.builtin.package:
name:
- python
- python-packaging
- git
state: present
- name: Create open3e system user
ansible.builtin.user:
name: open3e
system: true
create_home: false
shell: /usr/bin/nologin
# Owned by the service user: udsoncan writes its log file into the
# working directory.
- name: Create open3e directory
ansible.builtin.file:
path: "{{ edge_base_dir }}/open3e"
state: directory
owner: open3e
group: open3e
mode: "0755"
- name: Install open3e into a venv
ansible.builtin.pip:
name: "{{ edge_open3e_pkg }}"
virtualenv: "{{ edge_base_dir }}/open3e/venv"
virtualenv_command: python -m venv
notify: Restart open3e
- name: Deploy open3e vx3 datapoint overlay (battery-module DIDs)
ansible.builtin.copy:
src: open3e-vx3plus.py
dest: "{{ edge_base_dir }}/open3e/vx3plus.py"
mode: "0644"
notify: Restart open3e
- name: Deploy open3e devices.json
ansible.builtin.copy:
content: "{{ edge_open3e_devices | to_nice_json }}\n"
dest: "{{ edge_base_dir }}/open3e/devices.json"
mode: "0644"
notify: Restart open3e
- name: Deploy open3e environment (broker credential)
ansible.builtin.template:
src: open3e.env.j2
dest: "{{ edge_base_dir }}/open3e/open3e.env"
owner: open3e
mode: "0600"
notify: Restart open3e
- name: Deploy open3e systemd unit
ansible.builtin.template:
src: open3e.service.j2
dest: /etc/systemd/system/open3e.service
mode: "0644"
notify: Restart open3e
- name: Enable open3e
ansible.builtin.systemd:
name: open3e
enabled: true
state: started
daemon_reload: true
# --- E3onCAN: direct E380 meter reads (passive; per-review decision not
# to trust the VX3's relayed meter values). Same broker user as open3e;
# topics land under the same open3e/ prefix as e380_<did>_<name>.
- name: Check out E3onCAN (pinned)
ansible.builtin.git:
repo: "{{ edge_e3oncan_repo }}"
dest: "{{ edge_base_dir }}/e3oncan/src"
version: "{{ edge_e3oncan_version }}"
notify: Restart e3oncan
- name: Create E3onCAN venv with requirements
ansible.builtin.pip:
requirements: "{{ edge_base_dir }}/e3oncan/src/requirements.txt"
virtualenv: "{{ edge_base_dir }}/e3oncan/venv"
virtualenv_command: python -m venv
notify: Restart e3oncan
- name: Deploy e3oncan systemd unit
ansible.builtin.template:
src: e3oncan.service.j2
dest: /etc/systemd/system/e3oncan.service
mode: "0644"
notify: Restart e3oncan
- name: Enable e3oncan
ansible.builtin.systemd:
name: e3oncan
enabled: true
state: started
daemon_reload: true
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host).
# Managed by ansible (roles/heating-can).
# PASSIVE listener for the E380 energy meter's CANopen broadcasts — no
# writes on the bus, so it may run alongside the (single) open3e
# instance. Reads the meter directly instead of trusting the VX3's
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host). Read by open3e.service and
# Managed by ansible (roles/heating-can). Read by open3e.service and
# e3oncan.service.
# Full user:password pair as one variable so the open3e unit can pass it
# as a single -muser argument without exposing it in the unit file;
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host).
# Managed by ansible (roles/heating-can).
# Read-mostly listener — no -w flag, no service 0x77. The ONE sanctioned
# write path is the MQTT command topic (-l): HA publishes the DHW
# one-time-charge command there (scope decision 2026-07-02, see vican
@@ -4,6 +4,29 @@
keepalived_ntp_router_id: 4
keepalived_ntp_vip: "10.2.40.4"
keepalived_ntp_interface: "vmbr1.40"
# IPv6 service address for NTP (ntp.srv.renner.uno). Empty disables the v6
# instance entirely. IPv6 is the primary address family on this network, so
# this is the path that actually matters -- the v4 VIP above is legacy.
#
# Held by keepalived rather than assigned statically by networkd so that it is
# gated on check_ntp: present only while this host is fit to serve time, and
# withdrawn the moment it is not. A statically configured address is carried
# unconditionally from boot, which is how edge-1 came to answer NTP queries on
# 2026-08-04 while its clock was ~16 days in the past.
#
# Needs its own virtual_router_id: it is a distinct VRRP instance, and a VRID
# collision with the IPv4 instance on the same interface would have the two
# fighting over a single election.
keepalived_ntp_vip6: ""
keepalived_ntp_router_id6: 6
# Link-local address for the IPv6 VRRP instance, carried as its FIRST
# virtual_ipaddress. RFC 5798 gives an IPv6 virtual router a link-local primary,
# and keepalived warns "the first IPv6 VIP address should be link local" without
# one. Chosen to mirror the service address rather than derived from the virtual
# MAC, since no real NIC generates fe80::123 so there is nothing to collide with.
keepalived_ntp_vip6_linklocal: "fe80::123/64"
# NOTE: plaintext VRRP auth, unvaulted. Carried over as-is from the original
# role; VRRP PASS auth is weak regardless (sent in the clear on the segment),
# so this is a documented weakness rather than a secret worth vaulting.
@@ -18,6 +41,17 @@ keepalived_ntp_check_script: "/usr/local/bin/check-ntp-health.sh"
keepalived_ntp_check_interval: 2
keepalived_ntp_check_weight: -20
# Maximum |system clock offset| in seconds before the host refuses to hold the
# VIP. chronyd reports this as "System time : <n> seconds slow/fast of NTP
# time". Anything above this means the host knows its clock is wrong and must
# not serve time to clients -- see the reasoning in check-ntp-health.sh.j2.
#
# A serving NTP host should be within milliseconds of real time, so 1s is
# already several orders of magnitude beyond normal. It is deliberately not
# tighter: a brief excursion at boot, before makestep lands, should not flap
# the VIP (fall 2 at a 2s interval also requires the failure to persist).
keepalived_ntp_max_offset: 1.0
# sysctl applied for VRRP. Default preserves the PVE hosts' existing behaviour
# exactly. ip_forward is NOT appropriate everywhere -- on a host with legs in
# two VLANs it turns the box into a router between them, bypassing the gateway's
@@ -1,22 +1,39 @@
#!/bin/bash
# Keepalived State Change Notification Script
#
# Two VRRP instances call this: NTP_VIP (IPv4, legacy) passes MASTER/BACKUP/
# FAULT, and NTP_VIP6 (IPv6, the primary path) passes the same suffixed with
# '6'. They are logged distinctly because the two can legitimately diverge --
# a host can hold one family and not the other -- and a log line that does not
# say which family it means is useless during an incident.
TYPE=$1
NAME=$(hostname)
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
log() {
echo "$TIMESTAMP: $NAME $1" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME $1"
}
case $TYPE in
MASTER)
echo "$TIMESTAMP: $NAME transitioned to MASTER state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now NTP MASTER"
log "transitioned to MASTER state (NTP, IPv4)"
;;
BACKUP)
echo "$TIMESTAMP: $NAME transitioned to BACKUP state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now NTP BACKUP"
log "transitioned to BACKUP state (NTP, IPv4)"
;;
FAULT)
echo "$TIMESTAMP: $NAME transitioned to FAULT state (NTP)" >> /var/log/keepalived-state.log
logger -t keepalived "$NAME is now in NTP FAULT state"
log "transitioned to FAULT state (NTP, IPv4)"
;;
MASTER6)
log "transitioned to MASTER state (NTP, IPv6) - now serving ntp.srv.renner.uno"
;;
BACKUP6)
log "transitioned to BACKUP state (NTP, IPv6) - released ntp.srv.renner.uno"
;;
FAULT6)
log "transitioned to FAULT state (NTP, IPv6) - withdrew ntp.srv.renner.uno, unfit to serve time"
;;
*)
echo "$TIMESTAMP: $NAME unknown state: $TYPE" >> /var/log/keepalived-state.log
+115
View File
@@ -35,6 +35,121 @@
state: present
when: ansible_facts['os_family'] == 'Archlinux'
# --- Verify nothing else is assigning the IPv6 service address ---------------
#
# The removal itself lives in withdraw-static.yml, run as a separate fleet-wide
# play before any host reaches this role (see install-ntp-ha.yml). It has to
# complete everywhere first: DAD is enabled on the VRRP address, so a static
# copy on one host and a VRRP-managed copy on another cannot coexist -- the
# kernel marks the newcomer `dadfailed` and its master serves nothing.
#
# What remains here are the guards. They run on every execution, not just the
# migration, and fail the play if any static assignment survives -- which would
# both defeat the health gating (a static copy does not disappear when
# keepalived decides the host is unfit) and break the VRRP copy via DAD.
#
# The config file is what gets checked, not the running address, and that is
# what makes it safe to run every time: keepalived never writes to these files,
# so anything found is by definition static, whereas the live address is
# legitimately present on whichever host is master.
#
# Prefix-agnostic and \b-anchored on purpose. Matching the full
# address/prefixlen would fail open against a hand-written assignment using any
# other mask.
- name: Detect any remaining static assignment of the service address (ifupdown)
ansible.builtin.command:
argv:
- grep
- -rlE
- "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
- /etc/network/interfaces
- /etc/network/interfaces.d/
register: keepalived_ntp_static6_ifupdown
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
- name: Assert ifupdown is not also assigning the service address
ansible.builtin.assert:
that:
- keepalived_ntp_static6_ifupdown.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned statically by ifupdown on
this host
({{ keepalived_ntp_static6_ifupdown.stdout_lines | default([]) | join(', ') }}).
The automatic removal only matches the exact form
`up ip -6 addr replace {{ keepalived_ntp_vip6 }} dev <if> nodad`, so a
different prefix length or layout is left in place. Handing the address
to keepalived while ifupdown also assigns it means it would NOT be
withdrawn when check_ntp fails -- the host would keep serving time on an
address VRRP believes it has released. Remove the assignment from the
file(s) above, then re-run.
success_msg: >-
No static ifupdown assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
# On Arch (edge-1) the static assignment lives in a systemd-networkd file owned
# by the edge-vlan role, so this role must not rewrite it -- but it must not
# quietly proceed either. install-edge-host.yml is a separate play with no
# ordering relationship to this one, so nothing guarantees it has been re-run
# to drop the [Address] block. If it has not, edge-1 would end up holding
# fd40::123 statically AND running a VRRP instance for the same address:
# keepalived would believe it had withdrawn the address while networkd kept it
# assigned, and the host would go on answering NTP with a clock the health
# check had already rejected.
#
# That is the precise failure of 2026-08-04, on the host it happened to, so it
# fails fast rather than being left to inspection.
# Matched on the bare address, not the full address/prefixlen, and anchored
# with \b so it cannot also hit a longer address sharing the prefix (fd40::1234).
#
# Searching for the literal "fd40::123/128" would fail open: a hand-written
# [Address] using any other mask -- /64, say -- would not match, the assert
# below would pass, and keepalived would be handed an address networkd is
# still assigning. Since this check is detect-only, with no delete afterwards
# to catch the mistake, a miss here is silent. The Debian path does not share
# the problem: `ip -6 addr show ... to <prefix>` matches by range regardless of
# the assigned prefix length, and its delete is re-verified by assert.
- name: Detect a static assignment of the service address (systemd-networkd)
ansible.builtin.command:
argv:
- grep
- -rlE
- "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
- /etc/systemd/network/
register: keepalived_ntp_static6_networkd
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Archlinux'
- name: Assert networkd is not also assigning the service address
ansible.builtin.assert:
that:
- keepalived_ntp_static6_networkd.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned statically by
systemd-networkd on this host
({{ keepalived_ntp_static6_networkd.stdout_lines | default([]) | join(', ') }}).
Handing the address to keepalived while networkd also assigns it means
it would NOT be withdrawn when check_ntp fails -- the host would keep
serving time on an address VRRP believes it has released.
Re-apply the edge-vlan role first, whose template no longer carries the
[Address] block:
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
success_msg: >-
No static networkd assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Archlinux'
- name: Create keepalived configuration
ansible.builtin.template:
src: keepalived.conf.j2
@@ -0,0 +1,134 @@
---
# Withdraw a static assignment of the IPv6 NTP service address.
#
# Included by install-ntp-ha.yml as a FIRST play across the whole ntp_cluster,
# before any host is given the NTP_VIP6 instance. That ordering is the point of
# splitting it out.
#
# DAD is enabled on the VRRP address (see keepalived.conf.j2), so a static copy
# on any host and a VRRP-managed copy on another cannot coexist: the kernel sees
# the duplicate and marks the newcomer `dadfailed`, leaving a master that
# believes it is serving while carrying an unusable address. Observed on edge-1
# on 2026-08-07, where keepalived reported MASTER while the address sat
# `dadfailed deprecated tentative` because three PVE nodes still held it
# statically.
#
# Doing the withdrawal per host inside the main role could not avoid that: with
# serial: 1 the first host would claim the address via VRRP while the rest were
# still static. Fleet-wide first, then claim.
#
# fd40::123 was assigned by hand on all four servers, in two different ways,
# none of it in version control:
#
# PVE nodes an `up ip -6 addr replace ... nodad` hook in
# /etc/network/interfaces
# edge-1 an [Address] block in /etc/systemd/network/, owned by the
# edge-vlan role and removed by re-applying it -- not touched
# here, since this role must not rewrite another role's file
#
# There is a service gap between this play and the next, while no host holds
# the address. It is bounded by the run time of one play and clients simply
# retry; the alternative is the dadfailed state above, which does not resolve
# on its own.
- name: Remove static assignment of the NTP service address (ifupdown)
ansible.builtin.lineinfile:
path: /etc/network/interfaces
regexp: '^\s*up\s+ip\s+-6\s+addr\s+(replace|add)\s+{{ keepalived_ntp_vip6 | regex_escape }}\b'
state: absent
when:
- keepalived_ntp_vip6 | length > 0
- ansible_facts['os_family'] == 'Debian'
register: keepalived_ntp_static6_removed
# Everything below is gated on the config line having just been removed, so the
# runtime delete happens once -- on the migration -- and never again.
#
# The gate is load-bearing. Once this has landed, the address is legitimately
# present on whichever host is the elected v6 master, placed there by
# keepalived, and a delete cannot tell that apart from a static assignment (the
# address looks identical either way). Ungated, every later run would delete the
# master's address and restart keepalived, dropping the service address and
# forcing a re-election each time -- the flapping this whole change exists to
# prevent.
- name: Check whether the service address is still assigned statically
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ keepalived_ntp_interface }}"
- to
- "{{ keepalived_ntp_vip6 }}"
register: keepalived_ntp_vip6_present
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
# failed_when: false because the desired end state is "address absent". An
# `ip addr del` against an address that has already gone is the outcome this
# task wants, not a reason to abort the play for that host.
- name: Withdraw the statically assigned service address
ansible.builtin.command:
argv:
- ip
- -6
- addr
- del
- "{{ keepalived_ntp_vip6 }}"
- dev
- "{{ keepalived_ntp_interface }}"
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
- keepalived_ntp_vip6_present.stdout | default('') | length > 0
failed_when: false
# The withdrawal gets exactly one attempt -- the gate that makes it idempotent
# also means it never retries -- and the delete swallows its own errors. So the
# result is asserted rather than assumed.
#
# `ip addr del` matches on address AND prefix length: if a host's static
# assignment used a different mask or interface than ntp_vip6 /
# keepalived_ntp_interface, the delete is a silent no-op and the address stays
# outside VRRP control permanently, with the play reporting success.
- name: Confirm the static service address was actually withdrawn
ansible.builtin.command:
argv:
- ip
- -6
- addr
- show
- dev
- "{{ keepalived_ntp_interface }}"
- to
- "{{ keepalived_ntp_vip6 }}"
register: keepalived_ntp_vip6_after
changed_when: false
failed_when: false
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
- name: Assert the service address is no longer assigned outside keepalived
ansible.builtin.assert:
that:
- keepalived_ntp_vip6_after.stdout | default('') | length == 0
fail_msg: >-
{{ keepalived_ntp_vip6 }} is still assigned on
{{ keepalived_ntp_interface }} after the static assignment was removed.
`ip -6 addr del` matches address AND prefix length, so this usually means
the on-host assignment differs from ntp_vip6 or uses another interface.
Leaving it would both keep the address outside VRRP control and make the
VRRP-managed copy fail DAD elsewhere. Inspect with
`ip -6 addr show dev {{ keepalived_ntp_interface }}` and
`grep -rn {{ keepalived_ntp_vip6 }} /etc/network/ /etc/systemd/network/`.
success_msg: >-
{{ keepalived_ntp_vip6 }} withdrawn; safe for keepalived to claim it.
when:
- keepalived_ntp_vip6 | length > 0
- keepalived_ntp_static6_removed.changed | default(false)
@@ -6,23 +6,80 @@
# The chrony unit is named 'chrony' on Debian and 'chronyd' on Arch. This is
# templated rather than hardcoded: a wrong name here fails every check, which
# parks the host in permanent FAULT so it never takes the VIP.
#
# This check must answer "is it safe for clients to take time from this host",
# NOT merely "is chronyd alive". Those came apart on 2026-08-04: edge-1's dead
# CMOS battery left its clock ~16 days in the past, chronyd was running,
# answering queries and listening on 123, and the `local stratum` directive made
# it advertise itself to clients as a valid reference. Every liveness check
# passed for ~18h while the host stood ready to serve a clock from July. The
# earlier version of this script ran `chronyc tracking` and discarded its
# output -- the one command that knew the answer.
#
# Deliberate distinction, because these must not be conflated:
# * "provably wrong" -> chronyd HAS upstream measurements and they say the
# clock is far off. Fail: drop the VIP so a host with a
# sane clock serves instead.
# * "no upstream yet" -> nothing reachable / not yet measured. Also fail, but
# note that when the WAN is down every host fails
# together, they all lose the same weight, relative
# priority is unchanged and the VIP stays put. So the
# local-reference fallback still works; what this
# prevents is one host with a broken clock winning.
LOG=/var/log/keepalived-ntp-health.log
MAX_OFFSET={{ keepalived_ntp_max_offset }}
fail() {
echo "$(date): $1" >> "$LOG"
exit 1
}
# Check 1: Is chrony service running?
if ! systemctl is-active --quiet {{ chrony_service_name }}; then
echo "$(date): {{ chrony_service_name }} service not active" >> /var/log/keepalived-ntp-health.log
exit 1
fail "{{ chrony_service_name }} service not active"
fi
# Check 2: Is chrony synchronized or at least running?
if ! chronyc tracking >/dev/null 2>&1; then
echo "$(date): Chrony not responding to tracking query" >> /var/log/keepalived-ntp-health.log
exit 1
# Check 2: Is chrony responding, and what does it actually say?
# Captured, not discarded -- the offset below is the whole point of this script.
if ! TRACKING=$(chronyc tracking 2>/dev/null); then
fail "Chrony not responding to tracking query"
fi
# Check 3: Is NTP serving clients (port 123)?
if ! ss -lun | grep -q ':123 '; then
echo "$(date): NTP port 123 not listening" >> /var/log/keepalived-ntp-health.log
exit 1
fail "NTP port 123 not listening"
fi
# Check 4: Is the clock provably wrong?
#
# "System time : <n> seconds (slow|fast) of NTP time" is chronyd's own estimate
# of how far the system clock is from real time, derived from upstream
# measurements. On edge-1 during the incident this line read
# "1376235.375000000 seconds slow of NTP time" while every other check passed.
OFFSET=$(echo "$TRACKING" | awk '/^System time/ {print $4}')
if [ -n "$OFFSET" ]; then
if awk -v o="$OFFSET" -v m="$MAX_OFFSET" 'BEGIN {exit !(o > m)}'; then
fail "System clock off by ${OFFSET}s (max ${MAX_OFFSET}s) - refusing to serve time"
fi
fi
# Check 5: Has chronyd ever validated its clock against a real source?
#
# With `local stratum N` configured, chronyd advertises itself as a usable
# reference even when it has never synchronised to anything -- the upstream
# chrony documentation is explicit that the local reference activates
# immediately in that case. So a host that has never reached an upstream still
# looks authoritative to clients. Refuse the VIP until a real source has been
# selected.
#
# `chronyc sources` marks the selected source with '*' (best) or '+' (combined)
# in the second column. Neither present means nothing upstream is accepted.
if ! SOURCES=$(chronyc sources 2>/dev/null); then
fail "Chrony not responding to sources query"
fi
if ! echo "$SOURCES" | grep -qE '^.\*|^.\+'; then
fail "No upstream source selected - clock not validated, refusing to serve time"
fi
# All checks passed
@@ -39,3 +39,72 @@ vrrp_instance NTP_VIP {
notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP"
notify_fault "/usr/local/bin/keepalived-notify.sh FAULT"
}
{% if keepalived_ntp_vip6 %}
# IPv6 service address for NTP (ntp.srv.renner.uno). IPv6 is the primary
# address family here, so this instance -- not the IPv4 one above -- is the
# path that matters.
#
# A separate instance is mandatory, not stylistic: keepalived requires that
# "all addresses in virtual_ipaddress must be of the same family", so the v6
# address cannot be added to NTP_VIP.
#
# No authentication block: VRRPv3 removed authentication (RFC 5798), and
# keepalived always uses version 3 for IPv6 instances. Configuring auth here
# is rejected rather than ignored.
#
# The first address is link-local, and must be. RFC 5798 gives an IPv6 virtual
# router a link-local address as its primary, and keepalived warns
# "the first IPv6 VIP address should be link local" otherwise. It carries on,
# but the instance is then sourcing VRRPv3 from something the protocol does not
# expect.
#
# DAD is deliberately left ON (no -nodad). The flag was inherited from the
# previous design, where every NTP server carried fd40::123 simultaneously and
# duplicate detection had to be suppressed for that to work at all. Under
# single-master VRRP the opposite is true: exactly one host should hold the
# address, so a duplicate is a genuine fault and detecting it is the point.
#
# This does mean the migration cannot leave a static copy anywhere while a
# VRRP-managed one exists -- DAD sees the two and marks the newcomer
# `dadfailed`, leaving a master that believes it is serving while carrying an
# unusable address. Observed on edge-1 on 2026-08-07. install-ntp-ha.yml
# therefore withdraws every static assignment fleet-wide in a first play
# before any host configures this instance.
#
# track_script is what makes this health-gated: the address is only held while
# check_ntp passes, and is withdrawn as soon as it fails. On withdrawal
# keepalived sends unsolicited Neighbour Advertisements so client neighbour
# caches are updated immediately -- a statically assigned address cannot do
# this, and clients would keep sending to the dead holder until their own
# neighbour unreachability detection expired.
vrrp_instance NTP_VIP6 {
state {{ keepalived_ntp_state }}
interface {{ keepalived_ntp_interface }}
virtual_router_id {{ keepalived_ntp_router_id6 }}
priority {{ keepalived_ntp_priority }}
advert_int 1
virtual_ipaddress {
{{ keepalived_ntp_vip6_linklocal }} dev {{ keepalived_ntp_interface }}
{{ keepalived_ntp_vip6 }} dev {{ keepalived_ntp_interface }}
}
# weight 0 overrides the script's own weight, and the distinction decides
# whether this works at all. A non-zero weight only SUBTRACTS from priority
# on failure -- so the last remaining host, having no one to lose the
# election to, stays MASTER and keeps serving time with a clock it has
# already reported as wrong. Weight 0 instead transitions the instance to
# FAULT after <fall> failures, which releases the address unconditionally.
#
# That is the required behaviour here: a host unfit to serve time must stop
# answering, whether or not a replacement exists.
track_script {
check_ntp weight 0
}
notify_master "/usr/local/bin/keepalived-notify.sh MASTER6"
notify_backup "/usr/local/bin/keepalived-notify.sh BACKUP6"
notify_fault "/usr/local/bin/keepalived-notify.sh FAULT6"
}
{% endif %}
@@ -0,0 +1,14 @@
---
# --- MQTT: edge clients (open3e, e3oncan, sml2mqtt) publish to a LOCAL
# mosquitto on loopback, which bridges to the central cluster broker over TLS
# (open3e/e3oncan have no native MQTTS). This keeps plaintext on loopback only
# — off the untrusted IoT VLAN — and buffers telemetry in RAM during cluster
# blips. The local broker writes nothing to the eMMC (persistence off, logs to
# the volatile journal). See edge-mqtt-bridge.conf.j2 and README "MQTT bridge".
edge_mqtt_host: 127.0.0.1
edge_mqtt_port: 1883
# Upstream cluster broker the local bridge connects to (TLS; the public
# Let's Encrypt cert is validated against the system CA bundle).
edge_mqtt_cluster_host: mqtt.renner.uno
edge_mqtt_cluster_tls_port: 8883
edge_mqtt_bridge_cafile: /etc/ssl/certs/ca-certificates.crt
@@ -0,0 +1,7 @@
---
# The MQTT clients (zwave-js, sml2mqtt, open3e, e3oncan) reconnect on their
# own once the local broker is back.
- name: Restart mosquitto
ansible.builtin.service:
name: mosquitto
state: restarted
+27
View File
@@ -0,0 +1,27 @@
---
# Local MQTT bridge broker.
#
# Edge clients (open3e, e3oncan, sml2mqtt) publish here over loopback; this
# broker bridges to the cluster broker over TLS, because those clients have no
# native MQTTS. Persistence is off and logging goes to stdout so it writes
# nothing to local flash.
- name: Install mosquitto
ansible.builtin.package:
name: mosquitto
state: present
- name: Deploy local mosquitto bridge config
ansible.builtin.template:
src: edge-mqtt-bridge.conf.j2
dest: /etc/mosquitto/mosquitto.conf
owner: root
group: mosquitto
mode: "0640" # carries the cluster bridge credentials
notify: Restart mosquitto
- name: Enable local mosquitto bridge
ansible.builtin.systemd:
name: mosquitto
enabled: true
state: started
@@ -1,4 +1,4 @@
# Managed by roles/edge-host. Local mosquitto that the edge clients (open3e,
# Managed by roles/mqtt-bridge. Local mosquitto that the edge clients (open3e,
# e3oncan, sml2mqtt) publish to over loopback; it bridges to the central
# cluster broker over TLS. open3e/e3oncan have no native MQTTS, so this is how
# edge-1's MQTT gets off plaintext on the untrusted IoT VLAN, and it buffers
@@ -0,0 +1,23 @@
---
# Power-meter reader: sml2mqtt against the FTDI serial meter head.
# Service root. Kept per-role so the role is usable on its own.
edge_base_dir: /opt/edge-host
# Own compose project directory. Unchanged from the path used under the
# previous shared project.
sml_base_dir: "{{ edge_base_dir }}/sml2mqtt"
# The compose file that used to hold both this service and zwave-js. Removed
# once, so the fixed container_name is released for the new projects.
edge_legacy_compose: "{{ edge_base_dir }}/docker-compose.yml"
# Stable by-id symlink -- see the zwave-gateway role.
edge_stromzaehler_device: /dev/serial/by-id/usb-FTDI_FT230X_Basic_UART_DK0E1GP5-if00-port0
# Hand-pinned on purpose: critical meter bridge, deliberately excluded from
# image automation.
edge_sml2mqtt_image: spacemanspiff2007/sml2mqtt:3.6.0
# Secret from inventory/group_vars/edge_host/vault.yml:
# edge_sml2mqtt_mqtt_password
@@ -0,0 +1,6 @@
---
- name: Restart sml2mqtt
ansible.builtin.command:
cmd: docker compose up -d --force-recreate
chdir: "{{ sml_base_dir }}"
changed_when: true
@@ -0,0 +1,55 @@
---
# Power-meter reader: sml2mqtt against the FTDI serial meter head.
#
# Its own compose project. It shared a compose file with the Z-Wave gateway,
# which meant a change to either recreated both -- two unrelated services bound
# to different USB devices. Data paths are unchanged.
- name: Create sml2mqtt directory
ansible.builtin.file:
path: "{{ sml_base_dir }}"
state: directory
mode: "0755"
# sml2mqtt reads only a config file and has no env override for the password,
# so the rendered config carries it -- 0600, root-owned, same trust boundary as
# the k8s init-container approach it replaced.
- name: Deploy sml2mqtt config
ansible.builtin.template:
src: sml2mqtt-config.yml.j2
dest: "{{ sml_base_dir }}/config.yml"
mode: "0600"
notify: Restart sml2mqtt
- name: Deploy sml2mqtt compose project
ansible.builtin.template:
src: docker-compose.yml.j2
dest: "{{ sml_base_dir }}/docker-compose.yml"
mode: "0644"
notify: Restart sml2mqtt
- name: Pull sml2mqtt image
ansible.builtin.command:
cmd: docker compose pull
chdir: "{{ sml_base_dir }}"
changed_when: false
# Tolerates a missing USB device, as for the Z-Wave gateway.
- name: Start sml2mqtt
ansible.builtin.command:
cmd: docker compose up -d
chdir: "{{ sml_base_dir }}"
register: sml_compose_up
failed_when: >-
sml_compose_up.rc != 0
and 'error gathering device information' not in sml_compose_up.stderr
changed_when: >-
'Started' in sml_compose_up.stderr or 'Created' in sml_compose_up.stderr
or 'Started' in sml_compose_up.stdout or 'Creating' in sml_compose_up.stdout
- name: Warn if sml2mqtt is waiting for its USB device
ansible.builtin.debug:
msg: >-
sml2mqtt could not start -- the FTDI meter head is not present:
{{ sml_compose_up.stderr_lines | select('search', 'device') | list }}
when: sml_compose_up.rc != 0
@@ -0,0 +1,21 @@
# Managed by ansible (roles/power-meter-sml).
#
# Its own compose project -- see the role tasks for why it no longer shares a
# file with the Z-Wave gateway.
#
# Outbound-only (serial in, MQTT out). Host networking sidesteps the docker
# NAT66 gap: the broker LB is IPv6-first and bridge-network v6 SYNs leave
# unmasqueraded and die. The host stack reaches it fine (open3e proves it
# continuously).
services:
sml2mqtt:
image: {{ edge_sml2mqtt_image }}
container_name: sml2mqtt
restart: unless-stopped
network_mode: host
# Serial exclusivity: this is the ONLY reader of the meter port --
# HA's edl21 integration stays disabled.
devices:
- "{{ edge_stromzaehler_device }}:/dev/stromzaehler"
volumes:
- ./config.yml:/sml2mqtt/config.yml:ro
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host). Ported 1:1 from the k8s ConfigMap
# Managed by ansible (roles/power-meter-sml). Ported 1:1 from the k8s ConfigMap
# sml2mqtt-config; only host and password delivery differ (broker is now
# reached via its LoadBalancer hostname, password rendered by ansible
# instead of an init container).
@@ -7,19 +7,19 @@
# PVE cluster nodes - need direct scraping (no Alloy agent)
- targets:
- '10.2.40.201:9100'
- 'jupiter.renner.uno:9100'
labels:
job: 'node'
group: 'pve_cluster'
hostname: 'jupiter'
- targets:
- '10.2.40.202:9100'
- 'neptune.renner.uno:9100'
labels:
job: 'node'
group: 'pve_cluster'
hostname: 'neptune'
- targets:
- '10.2.40.203:9100'
- 'saturn.renner.uno:9100'
labels:
job: 'node'
group: 'pve_cluster'
@@ -51,9 +51,9 @@ scrape_configs:
scrape_timeout: 55s
static_configs:
- targets:
- '10.2.40.201' # jupiter
- '10.2.40.202' # neptune
- '10.2.40.203' # saturn
- 'jupiter.renner.uno'
- 'neptune.renner.uno'
- 'saturn.renner.uno'
metrics_path: /pve
params:
module: [default]
@@ -132,28 +132,29 @@ scrape_configs:
refresh_interval: 60s
# Chrony Exporter - NTP metrics from PVE cluster
# One static_config per host so each carries its own hostname label. The
# previous form listed all three targets together and recovered the hostname
# in relabel_configs by regex-matching __address__ against the node's IP --
# which silently coupled the labels to the addressing. Changing a target
# would have left the regex unmatched and the hostname label simply absent,
# with the scrape still succeeding.
- job_name: 'chrony'
static_configs:
- targets:
- '10.2.40.201:9123'
- '10.2.40.202:9123'
- '10.2.40.203:9123'
- targets: ['jupiter.renner.uno:9123']
labels:
group: 'ntp'
type: 'time-server'
relabel_configs:
- source_labels: [__address__]
regex: '10\.2\.40\.201:.*'
target_label: hostname
replacement: 'jupiter'
- source_labels: [__address__]
regex: '10\.2\.40\.202:.*'
target_label: hostname
replacement: 'neptune'
- source_labels: [__address__]
regex: '10\.2\.40\.203:.*'
target_label: hostname
replacement: 'saturn'
hostname: 'jupiter'
- targets: ['neptune.renner.uno:9123']
labels:
group: 'ntp'
type: 'time-server'
hostname: 'neptune'
- targets: ['saturn.renner.uno:9123']
labels:
group: 'ntp'
type: 'time-server'
hostname: 'saturn'
# Immich photo management metrics
- job_name: 'immich'
@@ -0,0 +1,5 @@
---
# SocketCAN interface. Consumed by heating-can.
# --- CAN / open3e (Viessmann E3 external bus) ---
edge_can_interface: can0
edge_can_bitrate: 250000
@@ -0,0 +1,7 @@
---
# reload alone only re-reads config files; existing links keep their old state
# until reconfigured.
- name: Reload networkd
ansible.builtin.shell:
cmd: networkctl reload && networkctl reconfigure {{ edge_can_interface }}
changed_when: true
+20
View File
@@ -0,0 +1,20 @@
---
# SocketCAN interface, owned by systemd-networkd.
#
# gs_usb is an in-tree module and autoloads when the adapter enumerates;
# networkd then matches the interface and applies bitrate + up. Separate from
# the services that consume the bus (heating-can, e380-can-meter) because the
# interface is host network state shared by both.
- name: Configure can0 via systemd-networkd
ansible.builtin.template:
src: 80-can0.network.j2
dest: /etc/systemd/network/80-can0.network
mode: "0644"
notify: Reload networkd
- name: Enable systemd-networkd
ansible.builtin.service:
name: systemd-networkd
state: started
enabled: true
@@ -1,4 +1,4 @@
# Managed by ansible (roles/edge-host).
# Managed by ansible (roles/socketcan).
# Viessmann E3 external CAN bus via Innomaker USB2CAN (gs_usb).
# The adapter is a stub tap on an already-terminated bus (120R jumper OFF).
[Match]
@@ -0,0 +1,24 @@
---
# Z-Wave gateway: zwave-js-ui against the ZWA-2 USB radio.
# Service root. Kept per-role so the role is usable on its own.
edge_base_dir: /opt/edge-host
edge_tz: Europe/Berlin
# Own compose project directory. Unchanged from the path used under the
# previous shared project, so the store is not migrated.
zwave_base_dir: "{{ edge_base_dir }}/zwave-js"
# Stable by-id symlink: the name travels with the device, not the host, so it
# survives moving the stick between machines.
edge_zwave_device: /dev/serial/by-id/usb-Nabu_Casa_ZWA-2_80B54EE15720-if00
edge_zwavejs_image: ghcr.io/zwave-js/zwave-js-ui:11.21.1
# Alpine version the zwave-js-ui image is built on -- the source for the musl
# libeatmydata.so preloaded into that container. Keep in sync if the image
# bumps its Alpine major (check: docker exec zwave-js cat /etc/os-release).
edge_zwave_alpine_base: alpine:3.22
# Secret from inventory/group_vars/edge_host/vault.yml:
# edge_zwavejs_session_secret
@@ -0,0 +1,6 @@
---
- name: Restart zwave-js
ansible.builtin.command:
cmd: docker compose up -d --force-recreate
chdir: "{{ zwave_base_dir }}"
changed_when: true
@@ -0,0 +1,95 @@
---
# Z-Wave gateway: zwave-js-ui against the ZWA-2 USB radio.
#
# Its own compose project, in its own directory, so the Z-Wave gateway and the
# power-meter reader are not coupled through a shared compose file. Data paths
# are unchanged -- the store stays at {{ zwave_base_dir }}/store -- so this is a
# project boundary change, not a data migration.
- name: Create zwave-js directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- "{{ zwave_base_dir }}"
- "{{ zwave_base_dir }}/store"
- name: Extract musl libeatmydata.so (matches the zwave-js-ui Alpine base)
ansible.builtin.shell:
cmd: |
set -euo pipefail
tmp="$(mktemp)"
trap 'rm -f "$tmp"' EXIT
docker run --rm --network=host {{ edge_zwave_alpine_base }} \
sh -c 'apk add --no-cache libeatmydata >/dev/null 2>&1 && cat /usr/lib/libeatmydata.so' > "$tmp"
test "$(stat -c%s "$tmp")" -ge 8000 # sanity: real .so is ~14 KB, guards against a failed/empty pull
install -m 0644 "$tmp" {{ zwave_base_dir }}/libeatmydata.so
creates: "{{ zwave_base_dir }}/libeatmydata.so"
notify: Restart zwave-js
# Because fsync is neutralized, a hard power-loss during the (rare) settings
# write could truncate settings.json — which holds the Z-Wave network keys.
# Keep a validated rolling backup on the eMMC so the keys stay recoverable.
- name: Install zwave-js settings.json backup script
ansible.builtin.template:
src: edge-zwave-settings-backup.j2
dest: /usr/local/bin/edge-zwave-settings-backup
mode: "0755"
- name: Install zwave-js settings backup service + timer
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- edge-zwave-settings-backup.service
- edge-zwave-settings-backup.timer
- name: Enable zwave-js settings backup timer
ansible.builtin.systemd:
name: edge-zwave-settings-backup.timer
enabled: true
state: started
daemon_reload: true
- name: Deploy zwave-js env (session secret)
ansible.builtin.template:
src: zwavejs.env.j2
dest: "{{ zwave_base_dir }}/zwavejs.env"
mode: "0600"
notify: Restart zwave-js
- name: Deploy zwave-js compose project
ansible.builtin.template:
src: docker-compose.yml.j2
dest: "{{ zwave_base_dir }}/docker-compose.yml"
mode: "0644"
notify: Restart zwave-js
- name: Pull zwave-js image
ansible.builtin.command:
cmd: docker compose pull
chdir: "{{ zwave_base_dir }}"
changed_when: false
# Tolerates a missing USB device: if the ZWA-2 is absent the container cannot
# start, and that must not fail the run for everything else on the host.
- name: Start zwave-js
ansible.builtin.command:
cmd: docker compose up -d
chdir: "{{ zwave_base_dir }}"
register: zwave_compose_up
failed_when: >-
zwave_compose_up.rc != 0
and 'error gathering device information' not in zwave_compose_up.stderr
changed_when: >-
'Started' in zwave_compose_up.stderr or 'Created' in zwave_compose_up.stderr
or 'Started' in zwave_compose_up.stdout or 'Creating' in zwave_compose_up.stdout
- name: Warn if zwave-js is waiting for its USB device
ansible.builtin.debug:
msg: >-
zwave-js could not start -- the ZWA-2 radio is not present:
{{ zwave_compose_up.stderr_lines | select('search', 'device') | list }}
when: zwave_compose_up.rc != 0
@@ -0,0 +1,39 @@
# Managed by ansible (roles/zwave-gateway).
#
# Its own compose project. zwave-js and the power-meter reader used to share
# one file under {{ edge_base_dir | default('/opt/edge-host') }}; they are
# unrelated services bound to different USB devices, so a change to one had no
# business recreating the other.
#
# No `privileged` needed: docker's --device handles the device-cgroup grant
# that hostPath in k8s could not (that was the only reason the pod ran
# privileged).
#
# Host networking: real host addresses (v4+v6), no bridge, no NAT. The broker
# LB is IPv6-first and docker bridge NAT66 is a tarpit (unmasqueraded SYNs die
# silently); the host stack just works.
services:
zwave-js:
image: {{ edge_zwavejs_image }}
container_name: zwave-js
restart: unless-stopped
stop_signal: SIGINT
tty: true
environment:
- TZ={{ edge_tz }}
# Batch the value-DB fsync storm to spare the eMMC (see the edge-flash-wear
# role). Neutralizes fsync only for this container; the WS/MQTT value
# stream to HA and cache persistence are unchanged.
- LD_PRELOAD=/usr/lib/libeatmydata.so
env_file:
- zwavejs.env
devices:
- "{{ edge_zwave_device }}:/dev/zwave"
volumes:
# musl libeatmydata.so extracted from the image's Alpine base (ro)
- ./libeatmydata.so:/usr/lib/libeatmydata.so:ro
# Persistent store: settings.json, device DB, network keys. Path is
# unchanged from the previous shared project, so nothing is migrated.
- ./store:/usr/src/app/store
# Binds 3000 (Z-Wave JS websocket for HA) and 8091 (web UI) on the host.
network_mode: host
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Rolling backup of zwave-js settings.json (holds the Z-Wave network keys).
# Managed by roles/edge-host. edge-1 runs zwave-js under libeatmydata (fsync
# Managed by roles/zwave-gateway. edge-1 runs zwave-js under libeatmydata (fsync
# neutralized to spare the eMMC), so a hard power-loss during the rare
# settings write could truncate settings.json. Keep a validated backup on the
# eMMC so the keys are always recoverable. Runs hourly via a systemd timer.
+17 -6
View File
@@ -1,7 +1,12 @@
# edge-host installer ISO
Bootable Arch installer image for `edge-1.srv.renner.uno` (the bare-metal
N100 that replaces the k8s-edge VM — see `ansible/roles/edge-host`).
Bootable Arch installer image for the bare-metal edge hosts — the machines
that replaced the k8s-edge VM, because Talos cannot do CAN or USB
passthrough. See `ansible/docs/edge-host.md`.
Not tied to one box or one model: the installer autodetects the target disk
(NVMe, else eMMC, else the first non-USB disk), which covers both the Wyse
3040 thin clients and larger hardware.
The image is the official archiso **releng** profile plus:
- `edge-install` in the live environment — partitions the NVMe
@@ -12,7 +17,7 @@ The image is the official archiso **releng** profile plus:
Division of labour: the ISO produces an **ansible-ready base system**;
all service configuration (zwave-js-ui, sml2mqtt, open3e/CAN) stays in
`ansible/roles/edge-host` as the single source of truth. Rebuilding the
`ansible/docs/edge-host.md` as the single source of truth. Rebuilding the
box is: boot ISO → `edge-install` → run the play.
## Build
@@ -38,10 +43,16 @@ archiso updates flow in automatically.
(NVMe, else eMMC — covers the Wyse 3040 — else first non-USB disk),
shows the target and requires typing the hostname before wiping
anything. Override with `edge-install /dev/sdY`.
4. Reboot, then from the workstation:
4. Reboot. Confirm the box is reachable by name — `ssh root@<host>` — before
going further; it is identified by name everywhere and its address is
never recorded. Then add it to `[edge_host]` in
`ansible/inventory/pve_hosts.yml` and, from the workstation:
```
cd ansible
ansible-playbook -i inventory/pve_hosts.yml plays/common.yml -l edge_host
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
```
5. Continue with the cutover checklist in `ansible/roles/edge-host/README.md`.
One run over the whole group, no `--limit`: the play carries
`certificates` and `common` itself, so there is no separate `common.yml`
step. Running it converges every edge host, which is the point — a run
that touches only the new box tells you nothing about the others.
5. Continue with the cutover checklist in `ansible/docs/edge-host.md`.
@@ -50,7 +50,7 @@ mount "$(part 2)" /mnt
mount --mkdir "$(part 1)" /mnt/boot
# Everything the edge-host ansible role expects to find; the role itself
# is applied over SSH after first boot (see ansible/roles/edge-host).
# is applied over SSH after first boot (see ansible/docs/edge-host.md).
pacstrap -K /mnt base linux linux-firmware intel-ucode openssh python git \
docker docker-compose
@@ -99,7 +99,13 @@ EOF
umount -R /mnt
echo
echo "${HOST} installed. Remove the USB stick, reboot, then from the"
echo "workstation: cd ansible &&"
echo " ansible-playbook -i inventory/pve_hosts.yml plays/common.yml -l edge_host"
echo " ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml"
echo "${HOST} installed. Remove the USB stick and reboot."
echo
echo "Then, from the workstation:"
echo " 1. confirm 'ssh root@${HOST}' works (it must be reachable by name)"
echo " 2. add ${HOST} to [edge_host] in ansible/inventory/pve_hosts.yml"
echo " 3. cd ansible && ansible-playbook -i inventory/pve_hosts.yml \\"
echo " plays/install-edge-host.yml"
echo
echo "One run, whole group, no --limit: the play carries certificates and"
echo "common itself, so there is no separate common.yml step."
@@ -38,7 +38,7 @@ spec:
# gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.7.4 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.8.0 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
@@ -52,7 +52,7 @@ spec:
# stdlib only; Flux keeps the tag in lockstep via the shared image
# policy) — no extra dependency.
- name: reconcile-publish-ip
image: ghcr.io/music-assistant/server:2.9.10 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
volumeMounts:
- name: data
@@ -62,7 +62,7 @@ spec:
readOnly: true
containers:
- name: music-assistant
image: ghcr.io/music-assistant/server:2.9.10 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
env:
- name: LOG_LEVEL
value: info
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
type: RuntimeDefault
containers:
- name: redis
image: redis:8.8.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
args:
- redis-server
- --save
+1 -1
View File
@@ -34,7 +34,7 @@ spec:
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
# track the same range/filter so they bump together on each release.
image: woodpeckerci/woodpecker-agent:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
env:
- name: WOODPECKER_SERVER
value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+1 -1
View File
@@ -40,7 +40,7 @@ spec:
# minor + patch auto-update within 3.x; only the 4.0 major is held for
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
# -bump hazard is gone). Stays in lockstep with the agent.
image: woodpeckerci/woodpecker-server:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
env:
- name: WOODPECKER_HOST
value: https://ci.renner.uno
+1 -1
View File
@@ -25,7 +25,7 @@ ACCESS_STAGING="$REPO_ROOT/kubernetes/access-staging"
NODES=(k8s-1 k8s-2 k8s-3)
# No dedicated workers since the k8s-edge VM was retired (2026-07-02) —
# hardware-bound services live on the bare-metal edge-1 Arch host
# (ansible/roles/edge-host).
# (ansible/docs/edge-host.md).
WORKERS=()
ALL_NODES=("${NODES[@]}" "${WORKERS[@]}")
DOMAIN="srv.renner.uno"