homelab/ansible/docs/edge-host.md

edge-host — bare-metal Arch thin client (replaces the k8s-edge Talos VM)

Runs the hardware-bound services that used to be pinned to the k8s-edge Talos node (opentofu vm_153_k8s_edge.tf), plus the new CAN bridge:

Service	How it runs	Hardware	Talks to
zwave-js-ui	docker compose	ZWA-2 Z-Wave USB radio	HA zwave_js via ws://:3000, UI :8091
sml2mqtt	docker compose	FTDI FT230X meter reader	Mosquitto (sml2mqtt user)
open3e	venv + systemd	Innomaker USB2CAN (gs_usb)	Mosquitto (open3e user)

Everything else that once lived on k8s-edge (HA core, matter-server, OTBR, music-assistant) already moved to the general node pool and stays in the cluster — they need VLAN L2 presence, not USB, so a single LAN NIC is sufficient.

The deployed unit is a Dell Wyse 3040 (Atom x5-Z8350, 2 GB RAM) on a ~7.3 GB internal eMMC (/dev/mmcblk0) — see "Logging & eMMC" for the flash-wear handling that hardware implies. The installer also supports an N100/NVMe box; it autodetects the target disk, so either works.

Rationale for leaving Kubernetes for the CAN part: the stock Talos kernel ships with CONFIG_CAN unset (siderolabs/talos#7686), so SocketCAN cannot work in any pod without a custom kernel image per Talos release.

Base install (once, from the edge-host installer ISO)

Build and boot the dedicated installer image — see images/edge-host-iso/README.md. Short version:

1. sudo pacman -S archiso && sudo images/edge-host-iso/build.sh
2. Write ISO to a USB stick, boot the target box from it, run edge-install (partitions the internal disk — NVMe if present, else the eMMC; the installer autodetects — installs the ansible-ready base system with SSH keys, DHCP, docker — asks for confirmation before wiping).
3. Confirm the box resolves and is reachable by name — ssh root@<host> must work before anything else. The UDM registers DHCP leases in DNS, so the record appears on its own; the host is identified by name everywhere (inventory, plays, this file) and its address is never recorded. Do not pin it to a reservation.
4. Add the host to [edge_host] in inventory/pve_hosts.yml, by name. That group is the base: every edge thin client belongs to it, and it assumes no peripherals.
5. From the workstation:

   ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
   

   One unrestricted run over the whole group. The play is self-contained -- it carries certificates and common itself, so there is no second command and no --limit. Running it converges every edge host, which is the point: a run that touches only the new box tells you nothing about the others.

Peripherals

Each attached device is its own inventory group and its own playbook. Group membership states which hardware a host physically has -- there are no flags and no when: on the roles, so moving a stick between boxes is a one-line inventory change.

device	group	playbook
Innomaker USB2CAN (Vitocal/VX3 + E380)	heating_can	plays/install-heating-can.yml
Nabu Casa ZWA-2 Z-Wave radio	zwave_gateway	plays/install-zwave-gateway.yml

Add the host to the matching group, then run that playbook. They depend on the base play having run: all three need the local MQTT bridge, and two of them need docker.

A host with nothing attached is simply absent from all three groups and needs none of these commands.

Secrets (before first play run)

Create inventory/group_vars/edge_host/vault.yml with ansible-vault; the current live values come straight out of the cluster:

# sml2mqtt broker password
kubectl -n home-assistant get secret mosquitto-sml2mqtt-credentials -o jsonpath='{.data.password}' | base64 -d
# open3e broker password
kubectl -n home-assistant get secret mosquitto-open3e-credentials -o jsonpath='{.data.password}' | base64 -d
# zwave-js session secret
kubectl -n home-assistant get secret zwave-js-session -o jsonpath='{.data.session-secret}' | base64 -d

# vault.yml
vault_edge_open3e_mqtt_password: "..."
vault_edge_zwavejs_session_secret: "..."
vault_edge_loki_password: "..."   # dedicated edge-1 Loki push cred (see "Logging & eMMC")

vault_edge_loki_password is a freshly-minted secret, not copied from the cluster: a dedicated edge-1 htpasswd user whose hash lives in the cluster secret kubernetes/apps/monitoring/loki-basic-auth.sops.yaml. To rotate, regenerate openssl rand -hex 24, update both the htpasswd line in that sops secret and this vault key, then re-run the play.

The Z-Wave network keys do NOT need separate handling — they live in settings.json inside the store that gets migrated below.

Cutover checklist (service downtime: minutes, do per-service)

Preparation:

• vault.yml populated (above), play runs green against edge-1
• CAN: Innomaker adapter connected to the VX3 external CAN port, ip -details link show can0 shows state ERROR-ACTIVE bitrate 250000, candump can0 -n 5 shows frames

zwave-js (order matters — never two servers on one radio):

• Copy the store out of the PVC while the pod still runs: kubectl -n home-assistant exec deploy/zwave-js -- tar cf - -C /usr/src/app store | tar xf - -C /opt/edge-host/zwave-js/
• Scale down in-cluster: kubectl -n home-assistant scale deploy/zwave-js --replicas 0
• Move the ZWA-2 stick to edge-1, docker compose up -d zwave-js
• Point HA zwave_js integration at ws://edge-1.iot.renner.uno:3000
• Nodes alive in the UI on :8091 → remove the flux kustomization home-assistant-zwavejs + PVC in a follow-up PR

sml2mqtt:

• Scale down in-cluster: kubectl -n home-assistant scale deploy/sml2mqtt --replicas 0
• Move the FTDI stick, docker compose up -d sml2mqtt
• HA meter entities update again (same entity_ids — config is a 1:1 port, Wh stays Wh) → remove sml2mqtt from the home-assistant kustomization in the follow-up PR

open3e:

• Stop any workstation listener (ONE instance per bus, hard rule)
• Move the USB2CAN adapter, systemctl start open3e
• Topics under open3e/# update on the broker

Afterwards:

• Retire VM 153: remove vm_153_k8s_edge.tf, talos/patches/k8s-edge.yaml, node from the cluster (kubectl delete node k8s-edge, tofu apply)
• Update wiki (k8s-edge-node-pattern note is then historical)

Day-2 notes

• Image bumps: edit edge_zwavejs_image / edge_sml2mqtt_image in defaults and re-run the play (no flux image-automation out here — deliberate for sml2mqtt, acceptable for zwave-js). If a zwave-js bump changes its Alpine major, also update edge_zwave_alpine_base and refresh the preloaded .so (see "Logging & eMMC" → zwave-js).
• open3e / E3onCAN updates: both are pinned to commits in roles/heating-can/defaults/main.yml, so re-running the play does NOT pick up upstream changes -- that is deliberate, these drive the heat pump. The weekly update-audit reports when either falls behind (see ci/agents/update-audit/prompt.md); bump the pin and re-run. Pinning to a tag is not an option for open3e: it publishes none.
• Ad-hoc CAN scans: systemctl stop open3e first (single-instance rule), restart afterwards.
• can-utils (candump) is AUR-only on Arch — install manually via pamac build can-utils if needed for debugging; the role deliberately doesn't manage AUR packages.

Logging & eMMC

edge-1 boots off a small internal eMMC (Dell Wyse 3040, ~7.3 GB), so the role keeps logs off local flash and monitors wear:

• journald is volatile (Storage=volatile, journald-edge.conf) — the journal lives in /run/log/journal (RAM), capped at edge_journald_runtime_max. The old persistent store /var/log/journal is purged on first apply.
• Docker logs to journald (docker-daemon.json, log-driver: journald) instead of unbounded json files under /var/lib/docker/containers. Both compose services inherit it; docker logs still works.
• Grafana Alloy ships the journal to Loki (edge-alloy.service). Its WAL/positions sit on /run/alloy (tmpfs), so the agent writes nothing to the eMMC either. Net effect: no log writes hit local flash — the trade-off is that logs are RAM-only and lost on reboot unless already shipped.
• Loki endpoint: the old srv-network Loki (loki.srv.renner.uno:3100) no longer exists — Loki moved into the cluster, so Alloy pushes to the authenticated cluster ingress https://loki.renner.uno/loki/api/v1/push with basic-auth (edge_loki_username / vault_edge_loki_password). Delivery only works once the edge-1 htpasswd line has reconciled into the cluster (Flux) — before that Alloy gets 401 and buffers in RAM.
• Flash longevity: noatime + lazytime on root, weekly fstrim.timer, and a weekly edge-emmc-health.timer that logs the JEDEC life-time / pre-EOL estimators and warns (journal priority) once the flash leaves the healthy band. Check by hand any time with /usr/local/bin/edge-emmc-health or cat /sys/block/mmcblk0/device/life_time (0x01 0x01 = <10 % consumed). mmc extcsd read /dev/mmcblk0 (from mmc-utils) gives the full detail.

zwave-js value-DB writes (the dominant residual writer)

With logs off flash, the biggest remaining eMMC writer is zwave-js-ui: its value database (store/*.jsonl) is fsync'd on every value change (jsonl-db, and zwave-js-ui doesn't expose the driver's throttle). On a busy network that fsync storm churned ~2.6 GiB/day for ~0.1 KiB/s of actual new data — near-pure write amplification (each fsync forces a block write + ext4 journal commit).

• Fix: the zwave-js container preloads libeatmydata (LD_PRELOAD=/usr/lib/libeatmydata.so, mounted read-only from zwave-js/libeatmydata.so). This makes fsync/fdatasync no-ops for that one process, so the kernel batches the writes via normal writeback (~30 s) instead of forcing each one. The WS/MQTT value stream to Home Assistant and the on-disk cache are unchanged — full data resolution, no re-interview on restart. No mount option can do this: fsync is a durability contract the filesystem must honor (commit=/data=writeback/nobarrier don't batch it); the only mount-level alternative is putting the cache on tmpfs, which makes the whole store volatile → re-interview on every restart.
• The .so is musl (the image is Alpine-based), extracted from edge_zwave_alpine_base — keep that var in sync with the image's Alpine version; to refresh after a major bump, rm zwave-js/libeatmydata.so and re-run.
• Durability guard: because fsync is neutralized, a hard power-loss during the (rare) settings write could truncate settings.json, which holds the Z-Wave network keys. edge-zwave-settings-backup.timer keeps a validated hourly rolling backup (settings.json.bak) on the eMMC so the keys stay recoverable. Value-cache loss on crash is harmless (devices re-report; jsonl-db tolerates a partial trailing line).
• Verify the drop with the eMMC-write sampling described above; roll back by removing the LD_PRELOAD env + .so mount from the compose file.

MQTT bridge (TLS to the cluster broker)

open3e, e3oncan and sml2mqtt do not publish to the cluster broker directly. They connect to a local mosquitto on loopback (127.0.0.1:1883, edge_mqtt_host), which bridges to the cluster broker over TLS (mqtt.renner.uno:8883). Two reasons:

• Security: open3e and e3oncan have no native MQTTS support (their MQTT code does a plain connect()), so they can't use the broker's 8883 TLS listener. Terminating TLS in the local broker keeps their plaintext MQTT on loopback only — off the untrusted IoT VLAN — while the actual network hop is encrypted. The public Let's Encrypt cert is validated against the system CA bundle (edge_mqtt_bridge_cafile); no cert distribution needed.
• Availability: the local broker buffers telemetry (in RAM) if the cluster broker is briefly unreachable, and flushes on reconnect — edge-1 no longer hard-depends on cluster/network uptime for every publish.

Two bridge connections preserve the existing cluster identities, so no cluster-side change was needed: one authenticates as the open3e user (carries open3e/# out + open3e/cmnd back), one as sml2mqtt (sml2mqtt/# out). The local broker is allow_anonymous on loopback — the trust boundary is the host; nothing off-host can reach it.

Bridge credential (REN-1307). The two vault-password bridges are being replaced by one bridge as the register account edge-1 (kubernetes/apps/home-assistant/mqtt-accounts.yaml). Its password is pushed by mqtt-account-controller to edge-mqtt-cred (:8443, the controller's esp32 API over HTTPS). That endpoint accepts only a client certificate from the MQTT CA with CN mqtt-account-controller; edge-1 holds the public CA and its own server key, nothing else. On a push it writes /etc/mosquitto/bridge.d/edge-1.conf, deletes bridge.d/legacy.conf and restarts the local mosquitto. The controller sets a new password whenever the bridge reports "not connected" (also after a reinstall), so no play run is needed for a rotation. .woodpecker/deploy-edge-mqtt-bridge.yml runs plays/edge-mqtt-bridge.yml on merge. Diagnosis: journalctl -u edge-mqtt-cred, and mosquitto_sub -t '$SYS/broker/connection/edge-1/state' (retained 1 = connected).

eMMC: the local broker writes nothing to flash — persistence false (retained/queued messages in RAM) and log_dest stdout (to the volatile journal → Loki). Do not add persistence or log-file settings to edge-mqtt-bridge.conf.j2.

Rollback: point edge_mqtt_host back to mqtt.renner.uno and re-run (clients reconnect directly on 1883). Config: edge-mqtt-bridge.conf.j2 → /etc/mosquitto/mosquitto.conf.