fix(heating-can): give supervisord a control socket

The image shipped without [unix_http_server], [rpcinterface:supervisor] or
[supervisorctl], so supervisorctl could not connect:

  Error: .ini file does not include supervisorctl section

That means no way to query or restart an individual service inside the
container -- docker logs was the only introspection. Per-service lifecycle
management was the whole argument for running a supervisor instead of one
process per container, and the first revision did not actually deliver it.

Unix socket only, inside the container. Nothing outside needs it, and this
container runs with host networking, so a TCP listener would be exposed on
the host.

The build-time self-test now asserts the section exists and supervisorctl
runs, so this cannot regress silently the way it shipped.

NOT fixed here, because my diagnosis was wrong: the repeated
"mqtt broker disconnected. reason_code = Unspecified error" is not a client
ID collision. E3onCAN has no -mcid option at all -- verified against the
pinned source -- and generates its own ID; the native broker log showed
E3onCANclient.py-e380-7f847281 alongside open3e-edge, so the two never
collided. I nearly "fixed" it by passing a flag that does not exist.

Telemetry is flowing (each reconnect succeeds), so this is churn rather than
an outage, but the cause is still unidentified. The next step is the broker's
own log on edge-1 -- mosquitto records the disconnect reason, which the
client side reports only as "Unspecified error".

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/docs/edge-host.md

@@ -39,6 +39,8 @@ Build and boot the dedicated installer image — see
    (inventory, plays, this file) and its address is never recorded. Do not
    pin it to a reservation.
 4. Add the host to `[edge_host]` in `inventory/pve_hosts.yml`, by name.
+   That group is the base: every edge thin client belongs to it, and it
+   assumes no peripherals.
 5. From the workstation:
    ```
    ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
@@ -49,6 +51,26 @@ Build and boot the dedicated installer image — see
    the point: a run that touches only the new box tells you nothing about
    the others.
 
+## Peripherals
+
+Each attached device is its own inventory group and its own playbook. Group
+membership states which hardware a host physically has -- there are no flags
+and no `when:` on the roles, so moving a stick between boxes is a one-line
+inventory change.
+
+| device | group | playbook |
+|---|---|---|
+| Innomaker USB2CAN (Vitocal/VX3 + E380) | `heating_can` | `plays/install-heating-can.yml` |
+| Nabu Casa ZWA-2 Z-Wave radio | `zwave_gateway` | `plays/install-zwave-gateway.yml` |
+| FTDI serial power-meter head | `power_meter` | `plays/install-power-meter.yml` |
+
+Add the host to the matching group, then run that playbook. They depend on
+the base play having run: all three need the local MQTT bridge, and two of
+them need docker.
+
+A host with nothing attached is simply absent from all three groups and needs
+none of these commands.
+
 ## Secrets (before first play run)
 
 Create `inventory/group_vars/edge_host/vault.yml` with ansible-vault; the
@@ -121,8 +143,12 @@ Afterwards:
   deliberate for sml2mqtt, acceptable for zwave-js). If a zwave-js bump
   changes its Alpine major, also update `edge_zwave_alpine_base` and refresh
   the preloaded `.so` (see "Logging & eMMC" → zwave-js).
-- open3e updates: re-run the play; the pip task tracks the git default
-  branch (pin `edge_open3e_pkg` to a tag if that ever bites).
+- open3e / E3onCAN updates: both are pinned to commits in
+  `roles/heating-can/defaults/main.yml`, so re-running the play does NOT pick
+  up upstream changes -- that is deliberate, these drive the heat pump. The
+  weekly update-audit reports when either falls behind (see
+  `ci/agents/update-audit/prompt.md`); bump the pin and re-run.
+  Pinning to a tag is not an option for open3e: it publishes none.
 - Ad-hoc CAN scans: `systemctl stop open3e` first (single-instance rule),
   restart afterwards.
 - can-utils (`candump`) is AUR-only on Arch — install manually via

ansible/inventory/group_vars/edge_host/vars.yml

@@ -36,3 +36,18 @@ keepalived_ntp_sysctl:
 
 # Realtek NIC on the Wyse 3040 has no hardware timestamping.
 chrony_hwtimestamp: false
+
+# --- Local MQTT contract -------------------------------------------------
+# Shared between the broker and its clients, so it belongs to the inventory
+# rather than to either side's role. mqtt-bridge listens here; heating-can and
+# power-meter-sml publish here; and the bridge authenticates to the cluster
+# broker AS the open3e user, so it needs that identity too.
+#
+# Keeping these in one role's defaults only worked while a single play loaded
+# every role at once. Splitting the plays by capability exposed it: the bridge
+# could not render without a heating-can default, and the CAN services could
+# not render without an mqtt-bridge default.
+edge_mqtt_host: 127.0.0.1
+edge_mqtt_port: 1883
+edge_open3e_mqtt_user: open3e
+edge_open3e_mqtt_topic: open3e

ansible/inventory/pve_hosts.yml

@@ -41,9 +41,36 @@ nas-1.renner.uno ansible_user=ladmin ansible_python_interpreter=/usr/bin/python3
 [grafana]
 grafana.srv.renner.uno ansible_user=ladmin
 
+# Every bare-metal edge thin client. Gets the host-generic roles only:
+# flash-wear handling, the log shipper, the MQTT bridge, the VLAN
+# sub-interface. Nothing here assumes any peripheral is attached.
 [edge_host]
 edge-1.iot.renner.uno ansible_user=root
 
+# Capability groups. Membership states which hardware a host physically has,
+# and each service playbook targets the group for its own concern. No flags and
+# no `when:` on the roles -- moving a stick to another box is a one-line change
+# here, which is what it should be.
+#
+# These are children of [edge_host], declared below: a peripheral is added to an
+# edge client, it does not replace the base. Making that a :children relation
+# rather than a comment means a host added only to a capability group still
+# resolves group_vars/edge_host -- where the local MQTT contract lives -- so it
+# cannot silently come up without edge_mqtt_host and friends.
+[zwave_gateway]
+edge-1.iot.renner.uno
+
+[heating_can]
+edge-1.iot.renner.uno
+
+[power_meter]
+edge-1.iot.renner.uno
+
+[edge_host:children]
+zwave_gateway
+heating_can
+power_meter
+
 # NTP servers. Includes edge-1 so the role survives the PVE hosts being
 # retired for bare-metal Talos, which cannot run chrony or keepalived itself.
 # See plays/install-ntp-ha.yml for why keepalived is scoped more narrowly

ansible/plays/install-edge-host.yml

@@ -3,10 +3,18 @@
 # hardware-bound services, because Talos supports neither SocketCAN nor USB
 # passthrough.
 #
-# One role per concern. Previously all of this lived in a single `edge-host`
-# role, which meant a host could not take the flash-wear handling or the log
-# shipper without also being handed the Z-Wave radio and the CAN bus. That is
-# what blocked a second thin client from joining as an NTP peer.
+# This play is the BASE only: what every edge thin client gets regardless of
+# what is plugged into it. The peripherals live in their own playbooks against
+# their own inventory groups:
+#
+#   plays/install-zwave-gateway.yml   [zwave_gateway]   ZWA-2 radio
+#   plays/install-heating-can.yml     [heating_can]     Innomaker USB2CAN
+#   plays/install-power-meter.yml     [power_meter]     FTDI meter head
+#
+# Splitting the roles without splitting the plays would have kept the coupling:
+# one play applying every role to one group is the monolith again, a layer up.
+# A host with no peripherals is simply absent from those groups -- no flags, no
+# `when:` on the roles.
 #
 # Self-contained: `certificates` and `common` are the parts of plays/common.yml
 # that apply to an edge host, carried here so bootstrapping one is a single
@@ -14,76 +22,20 @@
 # metal, not VMs) and monitoring-agent (Debian-oriented; these hosts ship logs
 # via edge-log-shipper).
 #
-# Order is dependency order, not alphabetical:
-#   mqtt-bridge      before its clients, so they have somewhere to publish
-#   edge-flash-wear  before docker, so the journal is volatile before
-#                    containers start logging into it
-#   socketcan        before heating-can, which consumes can0
-- name: Install Arch edge host
+# Order is dependency order: edge-flash-wear before docker, so the journal is
+# volatile before containers start logging into it.
+- name: Install Arch edge host (base)
   hosts: edge_host
   # Connection is already root (ansible_user=root), so this is a root-to-root
   # no-op today. Kept explicit, matching plays/common.yml, so the roles below
   # do not silently depend on the connection user staying privileged.
   become: true
 
-  # Defined on the play, not taken from a role's defaults: these are used by
-  # pre_tasks, which must not depend on role default resolution.
-  #
-  # heating-can, zwave-gateway and power-meter-sml each declare edge_base_dir
-  # in their own defaults too, so they stay usable standalone. Play vars win,
-  # so these are the effective values -- keep them in step if the path moves.
-  vars:
-    edge_base_dir: /opt/edge-host
-    edge_legacy_compose: "{{ edge_base_dir }}/docker-compose.yml"
-
-  # One-time migration off the shared compose project, BEFORE any role tries
-  # to start a container.
-  #
-  # zwave-js and sml2mqtt used to live in one project at edge_legacy_compose,
-  # both with a fixed container_name. Docker enforces container names globally,
-  # not per project, so while the old project still owns those names the new
-  # per-service projects cannot create them -- `docker compose up -d` fails with
-  # "Conflict: container name already in use", which is not the missing-USB-device
-  # error the roles tolerate, so the play would abort.
-  #
-  # It belongs here rather than in either role: it is a migration step for both
-  # of them, and putting it in one meant whichever role ran first hit the
-  # conflict. Guarded on the legacy file, so it runs once and then never again.
-  #
-  # check mode cannot verify this: command tasks are skipped, not simulated, so
-  # a --check run neither hits the conflict nor proves the teardown works.
-  pre_tasks:
-    - name: Retire the legacy shared compose project
-      ansible.builtin.command:
-        cmd: docker compose -f "{{ edge_legacy_compose }}" down
-      args:
-        removes: "{{ edge_legacy_compose }}"
-      failed_when: false
-
-    - name: Remove the legacy shared compose file
-      ansible.builtin.file:
-        path: "{{ edge_legacy_compose }}"
-        state: absent
-
-    - name: Remove the legacy zwave-js env from the shared project root
-      ansible.builtin.file:
-        path: "{{ edge_base_dir }}/zwavejs.env"
-        state: absent
-
   roles:
     - certificates
     - common
-
-    # Host-generic. An edge host with no attached peripherals takes these and
-    # nothing below them.
     - edge-flash-wear
     - docker-install
     - edge-log-shipper
     - mqtt-bridge
     - edge-vlan
-
-    # Hardware-bound. Each needs its own device present.
-    - socketcan
-    - heating-can
-    - zwave-gateway
-    - power-meter-sml

ansible/plays/install-heating-can.yml

@@ -0,0 +1,18 @@
+---
+# Heating system over the Viessmann E3 CAN bus, for hosts with the Innomaker
+# USB2CAN adapter attached.
+#
+# Targets [heating_can]: membership means the adapter is physically present.
+# Two roles because they are two concerns -- the interface is host network
+# state, the services consume it -- but one group, because both need the same
+# hardware.
+#
+# Requires the base play (plays/install-edge-host.yml) to have run: these
+# services publish to the local MQTT bridge it sets up.
+- name: Install heating CAN services
+  hosts: heating_can
+  become: true
+
+  roles:
+    - socketcan
+    - heating-can

ansible/plays/install-power-meter.yml

@@ -0,0 +1,14 @@
+---
+# Power-meter reader (sml2mqtt) for hosts with the FTDI serial meter head
+# attached.
+#
+# Targets [power_meter]: membership means the meter head is physically present.
+#
+# Requires the base play (plays/install-edge-host.yml) to have run: it needs
+# docker and publishes to the local MQTT bridge.
+- name: Install power-meter reader
+  hosts: power_meter
+  become: true
+
+  roles:
+    - power-meter-sml

ansible/plays/install-zwave-gateway.yml

@@ -0,0 +1,15 @@
+---
+# Z-Wave gateway (zwave-js-ui) for hosts with the ZWA-2 USB radio attached.
+#
+# Targets [zwave_gateway]: membership means the radio is physically present.
+# Moving the stick to another edge host is a one-line inventory change.
+#
+# Requires the base play (plays/install-edge-host.yml) to have run: it needs
+# docker, and the flash-wear handling that the libeatmydata preload exists to
+# complement.
+- name: Install Z-Wave gateway
+  hosts: zwave_gateway
+  become: true
+
+  roles:
+    - zwave-gateway

ansible/roles/heating-can/tasks/main.yml

@@ -34,17 +34,42 @@
 # stopped and masked rather than merely disabled -- masked so a stray
 # `systemctl start` or a leftover dependency cannot bring them back alongside
 # the containers.
-- name: Stop and mask the native CAN services
-  ansible.builtin.systemd:
-    name: "{{ item }}"
-    state: stopped
-    enabled: false
-    masked: true
+- name: Find which native CAN units still exist
+  ansible.builtin.stat:
+    path: "/etc/systemd/system/{{ item }}.service"
   loop:
     - open3e
     - e3oncan
-  failed_when: false
+  register: heating_can_native_units
 
+# Stop and disable BEFORE anything else, and do NOT swallow the result.
+#
+# The first version did stop + disable + mask in one call with
+# failed_when: false, then deleted the unit files. Masking a unit whose file
+# still exists conflicts, so the call failed, failed_when hid it, nothing was
+# stopped -- and the next task deleted the unit files out from under two
+# running processes. systemctl is-enabled then reported "not-found" and looked
+# clean while is-active still said "active".
+#
+# The result on 2026-08-10 was two open3e instances issuing UDS requests on the
+# same CAN bus for nine minutes, fighting over the open3e-edge client ID and
+# kicking each other off the broker.
+#
+# Only units that actually exist are touched, so a fresh host does not need an
+# error swallowed to get through this.
+- name: Stop and disable the native CAN services
+  ansible.builtin.systemd:
+    name: "{{ item.item }}"
+    state: stopped
+    enabled: false
+  loop: "{{ heating_can_native_units.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+  when: item.stat.exists
+
+# Masking is deliberately NOT used. It writes a symlink at the same path the
+# next task removes, so the two cancel out -- which is half of how the original
+# failed. With the unit files gone there is nothing left to start.
 - name: Remove the native unit files
   ansible.builtin.file:
     path: "/etc/systemd/system/{{ item }}.service"
@@ -54,10 +79,30 @@
     - e3oncan
   notify: Reload systemd
 
-# The venvs are the point of this change -- leaving them would leave the thing
-# that churned. Their configuration is NOT removed: devices.json, the overlay
-# and the env file are still rendered below and bind-mounted into the
-# containers.
+# The invariant that actually matters is "no native process is on the CAN bus",
+# and systemd reporting success is not the same claim. Two open3e instances
+# polling one bus is the failure this whole block exists to prevent, so it is
+# asserted directly rather than inferred from the calls above.
+- name: Check for surviving native CAN processes
+  ansible.builtin.command:
+    argv: [pgrep, -af, "edge-host/(open3e|e3oncan)"]
+  register: heating_can_survivors
+  changed_when: false
+  failed_when: false
+
+- name: Assert no native CAN process survived
+  ansible.builtin.assert:
+    that:
+      - heating_can_survivors.stdout | default('') | length == 0
+    fail_msg: >-
+      A native open3e/e3oncan process is still running after the units were
+      stopped and removed:
+      {{ heating_can_survivors.stdout_lines | default([]) | join('; ') }}.
+      Starting the container now would put two readers on the same CAN bus --
+      two open3e instances issue competing UDS requests and take over each
+      other's MQTT session. Kill the survivors, then re-run.
+    success_msg: No native CAN processes remain; safe to start the container.
+
 - name: Remove the native venvs
   ansible.builtin.file:
     path: "{{ item }}"

ansible/roles/mqtt-bridge/defaults/main.yml

@@ -5,8 +5,6 @@
 # — off the untrusted IoT VLAN — and buffers telemetry in RAM during cluster
 # blips. The local broker writes nothing to the eMMC (persistence off, logs to
 # the volatile journal). See edge-mqtt-bridge.conf.j2 and README "MQTT bridge".
-edge_mqtt_host: 127.0.0.1
-edge_mqtt_port: 1883
 # Upstream cluster broker the local bridge connects to (TLS; the public
 # Let's Encrypt cert is validated against the system CA bundle).
 edge_mqtt_cluster_host: mqtt.renner.uno

ci/agents/update-audit/prompt.md

@@ -78,6 +78,29 @@ For each, find the **current pin in the repo** (Read/Glob/Grep), then look up th
 6. **Ansible roles & OpenTofu providers** — `ansible/requirements.yml` (Galaxy
    roles/collections) and `opentofu/providers.tf` + `opentofu/.terraform.lock.hcl`.
    Report any with a newer published version.
+7. **Edge heating CAN services** — open3e (Vitocal heat pump, Vitocharge VX3)
+   and E3onCAN (E380 grid energy meter), shipped as one image built from
+   `ci/images/heating-can/Containerfile` and deployed by
+   `ansible/roles/heating-can`. Nothing else tracks them. They read and (for two
+   datapoints) write the heat pump over its CAN bus, so silent drift here
+   matters more than most.
+
+   Both upstream pins are `ARG`s in the Containerfile -- `OPEN3E_REF` and
+   `E3ONCAN_REF` -- not Ansible variables. Read them there, and note the
+   awkward bits so normal states are not reported as problems:
+   - open3e publishes **no tags at all** (its tags API returns an empty array),
+     so compare `OPEN3E_REF` against the default branch:
+     `https://api.github.com/repos/open3e/open3e/commits?per_page=1`. Report how
+     many commits behind and the dates. "No release found" is normal here.
+   - E3onCAN tags, but sparsely, and `E3ONCAN_REF` is an untagged commit later
+     than the newest tag. Compare against both
+     `https://api.github.com/repos/MyHomeMyData/E3onCAN/tags` and the default
+     branch, and say which the pin sits between.
+
+   The deployed image tag is `heating_can_image` in
+   `ansible/roles/heating-can/defaults/main.yml`. It is `:latest` today, so the
+   node follows whatever CI last built -- the pins above are what actually fix
+   the versions, and bumping them is what requires a rebuild.
 
 ## Method & rigor
 

ci/images/heating-can/Containerfile

@@ -103,6 +103,8 @@ RUN open3e --help > /dev/null \
  && supervisord --help > /dev/null \
  && grep -q '^\[program:open3e\]' /etc/supervisord.conf \
  && grep -q '^\[program:e3oncan\]' /etc/supervisord.conf \
+ && grep -q '^\[supervisorctl\]' /etc/supervisord.conf \
+ && supervisorctl --help > /dev/null \
  && echo "self-test OK: both services execute; supervisord has both programs"
 
 # CAN_RAW needs no capabilities, so nothing here runs as root.

ci/images/heating-can/supervisord.conf

@@ -10,6 +10,24 @@
 # expands %(ENV_NAME)s at start, so neither the compose file nor a wrapper
 # script constructs open3e's or E3onCAN's argv -- those CLIs stay an
 # implementation detail of this image.
+# Control socket. Without these two sections supervisorctl cannot connect, so
+# there is no way to query or restart an individual service inside the
+# container -- which is the entire reason for running a supervisor rather than
+# one process per container. The first revision shipped without them.
+#
+# Unix socket only, inside the container: nothing outside needs to reach it,
+# and the container runs with host networking, so a TCP listener would be
+# exposed on the host.
+[unix_http_server]
+file=/tmp/supervisor.sock
+chmod=0700
+
+[rpcinterface:supervisor]
+supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
+
+[supervisorctl]
+serverurl=unix:///tmp/supervisor.sock
+
 [supervisord]
 nodaemon=true
 user=open3e

kubernetes/apps/monitoring/pve-exporter-deployment.yaml

@@ -27,7 +27,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: pve-exporter
-          image: prompve/prometheus-pve-exporter:3.9.0 # {"$imagepolicy": "flux-system:pve-exporter"}
+          image: prompve/prometheus-pve-exporter:3.10.0 # {"$imagepolicy": "flux-system:pve-exporter"}
           # Image ENTRYPOINT is already `pve_exporter`; pass flags only.
           args:
             - --config.file=/etc/pve-exporter/pve.yml