Replaces the k8s-edge Talos VM (vm_153) with a dedicated Arch host
running the hardware-bound services: zwave-js-ui and sml2mqtt via
docker compose (configs ported 1:1 from the k8s manifests), plus the
new open3e CAN bridge (SocketCAN via systemd-networkd, venv + systemd
unit, read-only datapoint set).
Driver for leaving k8s here: stock Talos ships CONFIG_CAN unset
(siderolabs/talos#7686), so the Vitocal/VX3 CAN adapter cannot work in
any pod without a custom kernel per release.
Base-install procedure, secret bootstrap (vault.yml from cluster
secrets) and per-service cutover checklist in roles/edge-host/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fifth broker user for the Viessmann E3 CAN bridge (open3e, Vitocal
250-A + Vitocharge VX3). Same in-cluster bootstrap-Job pattern as the
other users; ACL scoped to open3e/#.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
status.renner.uno will move off-cluster, so it's a bad canary. Probe two
permanently cluster-hosted domains instead and count a round as failed
only when EVERY domain fails: one dead backend 502s on its own domain,
while the dead-route state kills all domains at once — this keeps single
app outages from restarting the proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The proxy's embedded client pins each proxied domain's route to a
routing-peer identity. netbird-ingress rollouts mint fresh identities by
design, and upstream's dns-interceptor never rebinds a route whose owning
peer is gone ('already routed by peer [...]. HA routing disabled') — the
proxy then serves only 502/timeouts until restarted (2026-07-02 outage).
Add a shared role deploying a systemd timer (1min) on both cluster
members (VPS-1, proxy-1): probe status.renner.uno through the local
proxy, restart the container after 3 consecutive 502/000 results, with a
10min cooldown against restart loops. Every probe outcome is journaled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
www/ was excluded (except the bespoke heizungsanlage card) because the old
one-way force-overwrite fought HACS's runtime rewrites. With two-way sync that
no longer applies: HACS drift is committed to `live`, PR'd, and merges cleanly.
Widen PATHS from www/heizungsanlage-card.js to the whole www/ directory.
Bonus: this also removes the merge-refusal edge case — www changes are now
committed before the merge step, so there's no uncommitted www drift left to
block a merge.
First run captures the current HACS drift (~6 frontend files: kiosk-mode,
xiaomi-vacuum-map-card) as one small live PR, then stays current.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
The sidecar was one-way and git-authoritative: it force-overwrote local edits to
the synced config paths, so changes made inside HA (code-server, the UI
automation/script editors) were reverted on the next tick.
Now it is two-way:
- PUSH: changes to the tracked config paths are committed (bot identity) onto a
dedicated `live` branch, pushed, and a PR `live -> main` is kept open (via the
gitea token already in the remote URL). Live edits stay live AND are captured
for review.
- PULL: when `main` advances (merged PR / external push) the commits are merged
back into `live`/the working tree and HA reloads.
Conflicts are left to git: a conflicting merge is ABORTED (never written into the
live files, which would break HA) and logged to resolve in the PR. Reloads fire
only on changes pulled from main (local edits are already applied by HA).
Also fixes the reload race: reload_all now retries until HA is serving, instead
of failing once right after a pod start. Scope unchanged (config paths only; rest
of www/ and untracked runtime state untouched); auto-gc still disabled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
The config-sync sidecar intentionally leaves www/ to HACS, but
www/heizungsanlage-card.js is a bespoke, non-HACS custom card committed to the
HA config repo (k.renner/homeassistant). Add that exact path to PATHS so the
sidecar force-syncs just that one file into /config/www; the rest of www/ stays
HACS-managed.
Note: extra_module_url is read only at HA startup, so a one-time full restart is
still needed to load the card after the file first lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only the "starting" line surfaced in `kubectl logs`: busybox sh block-buffers
stdout when it's a pipe, so subsequent lines could sit unflushed for the life of
the loop. Log to stderr instead (unbuffered; still captured in the container
log). Also emit a one-time "initial check: config in sync" line so an idle loop
is observably alive even when there's nothing to sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
The config-sync sidecar OOMKilled (exit 137) at the 64Mi limit and crashlooped,
so it never synced. Root cause: this repo's .git is ~1.1 GB (a 1 GB backup tar
lives in history), and git auto-gc tried to repack that whole pack inside the
tiny sidecar on first fetch.
- Disable gc.auto in the sync script and cap pack window / delta caches, so
fetch/checkout only ever touch the small config blobs and never repack.
- Raise the sidecar memory limit 64Mi -> 256Mi for index-pack headroom on the
first fetch.
Follow-up (not here): purge the 1 GB backup tar from the homeassistant repo
history and .gitignore backups/ — it bloats every clone and the config PVC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
The previous loop reloaded on any advance of origin/main, including commits that
touch only unsynced paths (www/, shelly_scripts/, docs). Now the reload trigger
is `git diff --quiet <remote> -- <config paths>`: HA is reloaded only when the
synced config paths differ. Non-config commits just advance the branch ref with
no reload. As a side effect the loop now also reverts local drift on synced
paths continuously (reloading only when something actually changed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
HA's /config is an RWO PVC that is a git checkout of the homeassistant repo,
edited via the code-server sidecar. There was no automated path from a merge to
the running instance, so merged config (e.g. the notification-queue work) sat
undeployed until a manual pull+reload.
Adds a `config-sync` sidecar (alpine/git) sharing the config PVC. Every 30s it
fetches origin/main and, on change, force-syncs ONLY the HA YAML config paths
(automations(.yaml+dir), packages/, scripts.yaml, templates.yaml,
configuration.yaml, notify.yaml, groups.yaml, powercalc.yaml, blueprints) into
the working tree, advances the branch ref without touching the working tree, and
calls homeassistant.reload_all so the change goes live with no restart.
Deliberately left untouched: www/ (HACS rewrites those tracked files at runtime)
and untracked runtime state (.storage/, *.db, secrets.yaml). git auth reuses the
PAT already embedded in the PVC's remote URL; reload auth uses a dedicated
long-lived token in a sops secret.
Consequence (git becomes source of truth): uncommitted edits to a synced path
made directly in code-server OR via HA's UI automation editor are reverted on the
next sync. Commit + push to persist.
NOTE: the reload token currently reuses the existing HA long-lived token; rotate
it to a dedicated `config-sync` token (revocable independently) — see PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5
The immich-library DataUpload (~250 GiB, millions of small files) now
runs ~3h50m–4h+ and was intermittently Canceled at exactly the 4h
`defaultItemOperationTimeout`. A canceled upload writes no kopia
snapshot, so the daily backup ends PartiallyFailed; the weekly
restore-test then selects that backup and fails restoring immich-library
with "Unable to load snapshot: snapshot not found", firing KubeJobFailed.
Observed cancellations:
2026-06-26 immich-library DataUpload Canceled 4h00m09s
2026-06-27 immich-library DataUpload Canceled 4h00m12s
2026-06-28 immich-library DataUpload Completed 3h50m18s
Raise both dataMoverPrepareTimeout and defaultItemOperationTimeout to 8h
for headroom as the library keeps growing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Home Assistant was pinned to k8s-edge solely for the USB power meter it
read via edl21. edl21 is now replaced by the sml2mqtt bridge (publishes to
Mosquitto), so HA:
- drops nodeSelector zone=edge + the workload-class=pinned toleration →
the heavy 8Gi pod schedules on the general cluster pool, off the
soon-to-be-tiny edge box;
- drops the /dev/stromzaehler USB hostPath mount + volume (bridge owns the
device now);
- drops securityContext.privileged — the USB device was its only
justification.
This also frees the serial fd the running HA process still held after
edl21's removal (HA recreates without the device), giving the sml2mqtt
bridge exclusive access to the meter.
Meter data in HA resumes once the MQTT sensors (reusing the old
sensor.smart_meter_* entity_ids for history continuity) land in the
homeassistant repo — brief expected gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
edl21 has been disabled in HA, freeing the meter's serial port, so the
bridge is activated for the cutover.
Three fixes from bring-up testing:
- logging.file: stdout (was ''). Empty string made sml2mqtt try to open
the /sml2mqtt config dir as a log file (IsADirectoryError crash loop).
'stdout' routes logs to the container console for kubectl/Loki.
- securityContext.privileged: true (with runAsUser 0). The meter device
is root:root 0600, AND the device cgroup blocks a hostPath-mounted char
device for non-privileged containers even as root. There is no
device-plugin in the cluster; the existing USB pods (zwave-js, HA) use
the same privileged pattern. User-approved.
- replicas: 0 -> 1 (port is now free; no edl21 contention).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First phase of moving the resource-hungry Home Assistant pod off k8s-edge.
HA is pinned to edge solely by the USB power meter (FTDI FT230X) it reads
via the built-in edl21 integration. This adds an sml2mqtt bridge that will
read the meter and publish to the in-cluster Mosquitto broker, so HA can
read it over MQTT and stop needing the USB device / the edge node.
Shipped DORMANT (replicas: 0): the meter's serial port is a single
stream, so sml2mqtt and HA's edl21 cannot both read it. Merging this
changes nothing at runtime. The cutover (separate steps) is: disable
edl21 -> scale this to 1 -> add HA MQTT sensors reusing the existing
entity_ids (preserves long-term-statistics history) -> evict HA from edge.
Contents:
- sml2mqtt-deployment.yaml: edge-pinned, replicas:0, runs
spacemanspiff2007/sml2mqtt:3.6.0 as root (meter dev is root:root 0600;
no privileged needed), USB hostPath mount, init container renders the
broker password into config.yml from a Secret.
- sml2mqtt-config.yaml: v3 config; Wh kept (not kWh) to match edl21 so
history is continuous; publishes to mosquitto:1883 as user sml2mqtt.
- mosquitto: 4th broker user `sml2mqtt` (publish-only sml2mqtt/#) via a
new bootstrap Job + passwd/ACL/env entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These three only need v30iot L2 presence, which exists on every node
(eth1 is the v30iot interface on k8s-1/2/3 and k8s-edge alike), not the
edge node's USB devices. Removing their topology.kubernetes.io/zone=edge
nodeSelector and workload-class=pinned toleration lets the edge
NoSchedule taint keep them off the (soon-to-be dedicated, tiny N100) edge
box, reserving it for the only genuinely USB-bound workloads:
home-assistant (FTDI power meter) and zwave-js (ZWA-2 radio).
- matter-server / otbr: macvlan NAD already uses master: eth1, which
resolves to v30iot on all nodes (verified live on k8s-1/2/3) — no NAD
change needed.
- music-assistant: hostNetwork gets v30iot L2 on any node; it is the
heaviest of the three (2Gi limit), so this is the main capacity win.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The netbird-ingress rootless routing peer re-originates every mesh + WAN-via-proxy connection from its own pod IP (e.g. fd10:244::2de1 = netbird-ingress-qxxc5), so Traefik's TCP peer for all that traffic is one shared pod IP — which is what let a single CrowdSec ban 403 every IPv6 client (2026-06-27 outage). The NetBird reverse-proxy (HTTP mode) already sets X-Forwarded-For with the real WAN client IP; Traefik just wasn't reading it.
Set ports.websecure.forwardedHeaders.trustedIPs to the pod CIDRs (10.244.0.0/16, fd10:244::/56 — the re-originators), so Traefik resolves the real client IP from XFF. Mirror it in the (still-inert) crowdsec-bouncer Middleware via forwardedHeadersTrustedIPs so it's ready when enforcement is re-enabled. Safe no-op where no XFF exists (direct LAN/mesh keep their RemoteAddr). Enforcement stays OFF — re-enabling still needs the internal allowlist (prereq #2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global bouncer middleware banned all IPv6 ingress: Traefik sees one shared/SNAT'd v6 source, so a single http-admin-interface-probing hit (the e2e scan) 403'd every v6 client. Remove the global entrypoint-middleware attachment; keep the plugin + Middleware CRD inert so re-enabling is one line after fixing client-IP visibility + adding an internal whitelist. Detection (agents) stays on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These checks probed resources retired during the PVE->K8s migration and have been failing since e2e could run again (the fetch-kubeconfig path was broken ~45h). Verified each against live cluster + repo before removing:
- migration-bucket OBC Bound (authentik/paperless/audiobookshelf): no such OBCs exist in the cluster or repo (kubectl get obc -A is empty).
- Ceph RGW S3 round-trip: mounts the migration-bucket Secret/ConfigMap that the (gone) OBC generated; can't run. (The CephObjectStore rook-ceph/migration is itself now vestigial — separate cleanup.)
- TLS for auth.k.renner.uno: retired hostname, no Traefik router (serves the default cert / 404). Kept dash.k/audiobooks.k.renner.uno, which are real.
- authentik OIDC well-known: paperless: paperless uses an Authentik proxy outpost, not OIDC, so it has no /application/o/paperless provider (404). Kept audiobookshelf + grafana, which are real OIDC apps (200).
Left intact: loki/prometheus/alertmanager/paperless-docs and all storage/velero/cert/db checks — those are valid and merely flaked transiently during the recent cluster churn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #77 used ports.websecure.middlewares, which chart 40.x's values schema rejects ('additional properties middlewares not allowed'), so the whole Helm upgrade failed and degraded the infrastructure kustomization. Attach the entrypoint default middleware via additionalArguments CLI flag instead (schema-free). experimental.plugins was fine and is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent's acquisition still targeted podName ingress-nginx-controller-*
in the ingress namespace — a selector that has matched no pod since
ingress-nginx was retired (2026-05-17), so the agents have been parsing
nothing and no decisions were being generated.
Repoint to the Traefik controller (podName traefik-*, program traefik)
and swap the nginx collection/parser for crowdsecurity/traefik (which
bundles its own log parser). This restores attack detection. Enforcement
(a Traefik bouncer acting on the decisions) is a separate change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All 13 services run in k8s and were verified healthy/Running before
removal (audiobookshelf, immich, loki, prometheus, ical-proxy,
music-assistant, authentik full stack, home-assistant, influxdb [in
cluster as influxdb-influxdb2], grafana [kube-prometheus-stack]). These
stopped Proxmox guests were the PVE->K8s migration rollback net; with the
replacements serving 1-7 weeks, retire them.
Removed tofu modules (destroys the stopped guests + their stale DNS):
- vm_103_homeassistant, vm_108_influxdb, vm_109_grafana
- lxc_121_ical_proxy, lxc_212_audiobookshelf, lxc_225_immich,
lxc_226_loki, lxc_227_prometheus, lxc_229_music_assistant
- lxc_authentik_ha (130/131/132/133) + its stale authentik-db.renner.uno
-> 10.2.40.135 (dead keepalived VIP) DNS record
- backup_jobs.tf: drop VMIDs 103/108/109/121/130/131/132/133/212/225/226/227/229
Kept: fortress VM (102, not a migrated service) and paperless VM (111).
IRREVERSIBLE: applying destroys these guests and wipes their disks.
Review the plan before apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Fable 5
security-bot