This reverts #1111 (3f3305cf). With the fingerprint stable, the ACPX engine
now resumes the previous Claude session, but the Claude transcript lives in
the per-run CLAUDE_CONFIG_DIR, which the server deletes after each run. acpx
uses resumePolicy same-session-only for persistent sessions, so the turn
fails ("Persistent ACP session ... could not be resumed: Resource not
found") instead of falling back to a fresh session (run 12a2a911).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude auto memory defaults to $CLAUDE_CONFIG_DIR/projects/<p>/memory.
Paperclip makes CLAUDE_CONFIG_DIR fresh per run and deletes it, so every
memory is lost. New image patch: the ACPX engine writes
autoMemoryDirectory=<AGENT_HOME>/memory into the per-run
.claude/settings.local.json. AGENT_HOME is restored and saved around each
run, local and in k8s pods. Not part of the session fingerprint, so
resume is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Built static in the go stage with go install (module checked against
sum.golang.org); added to the default-PATH check and the README table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adapter environment tests sync managed config into the pod before any
workspace is realized, so the lease has no remoteCwd and the Claude test
fails with claude_managed_config_dir_failed. #13733 fixes only the run
path. #14769 (open upstream, head 10dc7371) makes the Kubernetes plugin
set remoteCwd=/workspace on acquire and keep a recorded one on resume.
The plugin build from REN-405 is unchanged, so dist/ carries the fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream's production target ships the plugin source without dist/, so the
plugin loader reports paperclip.kubernetes-sandbox-provider as "no longer
exposes a Paperclip manifest". prepare.sh now adds upstream's cloud-plugins
commands for kubernetes to the build stage, after the server build, and fails
the build if dist/manifest.js is missing. The #13733 patch is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k8s sandbox runs fail because the run path never sets the lease remoteCwd
(paperclipai/paperclip#13864). The fix, paperclipai/paperclip#13733, is not
merged upstream yet. Build our own image from the master commit the nightly
runs now plus that patch, and point Flux image automation at it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
npm install -g ignored Paperclip's pnpm override and pulled
claude-agent-sdk 0.3.257. Install from a package.json with the override
(0.3.280) into /opt/claude-agent-acp and link the bin into /usr/local/bin.
The build fails if the SDK version is not 0.3.280.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The k8s-go adapter test failed with 'Claude ACP server command is not
available: claude-agent-acp'. Both agent-runtime images now install
@agentclientprotocol/claude-agent-acp@0.73.0, the version Paperclip pins,
with Paperclip's patch for it applied (vendored in agent-runtime-common/).
go and gofmt are also linked into /usr/local/bin, so every tool is found
with the default PATH, not only with the image ENV. The build checks that
with env -i. Kaniko context is now ci/images so both images can COPY the
shared patch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two pod images for the Paperclip Kubernetes sandbox (REN-284 step 2):
the stock agent-runtime-claude, pinned by digest, plus
- agent-runtime-go: Go 1.27.1, golangci-lint v2.14.0
- agent-runtime-ops: kubectl v1.36.5, flux v2.9.5, kustomize v5.8.1, helm 4.3.0
Tools are copied from their upstream images pinned by tag and digest, so
the existing Renovate dockerfile manager bumps them. Pushed to
konairius/agent-runtime-{go,ops}/agent-runtime-claude:v1 because the plugin
keeps that name and swaps only the registry prefix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The .NET wiki service has parity (same recall on the labelled set), serves
wiki-index.renner.uno and the notes webhook, and the workstation scripts
use it. Removes the app, its Flux Kustomization, image source and build
step, its CNP entries and the OpenBao policy file. The OpenBao role and
the old Gitea webhook are removed live after the merge.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 10608 aborted before its first edit: the agent calls
~/Notes/bin/wiki-snapshot.sh and git -C ~/Notes rev-parse/status, the
rules carried only /root/Notes. Both spellings are allowed for the vault
scripts and the read-only git subcommands on the vault checkout, config is
denied there too, and the consolidate step derives its --allowedTools
from the settings file instead of a second copy. Verified headless:
rev-parse, status and wiki-delete.sh allowed, config denied.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
All three nodes run v1.13.10 (applied by hand, one node at a time:
drain, talosctl upgrade --image, wait for 217/217 active+clean). This
brings the repo in step so a rebuild or a new node installs the same
version.
Why this release: its pkgs include siderolabs/pkgs@2da2cdc, 'backport
aes256k support (Ceph)', the libceph patches from Linux 7.x on top of
6.18.48. Kernel config on every node now has CONFIG_CRYPTO_KRB5=y and
CONFIG_CRYPTO_KRB5ENC=y. That was the blocker for rotating the CSI
CephX keys to aes256k (#527).
Patch release within 1.13; Kubernetes stays v1.36.3. Schematic is
unchanged; factory installer and ISO verified to exist for the tag.
talosctl checksum matches the release's sha256sum.txt and a local
download.
Refs #527
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review on #927: the git -C /root/source/ prefix let config and push
through; rm /root/Notes/wiki/ matched path traversal. Allowed git forms
are now log/show/diff/status/rev-parse/ls-files per repo (both the ~ and
/root spelling), config is denied for those forms too, rm is denied and
notes are deleted through the vault's wiki-delete.sh, which validates the
slug. Verified headless: log allowed, config denied, push not allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pass that verifies notes against sources, repos and the live cluster
and edits them still ran on the workstation timer, with no TypeSafe key,
no cost record, and no run on days the machine was off. It joins the
nightly pipeline after the audit: full clones of homelab and homeassistant
under ~/source, read-only kubectl via the ci-diagnostics token, its own
permission scope, JSON output so the report and total_cost_usd land in
the step log. push-notes also commits the rotation state and pushes the
agent's own snapshot commits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The audit agent had no executable path in CI (Bash limited to the two
retrieval tools), wrote itself scripts it could not run and reported one
of seven checks (runs 10262, 10362). The checks are mechanical, so
notes/bin/wiki_audit.py does them without a model. push-notes adds only
wiki/, raw/ and output/, so scratch files in the checkout root are no
longer committed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Additive; the running controller is untouched:
- ci/images/netbird-proxy-services -> ci/images/netbird-service-sync,
build step and image name renamed, so the merge builds
gitea_admin/netbird-service-sync.
- Namespace netbird-service-sync with a baseline policy: probes in;
out only to kube-dns, kube-apiserver, OpenBao :8200 and world :443
(NetBird management API).
- openbao-ns admits netbird-service-sync on :8200, like wiki-index.
- policies/netbird-service-sync.hcl documents the new policy and role
(applied by hand, as before).
Refs #876
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Replaces the workstation's wiki-maintenance.timer. The pipeline clones
k.renner/notes, runs wiki-distill over raw/ and wiki-audit over wiki/
as two separate `claude --agent` runs (no background tasks in print
mode, which killed the workstation job at the 600 s ceiling), then
commits and pushes as Claude Bot; the push webhook refreshes wiki-index.
The agent definitions now live in the vault (.claude/agents/), so CI
and workstation share them. The agents and their retrieval scripts
address the vault as ~/Notes; each agent step links $HOME/Notes to the
checkout. ci-claude gains python3 and py3-yaml for wiki_ask.py.
Secrets on the repo: claude_code_oauth_token (existing), typesafe_api_key,
wiki_bot_token (claude-bot, write:repository), wiki_index_basic_auth.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/health answered 503 until an index existed and fed startup and
liveness, so a live process without an index counted as dead and was
restarted every 15 min, and a later failed sync would have pulled a pod
with a good index out of the Service. /health is now always 200
(process serves); /ready is 200 once an index exists and stays so
across failed syncs, which keep the last good index. Probes follow:
startup 60 s on /health, readiness on /ready, liveness on /health.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The notes repo tree has 3192 entries. Gitea reports truncated=true on
every page including the last one (page 4, 192 entries) and answers
"tree": null one page further, which the sync hit:
"sync failed: 'NoneType' object is not iterable". Paging now stops on a
short page or when total_count is reached, and a null tree is empty.
Unit test with the observed page shapes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`bao kv put key=@file` stores the file's trailing newline; the first
sync after #875 failed with "Invalid header value b'token ...\n'" and a
webhook signed with the newline-free secret would never have verified.
Credentials are stripped on read. A failed sync now retries after 60 s
instead of waiting the full RESYNC_SECONDS.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Runtime image for a wiki-index service in namespace ai (manifests follow
once the image exists): standard-library Python that mirrors the notes
repository's wiki/ from Gitea into a bge-m3 index and serves POST /rank
to ~/Notes/bin/wiki_rank.py.
Sync fetches the branch head and tree via the Gitea API, embeds only notes
whose blob sha changed (through the in-cluster wiki-embed), drops notes
that left the tree, and persists the index on a PVC. It runs at start, on
every push webhook (HMAC-SHA256 verified) and every RESYNC_SECONDS. Gitea
token and webhook secret come from OpenBao through the Kubernetes auth
role on every use. Five unit tests cover parsing, incremental sync,
persistence, ranking and error handling.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.
ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
the upstream release assets (talos sha256sum.txt, flux checksums.txt)
and against a local download.
.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
already run; the lint step was one minor behind its own server.
Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TLS-passthrough services hand Traefik a raw TCP stream, so every WAN
request now logs the routing peer's pod IP (10.244.4.64/.112 in the
access log) instead of the client. CrowdSec's detection parses exactly
that log; with the HTTP-mode services gone it would attribute every
external request to two pod IPs, and a re-armed bouncer would repeat the
2026-06-27 mass ban.
The controller now sets options.proxy_protocol on every target (PROXY
protocol v2, TCP/TLS modes only per NetBird) and treats a service
without it as out of date, so the eleven existing services are updated
on the next pass. Traefik's websecure entrypoint accepts the header from
the pod CIDRs, mirroring forwardedHeaders.trustedIPs. Traefik takes
connections with or without a header from a trusted IP, so mesh and LAN
traffic that carries none is unaffected and the Traefik change can land
before the controller image.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
netbird-proxy-services reconciles the NetBird reverse-proxy services that
nobody managed so far: one TLS-passthrough service on :443 per host of every
Ingress carrying dns.renner.uno/public, targeting Traefik ClusterIP through
a subnet resource over the service CIDR. SNI and Host reach Traefik
untouched, the proxy terminates nothing, no domain resource is involved --
so the dns-interceptor pinning bug that wedges the proxies on every
routing-peer loss has nothing to pin on.
Ownership is derived from NetBird, not from local state: managed iff mode is
tls and the target is the controller subnet resource. The ten hand-made
HTTP/domain services are reported as held and never touched; migration is
deleting one of them, the controller creates the passthrough on its next
pass. The PAT is read from OpenBao per pass through a read-only role
(policies/netbird-proxy-services.hcl, applied once by hand like the rotator
roles). Standard library only; unprivileged image built alongside heating-can.
Validated against the live management (ONCE=1 through the rotator role):
network, resource, router and policy created, 10 hosts held, one host that
had no service at all (llm.renner.uno) created and answering behind its
Authentik gate; second pass 0 created / 0 updated / 0 deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Gitea runs inside the cluster, so during a cold rebuild every image that
only exists at gitea.renner.uno is unpullable — a rehearsal died with the
cni-plugins installer in ImagePullBackOff, no macvlan binary, no ceph-san
attachment, CephCluster stuck at "Configuring Ceph Mons", OpenBao waiting
forever on a PVC. The failure surfaced four layers from its cause.
The CI images move to konairius/, published by the build pipeline to
Gitea AND ghcr.io with one digest. The manifests keep pointing at Gitea;
the node-level fallback (machine.registries.mirrors, written by
bootstrap.sh at install time) tries Gitea first and ghcr second, and
containerd keeps the repository path when it falls through — so the same
path resolves on both. Steady state never leaves the network.
The Multus, whereabouts and cni-plugins-installer DaemonSets also gain
priorityClassName system-node-critical. Upstream ships them at priority
0: on a rebuild two nodes sat at 99% memory requests when the DaemonSet
rolled, both multus pods stayed Pending, and every pod needing a
secondary attachment waited on an annotation that could never come — with
nothing reporting why.
Rollout effect on the live cluster: the three CNI DaemonSets restart once
(existing pods keep their networking; only new attachments wait), and the
three init containers pull the new tag on their next pod restart. The
ghcr side is proven by the Hetzner throwaway runs, which pull through the
fallback; eyeball gitea.renner.uno/konairius packages before merging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image shipped without [unix_http_server], [rpcinterface:supervisor] or
[supervisorctl], so supervisorctl could not connect:
Error: .ini file does not include supervisorctl section
That means no way to query or restart an individual service inside the
container -- docker logs was the only introspection. Per-service lifecycle
management was the whole argument for running a supervisor instead of one
process per container, and the first revision did not actually deliver it.
Unix socket only, inside the container. Nothing outside needs it, and this
container runs with host networking, so a TCP listener would be exposed on
the host.
The build-time self-test now asserts the section exists and supervisorctl
runs, so this cannot regress silently the way it shipped.
NOT fixed here, because my diagnosis was wrong: the repeated
"mqtt broker disconnected. reason_code = Unspecified error" is not a client
ID collision. E3onCAN has no -mcid option at all -- verified against the
pinned source -- and generates its own ID; the native broker log showed
E3onCANclient.py-e380-7f847281 alongside open3e-edge, so the two never
collided. I nearly "fixed" it by passing a flag that does not exist.
Telemetry is flowing (each reconnect succeeds), so this is churn rather than
an outage, but the cause is still unidentified. The next step is the broker's
own log on edge-1 -- mosquitto records the disconnect reason, which the
client side reports only as "Unspecified error".
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#476 merged first, which removed edge_open3e_pkg, edge_e3oncan_repo and
edge_e3oncan_version in favour of the image. The audit entry added here still
told the agent to read those, so it would have gone looking for variables
that no longer exist and reported nothing -- the same silent non-auditing the
entry was written to prevent.
Repointed at where the pins actually live now: OPEN3E_REF and E3ONCAN_REF as
ARGs in ci/images/heating-can/Containerfile. The awkward specifics carry over
unchanged, since they are properties of the upstreams rather than of how we
package them: open3e publishes no tags so the comparison must be against
commits, and E3onCAN's pin is an untagged commit later than its newest tag.
Added a note the previous version could not have made: heating_can_image is
:latest, so the node follows whatever CI last built. The Containerfile ARGs
are what actually fix the versions, and bumping one is what requires a
rebuild -- otherwise a reader could reasonably conclude the tag is the pin.
The pin commit itself is now a no-op against master and survives only as the
audit registration; its conflict on heating-can/defaults resolved in favour of
the image.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
open3e was installed from an unpinned git URL, so the heating bridge
silently tracked upstream main: whatever HEAD happened to be on the day a
converge ran was what ended up driving the heat pump. Now pinned to
6553c104, which was simultaneously upstream HEAD and the copy already
installed on edge-1, so the pin froze the running state and changed nothing.
A commit rather than a tag because open3e publishes none -- its tags API
returns an empty array. The package self-reports 0.7.5 at that commit, which
is useful context but is not a tag.
Honest about what this does NOT fix: pip still re-clones and rebuilds a VCS
requirement even when the SHA is pinned, because it cannot compare an
installed VCS dist against a requested URL without fetching it first.
Verified directly on the host:
pip install --dry-run "git+...@6553c104"
-> Cloning ... Resolved ... to commit 6553c104
Installing build dependencies: started
So the task still reports changed and still restarts open3e on every
converge. Pinning buys deliberate versioning, not idempotence.
Pinning without an update path only converts churn into silent rot, so both
services are now registered with the weekly update-audit -- which had been
auditing neither, despite E3onCAN having been pinned since July. The audit
prompt warns that anything not listed is silently not audited, and these two
were exactly that.
The new scope entry records the awkward specifics so the agent does not
report normal states as problems: open3e has no tags at all, so it must
compare commits against the default branch; E3onCAN does tag, but sparsely,
and its pin sits between the newest tag and HEAD.
Also corrected while there: edge_e3oncan_version was commented "# V0.5.2",
which does not exist upstream. The repo's only tag is v0.5.1 (6f43b3fb), and
461dcbdf is a later untagged commit from 2026-06-21, "Fixed: MQTT client ID
collision caused stale retained values; added publish error logging".
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review on #476 found a real regression plus three things worth fixing before
this reaches edge-1.
Blocking: edge_open3e_devices.vx3.dpList still pointed at the old host path
/opt/edge-host/open3e/vx3plus.py. The compose file mounts that overlay at
/config/vx3plus.py and nothing mounts anything under /opt/edge-host inside
the container, so open3e would have looked for it at a path that does not
exist -- silently falling back to the stock vx3 datapoint list and dropping
the battery-module DIDs (vx3.1577-1582 and friends) the overlay exists to
add. Now /config/vx3plus.py.
Worth noting the build-time self-test could not have caught this: it proves
the binaries execute, not that devices.json resolves against the mounted
files at runtime.
Writable working directory. WORKDIR was /opt, which the COPY layers create
root-owned, while the process runs as open3e. udsoncan writes its log file
into the CWD -- the native install gave open3e's directory `owner: open3e`
for exactly that reason, and I dropped that when containerising. Now a
dedicated /var/lib/open3e, owned by the service user.
Secret mode. MQTT_PASSWORD was inline in docker-compose.yml, which is
rendered 0644. The env file this replaced was 0600, and the sibling roles on
this host keep secrets in a 0600 file beside a world-readable compose file.
Restored: credentials move to heating-can.env at 0600, referenced via
env_file.
edge_e3oncan_mqtt_format had become dead -- both services were fed
MQTT_FORMAT from edge_open3e_mqtt_format. The two default to the same string
so behaviour was unchanged, but overriding it did nothing. Wired through as
E3ONCAN_MQTT_FORMAT, which supervisord passes to e3oncan only.
Also from review: dropped the registered-but-unread heating_can_native_stopped,
and the migration now removes the emptied {{ edge_base_dir }}/open3e and
/e3oncan directories rather than leaving them behind.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The smoke-test step I added errored the whole pipeline before any step ran.
It used the freshly built image as its own step image, but the Gitea registry
is private: the build steps authenticate kaniko for PUSH, nothing gives the
runner credentials to PULL. So the step could never have worked.
Moved into the Containerfile as a build-time RUN, which is better anyway --
it needs no registry credentials, and a broken image is never published
rather than published and then flagged.
It asserts what the previous revision got wrong: both services actually
execute, and supervisord's config carries both programs. That is exactly the
failure that shipped green -- supervisord pointed at
/opt/e3oncan/src/E3onCANcollect.py, which does not exist (the file is at the
repo root), so the image would have run open3e while e3oncan crash-looped
and the E380 meter went silently missing.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image shipped a path that does not exist. supervisord ran
`python /opt/e3oncan/src/E3onCANcollect.py`, but that file is at the repo
ROOT -- there is no src/ directory. The `src` came from the old Ansible task,
which cloned into `{{ edge_base_dir }}/e3oncan/src`; that was the clone
DESTINATION, not part of the upstream layout, and I carried it into the
Containerfile without checking. Confirmed against the upstream tree at the
pinned commit: E3onCANcollect.py sits at the root alongside requirements.txt,
which is why the build succeeded while the runtime path was wrong.
So the image built green and would have started open3e and failed e3oncan on
every restart -- the E380 meter silently absent, with supervisord retrying
forever.
Nothing caught that, because nothing ran the image. Now CI does: a smoke-test
step uses the freshly built image as its own step image and asserts both
services execute and supervisord's config carries both programs.
It cannot exercise the CAN bus or the broker, neither of which exists on a
runner, so it verifies exactly what is verifiable without hardware. That is
still the difference between "built" and "works" -- and it belongs on a
throwaway runner, not on the production edge host, which is where I wrongly
went looking for this the first time.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts my own reinterpretation. The instruction was to put both in one
container because they belong together; I argued against it on crash-isolation
grounds, then read the correction "two services, one image" as agreement with
my version rather than as a restatement of the original. It was not.
A container runs one PID 1. With two processes that PID 1 is a supervisor,
which is how a container handles multiple services. supervisord now runs both
readers: it restarts either on its own if it dies, reaps children and forwards
shutdown -- none of which a bare process as PID 1 can do for a sibling, which
was the actual answer to the concern I raised.
Also gone: the dispatch entrypoint and the argv in the compose file. Both were
the same mistake in different places -- the invocation living outside the
artifact. open3e's and E3onCAN's command lines are an implementation detail of
this image now; supervisord builds them from environment variables it expands
itself. Nothing outside the image knows either CLI, so an upstream flag rename
cannot break the compose file or an Ansible template.
The compose file supplies configuration and nothing else: one service, one
container, environment only. The credential env file is dropped -- the
credentials are environment already.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing is installed on the host any more. The image is built in CI with
both sources pinned, published to the registry, and edge-1 pulls and runs
it. That is the whole point of building it -- installation happens at build
time, once, not on every converge.
Removed from the node: both venvs, the E3onCAN checkout, and the two systemd
units. Kept and still rendered: devices.json, the vx3 datapoint overlay and
the broker credentials, now bind-mounted into the containers rather than
read from a venv working directory.
Two services from one image, as before: open3e drives the Vitocal/VX3 over
UDS, e3oncan listens passively for the E380 meter. Separate containers so
either can crash and restart without a supervisor as PID 1.
The migration stops AND masks the old units before the containers start.
Masked rather than disabled because two open3e instances issuing UDS
requests on one CAN bus is the failure mode to avoid, and a stray
`systemctl start` or leftover dependency should not be able to bring the
native one back alongside the container.
No missing-device tolerance here, unlike the zwave and meter roles: can0 is
this host's purpose. If it is absent the heating telemetry is down and the
run should say so rather than pass quietly.
heating_can_image replaces edge_open3e_pkg / edge_e3oncan_repo /
edge_e3oncan_version. The source pins move into the Containerfile, so what
is deployed here is a tag -- and `docker compose up -d` on an unchanged tag
is a genuine no-op, which is what removes the restart-per-converge that
pinning the pip URL could not.
Verification is limited and worth stating: plays/install-heating-can.yml
lives in #475 and is not on this branch, so only the master-side play
syntax-checks here. A --check run cannot exercise this at all -- command
tasks are skipped under check mode, so neither the pull nor the compose up
would run. It cannot be meaningfully verified until the image has been built
and #475 is merged.
Depends on #475 (the capability-split playbooks, and the MQTT contract
moving to group_vars). This branch is cut from master, so heating-can's
defaults will conflict there; #475 should merge first and this be rebased
onto it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image had no ENTRYPOINT and no CMD, with a comment claiming that was
deliberate because the commands are rendered by Ansible. That was
rationalising a defect.
An image with nothing runnable is not a self-contained artifact. The
knowledge of how to START each service leaked into Ansible, and the two were
not even symmetric: open3e installs a console script via pip, while E3onCAN
is an unpackaged script in a checkout, so the compose file would have had to
carry `python /opt/e3oncan/src/E3onCANcollect.py` -- an implementation detail
of this image's internal layout, in a file that has no business knowing it.
Move the checkout and every caller breaks.
E3onCAN now gets a wrapper on PATH, so both services are invoked the same
way and `docker run <image> e3oncan --help` works.
CMD prints what the image provides and exits 64 (EX_USAGE) rather than
silently doing nothing, so `docker run <image>` is self-describing. Which
service to run, and its arguments, stays the caller's choice -- the compose
file renders those, because the datapoint list and broker settings live
there.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds one image containing open3e (Vitocal heat pump, Vitocharge VX3) and
E3onCAN (E380 grid energy meter), both pinned in the Containerfile.
Why: both were pip-installed into venvs on edge-1, open3e straight from an
unpinned git URL. pip cannot compare an installed VCS dist against a
requested one without fetching it, so `pip install` re-cloned and rebuilt on
every Ansible converge -- reporting changed, firing the restart handler, and
putting a gap in heating telemetry each time. Pinning the SHA does not fix
that; verified directly on the host that pip still clones and rebuilds a
pinned VCS requirement. A tagged image does: `docker compose up -d` on an
unchanged tag is a genuine no-op.
It also gives the weekly update-audit a normal artifact instead of two
awkward special cases. open3e publishes no tags at all, and E3onCAN is
pinned to an untagged commit sitting between its newest tag and HEAD -- both
of which the audit currently has to be told about in prose.
One image, two services. They read the same physical installation over the
same bus and are versioned together, so they are one artifact; but they will
run as two containers from it, so either can crash and restart on its own
without a supervisor as PID 1.
Fixes a long-standing blind spot in passing: e3oncan.service set
PYTHONUNBUFFERED=1 and open3e.service did not, so open3e's stdout was
block-buffered and effectively never reached the journal or Loki. The image
sets it for both.
Build follows the existing kaniko pattern exactly -- path-scoped `when:` so
only this image rebuilds, date tag plus latest, pushed to the Gitea registry
alongside the ci-* images.
Deliberately artifact-only. The heating-can role still deploys venvs and
systemd units; moving it to a compose project is a separate change, once
this image exists and has been pulled and run once. Rewriting the role that
drives the heat pump against an image that has never been built would be the
wrong order.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I proposed widening ImageUpdateAutomation's `update.path` to `./` and adding
a woodpecker-cli ImagePolicy so automation would own the lint pin the way it
owns the server and agent. Checking that before building it killed the plan,
and corrected two things I had asserted without evidence.
WRONG #1 — my reason for hesitating. I claimed the 7 ansible-vault encrypted
`.yml` files would break the walk and take all 33 automated images down with
them. That was a guess about kyaml I never verified. Objects that are not
Kubernetes manifests are SKIPPED, not fatal.
WRONG #2 — and this is the one that matters: the plan cannot work at all.
Image-automation only patches YAML that is a Kubernetes manifest. Maintainer,
fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
YAML manifest must have `apiVersion` and `kind` fields." The docs agree —
markers go "in any Kubernetes custom resource stored in Git". A Woodpecker
pipeline file is plain YAML with no apiVersion/kind, so a marker in it is
silently ignored.
That is a property of the FILE, not of `update.path`. Widening the path buys
exactly nothing and would drag 197 unrelated ansible/esphome YAMLs into the
walk. So no policy, no second automation, no path change — all three would
have been motion that looked like a fix and changed nothing. The failure mode
would have been the quiet kind this repo keeps getting bitten by: a marker
sitting in the file looking maintained, never once updated.
What actually closes the gap, given the pin must stay manual:
* README gains a "Not enrolled (structurally impossible)" section holding
the quote and the explicit "do not try widening the path" warning, so the
next person does not re-derive this.
* update-audit item 5 now covers `.woodpecker/*.yml` step images and is told
to flag woodpecker-cli whenever it trails the DEPLOYED SERVER, not merely
the newest CLI release — the server auto-bumps and the CLI does not, so
that skew is the thing worth reporting. Also told to compare only the
vX.Y.Z part (the tag carries `-alpine`; the plain tag is scratch with no
shell and would break the step) and to ignore the `next-*`/`pull_*` dev
tags, which are the newest by date on that repo and are not releases.
* the step comment states the real constraint instead of my earlier wrong
one, which told the reader that widening the path was the fix.
Verified: v3.16.0-alpine is the newest 3.x alpine release tag, and matches the
deployed server v3.16.0. Config still parses; step image unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner directive: tools belong in the image, not installed at container
start. This init container ran `apk add --no-cache openssl` on EVERY
paperless pod start.
Unlike the multus CNI installer, this one genuinely needs an image: the
openssl CLI is absent from the alpine base (verified — the base carries the
libraries but not the command), and busybox has no `s_client` equivalent,
so the STARTTLS certificate capture cannot be done with what is already
there.
Adds ci/images/alpine-openssl, named for what it is rather than for this
consumer, so anything else needing s_client/x509 from a shell script can
reuse it.
Script logic is unchanged — only the apk add line is removed and the image
swapped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review point on #447. The build-time check asserted macvlan, static and
host-local were executable but omitted dhcp, which the sibling
cni-dhcp-daemon DaemonSet runs and which its own comment says ships from
this installer ("the dhcp binary ships from the sibling
cni-plugins-installer DaemonSet ... whole tarball extracted").
Verified that dependency in kubernetes/infrastructure/multus/
cni-dhcp-daemon.yaml before changing anything, rather than taking the
review's word for it.
Consequence of the omission: since the whole /plugins directory is copied
regardless of what the loop checks, nothing is broken today — but the
"fail loudly if upstream restructures the archive" guarantee only covered
three of the four binaries this repo depends on. Upstream dropping dhcp
while keeping the others would have built green and surfaced later as a
broken DHCP daemon on whichever node rolled first.
Confirmed all four are executable in the already-published image, so this
tightens the assertion without changing what ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Supersedes the wget-instead-of-curl approach in #446, which removed the
`apk add` but left the real problems in place.
The installer fetched the containernetworking/plugins release tarball from
GitHub on the node and extracted it into /opt/cni/bin with NO integrity
verification of any kind. That directory is executed as root in the CNI
path on every node, so a tampered or MITM'd artifact would have obtained
privileged execution cluster-wide with nothing to detect it. It also put
github.com on the critical path of every node rebuild.
Now the binaries are baked into ci/images/cni-plugins, where the download
happens once and the sha256 IS verified, and the installer is a local file
copy with zero network calls (verified: no wget/curl/apk/https in the
rendered script).
The pinned digest was cross-checked against upstream's own published
cni-plugins-linux-amd64-v1.5.1.tgz.sha256 asset, not merely recorded from a
local download — a download can be tampered with, so comparing against
what upstream states is the point:
77baa2f669980a82255ffa2f2717de823992480271ee778aa51a9c60ae89ff9b
Verified the gate actually gates: with the correct digest the layer logic
downloads, verifies, extracts and asserts macvlan/static/host-local are
executable; with a deliberately wrong digest `sha256sum -c` FAILS and the
extract step is never reached.
Install script changes beyond dropping the fetch:
- Copies unconditionally instead of the old "already installed?" guard.
The guard existed to avoid re-downloading; a local copy is cheap, so
copying every time means a node always matches the version pinned in the
manifest rather than keeping whatever it first installed.
- Copies to a temp name in the same directory then rename(2)s over the
target. `cp` onto a plugin the kubelet is currently executing can fail
with ETXTBSY; rename never does and is atomic, so a concurrent CNI
invocation sees either the old or the new binary, never a partial one.
- LICENSE and README.md are no longer installed into /opt/cni/bin; the old
tar extraction put them there.
The chown 0:0 is kept but re-framed as a REPAIR. Its original cause is
gone — extraction applied the archive's `./` uid/gid (1001/127, the
github-actions runner) to the destination, and `cp` into an existing
directory does not touch its ownership. It still runs because nodes
provisioned by the old installer may already carry the wrong ownership,
which breaks Cilium's apply-sysctl-overwrites init container.
Verified end-to-end with the rendered script, including that repair path:
pre-seeded owner 1001:127 -> 0:0 after run
fresh install -> host-local/macvlan/static present, macvlan executes
and reports "CNI macvlan plugin v1.5.1", exit 0
re-run -> idempotent, exit 0, 18 plugins, no .tmp files left
Note the image carries ~80MB of extracted plugins (18 binaries), matching
what the old installer put on each node. All are kept rather than trimming
to the few in use, to avoid removing plugins already present on nodes as a
side effect of this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner directives, both applied here:
- "z2m-seed is a shit name for the image — name it for what it is, not for
how it is used." Renamed to alpine-jq. Nothing in it is
zigbee-specific; it is alpine plus jq, useful to anything that needs to
read JSON from a shell script. The Containerfile now describes it that
way, with the zigbee2mqtt case recorded as its first consumer rather
than its identity.
- "You do not need to pin the digest for our own images." Referenced by
tag. This also collapses what was going to be two PRs into one, since
the digest no longer has to exist before the Deployment can name it.
The seed-data init container now uses that image instead of
`apk add --no-cache jq` on every pod start.
Two consequences of dropping digest pinning, both good:
- The whole --image-name-with-digest-file / "PIN THIS:" apparatus is gone
from the pipeline. That machinery existed only to hand a digest to a
follow-up commit, and it is what broke every pipeline in the repo (#443)
via an unquoted ": " in a YAML scalar. Less surface, same outcome.
- :latest means Kubernetes defaults imagePullPolicy to Always, so a
rebuilt image is picked up without editing the manifest.
Kept the fail-loudly validation, since a wrong network_key costs the whole
Zigbee network: explicit checks for an absent key (jq prints "null" for a
missing path, so that is checked separately rather than being reported as
merely "non-hex"), non-hex characters, and a length other than 32.
Verified by running the manifest's RENDERED script, against the real
coordinator backup and synthetic fixtures. The alpine-jq tag does not
exist until this merges and builds, so the run used the existing image of
identical content (alpine 3.21.7 + jq 1.7.1, verified present):
real backup -> exit 0, network_key line byte-identical to the
live value (sha256 27c5c77a945bcecc)
non-hex key -> exit 1, "contains non-hex characters"
4-char key -> exit 1, "length 4, expected 32 hex chars"
absent network_key -> exit 1, ".network_key.key absent from ..."
device link_key also
present in the doc -> correctly ignored; the network_key is used
Validated with `woodpecker lint` (not just `yq`, which is what let #443
through), plus kustomize build and a server-side dry run. No stale
z2m-seed references remain.
NOTE: the now-orphaned z2m-seed package is still in the registry and
should be deleted; that needs package-delete rights this account does not
have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 of 2 toward removing `apk add` from the zigbee2mqtt seed-data init
container, which currently installs jq on every pod start. Owner
directive: baking tools into a layered image is fine, adding them at
runtime is not.
This commit only adds the image and its build step; nothing consumes it
yet, so it is inert on merge. The Deployment cannot be switched in the
same commit because runtime images in this repo are pinned by DIGEST (as
the secret-rotator CronJobs are), and the digest does not exist until
this has built and pushed. The build therefore writes it out via kaniko's
--image-name-with-digest-file and echoes it as "PIN THIS:", so the
follow-up commit can copy the reference straight from the build log
without a separate registry query.
Why a real JSON parser rather than busybox sed/awk: "key" occurs 12 times
in the actual backup — once per device with an APS link_key — and the
FIRST occurrence is a device key, because "devices" sorts before
"network_key". An extraction that is not correctly anchored yields a
device link key, herdsman then forms a NEW network, and every device
drops off. This was reproduced while prototyping the awk approach, which
is what makes jq worth an image here rather than clever one-liners in the
manifest.
Follows the existing conventions: Containerfile alongside the other
ci/images entries, base pinned by digest to the same alpine:3.21 the
ci-k8s image uses, kaniko build gated on its own path so unrelated pushes
do not rebuild it, and date + latest tags pushed as the other images do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
