claude-botandClaude Opus 5.5
a17c4e8161
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trmnl
Terminus (self-hosted TRMNL BYOS server) plus the headless renderer that screenshots the Home Assistant entrance dashboard and pushes it in. One device: a TRMNL X at the entrance, 1872x1404, 16 greys.
Device firmware
Terminus mirrors one model-agnostic binary and it is the ESP32-C3 (OG)
build. This panel is a TRMNL X, i.e. ESP32-S3, so that file is not the
artefact for a manual flash here. Use
https://trmnl-fw.s3.us-east-2.amazonaws.com/trmnl_x/flash/FW<version>.bin
— a merged image at offset 0x0 — and run esptool image-info on
anything before writing it.
The panel runs 1.8.16, flashed over USB on 2026-09-05. It shipped on
1.8.6, where turning on firmware_update froze the screen for as long as
an update was offered (fixed upstream in 1.8.7, commit 9b7a0f7).
Image automation
Both images are enrolled (infrastructure/image-automation/terminus.yaml,
trmnl-renderer.yaml) and track the latest 0.x. A bad bump on either
surfaces as a blank or corrupted panel, not as a failed rollout — no
readiness probe can tell that a rendered screen became unreadable. Look at
the panel after a bump lands; revert the tag if it is wrong.
Device endpoint
trmnl-device.renner.uno is the device-facing name, TLS via Traefik
(device-ingress.yaml). API_URI must carry the same scheme the device
polls with, because Terminus bakes it into the absolute image_url it
returns. The LoadBalancer in device-service.yaml keeps 10.2.99.89
without a hostname as a cleartext fallback that depends on neither
Traefik nor certificate renewal — the captive portal accepts a bare
address.