A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the matrix-bridge Deployment, Service and ExternalSecret in the
matrix namespace, loads its appservice registration in Synapse, and opens
the bridge -> Paperclip API path (:3100) in both namespaces' policies.
Tokens and the board key come from OpenBao homelab/apps/matrix-bridge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Element X signs in only with a password or through MAS. Put MAS in front
of Synapse with Authentik (passkey) as the upstream provider, and enable
the QR rendezvous (MSC4108/MSC4388).
- apps/matrix/mas.yaml: MAS 1.26.0, `mas` database in mx-pg (CNPG
Database), account.matrix.renner.uno.
- synapse.yaml: matrix_authentication_service, password_config off; a
syn2mas init container moves the existing accounts and sessions once
(marker on synapse-data).
- Drop the owner-account Job; MAS owns accounts now.
- Authentik: public PKCE client `matrix-mas`, passkey flows, only the
Matrix Users group (k.renner).
- matrix-ns: MAS egress to Traefik :8443 (auth.renner.uno) only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New `matrix` namespace and Flux KS: Synapse v1.162.0 on a CNPG Postgres
(mx-pg, Barman to Hetzner), media store + signing key on synapse-data
(Velero matrix-daily), Element Web locked to matrix.renner.uno, and an
idempotent Job that creates the one owner account from OpenBao
apps/matrix. Federation off, registration and guests closed, LAN-only
ingress, egress limited to the in-cluster DB and the push gateway.
Appservice slot left empty for REN-565 Part 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-botandClaude Opus 5.5