A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remaining backed-up clusters still declared bootstrap.initdb, so a
cold rebuild came up with empty Home Assistant, Matrix, Paperclip and
TRMNL databases. Each now declares bootstrap.recovery from its
ObjectStore (serverName = cluster name). mx-pg drops its initdb
encoding/locale settings: the physical restore keeps the C collation.
RECOVERY.md lists all seven clusters; gf-pg has no ObjectStore and stays
on initdb.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Matrix room per card labelled 'matrix' (REN-668). Sets
PAPERCLIP_PUBLIC_URL for the card link in the room topic.
Rollback: build.1791177680 and drop the env var.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route agent comments on author account only, from konairius/matrix-bridge#5
(9a83e70, REN-668). No env, secret, port or network change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three Matrix Ingresses had no dns.renner.uno/public label, so
netbird-service-sync never created a NetBird proxy service for them and the
public proxy rejects their SNI (tlsv1 alert internal error). Label them like
audiobooks/immich and add Gatus checks for the three hosts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
as_token and hs_token come from an ESO Password generator (CreatedOnce)
instead of OpenBao, so no one has to seed them by hand. Synapse mounts
only the generated Secret and no longer depends on the OpenBao path;
OpenBao homelab/apps/matrix-bridge now holds only paperclip_api_key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the matrix-bridge Deployment, Service and ExternalSecret in the
matrix namespace, loads its appservice registration in Synapse, and opens
the bridge -> Paperclip API path (:3100) in both namespaces' policies.
Tokens and the board key come from OpenBao homelab/apps/matrix-bridge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner wants QR sign-in from the desktop app as well. Linking a phone from
a signed-in Element (Link new device) only needs the server side from the
previous commit (MSC4108 + MAS device-code grant). This flag also lets a
new browser sign in by QR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Element X signs in only with a password or through MAS. Put MAS in front
of Synapse with Authentik (passkey) as the upstream provider, and enable
the QR rendezvous (MSC4108/MSC4388).
- apps/matrix/mas.yaml: MAS 1.26.0, `mas` database in mx-pg (CNPG
Database), account.matrix.renner.uno.
- synapse.yaml: matrix_authentication_service, password_config off; a
syn2mas init container moves the existing accounts and sessions once
(marker on synapse-data).
- Drop the owner-account Job; MAS owns accounts now.
- Authentik: public PKCE client `matrix-mas`, passkey flows, only the
Matrix Users group (k.renner).
- matrix-ns: MAS egress to Traefik :8443 (auth.renner.uno) only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New `matrix` namespace and Flux KS: Synapse v1.162.0 on a CNPG Postgres
(mx-pg, Barman to Hetzner), media store + signing key on synapse-data
(Velero matrix-daily), Element Web locked to matrix.renner.uno, and an
idempotent Job that creates the one owner account from OpenBao
apps/matrix. Federation off, registration and guests closed, LAN-only
ingress, egress limited to the in-cluster DB and the push gateway.
Appservice slot left empty for REN-565 Part 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-botandClaude Opus 5.5
renovate-bot