Both rook-ceph-mgr-a and -b were CrashLoopBackOff with exit 137
(OOMKilled) at the previous 512Mi/1Gi sizing, which stalled Ceph
CSI provisioner calls and left PVCs stuck in FailedMount. Dashboard
+ volumes + rbd_support modules need more headroom than the initial
default.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Garage rejects HeadBucket calls with 400 Bad Request when boto3 sends
them without a signed region, which makes CNPG's bootstrap.recovery
fail at the barman-cloud-check-wal-archive step. Add REGION=garage to
the barman-cloud-creds Secret and reference it from the ObjectStore's
s3Credentials.region so barman-cloud-* invocations sign with the
correct region.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The barman-cloud Deployment mounts barman-cloud-{client,server}-tls
Secrets that are created by cert-manager Certificates in post-crds. In
a single-Kustomization infrastructure reconcile this is circular: the
Deployment can't start without the Secrets, and post-crds can't run
until infrastructure reports ready. Move the plugin into its own
Kustomization that dependsOn infrastructure-crds so the Secrets exist
before the plugin Pod schedules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same CRD-ordering fix as the barman Certificates: Velero Schedule CRs
need velero.io CRDs, which don't exist until the Velero HelmRelease
finishes installing. Moving them into post-crds/ lets the dry-run
succeed on a fresh bootstrap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On a fresh cluster bootstrap the `infrastructure` Kustomization
reconciles before cert-manager's HelmRelease finishes installing its
CRDs, so Flux's dry-run fails on the barman-cloud Certificate /
Issuer with `no matches for kind Certificate in version
cert-manager.io/v1`. Move those three resources into post-crds/ which
runs after infrastructure and after cert-manager's CRDs are present.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
barman-cloud-backup reads PGDATA from the local filesystem — it is not
a pure network client — so connecting to a `services:` Postgres leaves
PGDATA unreachable and the backup fails on pg_control. Drop the
`services:` block and boot a local Postgres inside the seed-barman step
(initdb + pg_ctl start) so PGDATA is co-located with the barman
process. Loopback connection is trust-auth, ephemeral, step-scoped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The migrate-{paperless,audiobookshelf} pipelines now push legacy
tarballs directly to Garage (s3://k8s-migration/{paperless,audiobookshelf}/...)
using Woodpecker-scoped garage_migration_* secrets, so the seed
producer has no cluster dependency. Runs as a standalone `woodpecker
pipeline create` — no trigger-backup-job required.
In-cluster seed Jobs:
- Read the same tarballs from Garage via a new SOPS-encrypted
`garage-migration-creds` Secret in each seed namespace.
- paperless-data-seed-v2 and audiobookshelf-data-seed-v2 replace the
v1 Jobs (Jobs are immutable, so a name bump is needed).
- audiobookshelf-seed no longer owns the migration-bucket OBC; it
moves back to the audiobookshelf namespace where its legacy
trigger-backup-job still references it (retirement deferred).
Flux Kustomizations for *-seed gain decryption.provider=sops so the
new Secret can be applied.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cluster-independent migration seed flow needs a dedicated Garage bucket
that the standalone migrate-* pipelines write to (and the seed Jobs
read from at bootstrap). Separate from k8s-cnpg (DB-only) and k8s-velero
(ongoing PVC backups).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The postgres:16 image lacks gcc / libpq-dev, so pip build of barman's
psycopg2 dep fails. Install the compiled binary via apt and point the
venv at system site-packages so barman[cloud] reuses it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every cluster bootstrap now restores from the latest backup in Garage;
migration becomes a one-off seed that writes to the same paths the live
cluster uses.
DB path (ak-pg, pprl-pg):
- Cluster.bootstrap switched from initdb to recovery, reading the live
Barman ObjectStore via externalClusters.
- migrate-{authentik-db,paperless} pipelines stand up an ephemeral
Postgres, pg_restore the legacy dump into it, purge the Barman path,
then barman-cloud-backup against that instance so the seed matches
the format the live cluster writes.
- Obsolete post-bootstrap restore-job manifests removed.
App PVC path (paperless, audiobookshelf):
- defaultSnapshotMoveData=true on the Velero HelmRelease so CSI snapshot
contents are moved to Garage (survive Ceph loss).
- Daily Velero Schedules for paperless, audiobookshelf, authentik.
- velero-bootstrap-restore Kustomization: Job creates one Restore per
Schedule from latest Backup at rebuild; skips schedules with zero
Backups so first-ever bootstrap is a no-op.
- Transitional *-seed Kustomizations pull legacy tarballs into fresh
PVCs on first bootstrap (sentinel-idempotent). Velero has no
external-producer API so this bridge is unavoidable during migration;
retire once Velero has a verified backup.
Requires Woodpecker secrets garage_cnpg_key_id + garage_cnpg_secret_key
(cnpg-key.creds from /volume1/docker/garage/config/keys/ on nas-1).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies the full Barman plugin chain after the Velero block:
- barman-cloud plugin pod ready (must be in database ns, not
cnpg-system — see wiki/cnpg-plugin-same-namespace-constraint)
- ObjectStore CRs exist for ak-pg and pprl-pg
- Cluster conditions report ContinuousArchiving=True
- ScheduledBackup 'daily' jobs have produced at least one
completed Backup each
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ScheduledBackup CRs trigger a base backup via the Barman Cloud
plugin daily at 02:30 UTC (after the PBS client's 02:00 NAS
backup, so the S3-resident backup data is picked up by PBS on
the following day's 02:00 run via Garage's underlying /volume1
filesystem). `immediate: true` forces the first run at apply time
so we can verify the backup path end-to-end; `backupOwnerReference:
self` keeps historical Backups tied to the schedule lifecycle.
WAL archiving is already continuous via postgres.yaml
spec.plugins[].isWALArchiver=true.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CNPG 1.29's plugin controller ONLY discovers plugin Services and
their mTLS secrets in the operator's own namespace, per
internal/controller/plugin_controller.go:
// We only consider the secrets that are installed in the
// operator namespace because plugins need to be deployed
// in the same namespace as the operator.
if obj.GetNamespace() != r.OperatorNamespace {
return nil
}
[...]
client.InNamespace(r.OperatorNamespace),
The vendored manifest installed barman-cloud into cnpg-system, which
the operator (running in database) was blind to — hence the
phaseReason "Unknown plugin: 'barman-cloud.cloudnative-pg.io'. This
may be caused by the plugin not being loaded correctly by the operator."
Rewrote the manifest to target `database` (12 namespace fields
flipped). Removed the now-orphan cnpg-system namespace definition.
Reverted the temporary --log-level=debug arg on the operator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Temporary — needed to see why the plugin controller is silent on the
barman-cloud Service. Revert once discovered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CNPG operator refused to reconcile Clusters that reference the
Barman Cloud plugin with:
"Cluster cannot proceed to reconciliation due to an unknown plugin
being required"
Its loaded config showed includePlugins="" — the operator's plugin
allowlist was empty. Setting INCLUDE_PLUGINS in the controller-
manager ConfigMap (via Helm values.config.data) lets the operator
recognize and wire up the plugin's Service (cnpg-system/barman-cloud)
discovered via its cnpg.io/pluginName label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds per-cluster ObjectStore CRs pointing at Garage's k8s-cnpg bucket
(subpaths ak-pg/ and pprl-pg/) with gzip+AES256 for both WAL and data,
and a 30-day retention policy. Credentials stored as SOPS-encrypted
Secrets (barman-cloud-creds) in each namespace.
Each Cluster adds spec.plugins referencing barman-cloud.cloudnative-pg.io
as the WAL archiver pointing at its ObjectStore. Bootstrap remains
initdb for now (existing clusters ignore bootstrap changes) — switching
to bootstrap.recovery for disaster-recovery bootstrap is a follow-up
once the first base backup is confirmed in Garage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendored upstream manifest from
github.com/cloudnative-pg/plugin-barman-cloud releases/download/v0.12.0/manifest.yaml.
Installs the plugin controller into cnpg-system namespace with its
ObjectStore CRD (barmancloud.cnpg.io/v1) and cert-manager-issued
self-signed client/server TLS. cnpg-system namespace added to the
namespaces manifest.
Paired ObjectStore CRs + Cluster spec wiring for ak-pg / pprl-pg
come in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`kubectl get backup` was resolving to `backups.postgresql.cnpg.io`
(CNPG) instead of `backups.velero.io`, so the backup-phase and
item-count checks timed out against a non-existent CNPG Backup
even though Velero's own backup Completed (restore worked, marker
survived). Fully-qualifying to backup.velero.io / restore.velero.io.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a CSI snapshot data movement roundtrip to e2e-test.sh:
- BSL default reports Available
- node-agent DaemonSet has 3 pods ready
- VolumeSnapshotClass ceph-block-snapclass labelled for Velero
- Disposable velero-e2e namespace writes a marker into a ceph-block PVC
- Backup with snapshotMoveData=true reaches Completed with >=1 item
- Namespace deleted, Restore reaches Completed
- Restored pod comes back and the marker file matches
Left-behind state on pass is cleaned up in-line; on failure the Backup
and Restore YAML are dumped into check.out for inspection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two blockers surfaced when the Velero HelmRelease reconciled:
1. velero namespace defaulted to `baseline` PSS — node-agent
DaemonSet needs hostPath mounts for /var/lib/kubelet/{pods,plugins}
and was rejected with PodSecurity violations. Labelled the
namespace `privileged` to match the rook-ceph/crowdsec/ingress
pattern.
2. The cluster had no `snapshot.storage.k8s.io` CRDs — Rook-Ceph
CSI does not ship them. Added the piraeus snapshot-controller
chart in kube-system; it provides the VolumeSnapshot,
VolumeSnapshotContent, and VolumeSnapshotClass CRDs plus the
controller that reconciles them. Required for Velero CSI
snapshot data movement and for our rook-ceph VolumeSnapshotClass
manifest to apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Velero via HelmRelease in a new `velero` namespace, configured
for CSI snapshot data movement to the Garage S3 endpoint on nas-1
(k8s-velero bucket). Credentials SOPS-encrypted via the repo's age
recipient; AWS plugin loaded as an initContainer; node-agent enabled
for data movement and FSB fallback; native VSL snapshots disabled in
favour of the CSI path.
Paired with a Rook-Ceph VolumeSnapshotClass for RBD (default) and
CephFS — labelled for Velero auto-discovery. Adds a `velero`
namespace definition so Flux has somewhere to put everything.
No checksum gymnastics required on Garage today, but keeps the
aws-sdk-go-v2 `checksumAlgorithm=""` workaround documented in-place
against regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
no_log: true hid the roundtrip's actual error. Now a separate
network-probe task (DNS/TLS/cert, no secrets) runs first with full
logging, then the secret-bearing aws-cli roundtrip runs with no_log
but a debug task prints only its stdout/stderr lines so errors
surface without leaking env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reads the velero-key creds from nas-1, then delegates a full S3
roundtrip (list/put/list/get+diff/delete) to the CI runner against
https://s3.nas-1.renner.uno. Exercises DNS, reverse proxy, TLS cert,
and Garage's path-style S3 API end-to-end. Secrets kept out of the
log via no_log.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends install-garage-synology with an idempotent step that creates
three buckets (k8s-velero, k8s-cnpg, k8s-secrets), a dedicated key
per bucket (velero-key, cnpg-key, secrets-key), and the matching
bucket allow grants. Garage does not expose an access-key secret
after creation, so the secrets are captured at create time and
persisted to chmod-600 files under /volume1/docker/garage/config/keys/
on nas-1 for later retrieval via sudo cat. no_log: true keeps them
out of the pipeline log.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Created by an unsanctioned agent session; not the intended deployment
path for Ansible playbooks in this repo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Needed for Synology playbooks — DSM does not support SSH key auth for
the ladmin user, so nas-1 host_vars sets ansible_ssh_pass. Without
sshpass in the CI container, those plays fail before the first task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-node Garage in DSM Container Manager as the on-prem S3 target
for Velero and the CNPG Barman Cloud Plugin. SQLite engine over LMDB
avoids unclean-shutdown corruption risk given replication_factor=1
with no peers to rebuild from. Ports bound to loopback; DSM reverse
proxy terminates TLS. Bucket + key bootstrap is a one-shot manual
step post-install.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
infrastructure-ceph-cluster was reporting Ready as soon as the
CephCluster/CephBlockPool/CephFilesystem manifests applied, because
kstatus can't interpret Rook CR conditions by default. Downstream apps
(which `dependsOn: infrastructure-ceph-cluster`) then reconciled too
early, scheduled pods, and crash-looped init containers against storage
that wasn't yet serving I/O — resolving eventually, but noisily.
Add healthCheckExprs that block the Kustomization until each Rook CR's
status.phase == 'Ready'. Apps now wait for a truly-healthy Ceph.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rebuild-k8s pipeline was hanging on the destroy phase of `tofu apply
-replace` because BPG issues a graceful shutdown and waits up to 1800s
(provider default `timeout_shutdown_vm`) for Talos guests that don't
respond. Setting `stop_on_destroy = true` on all 4 K8s VMs switches the
provider to `qm stop` (hard power-off), so destroy is immediate.
- Module gains `stop_on_destroy` variable (default false preserves
existing behavior for every other VM).
- Opted in on k8s-1/2/3 (controlplane) and k8s-edge (worker).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the bootstrap.sh kubectl-taint hack and fixes it properly in
the Talos machine config. Talos machine.nodeTaints is silently
ineffective on workers — NodeRestriction admission blocks post-join
taint writes by worker kubelets. Kubelets CAN register with taints via
the --register-with-taints flag, so pass it through kubelet.extraArgs
and the taint lands at node registration instead of post-hoc reconcile.
Also drops the workload-class nodeLabel (same NodeRestriction issue:
arbitrary-namespace labels aren't kubelet-settable); only the
kubernetes.io zone label stays since that one is on the allow-list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Talos nodeTaints in the per-node config patch don't always take effect
on first kubelet registration (pipeline 508 saw k8s-edge come up with
zero taints despite talosctl apply-config succeeding). Result: every
stateful workload the scheduler cared about landed on the edge node —
Grafana, Prometheus, Loki, Authentik pods, etc.
Apply taints + labels via kubectl after the node is Ready, which
survives future reconciliation regardless of how Talos applies the
config patch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First measurement on a 70 GB library took ~70 min end-to-end: 17 min
LXC tar + upload, 52 min S3 download + extract into PVCs. 30 min was
too tight — e2e would fire before the restore job wrote its sentinel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The namespace had no CNP, but ingress-nginx's egress policy is an
allow-list that only permits specific target namespaces. Requests to
audiobooks.k.renner.uno hit 504 because Cilium dropped ingress-nginx →
audiobookshelf-pod connects.
Adds the same default-deny-ingress pattern used by paperless/authentik
and extends ingress-nginx-egress with the new namespace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a 4th Talos node (VMID 153, k8s-edge) outside the Ceph quorum
and general scheduling pool. Tainted workload-class=pinned:NoSchedule so
only tolerating pods schedule here; intended for Home Assistant (Z-Wave +
EDI21 USB sticks) and Music Assistant (latency-sensitive; moved out of
the HA VM originally because in-VM scheduling caused audio stutters).
Rationale: NoSchedule-tainting one of the existing 3 nodes would shrink
general+Ceph capacity from 3 to 2 and compromise redundancy for every
other workload. A dedicated node keeps the symmetric 3-node layout
intact, and USB passthrough pins this node to a physical host anyway so
non-HA is the correct model.
target_node is a placeholder until the Proxmox host carrying the USB
dongles is chosen; module already ignores node_name. USB passthrough
(usb = [...]) and the workload manifests get added in follow-up commits.
bootstrap.sh WORKERS array and e2e-test.sh 4-node check already landed
with the Audiobookshelf commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
regex_search(..., multiline=True) returns a list, so the tar command
was expanding to tar -C ['/usr/share/audiobookshelf/config']. Drop the
auto-detection and just pin the Debian defaults — the LXC's
/etc/default/audiobookshelf confirmed those are the values in use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the Paperless pattern: new namespace, RWX PVCs, OBC-based S3
migration bucket, trigger-backup-job that kicks a Woodpecker pipeline,
restore-data-job that streams tarballs directly from S3 into PVCs.
The Deployment uses an initContainer + sentinel file so the app never
boots on an empty /config while the SQLite tarball is still extracting,
and the restore job is idempotent (skipped when the sentinel exists).
Also extends bootstrap.sh wait-apps + e2e-test.sh to cover the new
Kustomization, OBC, TLS cert, and /healthcheck on audiobooks.k.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Flip all ingress cert-manager.io/cluster-issuer annotations from
letsencrypt-prod to letsencrypt-staging. Five rebuilds in a week
tripped LE's per-identifier rate limit (5 certs / 168h) and left 3
hostnames with the ingress-nginx fake cert. Staging has ~30k/week,
so rebuilds stop being rate-limited. ClusterIssuer definitions for
both remain in post-crds; flip back per-ingress when ready for prod.
- Rework e2e-test.sh TLS check to validate the cert *issuer* contains
"Let's Encrypt" rather than chain-validating against the public CA
bundle. Same check works for both staging and prod, and still
catches the real failure mode (fake cert = cert-manager broken).
- HTTP checks now use -k since staging certs aren't publicly trusted;
the separate TLS check is what asserts cert-manager worked.
- Chain the e2e step onto .woodpecker/rebuild-k8s.yml so every rebuild
validates the resulting cluster end-to-end before reporting success.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a manual Woodpecker pipeline (`PIPELINE=e2e-test`) that drives a
real test of the cluster end-to-end, beyond what `rebuild-k8s`'s
wait-for-reconciliation covers:
- All nodes Ready; all Flux Kustomizations and HelmReleases Ready; zero
failing pods cluster-wide.
- Ceph: CephCluster Ready, 3 OSDs Running, CephObjectStore Ready.
- Storage: provisions a `ceph-block` PVC, mounts it in a short-lived
pod, writes+reads a file to prove the CSI path works, then cleans up.
- OBCs: authentik + paperless migration buckets Bound.
- Databases: CNPG ak-pg and pprl-pg both have 2 ready instances.
- TLS: validates each ingress hostname's cert with OpenSSL against the
public CA bundle (no `-k`) so we actually catch fake certs / LE
failures.
- App health endpoints: Authentik live/ready probes, Grafana /api/health
reports database=ok, paperless/docs reachable (not 5xx).
- External DNS: verifies public resolution via 1.1.1.1.
Script lives at kubernetes/e2e-test.sh so it can also be run locally
once kubeconfig is loaded. The pipeline decrypts kubeconfig.sops from
kubernetes/access/ (committed by rebuild-k8s) and runs it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop duplicate HelmRepository external-dns in apps namespace (owned by
infrastructure/external-dns-unifi) to end SSA ownership flapping.
- Infrastructure Kustomization timeout 5m -> 15m; first-boot Rook +
cert-manager + cnpg + cilium do not fit in 5m on a cold cluster.
- authentik-db timeout 10m -> 30m; the restore job polls S3 for up to
10min and needs margin against the trigger-backup job.
- Remove orphaned sealed-secrets / external-secrets / metallb dirs that
were never referenced (Flux decrypts SOPS natively; Cilium handles L2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the OBC CEL health check lived on infrastructure-ceph-cluster,
which holds OBCs for every app. A slow bind in one namespace stalled
every other app that depends on infrastructure-ceph-cluster. Move each
OBC into its own app Kustomization (paperless-db, authentik-db) and
apply the health check there so each app only waits on its own bucket.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
