pg_restore commits the data; the CNPG Backup CR is async work for the
operator. The Job doesn't need to wait for it. Drops the 1h–2h poll
loop that was the likely cause of the immich-seed Pod terminating
mid-poll on the first cold-start run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the split seed-barman / seed-pvcs flow in migrate-paperless,
migrate-immich, and migrate-authentik-db with a single in-cluster Job
per app. Each Job gates on an existing completed CNPG Backup
(no-op if already seeded), rsyncs legacy PVCs into the live cluster,
pg_restores the legacy DB into the live CNPG cluster, and creates a
fresh CNPG Backup so future cold-start bootstrap.recovery has a base
backup. Pipelines collapse to "stage SSH secret, quiesce, apply Job,
wait Complete".
Adds Flux Kustomizations for each seed (kubernetes/clusters/homelab/
<app>-seed.yaml) plus SOPS-encrypted SSH secrets, so a fresh
namespace post-rebuild auto-triggers the seed without operator
intervention. App Kustomizations gain <app>-seed in dependsOn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Delete kubernetes/apps/authentik/postgres.yaml: orphan CNPG Cluster with
bootstrap.initdb and no plugin declaration; would compete with the
working ak-pg in authentik-db/ if accidentally applied.
- Delete kubernetes/apps/owncloud/: subtree never wired into Flux (no
Kustomization in clusters/homelab/ targets it), would not deploy.
- Drop CLAUDE.md: superseded by ~/.claude/CLAUDE.md global instructions.
- rebuild-k8s.yml: track per-module retry failures and exit 1 if any
module didn't recover; add set -e to disk-wipe step. Stops masking.
- import-cluster-access.sh: drop "|| true" on git checkout so missing
access files surface immediately instead of silently continuing.
- Add scripts/trust-letsencrypt-staging.sh: helper to install Let's
Encrypt staging roots into system or Firefox trust stores.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ghcr.io/opentofu/opentofu:1.11 has an ONBUILD that prints a deprecation
warning and runs `exit 1` — the maintainers explicitly disabled using
that image as a base in 1.10+ (opentofu/opentofu#1931). Build pipeline
810 hit it with `error building stage: ... exit status 1`.
Switch the base to alpine:3.21 and install the tofu binary directly
from the GitHub release zip (TOFU_VERSION=1.11.0 ARG, matches what the
pipeline previously pinned).
deploy-infrastructure was the one pipeline still installing
ansible/proxmoxer/galaxy collections at job time, because tofu apply
invokes ansible playbooks via module ansible_playbook provisioners —
both tools have to be in the same image.
ci-tofu inherits FROM ghcr.io/opentofu/opentofu:1.11 and bakes ansible
+ community.{proxmox,docker} collections + python deps + utility CLIs
on top. ENTRYPOINT/CMD reset so Woodpecker can run shell commands.
Both plan and apply steps swap to ci-tofu:2026-04-29. Drops apk add
git from plan and apk/pip/galaxy from apply.
Drops the per-job apk/pip/galaxy/curl-kubectl/curl-velero/curl-flux
preamble across 14 pipelines. Setup phase eliminated; only the
credential-staging steps remain. Pilot on update-all (commit b990749)
proved the swap mechanics; this completes the rollout.
ci-k8s gains velero (v1.16.0 ARG) so the migrate-* seed-pvcs steps
don't have to download it on every run.
ci-ansible swaps:
- update-all (already landed in b990749 — listed for completeness)
- install-{garage-synology,woodpecker-server,registry-mirror}
- deploy-caddy
- test-garage-smoke
- migrate-{immich,paperless,authentik-db}.backup-legacy-db
- rebuild-k8s.tune-cp-zvols
ci-k8s swaps:
- migrate-{influxdb,immich,paperless,music-assistant,audiobookshelf}.seed-pvcs
- migrate-authentik-db.wipe-and-restore
- e2e, e2e-test
- rebuild-k8s: talos-bootstrap, wipe-osd-disks, export-access,
flux-bootstrap, wait-apps, e2e-test
Left alone:
- deploy-infrastructure (special tofu+ansible mix on tofu image)
- migrate-tfstate (alpine + aws-cli/jq is fine)
- build-caddy (golang + alpine for release)
- security-scan (trivy)
- migrate-*.seed-barman steps (postgres-version-pinned images)
Image refs pin :2026-04-29; pipelines also use shell `base64 -d` in
place of the prior `python3 -c "import base64..."` (matches the
deploy-infrastructure.yml pattern).
Pilots the swap from alpine:3.21 + apk/pip/galaxy preamble to the
prebuilt ci-ansible image. Pinned to :2026-04-29 so a future image
rebuild doesn't change behavior here without an explicit tag bump.
Also replaces the python3 -c base64 decode with shell base64 -d,
matching the pattern in deploy-infrastructure.yml.
804 paperless tried `kubectl delete restore "${SCHED}-bootstrap"` and
saw `kubectl delete restore -bootstrap` (interpreted as a flag). The
braced \${SCHED} form is interpolated by Woodpecker's own template
engine BEFORE the YAML reaches the shell — same bug class as the
earlier session lesson on `${VAR}` eating env vars.
s/\${SCHED}-bootstrap/\$SCHED-bootstrap/ across all 6 migrate-*.yml.
The bare \$SCHED-bootstrap form is unambiguous (no following
alphanumeric to extend the var name) and is what bash actually sees.
First run (pipeline 805) hit Docker Hub's anonymous rate limit pulling
alpine:3.21 directly from index.docker.io. The homelab already runs a
pull-through cache at registry-mirror.srv.renner.uno:5000 (LXC 250) for
exactly this. Wire kaniko through it via --registry-mirror; the mirror
serves plain HTTP, so --insecure-registry is paired.
apk add + pip install + ansible-galaxy ran on every job across ~12
pipelines and dominated wall time. Bake the toolsets into two images
pushed to Gitea's container registry; consuming pipelines pull instead.
ci-ansible: ansible + community.{proxmox,docker} + proxmoxer/requests
+ utility CLIs (sops, aws-cli, jq, sshpass, ...).
ci-k8s: talosctl (pinned to v1.12.6), kubectl (stable at build
time), flux + sops/openssl/bind-tools.
Tags YYYY-MM-DD plus :latest. Build pipeline triggers on changes under
ci/images/** or manual PIPELINE=build-ci-images.
paperless 803 produced a Backup that ended in
WaitingForPluginOperationsPartiallyFailed because the namespace's
CNPG-managed pprl-pg-1 PVC was Pending (CNPG plugin connection broken
on the live cluster) and Velero couldn't snapshot it. The wipe-and-
restore pre-flight then refused to proceed (`bphase != Completed`).
The live paperless-daily / immich-daily Schedules already exclude CNPG
PVCs via the labelSelector `cnpg.io/cluster DoesNotExist` (CNPG handles
those from Barman bootstrap.recovery, not from Velero). The migrate
pipelines weren't applying the same selector, so they were trying to
back up state Velero shouldn't be touching in the first place.
Add `--selector '!cnpg.io/cluster'` to both. authentik-db already had
this. influxdb / music-assistant / audiobookshelf don't have CNPG, so
no change there.
Mirror of install-registry-mirror.yml — manual-only, runs the existing
ansible/plays/install-woodpecker-server.yml playbook. Needed to apply
the SQLite WAL/busy_timeout datasource change (and the LXC memory bump
that lands via deploy-infrastructure won't trigger the playbook on
in-place memory edits).
- vm_113_netbox.tf removed. Apply 797 hit a bpg validator error on
the netbox in-place update ("Defined disk interface not supported.
Interface was , but only [ide sata scsi virtio] are supported"),
but the resource is being decommissioned anyway, so skip the
schema fix and let the next apply destroy the VM.
- backup_jobs.tf: prune VMIDs whose .tf files were deleted earlier
this session (112 ha-dev, 113 netbox, 124 kasm, 125 dev, 223 ollama,
235 wiki, 236 owncloud, 240 vault). Keeping them in the include
list would error on the next vzdump cycle once the VMs are gone.
Phase 4's cutover to Garage made backend + aws-provider share the
same AWS_ACCESS_KEY_ID env var, which now points at Garage. The aws
provider talks to RustFS (still — for caddy_certificates / migration /
tfstate buckets), and Garage credentials don't authenticate against
RustFS, so plan 794 regressed: refresh of the three aws_s3_bucket
resources returned "doesn't exist" → plan proposed to recreate them.
Give the aws provider explicit credentials via two new TF_VARs sourced
from the still-existing tf_rfs_* Woodpecker secrets. Backend keeps the
env-var path with Garage creds; aws provider gets RustFS creds via
TF_VAR. No state shape change.
This is a transitional separation. When RustFS is fully decommissioned,
the aws provider's endpoint also moves to Garage and the two cred
sources can collapse back together.
Phase 4 of the migration off RustFS. State file was copied + sha256
verified by pipeline 793. Switch:
- providers.tf backend "s3": endpoint http://s3.renner.uno:9000 →
https://s3.nas-1.renner.uno, region main → garage.
- deploy-infrastructure.yml (validate-secrets / plan / apply) and
rebuild-k8s.yml (recreate-vms): tf_rfs_access/tf_rfs_secret →
tf_garage_access/tf_garage_secret.
The aws provider block in providers.tf still points at RustFS — that
manages aws_s3_bucket.caddy_certificates which lives on RustFS, and
migrating that bucket is a separate concern.
This push will auto-trigger deploy-infrastructure plan against Garage;
expected to produce the same plan output as pipeline 787 (cleanup-only
baseline) plus the still-unfixed 8-resource provider misread, since
the actual rm+import payload couldn't persist last time and needs to
be re-run with a writable backend.
Phase 3 of the OpenTofu backend migration off RustFS. Reads source
state from RustFS (still works — only writes are wedged on quorum
failure), validates as JSON, writes to Garage, reads back, compares
sha256+bytes. Source is left in place on success so Phase 4
(providers.tf cutover) can be rolled back by flipping endpoint back
to RustFS if needed.
Manual-only, scoped via PIPELINE=migrate-tfstate. Aborts if the source
state is implausibly small or doesn't parse as the expected JSON shape.
Step 1 of migrating the OpenTofu S3 backend off RustFS (which has been
hitting "Quorum not reached" lock failures during pipeline 790's state
writes). Garage is single-node on nas-1 — same machine the K8s backups
already use — and has a clean idempotent bootstrap path via
`bootstrap_pair`.
After this, run install-garage-synology manually once. The new
credentials land in /volume1/docker/garage/config/keys/tfstate-key.creds
on nas-1 (chmod 600, root-owned). They feed into Woodpecker secrets
`tf_garage_access` / `tf_garage_secret` which the deploy-infrastructure
pipeline will pick up after the providers.tf backend swap.
pvesh /cluster/resources confirms neither VMID is in the cluster
anymore. Same shape as the prior 5-file cleanup: refresh marks them
"has been deleted", removing the .tf lets state drop them cleanly on
next apply.
Leaves 8 misread resources still in scope (gitea, audiobookshelf,
authentik_app_1/2, authentik_db_1, loki, registry_mirror, wp-agent-2),
all now confirmed migrated to nodes other than their .tf
target_node — fix is rm+import in a single pipeline run with current
node mapping.
Earlier theory was that 0.104 broke reads for 10 resources. User
clarified: those resources had real out-of-band changes (storage moved,
HA migrations between nodes) — state is stale relative to where the
resources actually live now, so the provider's `<node>/<vmid>` read
404s. Pinning the provider does not fix that; the state itself needs
to be re-imported from current locations.
Putting the constraint back to `~> 0.95` so init-upgrade keeps pulling
the latest within 0.x. The 5-file resource cleanup stands.
Plan 785 failed: 0.95.x doesn't have `proxmox_backup_job` (added in
0.96.0), so backup_jobs.tf wouldn't compile. The checked-in
.terraform.lock.hcl shows 0.96.0 as the last-good version, which the
pipeline silently moved off of via `tofu init -upgrade`. Bumping the
floor to 0.96.0 keeps the resource available while still excluding
the 0.104 read regression.
Pipeline 781/784 plans showed 16-add/3-change/0-destroy with the bpg
provider reporting 15 resources as "has been deleted" out-of-band.
Investigation found two distinct problems:
1. Five resources (wiki #235, vault #240, kasm #124, dev #125,
homeassistant-dev #112) were genuinely removed from Proxmox — the
provider read is correct for those. The .tf files were left behind,
so plan kept proposing to recreate them. Drop the files.
2. Ten resources (gitea, audiobookshelf, authentik_app_1/2,
authentik_db_1, loki, ollama, owncloud, registry_mirror,
wp-agent-2/911) are alive and serving traffic — gitea HTTP 200,
the rest resolve via the Proxmox-API-backed DNS — but bpg/proxmox
v0.104.0 (which `tofu init -upgrade` pulled under `~> 0.95`)
marks them as deleted on read. Same plan pre-fix produced the same
misread, so the regression sits between 0.96 and 0.104. Pin to
`~> 0.95.0` to restore the documented baseline; revisit upgrading
when there is a specific reason to.
The TOFU_IMPORT loop has been broken since it was added: Woodpecker's
variable interpolation eats `${line%% *}` and `${line##* }` at YAML
compile time, so the per-line resource/id parsing always evaluated to
empty strings. The compiled shell read literally:
resource=""
id=""
tofu import "" "" # → "Variable name required"
The `|| true` masked it, so plans appeared to succeed and nothing got
imported. Confirmed end-to-end on pipeline 782 (15-line TOFU_IMPORT
intended to backfill orphan LXC/VM resources — every line errored
identically).
Fix is two-line: escape with `$$` in both plan and apply blocks so
Woodpecker emits literal `${line%% *}` for the runtime shell.
Woodpecker server was hitting `database is locked` cascades under
concurrent pipeline load (multi-step pipelines + metrics + cron + queue
ops contending on the same SQLite handle). Failed writes turned into
`sql: no rows in result set` for queue acks and lost step UUIDs, which
cancelled otherwise-healthy long-running steps with `queue: task
expired` server-side. Symptom hit migrate-paperless 778-780 every time
seed-barman crossed the 6-10 min mark.
Two changes, applied together:
- LXC 231 memory 1 GiB → 4 GiB. SQLite mmap thrashes under low memory
with concurrent readers/writers; bumping leaves room for the page
cache to absorb contention.
- WOODPECKER_DATABASE_DATASOURCE now sets _journal=WAL,
_busy_timeout=10000, _synchronous=NORMAL on the SQLite connection.
WAL lets readers continue while a writer holds the lock; the busy
timeout makes contending writers wait 10 s instead of failing
immediately.
Apply via deploy-infrastructure pipeline (memory) and
install-woodpecker-server (env file + container restart).
task-expiration
778/779 paperless seed-barman steps both died ~6-10 min into the
barman-cloud-backup invocation with no error in the log. The cancel
came from the Woodpecker server side: `rpc error: code = Unknown desc =
queue: task expired`. Same failure class as the earlier
woodpecker-step-heartbeat-prevents-silent-kill lesson — barman is
silent for the entire upload (no stdout while compressing/encrypting/
streaming the base backup), so the task gets killed.
Run barman-cloud-backup in the background and emit a one-line
heartbeat every 30 s while it's alive. Capture its real exit code via
`wait "$BCB_PID"` so a true non-zero failure still aborts the step
(no `|| true` masking).
Applied to all 3 pipelines that invoke barman-cloud-backup
(paperless, immich, authentik-db). Variable name uniformity ($BCB_PID,
$BCB_START, $BCB_RC) keeps the wrapper copy-paste-identical.
777 failed in seed-pvcs because paperless was already in a broken-
namespace state — Flux Kustomization Ready=False (dependsOn paperless-db
was unhealthy due to a CNPG plugin NetworkPolicy denial), so manifests
including the Deployment had never been applied. `kubectl scale
deployment paperless --replicas=0` returned NotFound and `set -eu`
aborted the run before we could even reach the wipe-and-restore tail
that would have recovered the namespace.
The whole point of wipe-and-restore is that the namespace gets recreated
clean from manifests + Backup, so the pre-quiesce step shouldn't gate
on a Deployment that may legitimately not exist yet. Tolerate the
NotFound and proceed; if a Pod is running it'll be caught by the
namespace-cascade-delete in the wipe step.
Applied to all 5 PVC pipelines (audiobookshelf, paperless,
music-assistant, influxdb StatefulSet, immich).
Add a wipe-and-restore step matching the other migrate-* pipelines:
take a fresh authentik-daily Velero Backup (same selector as the live
Schedule — CNPG-managed PVCs excluded since CNPG bootstrap.recovery
handles those from Barman), wipe the namespace, force-reconcile
velero-bootstrap-restore, and wait for the Restore to land.
When the namespace is recreated by Flux:
- Velero's authentik-daily-bootstrap Restore brings back any non-CNPG
PVCs (currently none in this homelab — Authentik is otherwise
stateless, so the Backup is empty but still drives the chain)
- The ak-pg CNPG Cluster CR is recreated; CNPG operator runs
bootstrap.recovery against k8s-cnpg/ak-pg, pulling the Barman seed
produced earlier in this same pipeline
- Authentik server/worker Deployments come up against the fresh DB
A green pipeline now proves both seed paths end-to-end without a
cluster rebuild. Same 4 h Job-wait, same dependsOn-race fix as the
PVC migrations.
Roll the wipe-and-restore tail proven on migrate-audiobookshelf to
migrate-paperless / migrate-music-assistant / migrate-influxdb /
migrate-immich. Same shape, same parameter source ($APP/$NS/$SCHED/
$BACKUP_NAME), same 4 h Job-wait cap.
Each pipeline now:
1. Writes the seed Backup (existing behaviour, unchanged)
2. Verifies Backup phase=Completed
3. Deletes the bootstrap-restore Job + the stale ${SCHED}-bootstrap
Restore (idempotency-buster)
4. Deletes the namespace (cascades pods/PVCs; for paperless/immich
this also removes the CNPG Cluster CR so the DB is round-tripped
through Barman bootstrap.recovery)
5. Force-reconciles velero-bootstrap-restore Kustomization (annotate
reconcile.fluxcd.io/requestedAt) so Flux drops Ready=True before
the app Kustomization's dependsOn check
6. Waits for the recreated Job to Complete (which itself blocks on
`velero restore create --wait` Phase=Completed → PVCs populated)
7. Sanity-checks the Restore phase before returning
A green pipeline now proves the production restore chain reads the
just-written seed end-to-end — no separate cluster rebuild needed.
audiobookshelf-side parameterisation drops the hard-coded restore
name in favour of "${SCHED}-bootstrap" so a future schedule rename
doesn't require pipeline edits.
migrate-authentik-db is intentionally not touched: it writes only a
CNPG/Barman seed (no Velero Backup), so the pre-flight on
$BACKUP_NAME doesn't apply. Treat as a follow-up.
776 timed out 10 s before the bootstrap-restore Job's Restore landed
Phase=Completed: the books DataDownload (75 GiB at ~40 MiB/s from
Garage NAS) needed ~31 min, just over the 30 min cap. Restore was
otherwise healthy — Backup Completed, Restore Completed seconds after
the pipeline gave up, all PVCs Bound.
4 h matches the rsync and Backup-poll waits already in this script and
gives plenty of headroom for the immich-class workloads when the same
pattern is rolled to other migrate-* pipelines.
The bootstrap-restore Job is idempotent: it skips schedules whose
stable Restore CR is already Completed/PartiallyFailed. After this
pipeline deletes the old Backup, the existing audiobookshelf-daily-
bootstrap Restore (Phase=Completed but pointing at the deleted Backup)
makes the new Job exit without re-running the restore — and the test
trivially "passes" while leaving PVCs empty.
Drop the stale Restore alongside the Job before force-reconciling, so
the new Job sees no existing Restore and creates a fresh one against
the just-completed Backup.
774 produced an "initial setup" audiobookshelf because the
audiobookshelf Kustomization's dependsOn check on velero-bootstrap-
restore read a stale Ready=True (cached from before the Job delete) and
proceeded to create empty PVCs ahead of Velero's Restore. With
existing-resource-policy=update Velero only patches metadata, so the
PVCs stayed empty and the app came up uninitialised.
Fix: after deleting the namespace + Job, force-reconcile the velero-
bootstrap-restore Kustomization (annotate reconcile.fluxcd.io/
requestedAt) so its Ready drops, then wait for the recreated Job to
Complete. The Job's `velero restore create --wait` already blocks on
Phase=Completed, so Job-Complete = PVCs populated. Sanity-check the
audiobookshelf-daily-bootstrap Restore phase before exiting. Trap then
unsuspends the audiobookshelf Kustomization and Flux reconciles
against PVCs that already have data.
Replace the in-script delete-PVCs + velero-restore-create dance (which
kept racing with Flux's reconcile loop) with a one-liner: delete the
namespace + delete the velero-bootstrap-restore Job. Flux then
repopulates through its existing dependency chain — bootstrap-restore
Kustomization recreates the Job, the new Job submits the Restore from
the freshly-completed Backup, audiobookshelf Kustomization sees the
restored PVCs and rebuilds the namespace.
This is the same code path a cluster rebuild uses, so a green pipeline
proves the production restore chain works end-to-end on this Backup.
No more ad-hoc PVC deletion, no more recover_pvcs racing with
pvc-protection finalizers, no more scale=0 vs Flux suspend race —
namespace cascade handles all of it cleanly.
772 left the app Deployment Pod (audiobookshelf-76f6dddbfd-rxpks) Running
with the PVC mounted at the start of the round-trip section, blocking
PVC delete via pvc-protection finalizer. Initial `scale --replicas=0`
+ `wait --for=delete` happens at script start, but Flux's reconcile
loop has a race: the suspend patch and the scale=0 land in the same
moment, and Flux can reapply replicas=1 from manifest before suspend
takes effect.
Fix: re-issue scale=0 immediately before the round-trip wipe and use
a strict wait (no `|| true`). If a Pod still won't terminate within
180s, surface the Kustomization yaml + pod listing for diagnosis and
recover_pvcs before exiting non-zero.
Previously, if the round-trip Restore failed (Velero down, BSL unreachable,
Restore Phase=Failed, 4 h timeout, or post-restore PVC not Bound) the
script exited with the audiobookshelf PVCs deleted and Flux suspended —
leaving the namespace structurally broken. The trap unsuspends Flux but
that does not recreate the deleted PVCs in time, so the Deployment ends
up referencing a non-existent claim. 770 hit this exact mode and lost
audiobookshelf-config until manual recovery.
Now: every failure path after `velero backup get` confirms
Phase=Completed re-applies kubernetes/apps/audiobookshelf/pvcs.yaml so the
namespace lands back at "Flux-managed structural PVCs" before exiting.
Data is still gone until a successful Restore brings it back, but the
cluster shape is intact and the next migrate run (or rebuild) can repair.
Also: hard pre-flight check refusing to delete the only data copy if
Backup phase ≠ Completed.
770 timed out on `kubectl delete pvc --wait=true` because the rsync seed
Pod (status=Succeeded but still in the namespace) held the PVCs'
pvc-protection finalizer. Synchronously delete the Job + Pod first so
the finalizer clears, then PVC delete is instant.
Also surfaces describe + pod listing on PVC delete failure so we don't
have to chase the lock holder by hand next time.
After the seed Backup lands Phase=Completed, wipe the 3 audiobookshelf
PVCs and create a Velero Restore against the new Backup. Catches an
unreadable seed Backup the moment it happens, not at the next cluster
rebuild — and validates the shallow-snap → Kopia → restore chain
end-to-end on a real workload without taking down the cluster.
Flux stays suspended through the wipe/restore (existing trap re-enables
it on EXIT), so PVCs are recreated by Velero with their Kopia-restored
contents before the Deployment scales back up.
Drops the explicit image.repository override on the rook-ceph-operator
HelmRelease so it uses the chart default (docker.io/rook/ceph).
The override was added when we hit Docker Hub anonymous rate-limiting
during cold rebuilds. That risk is now absorbed by the local
registry-mirror (LXC 250) — rate limits bite at most once per image, on
the mirror's first cold-cache fetch.
Quay's S3-backed CDN turned out to be much worse than Docker Hub for
this workload: rebuild #764 hit a TLS handshake timeout to
cdn01.quay.io (S3 redirect), and rebuild #768 saw the mirror's first
cold quay fetch take longer than Helm's 5min install window, kicking
off a retry loop where every cycle wiped the in-progress pull. The
docker.io path through the same mirror has been reliably fast all
session.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ansible ignores the in-repo ansible.cfg in CI because the checkout dir
is world-writable, so the role search path defaults wipe out
roles_path=../roles:./roles. Pipeline 766 failed with
"the role 'common' was not found".
Match update-all.yml's pattern: set ANSIBLE_ROLES_PATH explicitly in the
step env block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-task workflow_dispatch pipeline for re-running the registry-mirror
role on the LXC after config changes (e.g. adding new upstream mirrors).
Modeled after install-garage-synology.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The current registry-mirror only proxies docker.io, so Cilium (quay.io)
and other non-docker.io images get pulled direct from upstream. Rebuild
#764 stalled for ~15 min on a quay.io S3 TLS handshake timeout — the
cluster's containerd has no fallback once upstream times out.
CNCF distribution/distribution can only proxy one upstream per instance,
so the role now runs four containers (docker→5000, quay→5001, ghcr→5002,
registry.k8s.io→5003) on the same LXC, each with its own data dir and
config. Talos config-patches list all four mirror endpoints; containerd
routes pulls accordingly.
Migration: tasks remove the legacy single-upstream "registry-mirror"
container before starting the new "registry-mirror-{docker,quay,ghcr,k8s}"
set so the port 5000 handover is clean.
Takes effect on the next rebuild after install-registry-mirror.yml runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an Ansible playbook that sets ZFS sync=disabled on the zvols backing
the K8s control-plane VM root disks (vm-150/151/152) and wires it into
the rebuild-k8s pipeline between recreate-vms and talos-bootstrap.
Why: the rebuild we just did surfaced multi-second etcd leader-election
stalls and a sustained 100-500 ms apply-request tail caused by fsync
contention on the shared local-zfs pool. Pool is already on NVMe, so a
SLOG on the same device class doesn't help; the fix is to bypass the
sync-write path for these zvols entirely. etcd's WAL fsync returns
immediately and ZFS still flushes via txgs (~5s default), so the worst
case on hard power loss is up to ~5s of writes per CP member.
Tradeoff accepted: 3-member quorum + Velero backups cover the durability
gap. Targeted only at the CP zvols — workers and other VMs untouched.
The playbook auto-discovers vm-15{0,1,2}-disk-N zvols on each PVE host
and is idempotent (skips zvols already at sync=disabled).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Velero's CSI Snapshot Data Movement clones the source PVC into a
backupPVC before Kopia reads. On CephFS that clone is metadata-bound:
pipeline 760 ran ~2.5 h on a 250 GiB immich-library subvolume at
~25 MiB/s while the NAS sat idle the entire time.
ceph-csi 3.10+ supports mounting a snapshot directly as ReadOnlyMany
(shallow read), and Velero 1.15+ supports skipping the backupPVC clone
when configured per source storage class. Cluster runs ceph-csi 3.16.2
and Velero 1.18.0, so both prerequisites are satisfied.
- New ConfigMap velero/node-agent-config sets readOnly:true for both
ceph-block and ceph-filesystem source storage classes
- node-agent picks it up via --node-agent-configmap=node-agent-config
Replace the single-node API endpoint (k8s-1.srv.renner.uno) with a
Talos shared VIP — etcd-elected leader hosts 10.2.40.60 on its virtio
NIC, so kubectl/Flux survive any single CP-node loss.
- Per-CP patches add interfaces[].vip on virtio_net via deviceSelector
- cluster.yaml: controlPlane.endpoint flipped, VIP IP + k8s.renner.uno
added to machine + apiServer certSANs
- bootstrap.sh: K8S_API_HOST drives talosctl gen-config and the
kubeconfig server: rewrite
- New UniFi A record k8s.renner.uno -> 10.2.40.60
Existing apiserver cert doesn't list the new SANs; rebuild-k8s or a
Talos cert rotation is needed before kubectl-via-VIP works.
Also drop vestigial hastate = \"started\" from the three CP VMs — the
proxmox_vm module never wired hastate to a haresource (no-op line),
and PCI passthrough on the Ceph OSD pins these VMs to their host
anyway, so HA migration was never a real option.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ceph health flagged MON_DISK_LOW during migrate-immich pipeline 760
(mon.a 27% free, mon.c 16% free). Mons store growing osdmap +
monstore on the host's /var/lib/rook, which on the K8s VMs is the
single 32 G root disk. Under heavy CSI snapshot / clone load the mon
db grew enough to trip the warning.
Bump all four K8s VMs (k8s-1/2/3/edge) to 64 G so the next rebuild
absorbs this — no in-place expand needed since rebuild-k8s recreates
the VMs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-immich pipeline 759 stalled — Velero CSI Snapshot Data
Movement creates a backupPVC per source PVC by provisioning from the
VolumeSnapshot, and CephFS subvolume clone defaulted to 4 concurrent
threads. With 3 immich PVCs (im-pg-1, immich-ml-cache, immich-library)
plus other in-flight clones, the cloner pool exhausted:
rpc error: code = Unknown desc = rados: ret=-11,
Resource temporarily unavailable: "all cloner threads are busy,
please try again later"
Two backupPVCs (immich-ml-cache, immich-library) sat Pending for 19+
min. Bumping mgr/volumes/max_concurrent_clones to 16 via
CephCluster.spec.cephConfig gives headroom for parallel multi-PVC
backups, including future cluster-bootstrap restores running multiple
schedules in parallel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-immich timed out repeatedly on Velero's data-mover-prepare
phase. While bumping dataMoverPrepareTimeout helps, the underlying
issue is CephFS metadata server thrashing under load:
- Active MDS at 835Mi / 1Gi limit (82%) — millions of small immich
files saturate the inode/dentry cache. CSI VolumeSnapshotContent
provisioning on a stressed MDS goes slow, blowing the 30-min
Velero prepare timeout.
- mgr-a at 1553Mi / 2Gi limit (76%) — also tight.
Bump:
- CephFilesystem metadataServer.resources: 512Mi/1Gi → 2Gi/4Gi
- CephCluster mgr resources: 1Gi/2Gi → 2Gi/4Gi
Hosts have plenty of free RAM (same justification as the 48 GB CP
nodes from commit 16f86e2) so this is a cheap fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7