Stage 1 of the offsite backup migration (paperless-first). Repoint the
pprl-pg ObjectStore from local Garage (s3.nas-1.renner.uno) to Hetzner
fsn1, and switch its credentials to the new hetzner-s3-creds secret.
- destinationPath: s3://k8s-cnpg/pprl-pg -> s3://renner-uno-k8s-cnpg/pprl-pg
- endpointURL: s3.nas-1.renner.uno -> fsn1.your-objectstorage.com
- s3Credentials: barman-cloud-creds -> hetzner-s3-creds
The ObjectStore CRD has no path-style knob; barman/boto3 default (auto)
addressing works against Hetzner's Ceph RGW. Garage creds/secret are left
in place for a one-revert rollback. Validate (WAL archive + base backup +
restore) before broadening to authentik/immich/velero.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New secrets holding the Hetzner Object Storage keypair, ready for the
per-app backup cutover:
- seed/{paperless,authentik,immich}/hetzner-creds.sops.yaml
→ Secret/hetzner-s3-creds (ACCESS_KEY_ID, ACCESS_SECRET_KEY, REGION=fsn1)
- infrastructure/velero/hetzner-creds.sops.yaml
→ Secret/cloud-credentials-hetzner (cloud INI blob)
These are not referenced by any kustomization yet, so they're inert —
the existing Garage creds keep driving backups until each ObjectStore /
BSL is repointed (staged, paperless-first). Also gitignore
*.plaintext.yaml so the pre-encryption template can never be committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The migrate-* PVC-tarball workflow is retired, so renner-uno-k8s-migration
is no longer needed. Remove it; the (empty) bucket is destroyed by the
targeted apply. Remaining backup buckets: cnpg, velero, secrets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First apply created k8s-cnpg/k8s-secrets/k8s-migration but k8s-velero
failed: "The requested bucket name is not available." Hetzner Object
Storage uses a global bucket namespace (virtual-hosted style → buckets
are endpoint subdomains), so generic k8s-* names collide with other
customers. Prefix all buckets with the domain (renner-uno-k8s-*) to stay
unique and consistent. The three already-created plain buckets are empty
and get recreated under the new names by the targeted apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy-infrastructure's `tofu plan` was failing for everyone (not just
this branch): CI runs `tofu init -upgrade`, and the loose `~> 0.41`
constraint resolved to unifi 0.50.0+, which changed the dns_record `ttl`
schema from an integer to a Go duration string. Every plan then died on
the existing integer TTLs:
Error: Invalid Time Duration String Value
unifi_dns_record.renner_uno (ttl = 300)
unifi_dns_record.authentik_db (ttl = 60)
Bisected the provider: 0.49.0 is the last version that accepts integer
TTLs; 0.50.0 introduced the break. Pin to ~> 0.49.0 so the existing
config keeps validating. (A future bump past 0.49 needs those TTLs
rewritten to duration strings first.)
Also regenerates the lock: adds aminueza/minio, drops the stale unused
hashicorp/aws entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the offsite S3 backup target that decouples cluster backups from
nas-1's local Garage, so nas-1 can be powered off nightly to save power
(and as a genuine DR improvement: backups + bootstrap secrets no longer
share a box with, and survive a rebuild of, the cluster).
Hetzner has no API to generate S3 credentials (Console-only, confirmed
empirically against the live hcloud API), so provisioning is two-part:
the keypair is created by hand in the Console, then the aminueza/minio
provider creates the buckets (k8s-cnpg, k8s-velero, k8s-secrets,
k8s-migration) over the S3 API.
- object_storage.tf: minio provider (fsn1) + for_each bucket resources
- variables_hetzner.tf: endpoint/region/keypair vars + enable flag
- providers.tf: aminueza/minio ~> 3.33
- deploy-infrastructure.yml: wire TF_VAR_hetzner_s3_{access,secret}_key
from Woodpecker secrets into plan + apply
Bucket names mirror the current Garage layout so the cluster-side change
is only an endpoint/region/addressing swap (handled in follow-up commits,
staged paperless-first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Moves the static ftp-scanner-config ConfigMap out of ftp-scanner.yaml
into a configMapGenerator in kustomization.yaml. Values are unchanged
(PASV_ADDRESS=10.2.99.82, ports 21100-21104, scanner, uid/gid 1000).
The content-hash name suffix is kept on purpose: vsftpd templates
pasv_address from PASV_ADDRESS at container start, so a changed value
must roll the pod. With the hash suffix, editing any value renames the
generated ConfigMap, mutates the Deployment pod template, and triggers
an automatic rolling restart. Without it, an IP change (e.g. an LB
re-allocation) would silently leave the pod advertising a stale PASV
address, breaking classic-PASV clients (older Brother scanners) while
EPSV clients keep working.
The Deployment consumes the CM via envFrom.configMapRef, so kustomize's
nameReference transformer rewrites the reference to the hashed name --
verified with `kubectl kustomize` (ftp-scanner-config-6d2597bt4d).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps the samba-bridge fork images 2026.2.3 -> 2026.5.3 (built from
authentik-samba-bridge#f02b945 / PR #1) and pins the Helm chart to
2026.5.3 so chart appVersion and the running app image are in lockstep.
- global.image.tag: 2026.2.3-b06668e -> 2026.5.3-f02b945
- ldap-outpost container_image: 2026.2.3-0140972 -> 2026.5.3-f02b945
- chart version: ">=2025.12.0" (floated to 2026.5.3) -> pinned "2026.5.3"
Both images verified present in the gitea registry. The fork bump
validated the samba patch and server build assumptions (TENANT_APPS
sed, ManagedAppConfig, python3.14 venv, ldap runtime contract) against
upstream version/2026.5.3 before building.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The HelmRelease used an unbounded range (>=37.0.0), which auto-resolved
to chart 41.0.0. That release removed the root 'logs' key from the
values schema, so the Helm upgrade failed validation
('additional properties \'logs\' not allowed'), the traefik HR never
went Ready, and the infrastructure Kustomization's health check stalled
in 'Reconciliation in progress' — gating every dependent app
(paperless, authentik, home-assistant, ...).
Constrain to the working 40.x line. The 41.x values migration is
deferred and should be done deliberately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert PR #19: the custom worktree wrapper scripts duplicated Claude
Code's native worktree support and are not the intended workflow.
Use the built-in worktree feature instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude-start.sh creates (or reuses) a sibling git worktree at
../homelab-<name> on a new branch off master and launches claude in it.
claude-stop.sh tears the worktree down, refusing on uncommitted changes
unless --force, with optional --delete-branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Roll out shutdown tuning across the barman-backed CNPG clusters:
- stopDelay: 60 (all three) — caps terminationGracePeriodSeconds. With the
barman-cloud plugin the demoted/restarting pod's instance-manager deadlocks
on shutdown: a plugin runnable misses the 30s controller-runtime grace, the
process never exits, and the native sidecar (which only stops after the main
container) keeps running -> pod hangs Terminating (upstream
cloudnative-pg#9770). Default 1800s leaves 2-instance clusters at 1/2 (and
single-instance im-pg fully down) for ~30min after each failover/restart;
60s caps that to ~1min. Postgres itself is cleanly down by ~12s, so the
trimmed time is only the wedged instance-manager shell.
- smartShutdownTimeout: 10 (pprl-pg, im-pg) — already on ak-pg (#16); caps the
smart-shutdown drain so -rw isn't backendless for 180s. Validated on ak-pg:
app downtime 162s -> ~10s.
Note: changing stopDelay alters terminationGracePeriodSeconds, so CNPG rolls
each cluster's pods once to apply it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ~10k-entity HA install OOM-looped at its 4Gi cgroup limit: working set
sits ~3.9Gi, ran cleanly overnight, then tipped over on morning load spikes
into a tight OOMKilled restart loop (recorder unclean-shutdown markers every
few minutes). The legacy HAOS VM 103 that ran this same workload had 16Gi.
Raise k8s-edge (VM 153) memory 6144 -> 12288 so the HA pod can have an 8Gi
limit with headroom for the other pinned pods (matter-server, otbr, zwave-js,
music-assistant). Freed RAM comes from the now-stopped VM 103.
Apply order: tofu apply + edge node reboot must land before the pod limit
change so the node actually has the RAM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ak-pg failover/switchover caused ~2.7min of app downtime. Root cause: on
demotion CNPG immediately drops the old primary from the -rw service
(role->unhealthy), but only promotes the replacement after the old primary is
fully down. The default smart shutdown waits smartShutdownTimeout=180s for
client connections to drain; Authentik's persistent connections never do, so
-rw stays backendless (clients get "No route to host") for the full window.
Postgres promotion itself is ~3s.
Capping smartShutdownTimeout at 10s bounds the backendless window to ~10-15s
(fast shutdown + promotion). Applied to ak-pg first per staged-rollout; pprl-pg
and im-pg to follow after validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Today's kubelet-restart-induced switchovers on ak-pg and pprl-pg each
took ~5 minutes from probe-failure to authentik/paperless reconnecting,
dominated by ~4 minutes of postgres-internal promote work. Most of
that time is the new primary chasing missing WAL via barman-cloud (the
old primary terminated mid-stream, queued WAL had to be replayed from
the archive) and the standby waiting through wal_retrieve_retry_interval
cycles before noticing the promote trigger.
With one synchronous standby (`synchronous.number: 1`), the standby is
always at the same LSN as the primary at commit time, so on promote
there is no missing WAL to chase and the standby is immediately ready
to accept connections. CNPG switchovers in this mode complete in single
digits of seconds.
`dataDurability: preferred` (CNPG v1.20+ modern API, replaces the
legacy minSyncReplicas/maxSyncReplicas fields) makes sync best-effort:
the primary stops waiting for the standby if it becomes unavailable,
so a standby outage does not block writes. The trade-off versus async
is ~1-5 ms of added commit latency on healthy LAN+Ceph, which is well
under the noise floor for authentik and paperless workloads.
Single-instance clusters (im-pg, gf-pg) are not touched - sync
replication requires >= 2 instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All three authentik components were sitting at 70-80% of their 1Gi
memory limit at steady state, with no headroom for spikes
(LDAP sync, blueprint apply, pg checkpoint/vacuum, gunicorn worker
restart). Today's outage during the kubelet config rollout on k8s-2
saw ak-pg-1 (the primary at the time) shut down on a readiness-probe
failure and not come back cleanly; suspected contributing cause is
that brief memory pressure tipped it past the 1Gi limit.
- ak-pg cluster: 512Mi req / 1Gi limit -> 768Mi req / 2Gi limit
(steady-state usage on primary is ~580MiB; vacuums and checkpoint
spikes were eating into the cap with no margin)
- authentik-server: 768Mi req / 1Gi limit -> 1Gi req / 2Gi limit
(live pods sitting at 720-745MiB, ~74% of cap)
- authentik-worker: 768Mi req / 1Gi limit -> 1Gi req / 2Gi limit
(workers run heavy background tasks - LDAP sync, policy eval,
blueprint application - and need headroom)
Limit bump is the safety net (no OOM). Request bump matches steady
state so scheduler accounts for real footprint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CP VMs are seeing Ceph `MON_DISK_LOW` on mons a,b because their
shared /var (xfs on /dev/vda5) sits at 77-79% used — past Ceph's
default `mon_data_avail_warn` line of <30% free. Mon DB itself is
118 MiB; the disk pressure is pod density (k8s-2 hosts 75 pods vs
k8s-1's 42), driving containerd image cache and /var/log/pods.
Kubelet's default image GC thresholds (85% high / 80% low) sit
above where we land, so GC never fires and unused image layers
accumulate indefinitely. Lowering to 70/60 forces eviction before
/var crosses the mon warn line, with Spegel's P2P pull-through cache
softening the re-pull cost on the rare case a GC'd image is needed
back.
Requires a node reboot to take effect (Talos `extraConfig` change).
Roll one CP at a time with `kubectl drain` + Ceph `HEALTH_OK` wait
between nodes, per the standing drain/Ceph-wait discipline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rook-ceph HelmRelease's floating `>=1.16.0` constraint silently
upgraded to v1.20.0, which extracts CSI driver management into the
upstream ceph-csi-operator subchart. That subchart's generated
ctrlplugin Deployments and nodeplugin DaemonSets reference per-driver
ServiceAccounts (`rbd-ctrlplugin-sa`, `cephfs-ctrlplugin-sa`,
`rbd-nodeplugin-sa`, `cephfs-nodeplugin-sa`) that the main rook-ceph
chart no longer ships — they moved to a separate ceph-csi-drivers
chart. With the SAs missing, both ctrlplugin Deployments sat at 0/2
with FailedCreate for 5 days, blocking PVC attach/detach across the
cluster (Grafana, Alertmanager, libation stuck; Home Assistant pinned
to the edge node which never got an RBD nodeplugin pod).
The ceph-csi-drivers chart cannot be installed cleanly here:
helm-controller v1.5.4 predates `disableTakeOwnership`, so Helm would
refuse to adopt the operator-managed `Driver` and `OperatorConfig` CRs
the v1.18 -> v1.20 auto-migration created. Instead, render only the
SAs + RBAC the operator needs from the chart's templates as static
manifests under kubernetes/infrastructure/ceph-csi-drivers/. The
existing operator-managed CRs continue to be the source of truth.
Also pin the rook-ceph chart to `~1.20.0` to prevent the same
floating-constraint surprise on the next minor bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8
security-bot