Commit Graph
4354 Commits
Author SHA1 Message Date
k.rennerandClaude Opus 4.7 e7c61028c0 fix(immich): align wait-for-restore sentinel with seed job (.restore-v2-done)
Seed job writes /library/.restore-v2-done; server's wait-for-restore
init was still polling /data/.restore-done (pre-v2 name). Server pod
stuck at Init:0/2 for 37 min with sentinel mismatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 10:00:26 +02:00
k.rennerandClaude Opus 4.7 44bbdc0405 fix(immich-db): single instance during pgvecto.rs→vchord migration
pg_basebackup uses tar format whose 100-byte filename limit is exceeded
by pgvecto.rs index paths (pg_vectors/indexes/<uuid>/segments/<uuid>).
All replica joins fail deterministically with "file name too long for
tar format". Scale to 1 instance until Immich's in-app migration drops
the vectors extension and the pg_vectors directory goes away, then bump
back to 2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 08:10:55 +02:00
k.rennerandClaude Opus 4.7 e97e33bd58 fix(immich-db): use shared_preload_libraries as top-level list
CNPG rejects shared_preload_libraries under spec.postgresql.parameters
as a "fixed configuration parameter" — it manages that GUC itself and
exposes spec.postgresql.shared_preload_libraries as a dedicated list
for operator-aware extension preloading. Flux apply failed with admission
webhook denial.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 07:56:15 +02:00
k.rennerandClaude Opus 4.7 448dd40830 feat(immich): K8s manifests — CNPG cluster + seed + app
Three Flux Kustomizations chained via immich-db → immich-seed → immich:

- immich-db: CNPG Cluster using corentingiraud dual-extension image;
  bootstrap.recovery from s3://k8s-cnpg/im-pg (seeded by the
  migrate-immich pipeline). Barman ObjectStore + daily ScheduledBackup.
- immich-seed: library+ml-cache PVCs (RWM ceph-filesystem, expandable),
  alpine Job that streams the 171 GiB tarball from s3://k8s-migration/
  immich/ and writes /library/.restore-done sentinel. Prune disabled so
  PVCs survive Kustomization retirement.
- immich: server + ML + redis Deployments, ConfigMap for non-secret env,
  Ingress with letsencrypt-staging (dev cluster), init container that
  blocks on the restore sentinel, LE-staging CA fetch for OIDC against
  auth.renner.uno. DB creds via CNPG-managed im-pg-app Secret.

DB_VECTOR_EXTENSION intentionally unset — Immich >=1.133 defaults to
vchord and auto-migrates pgvecto.rs → VectorChord on first boot against
the dual-extension image.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 07:51:38 +02:00
k.rennerandClaude Opus 4.7 6004a0df77 feat(seed): v3 seed Jobs on aws-cli, versioned sentinel
Bumps audiobookshelf + paperless seed Jobs to v3. Two changes:

- Swap mc (fetched from dl.min.io at runtime) for aws-cli v2 from
  Alpine's community repo. Cleaner tooling, better retry/timeout
  behavior, standard AWS env-var auth. Forces path-style addressing
  because the Garage TLS cert covers s3.nas-1.renner.uno only, not
  wildcarded bucket subdomains.

- Bump sentinel path to .restore-v3-done / .seed-v3-done so v3 Jobs
  re-run on existing PVCs rather than short-circuiting on the v2
  sentinel. Paired with the 20 MB block_size change on Garage and the
  migrate-* pipelines re-uploading tarballs, this lets seeds exercise
  the new chunk layout.

Immich seed mirrors these edits but is still part of the untracked
in-progress CNPG replatform work, so it stays out of this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 00:45:00 +02:00
k.rennerandClaude Opus 4.7 d387fcb78e perf(garage): 20M block_size + restart-on-change
ci/woodpecker/manual/install-garage-synology Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/migrate-audiobookshelf Pipeline was successful
ci/woodpecker/manual/migrate-immich Pipeline was successful
Raises Garage's block_size from the 1 MiB default to 20 MB to cut
HDD-seek overhead on reads of backup objects. On the RS1221+ RAID 5
volume1 the per-block cost was seek-dominated (~80 ms overhead vs
2.5 ms transfer at 1 MiB); 20 MB blocks push cycles into the
transfer-dominated regime and align one 16 MiB Postgres WAL segment
per block for CNPG archives. Only affects NEW uploads.

Adds shadow-file diff detection so the toml-write step only reports
"changed" on real changes, and a force-recreate step that fires when
garage.toml actually changed — Garage has no SIGHUP reload and
compose does not notice bind-mounted file changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 23:02:23 +02:00
k.rennerandClaude Opus 4.7 e0ffd8db64 fix(immich): seed-barman uses tensorchord image (root+bash), not CNPG operand
ci/woodpecker/manual/migrate-immich Pipeline failed
corentingiraud/cnpg-pgvector-vectorchord:16-migration inherits USER 26
from its CNPG operand base, so apt-get can't acquire the apt lock inside
a Woodpecker step (run #645 failed with EACCES on /var/lib/apt/lists/).

Swap to tensorchord/pgvecto-rs:pg16-v0.3.0 — same pgvecto.rs version as
LXC 225, based on postgres:16 (root default + bash), clean apt layer.
Drop vchord.so from the ephemeral's shared_preload_libraries (not shipped
there; only needed on the target CNPG cluster for Immich's in-app
migration). WAL recovery between the tensorchord ephemeral and the
corentingiraud target works because both ship vectors.so in the same
v0.3.x series.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:54:56 +02:00
k.rennerandClaude Opus 4.7 f1a98090aa fix(immich): drop -o pipefail from seed-barman — dash incompatible
ci/woodpecker/manual/migrate-immich Pipeline failed
corentingiraud/cnpg-pgvector-vectorchord:16-migration uses dash as
/bin/sh (unlike postgres:16 which ships bash), and dash doesn't
support `set -o pipefail`. First pipeline run failed at line 1.
None of the commands in this step rely on pipefail semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:43:20 +02:00
k.rennerandClaude Opus 4.7 baf092098b feat(immich): migration pipeline to seed Garage for CNPG replatform
ci/woodpecker/manual/migrate-immich Pipeline failed
Seeds the CNPG im-pg Cluster's Barman archive at s3://k8s-cnpg/im-pg
and the library tarball at s3://k8s-migration/immich/immich-library.tar.
Ephemeral Postgres uses the dual-extension image so pgvecto.rs-formatted
pg_restore succeeds. Library tar is uncompressed (already-compressed
media). Follows the pprl-pg / ak-pg precedent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:25:53 +02:00
rebuild-k8s pipeline 4513931e6a ci: Export cluster access files from rebuild 2026-04-22 18:20:04 +00:00
k.rennerandClaude Opus 4.7 8133d4f198 feat(velero): expose metrics + add Grafana dashboard
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Enable Velero ServiceMonitor and node-agent PodMonitor with
`release: kube-prometheus-stack` labels so kube-prom-stack's
selector picks them up, and import community dashboard 11055 via
Grafana's gnetId download path so backup inventory/status is
visible at dash.renner.uno.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 19:47:03 +02:00
rebuild-k8s pipeline 5307eff46c ci: Export cluster access files from rebuild 2026-04-22 17:29:15 +00:00
rebuild-k8s pipeline 89486e63d4 ci: Export cluster access files from rebuild
ci/woodpecker/manual/rebuild-k8s Pipeline failed
2026-04-22 15:40:50 +00:00
k.rennerandClaude Opus 4.7 a54e528c97 fix(authentik): drop auth.renner.uno from authentik-test Ingress
ci/woodpecker/manual/rebuild-k8s Pipeline failed
The chart-rendered `authentik-server` Ingress claims `auth.renner.uno`
(see helmrelease.yaml server.ingress). Having the same host+path on
the standalone `authentik-test` Ingress trips ingress-nginx's
validating admission webhook ("host ... is already defined in ingress
authentik/authentik-test"), which fails the chart install and blocks
the whole authentik Kustomization on rebuild.

Test-domain Ingress now owns only `auth.k.renner.uno`. Prod traffic
flows through the chart's Ingress, including its proxy-buffer
annotations and the cloudflare-proxied external-dns annotation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 17:35:43 +02:00
k.rennerandClaude Opus 4.7 c8f07de409 fix(velero): scope app schedules to PVCs only — Flux owns the rest
The paperless/audiobookshelf/authentik daily Schedules backed up the
whole namespace minus CNPG objects. That includes Deployments,
Services, Ingresses, ConfigMaps, Secrets — all of which Flux already
owns from git. On rebuild, Velero restored the snapshot-age manifests
and then app Kustomizations (correctly) clobbered them with current
git state — except when the Kustomization chain was blocked on the
restore CR completing, leaving the cluster temporarily serving the
stale spec.

Concrete failure surfaced after the 2026-04-22 rebuild: the paperless
Ingress was restored from a backup that pre-dated commit 53c0d47
(prod hostnames added). It had only `*.k.renner.uno` hosts; git has
prod + test. `docs.renner.uno` was unreachable until paperless-seed →
paperless reconciled, which itself was waiting on audiobookshelf's
restore plugin operations to finish. Stale shadowing, two separate
apps coupled, no benefit — Velero shouldn't have shipped the Ingress
at all.

Per the project memory rule "back up app data only; Flux rebuilds
cluster state": add `includedResources: [persistentvolumeclaims]` to
each Schedule. snapshotMoveData=true continues to do the heavy lift via
Kopia + node-agent — the actual PVC contents go to Garage and come
back independently of the K8s API surface. Manifests now come from one
authoritative source (git) and never get shadowed by yesterday's
snapshot.

Authentik's Schedule has no PVCs today and becomes a no-op until one
lands; the comment is updated to say so.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 17:33:53 +02:00
rebuild-k8s pipeline 9124343b7f ci: Export cluster access files from rebuild 2026-04-22 14:27:21 +00:00
k.rennerandClaude Opus 4.7 8eff01cae2 fix(audiobookshelf): trust LE staging CA for OIDC token exchange
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Every K8s ingress in the test plane serves letsencrypt-staging certs
via cert-manager. Node's default TLS stack doesn't trust staging —
so ABS's openid-client POST to https://auth.renner.uno/application/o/token/
fails with certificate verify errors.

Mirror the test-browser pattern (commit 6c05798): init container
fetches letsencrypt-stg-root-{x1,x2}.pem into an emptyDir, concatenated
into one bundle. Main container mounts the bundle at
/etc/ssl/extra-ca/bundle.pem and sets NODE_EXTRA_CA_CERTS accordingly.

NODE_EXTRA_CA_CERTS is honored by Node ≥7.3 — openid-client uses
Node's built-in TLS so no library-specific work is needed.

Pod restart re-fetches. If letsencrypt.org is unreachable at pod start
the init container fails and the pod won't come up — acceptable for a
test-cluster trust bootstrap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 16:14:22 +02:00
k.rennerandClaude Opus 4.7 342a76d93e fix(network-policies): allow any in-cluster source to reach ingress-nginx
The ingress-default-deny CNP enumerated `ingress`, `apps`, `paperless`
as allowed-from namespaces. Any other namespace's pods (e.g.
audiobookshelf) couldn't reach ingress-nginx in-cluster — surfaced
while verifying the k8s_gateway DNS fix: ABS resolved auth.renner.uno
to 10.2.40.80 correctly, but TCP connect timed out because the policy
dropped the packet.

Replace the per-namespace allow list with a `cluster` entity match.
Any pod in the cluster can now reach ingress-nginx; external entry is
still bounded by the existing Cloudflare/MetalLB/host-probe paths.
New app namespaces no longer need a policy edit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 16:11:05 +02:00
k.rennerandClaude Opus 4.7 6f8ddb9f07 fix(k8s-gateway): add health/ready plugins so chart probes pass
Chart's default livenessProbe is :8080/health and readinessProbe is
:8181/ready. Those endpoints are opened by CoreDNS's own `health` and
`ready` plugins, which I left out of extraZonePlugins — pods stayed
Unready with connection-refused probe failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 15:55:55 +02:00
k.rennerandClaude Opus 4.7 4fc9d78fae feat(dns): k8s_gateway for in-cluster *.renner.uno resolution
Deploys ori-edge/k8s_gateway (chart 2.4.0) watching Ingress resources
across the cluster and synthesising A/AAAA records pointing at the
ingress-nginx LB. Main CoreDNS delegates the renner.uno zone to it via
a pinned ClusterIP (10.96.0.11) — avoids DNS-name resolution during
CoreDNS startup.

Why: pods resolving *.renner.uno currently go via upstream DNS to the
legacy Caddy VIP (10.2.45.100), which silently routes server-side
traffic (OIDC token exchange, JWKS, webhook callbacks) to the old
prod stack instead of the in-cluster Service. Symptom: audiobookshelf
OIDC login fails with `invalid_grant` because the authorization code
issued by K8s Authentik is exchanged against legacy Authentik. The
wiki's `[[k8s-east-west-leak-after-host-split]]` names this failure
mode verbatim; the previously-attempted fix (`template IN A renner.uno`
synthesis) was too broad and got reverted in 7a09bad.

Shape of the fix:
- k8s_gateway watches only Ingress (resources: [Ingress]).
- `fallthrough renner.uno.` hands names without an in-cluster Ingress
  to a `forward . 10.2.40.53` appended via extraZonePlugins — the
  Proxmox DNS HA VIP serves everything else (gitea, immich, cloudflared
  LXCs) unchanged.
- Main CoreDNS gets one `renner.uno:53` server block forwarding to
  10.96.0.11. No wildcards, no manual FQDN list — adding a new app
  Ingress just works.

External/browser DNS is untouched. No per-pod hostAliases needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 15:46:25 +02:00
k.rennerandClaude Opus 4.7 6c05798209 feat(test-browser): pre-install Let's Encrypt staging roots in Firefox
Every K8s ingress in the test plane uses cert-manager's
letsencrypt-staging ClusterIssuer, so browsing any *.renner.uno
hostname from inside the test-browser Firefox produced a manual
SEC_ERROR_UNKNOWN_ISSUER click-through per host.

Install the two LE staging roots (X1 and X2) via Firefox Enterprise
Policy:

- New ConfigMap firefox-policies.yaml carries a policies.json that
  sets Certificates.Install to point at the two cert files the
  init container drops at pod start.
- Init container fetches the roots from letsencrypt.org into a
  shared emptyDir (read-only mount in the main container). Means
  LE's live staging hierarchy is always what's trusted — no stale
  embedded PEM to rotate.
- Main container mounts the ConfigMap at
  /etc/firefox/policies/policies.json and the cert emptyDir at
  /etc/firefox/certs/, both read-only.
- ImportEnterpriseRoots: true is also set as a belt-and-braces
  measure so any cert dropped into the system store (via future
  automation) is also trusted.

Pod-restart re-fetches. If letsencrypt.org is unreachable at
pod start the init container fails; the pod won't come up until
it can fetch, which is acceptable for a test harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 15:00:27 +02:00
k.rennerandClaude Opus 4.7 2568afb266 fix(ingress): preserve Authorization on HTTP/2 for test-browser
The crowdsec-patched ingress-nginx controller drops the Authorization
header when the incoming connection is HTTP/2. Browsers default to
HTTP/2, so the kasmweb noVNC basic-auth challenge never receives
credentials — backend always sees "BasicAuth required, but client
didn't send any" and returns 401.

HTTP/1.1 requests through the same path forward Authorization fine,
so this is specifically an HTTP/2 handling regression in the patched
controller image, not something stripped by visible Lua filters.

Fix:
- Enable allow-snippet-annotations globally + annotations-risk-level
  Critical so individual Ingresses can use configuration-snippet.
- Add configuration-snippet on the test-browser Ingress that
  explicitly re-injects the Authorization header into the upstream
  request via proxy_set_header.

allow-snippet-annotations is a global setting with cluster-wide
reach. For this homelab with a single operator it's acceptable; the
alternative (swapping to stock ingress-nginx + separate crowdsec
bouncer) is a much larger change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 14:12:54 +02:00
k.rennerandClaude Opus 4.7 d669580540 fix(test-browser): VNC_PW must be >=6 chars (kasmvnc requirement)
ci/woodpecker/cron/security-scan Pipeline was successful
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 13:18:48 +02:00
k.rennerandClaude Opus 4.7 545260c1ee feat(test-browser): add pod-scoped hostAliases for K8s view
Replaces the reverted cluster-wide CoreDNS template with a narrow
pod-scoped /etc/hosts override. Inside the test-browser Firefox,
the six currently-migrated prod hostnames resolve to the K8s
ingress LB (10.2.40.80) instead of the legacy Caddy VIP.

Blast radius: one pod. Cluster CoreDNS stays pristine — no risk
of a repeat of the 2026-04-22 template-plugin break.

Maintenance: when a new service migrates to K8s, append its
prod-hostname to the hostAliases list.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 13:17:43 +02:00
k.rennerandClaude Opus 4.7 7a09bad5b1 fix(coredns): revert template plugin — broke cluster DNS
Restores the original Corefile without the *.renner.uno template
synthesis. ConfigMap stays Flux-managed so CoreDNS has a valid
config to reload; reverting the whole kustomization would have
deleted the ConfigMap via Flux prune, leaving CoreDNS with no
Corefile at all.

kube-dns-lb Service and the coredns-config kustomization directory
are intentionally left in place — they're harmless in isolation
and useful if we later reintroduce a corrected template.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 12:58:51 +02:00
k.rennerandClaude Opus 4.7 765b252b7b feat(test-browser): in-cluster noVNC Firefox harness
Replaces the Kasm-VM-based test harness (blocked by VM 124's
localStorage-vs-cookie auth mismatch inside Kasm's reverse-proxy
configuration) with a single-container kasmweb/firefox pod
running directly in the K8s cluster.

- Deployment runs kasmweb/firefox with 500m/1Gi requests, 4Gi cap,
  512Mi tmpfs shm, baseline capability set matching other apps.
- ClusterIP Service fronts the container's HTTPS port 6901.
- Ingress exposes browser.k.renner.uno + browser.renner.uno (both
  via the standard test-domain pattern) with
  backend-protocol=HTTPS + ssl-verify=off so nginx accepts the
  self-signed cert baked into the kasmweb image. proxy-read/send
  timeouts raised to 3600s for long-running noVNC sessions.
- letsencrypt-staging issuer, matching the rest of the test stack.

The pod's /etc/resolv.conf uses cluster CoreDNS, so *.renner.uno
resolves to 10.2.40.80 (ingress LB) via the coredns-config
template plugin. Zero per-hostname maintenance; any newly-added
ingress is immediately browseable from this pod.

Default VNC_PW is "kasm"; rotate via env edit. Sessions are
stateless — nothing to back up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 12:47:59 +02:00
k.rennerandClaude Opus 4.7 33d3b6d9f7 feat(metrics-server): deploy so k9s/kubectl top show CPU/MEM
Cluster had no metrics-server, so the metrics.k8s.io API returned 404
and k9s columns stayed blank. Talos kubelets serve self-signed certs
(serverTLSBootstrap not enabled), so --kubelet-insecure-tls is required
and InternalIP is forced since nodes don't publish resolvable Hostname
addresses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 10:53:18 +02:00
k.rennerandClaude Opus 4.7 d769ef92bf feat(coredns): synthesize *.renner.uno → K8s ingress + expose via LB
Adds a Flux-managed ConfigMap that replaces the stock Talos CoreDNS
Corefile with one extra pair of template-plugin blocks: any A/AAAA
query for *.renner.uno is answered directly with the ingress-nginx
LoadBalancer IPs (10.2.40.80 / fd40::80). All other plugins (errors,
health, ready, log, prometheus, kubernetes, forward, cache, loop,
reload, loadbalance) are preserved verbatim.

Why this matters after the phase-2 app-config cleanup: K8s apps now
emit *.renner.uno hostnames. Without this synthesis, a pod doing a
server-side call to auth.renner.uno would hit cluster CoreDNS →
forward upstream → ingress.renner.uno CNAME → 10.2.45.100 (Caddy,
prod). That's an east-west leak into production. The template
forces every *.renner.uno lookup to stay inside the K8s LB. Nothing
left to leak.

Also adds a second Service (kube-dns-lb) in kube-system with the
same selector as the existing ClusterIP kube-dns, pinned via Cilium
LB-IPAM to 10.2.40.83 / fd40::83. Kasm workspace sessions can point
their Dns at this LB IP and get exactly the same view the cluster
pods have — no per-hostname ExtraHosts maintenance.

Existing kube-dns ClusterIP (10.96.0.10) is untouched so no pod
re-wires its /etc/resolv.conf.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 10:47:09 +02:00
k.rennerandClaude Opus 4.7 53c0d47843 feat(ingress): add prod-hostname rules + drop *.k. from app config
Adds <app>.renner.uno as a second host rule and TLS SAN on each test
Ingress (authentik, paperless/docs, audiobookshelf, ical-proxy, grafana).
With the external-dns domain filter now scoped to k.renner.uno, these
prod hostnames cannot be written back into UniFi by external-dns — the
prod CNAMEs managed by OpenTofu stay pointed at the Caddy VIP, while
the K8s ingress now accepts traffic arriving with a prod Host header
(e.g. from a Kasm session whose extra_hosts maps *.renner.uno to the
cluster LB 10.2.40.80).

Also removes *.k.renner.uno from app-level config where it was baked
in as cruft: PAPERLESS_URL, Grafana root_url, Grafana OIDC auth_url
are now prod hostnames — matching the architectural intent that the
K8s cluster IS the future prod. PAPERLESS_CSRF_TRUSTED_ORIGINS keeps
both variants during transition so existing *.k.renner.uno browsing
doesn't immediately break on form POSTs.

cert-manager will issue fresh letsencrypt-staging certs with the
expanded SAN set via DNS-01; no public DNS changes needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 10:15:36 +02:00
k.rennerandClaude Opus 4.7 d57fb67927 fix(external-dns): narrow domainFilters to k.renner.uno
Production *.renner.uno records are owned by OpenTofu (UniFi DNS CNAMEs
to ingress.renner.uno). With domainFilters set to the renner.uno apex,
external-dns would attempt to upsert those records the moment a K8s
Ingress lists a *.renner.uno host, which would break production routing.

Scoping the filter to the k.renner.uno subdomain means external-dns can
only manage records inside that subdomain — the physical guard that
lets us safely add prod hostnames as secondary host rules on the K8s
test ingresses in the follow-up commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 10:08:12 +02:00
k.rennerandClaude Opus 4.7 7692e2433a feat(caddy): add kasm.renner.uno reverse proxy and deploy-caddy pipeline
ci/woodpecker/push/deploy-caddy Pipeline was successful
ci/woodpecker/push/security-scan Pipeline was successful
- Adds kasm.renner.uno site block to Caddyfile.j2 pointing at
  https://kasm.srv.renner.uno:443 with tls_insecure_skip_verify
  and unbuffered flushing for Kasm's WebSocket streams.
- Adds .woodpecker/deploy-caddy.yml so changes under
  ansible/roles/caddy-ha/** run install-caddy-ha.yml on push
  (plus manual via PIPELINE=deploy-caddy).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 07:59:11 +02:00
rebuild-k8s pipeline 92df80992a ci: Export cluster access files from rebuild 2026-04-21 13:17:11 +00:00
k.rennerandClaude Opus 4.7 c236e531c1 fix(cnpg): Revert to authentik/paperless owner+database, not app
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/rebuild-k8s Pipeline failed
CNPG bootstrap.recovery.owner/database accepts any name — 'app' was
just CNPG's default when unspecified, not a requirement. Keeping the
conventional authentik/authentik and paperless/paperless names aligns
with the dump structure and the app configs, removing the forced
detour through 'app'.

- Cluster specs: set explicit owner/database to the app name
- Migration pipelines: TARGET_DB/OWNER back to authentik/paperless
  (pg_restore --role= still applied so tables land owned correctly)
- authentik HelmRelease: user/name back to authentik
- paperless ConfigMap: DBNAME back to paperless

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 14:48:53 +02:00
k.rennerandClaude Opus 4.7 8f704610c3 fix(migrate): Restore dump with --role=app so app owns its tables
ci/woodpecker/manual/migrate-paperless Pipeline failed
ci/woodpecker/manual/migrate-authentik-db Pipeline failed
Without --role, pg_restore runs all DDL as the connection user
(postgres) while --no-owner strips OWNER clauses, so objects end
up owned by postgres. After CNPG bootstrap.recovery the live
application connects as app and hits "permission denied" on its
own data.

Add --role=$TARGET_OWNER so pg_restore does SET ROLE before each
statement, and REASSIGN OWNED + GRANT ALL ON SCHEMA public as a
safety net for anything created outside that scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 14:41:25 +02:00
Flux fd4f6e55fd Add Flux v2.8.6 component manifests
ci/woodpecker/cron/security-scan Pipeline was successful
2026-04-21 11:59:08 +00:00
rebuild-k8s pipeline 6467d05e3d ci: Export cluster access files from rebuild 2026-04-21 11:58:48 +00:00
k.rennerandClaude Opus 4.7 8e7fc5a711 fix(apps): Point authentik + paperless at app/app DB after CNPG restore
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Authentik's helm values set AUTHENTIK_POSTGRESQL__USER/NAME=authentik
and paperless-config sets PAPERLESS_DBNAME=paperless, but the CNPG
Cluster bootstrap.recovery uses database=app/owner=app and the
migration now loads the dump into app/app to match the managed
<cluster>-app Secret. The apps were authenticating as the wrong user
against a CNPG-reset password, which fails.

Point both apps at app/app so the chain is consistent:
  migration dump → app DB as app user (with 'seed' password)
  CNPG bootstrap.recovery → recovers, then resets app's password to
  the one in <cluster>-app Secret
  app → reads user + password from <cluster>-app, connects to app DB

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 13:53:28 +02:00
rebuild-k8s pipeline dab1a437c4 ci: Export cluster access files from rebuild 2026-04-21 11:19:48 +00:00
k.rennerandClaude Opus 4.7 912981f42f fix(migrate): Load dump into app/app to match CNPG Cluster recovery
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Both Cluster specs declare bootstrap.recovery.owner=app /database=app,
and CNPG generates the <cluster>-app Secret with username=app /dbname=app.
The authentik and paperless Helm/Deployment manifests consume that
Secret directly, so the app must find its data under the app user's
default database.

The migration was loading the legacy dump into authentik/authentik
and paperless/paperless, which CNPG then bootstrapped correctly (PG
user "authentik" keeps its "seed" password from the dump) but the
application pods connect as "app" against the CNPG Secret password
and authentication fails.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 12:56:54 +02:00
rebuild-k8s pipeline 8ea65648b1 ci: Export cluster access files from rebuild 2026-04-21 10:18:28 +00:00
k.rennerandClaude Opus 4.7 934be05e68 fix(migrate): Bake AWS creds into archive_wal.sh + log to /tmp
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/rebuild-k8s Pipeline failed
postgres sanitizes env on daemon startup so the previous archive_wal.sh
lost AWS_ACCESS_KEY_ID/SECRET/REGION and GARAGE_ENDPOINT when PG
spawned it from archive_command. Embedding the values into the wrapper
at pipeline-run time lets subprocesses authenticate.

Also capture archive_wal.log and print it before stopping PG, and wait
for archive_status to show a .done file (not just zero .ready) so a
silent-failure archive_command doesn't pass the wait loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 11:47:12 +02:00
k.rennerandClaude Opus 4.7 7be13fa1bb fix(migrate): Use PG native archive_mode so end_wal ships to archive
ci/woodpecker/manual/migrate-authentik-db Pipeline failed
ci/woodpecker/manual/migrate-paperless Pipeline failed
The previous approach — pg_switch_wal + manual loop uploading pg_wal/*
after barman-cloud-backup — lost the end_wal segment. pg_switch_wal
triggers PG's WAL recycling: the just-closed segment file is renamed
in-place to become the next active segment, so by the time the loop
reads pg_wal/, the file named <end_wal> no longer exists.

Configure archive_mode=on with archive_command=barman-cloud-wal-archive
during initdb, then force pg_switch_wal after barman-cloud-backup and
wait for archive_status/*.ready files to drain. PG archives each
segment synchronously on close, so the end_wal segment uploads before
the recycle happens.

Fixes the CNPG recovery error "object storage or file not found
<wal>: WAL not found" with the skipEmptyWalArchiveCheck annotation
already in place.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 11:37:05 +02:00
rebuild-k8s pipeline 29e687e0be ci: Export cluster access files from rebuild 2026-04-21 08:52:44 +00:00
rebuild-k8s pipeline 59c938f295 ci: Export cluster access files from rebuild
ci/woodpecker/manual/rebuild-k8s Pipeline failed
2026-04-21 08:15:47 +00:00
k.rennerandClaude Opus 4.7 ab0aa6eddf fix(cnpg): Skip empty-WAL-archive pre-check on bootstrap.recovery
ci/woodpecker/manual/rebuild-k8s Pipeline failed
The barman-cloud plugin v0.12.0 pre-check fails with "Expected empty
archive" when bootstrap.recovery points at a pre-populated archive —
it invokes barman-cloud-check-wal-archive without a --timeline hint,
so any existing WAL segments trip it.

This contradicts the symmetric-restore design: producer==consumer,
seed + live backups share one path, and recovery must succeed against
whatever is already there (latest backup, else migration seed).

The cnpg.io/skipEmptyWalArchiveCheck=enabled annotation on the Cluster
metadata disables the pre-check while leaving the actual restore path
intact. Applied to ak-pg and pprl-pg. Velero follows the same pattern
naturally (restore picks latest backup; tarball seed fills the gap
when the Velero bucket is empty).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 09:41:53 +02:00
rebuild-k8s pipeline 5a44c87561 ci: Export cluster access files from rebuild 2026-04-21 06:48:20 +00:00
k.rennerandClaude Opus 4.7 5cdc41b64c fix(migrate): Archive WAL after base backup so CNPG recovery can replay
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
ci/woodpecker/manual/rebuild-k8s Pipeline failed
bootstrap.recovery failed with "object storage or file not found
<wal>: WAL not found" because the migration's ephemeral Postgres runs
without archive_command — the base backup lands in Garage but no WAL
segments ever ship to the archive, so CNPG's pre-recovery check can't
locate end_wal and aborts.

After barman-cloud-backup completes, force a CHECKPOINT + pg_switch_wal
to close the active segment, then upload every complete WAL file in
pg_wal/ via barman-cloud-wal-archive with matching compression +
encryption. Applied to both migrate-authentik-db and migrate-paperless.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 07:52:04 +02:00
k.rennerandClaude Opus 4.7 50f9a280cc chore: Remove local markdown docs; wiki is single source of truth
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
ci/woodpecker/push/security-scan Pipeline was successful
Documentation now lives in ~/Notes/wiki/ (Obsidian vault). The local
.md files duplicated wiki content and drifted from it. CLAUDE.md stays
(it's Claude Code agent instructions, not documentation) and had its
dangling "Related Documentation" + kubernetes/README.md +
dns-server/HA_DEPLOYMENT.md references stripped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 07:42:43 +02:00
k.rennerandClaude Opus 4.7 7ecf11938f chore(k8s): Remove obsolete in-cluster trigger-backup Jobs
The trigger-backup-job + woodpecker-api Secret in each app namespace
belonged to the old migration mechanism where an in-cluster Job called
the Woodpecker API to kick the migrate-* pipeline. Seed backups are now
produced externally by the migrate-* pipelines directly, so the Job and
its sole dependency (woodpecker-api Secret) are dead weight.

Removed per namespace: trigger-backup-job.yaml, woodpecker-api.sops.yaml,
and their refs in kustomization.yaml. migration-bucket OBCs are kept
(still referenced by kubernetes/e2e-test.sh).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 07:36:30 +02:00
rebuild-k8s pipeline 5634441a7c ci: Export cluster access files from rebuild
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
ci/woodpecker/manual/migrate-paperless Pipeline was successful
2026-04-21 05:16:44 +00:00