- protonmail-bridge: StatefulSet with 1Gi ceph-block PVC, internal
Service on imap/smtp. Initial login still requires `kubectl exec`
into the pod (one-time, state persists on PVC).
- ftp-scanner: vsftpd Deployment, LoadBalancer on 10.2.40.82.
Pre-creates Konstantin/Sarah folder layout in paperless-consume
PVC; paperless-ngx's recursive consumer picks up uploads.
- CiliumNetworkPolicy allows world/host/remote-node → FTP ports.
- Generated scanner password stored SOPS-encrypted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Paperless hits auth.k.renner.uno for OIDC discovery, which Cilium
socketLB redirects to the ingress-nginx backend pod. Both sides of
that connection need matching allow rules: paperless egress to
ingress-nginx, and ingress namespace ingress from paperless.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without socketLB, pod connections to LoadBalancer external IPs go
out the wire and hit the L2-announced VIP, which doesn't reply to
cluster-internal pods. socketLB redirects via eBPF at connect()
time to the backend pod.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With L2 announcements, the LoadBalancer VIP is advertised via ARP on
the LAN, so traffic from a pod to its own cluster's LB VIP times out.
Enable bpf.lbExternalClusterIP so Cilium redirects the socket straight
to the backend pod, fixing server-to-server OIDC from paperless to
auth.k.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Django-allauth uses one server_url for both discovery and browser
redirects. With an internal cluster URL, Authentik returns auth/token
endpoints on the cluster-internal host — browsers can't resolve that.
Point server_url at the ingress-facing auth.k.renner.uno so every
endpoint the discovery doc returns is externally routable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Authentik's default-deny-ingress only allowed traffic from authentik,
ingress-nginx, database and monitoring. Paperless resolves OIDC
discovery + token exchange via the internal authentik-server svc,
so it needs ingress from the paperless namespace too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kstatus doesn't recognize ObjectBucketClaim phase (OBC has no
status.conditions), so Flux's wait:true returned Ready even while
OBCs were still Pending. Add a CEL healthCheckExpr that blocks
until status.phase == Bound — this gates paperless-db and
authentik-db on the RGW bucket actually being provisioned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The migration-bucket ObjectBucketClaim lived in paperless-db's
kustomization, so the trigger-backup + restore Jobs started before
the OBC was Bound and their ConfigMap/Secret existed. Mirror the
authentik pattern: put the OBC in infrastructure/rook-ceph-cluster
so infrastructure-ceph-cluster's wait: true blocks dependent
kustomizations until the bucket is Bound.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rebuild pipeline commits a new talosconfig/kubeconfig each run.
Without a git pull beforehand, import-cluster-access imports stale
configs and kubectl fails with x509 CA mismatch against the newly
rebuilt cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Break bootstrap.sh into three phases (talos, flux, wait-apps) that
each CI step can invoke independently. Export cluster access files
immediately after the talos phase so users have kubeconfig /
talosconfig even if later phases fail.
Pipeline steps:
1. recreate-vms — tofu replace of all 3 Talos VMs
2. talos-bootstrap — generate + apply Talos, bootstrap K8s, get kubeconfig
3. export-access — SOPS-encrypt + commit talos/kube config
4. flux-bootstrap — install flux, wait for infrastructure + Ceph
5. wait-apps — wait for apps/authentik/paperless to reconcile
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Paperless data migration pulls ~3.5GB from S3 on first boot; give
it a longer window (15 minutes) than authentik before declaring
the bootstrap complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ansible fetch OOM'd on the 3.5GB media tarball. Install mc on the
paperless VM and stream pg_dump + tar outputs straight into S3
with mc pipe. Nothing touches the CI runner disk. Also drops the
per-pipeline S3 credential vars — credentials come from env.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ansible already has the SSH connection; pulling files back via
fetch module mirrors the authentik migration pattern and avoids
the separate ssh/scp dance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ssh cat-with-redirect pattern hit a remote shell quoting issue.
scp is simpler and purpose-built for file transfer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container name depends on compose project name and version (dashes
vs underscores). Discover dynamically via compose service label
instead of hardcoding paperless-ngx-db-1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Django raised DisallowedHost for kubelet probe requests (pod IP as
Host header), making the probe fail and the pod crashloop. Paperless
is behind ingress-nginx, which terminates the external Host header
and forwards to the pod by IP — so Django ALLOWED_HOSTS adds no
security. CSRF_TRUSTED_ORIGINS stays scoped to the ingress hostnames.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install community.proxmox collection in CI so dynamic inventory
resolves paperless.srv.renner.uno. SSH as ladmin (VM convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /api/ probes were working — DisallowedHost log entries are
informational noise, not readiness failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch liveness/readiness from httpGet to tcpSocket — avoids Django
ALLOWED_HOSTS check failing on kubelet's pod-IP probe requests
without needing to hardcode a Host header.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Django rejected probe requests with the pod IP as Host header since
ALLOWED_HOSTS only includes the ingress hostnames. Set Host header
on the probe to paperless.k.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
K8s auto-injects env vars like PAPERLESS_PORT=tcp://svc-ip:80 from
sibling Services, which paperless-ngx interprets as its listen port
and fails to start. Disabling enableServiceLinks strips these.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The paperless-ngx image runs s6-overlay as root and chowns mounted
volumes to uid 1000. Dropping ALL capabilities stripped CAP_CHOWN,
causing init-folders to fail. Add back CHOWN/DAC_OVERRIDE/FOWNER/
SETGID/SETUID and allow privilege escalation so s6 can switch users.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploy paperless-ngx with CNPG PostgreSQL, Redis, Gotenberg, Tika in
dedicated paperless namespace. OIDC via internal Authentik. Exposed
on paperless.k.renner.uno and docs.k.renner.uno for testing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/import-cluster-access.sh decrypts the SOPS-encrypted
talosconfig and kubeconfig from kubernetes/access/ into ~/.talos/
and ~/.kube/. Run after a rebuild-k8s pipeline completes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The python3 yaml filter was failing silently in the pipeline. Switch
to a pure-awk doc filter that splits on --- and drops any document
matching kind: HostnameConfig. Verified with talosctl validate.
Also drop stderr suppression on talosctl gen config so future failures
surface in the log.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Woodpecker steps use separate containers; home dirs don't persist.
Copy talosconfig/kubeconfig from ~ to access-staging/ in the bootstrap
step so the export step can read them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New export-access step encrypts both access files with SOPS and
commits them to kubernetes/access/. After a rebuild, a user can
pull and decrypt to get working talosctl/kubectl access.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI (Woodpecker) there's no terminal, so the \r-based progress bar
looks garbled. Fall back to plain per-step status lines when stdout
isn't a TTY.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Talos 1.12 gen config emits a HostnameConfig document that conflicts
with machine.network.hostname in per-node patches. Strip the auto-
generated HostnameConfig so existing patches continue to work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply all patches (cluster, cilium-cni, per-node) during apply-config
rather than during gen-config. Matches the previous working flow and
avoids merge issues with the generated base config.
Pin talosctl to v1.12.6 to match the Talos image used on VMs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With set -e, err=\$(failing-cmd) exits the script before the exit
code can be checked. Use if cmd; then ... form so the failure is
inside a conditional and set -e permits it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface the actual talosctl apply-config failure on each retry so we
can diagnose why the loop is silently exhausting 60 attempts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Proxmox DNS server is the canonical source of truth for node IPs.
Remove the Proxmox API fallback — trust the DNS and let its retry loop
handle transient boot delays.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alpine's BusyBox grep doesn't support Perl regex (-P).
Use sed for Proxmox API IP extraction instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After VM recreation, DNS is stale because the Proxmox DNS server
cache hasn't refreshed. Query the QEMU guest agent via Proxmox API
to get the correct IPv4, falling back to DNS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace static controlplane.yaml with talosctl gen config at bootstrap
time. Cluster settings (CIDRs, SANs, features) are in patches/cluster.yaml.
Certs and tokens are generated fresh each rebuild.
No more talosconfig_b64 CI secret needed — talosconfig is generated
alongside the node configs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI runner needs talosconfig for talosctl commands after apply-config.
Base64-encoded in CI secret talosconfig_b64.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After VM recreation, DHCP assigns new IPs. The Proxmox DNS cache
refreshes every 60s but the old A records have 300s TTL. Re-resolve
on each attempt and increase retries to 60 (10 min).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cilium dual-stack adds ULA IPv6 to nodes, which DNS returns.
CI runner prefers IPv6 but Talos API doesn't listen on ULA.
Resolve to IPv4 explicitly with dig for apply-config --insecure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VMs need time to boot into Talos maintenance mode after creation.
Add 60s wait and DNS resolution check for debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
k.renner
