Commit Graph
43 Commits
Author SHA1 Message Date
claude-botandClaude Opus 5.5 a17c4e8161 feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
  woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
  per incarnation (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest completed base backup (backup.info
                      status=DONE; barman uploads it as STARTED when a
                      backup begins, barman src/barman/cloud.py
                      coordinate_backup)
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool; the
  OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
   home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
   with prune: false (the stack's first PR) when this merges;
   cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix.

RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:18:14 +02:00
claude-botandClaude Opus 5.5 01213d6ee0 feat(authentik-gate): migrate the remaining 12 gates to the authentik-gate class
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
paperless, shelly-manager, zone-configurator, esphome, zigbee2mqtt,
alertmanager, prometheus, wiki, paperclip, navidrome, lidarr and wordpress
now gate through authentik-gate-sync: each source Ingress names its real
backend and carries the old provider's settings as annotations (group,
login-flow, authorization-flow where it was implicit consent, paperless'
skip-path).

Removed per gate: the ExternalName Service, the egress CNP in authentik,
the namespace CNP rule admitting the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). Groups stay declared without users lists;
membership is managed in Authentik.

Per-gate specifics:
- paperless: the docs.renner.uno alias backends the generated
  paperless-gate Service (its middleware redirects before the backend); the
  auth.renner.uno/outpost.goauthentik.io route is removed, proxy outposts
  take their callback on the app host.
- alertmanager / prometheus: source Ingresses renamed to alertmanager /
  prometheus; the controller generates the *-gate route Ingresses.
- wordpress-admin: gains a tls block for www.renner.uno (wordpress-tls,
  certificate still owned by the public Ingress), so the gate's external
  host stays https.
- wiki, paperclip: only the gate peer/rule leaves their shared CNPs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:37:26 +02:00
claude-botandClaude Opus 5.5 fb282baf6d feat(home-assistant): gate vscode.home.renner.uno through the authentik-gate class
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
First group gate on authentik-gate-sync. The Ingress switches to
ingressClassName authentik-gate with its real backend
(home-assistant-vscode:80) and annotations for the group (Home Assistant
Config Editors) and the code-server skip path; the default
passkey-stepup-authorization flow matches the old provider.

Removed: the ExternalName Service, the vscode-gate egress CNP, the
home-assistant-ns ingress rule for the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). The group stays declared in the blueprint but
without a users list: membership is now managed in Authentik, and current
members are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:21:52 +02:00
claude-botandClaude Opus 5.5 406b7b61f7 feat(llama-server): gate llm.renner.uno through the authentik-gate class
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
First gate on authentik-gate-sync. The Ingress keeps its host, TLS, public
label and cert-manager annotation, switches to ingressClassName
authentik-gate and names its real backend (llama-server:8080); the
passwordless login flow and the manifest/service-worker skip path move to
auth.renner.uno/* annotations. The controller now owns the outpost, the
Traefik route and both CNPs.

Removed: the hand-written ExternalName Service, the llama-gate egress CNP,
the ai-ns ingress rule for the old outpost, and the llama-gate blueprint
objects (state: absent, outpost -> application -> provider).

Expect a short gap on llm.renner.uno at switch-over: Traefik drops the route
when the class changes and serves the new one once the controller's next pass
has started the new outpost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:08:30 +02:00
claude-botandClaude Opus 5.5 bbb1582bcf feat(authentik): authentik-gate ingress class controller
Adds authentik-gate-sync, an in-house controller (ci/images/authentik-gate-sync,
stdlib Python like netbird-service-sync) that implements an `authentik-gate`
IngressClass. An Ingress of that class names its real backend; the controller
turns it into an Authentik proxy gate and wires the upstream ingress provider:

- Authentik: proxy provider, application (meta_publisher marks ownership),
  outpost on the local Kubernetes connection, optional group binding.
- App namespace: ExternalName Service <ingress>-gate -> outpost, Traefik
  Ingress <ingress>-gate (source hosts/paths/TLS/labels/annotations, backend
  swapped, cert-manager annotations dropped so only the source issues), and a
  CNP admitting only this outpost to the backend pods.
- authentik namespace: CNP letting the outpost reach only those pods.

Annotations: auth.renner.uno/group (members managed in Authentik; none = any
user), /login-flow, /authorization-flow (default passkey-stepup-authorization),
/skip-path. Fail closed: the Traefik route is applied last and removed when the
gate's config is invalid (missing group/flow/backend); transient API errors
leave the gate as the last good pass left it.

The API token is zero-touch: an ESO Password generator (CreatedOnce) feeds
both the controller and AUTHENTIK_GATE_SYNC_TOKEN on server/worker, which the
new gate-sync blueprint sets as the key of a non-expiring token for a
service account whose role covers only gate objects.

No existing gate moves yet; llama is migrated in a follow-up once this runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:39:50 +02:00
claude-botandClaude Opus 5.5 7518022b59 chore(openproject): decommission OpenProject
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
OpenProject Community hides every OIDC provider without an Enterprise
token (auth_plugin.rb, EnterpriseToken.allows_to?(:sso_auth_providers)),
so it cannot sign in through Authentik.

Removes the app, its Flux Kustomization, namespace, network policy,
op-pg PodMonitor, blackbox probe, Velero schedule and ClusterSecretStore
entry. Authentik drops the OIDC secret env and marks the OpenProject
application, provider and group absent. Gitea no longer allows
openproject.renner.uno as a webhook host.

Refs: REN-1363

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 14:04:56 +02:00
claude-botandClaude Sonnet 5.5 37c4287931 feat(openproject): sign in through Authentik OIDC
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Authentik gets an OpenProject confidential OAuth2 provider, application and
OpenProject Users group in the blueprint; the client secret is read from
OpenBao apps/openproject through an ExternalSecret and !Env. OpenProject
seeds the matching provider from env, and its egress policy allows
Authentik via Traefik.

Refs: REN-1363

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-09 13:10:16 +02:00
claude-botandClaude Opus 5.5 378e433086 feat(lidarr): missing-album view over the Navidrome library
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Lidarr in the navidrome namespace as a "what is missing" UI: artist
discographies (Wanted -> Missing) and Spotify followed artists / saved
albums / playlists as import lists.

- Mounts navidrome-music-bulk read-write (Lidarr has no read-only root
  folder mode); config on ceph-block.
- A reconcile sidecar (same image, curl+jq) reads the API key from
  /config/config.xml and on every start sets renameTracks=false,
  writeAudioTags=no, no recycle bin, no empty-folder deletes, no
  permission changes -- then adds /music as root folder only if absent,
  so the automatic library import never runs with file-mutating
  settings.
- No download client or indexer.
- AuthenticationMethod=External behind a dedicated lidarr-gate outpost
  restricted to a new "Music Librarians" group (k.renner).
- Image automation on the linuxserver build number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 19:33:54 +02:00
claude-botandClaude Opus 5.5 cb81157216 feat(authentik): paperclip-gate outpost and Paperclip Users group
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:07:14 +02:00
claude-botandClaude Opus 5.5 222d0263a4 feat(navidrome): self-hosted music streaming on the bulk pool
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Navidrome (Subsonic API) as the Spotify replacement for phone apps.

- Library PVC on ceph-filesystem-bulk (RWX, 300Gi, mounted read-only);
  SQLite data on ceph-block (RWO).
- Web UI behind a dedicated navidrome-gate Authentik proxy outpost;
  Navidrome trusts X-Authentik-Username (ExtAuth) and auto-creates the
  user on first SSO login (first user = admin).
- /rest and /share routed straight to Navidrome with the identity header
  stripped, so Subsonic clients use their Navidrome password.
- Namespace baseline CNP, gate egress CNP, Flux image automation,
  Velero daily schedule (09:15), bootstrap wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 11:12:08 +02:00
claude-botandClaude Fable 5.1 ae5de9bc84 feat(wiki): Authentik wiki-gate in front of wiki.renner.uno
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Proxy provider, application and outpost blueprint; outpost egress CNP;
ExternalName mirror; the ingress backends the outpost and the BasicAuth
middleware is gone. The ai CNP admits the outpost and Gitea only, so the
X-Authentik-Username header the service trusts from pod addresses cannot
arrive from the LAN.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-24 12:48:22 +02:00
claude-botandClaude Opus 5 939028fadd refactor(flux): CRD consumers move behind the CRDs they need
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
One unknown kind fails a server-side dry-run for the WHOLE apply set,
including the very HelmRelease that would have installed the missing
CRDs. On a running cluster every CRD exists, so this never showed; a
cold rebuild deadlocks on it immediately.

Three groups move, content unchanged:

ExternalSecrets and Traefik Middlewares out of infrastructure into
post-crds, which reconciles after the operators. The flux-git
ExternalSecret also changes creationPolicy Merge -> Orphan with
deletionPolicy Retain: Merge never CREATES the target secret, so on a
fresh cluster the Gitea credential handover silently did not happen.

ServiceMonitors/PodMonitors out of their charts and app sets into
apps/monitoring-rules, which applies after kube-prometheus-stack owns
the CRDs. Rook additionally stops creating its own two monitors
(spec.monitoring.enabled: false) — creating them is fatal to its
node-daemon reconcile when the CRD is missing, which held the
CephCluster in Progressing and with it the whole apps layer. Both
objects are reproduced field for field off the live cluster; the
ceph-exporter DaemonSet and the metrics endpoints are untouched.
Cilium is the worst case: its chart aborts rendering outright on the
missing CRD, and cilium is one of the seven infrastructure
healthChecks — the lowest layer in the cluster waiting on the highest.

Steady-state effect: object ownership moves between Kustomizations;
specs are identical. Verified on local vmtest runs and the Hetzner
throwaway cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-06 20:47:01 +02:00
claude-botandClaude Opus 5 aa53335c2b feat(llama-server): public WebUI behind an Authentik gate
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
llm.renner.uno becomes public and is fronted by a dedicated Authentik proxy
outpost. The gate is the entire access control: llama-server has no
authentication of its own and serves the OpenAI API on the same port as the
chat UI, which is exactly why the hostname was LAN-only until now.

Follows the vscode-gate pattern, including its passkey step-up, for the same
reason — what sits behind the gate is unprotected, so the gate is not a second
factor on top of an app login, it is the only one.

The API stays in-cluster. Home Assistant, Paperless and CI address
llama-server.ai.svc.cluster.local:8080 directly, never traverse Traefik or the
outpost, and so need no credentials. Nothing about them changes.

Traefik is deliberately removed from the ai namespace's ingress policy. It
routes the hostname to the outpost and never to llama-server, and dropping the
rule makes that structural: an Ingress pointing straight at the Service, by
accident or otherwise, would simply not connect.

Two details taken from prior gate failures rather than rediscovered:

  - No /outpost.goauthentik.io Ingress on auth.renner.uno. Only paperless-gate
    has one; the other gates serve those paths on their own host. Adding a
    second would have claimed a host+path paperless already owns and broken its
    login.
  - skip_path_regex covers manifest.webmanifest and sw.js, both of which the
    WebUI serves. A manifest is fetched with credentials omitted per the W3C
    spec, so it carries no cookie and the gate answers with a redirect, turning
    every page load into a round-trip. That is the failure that forced
    paperless-gate's carve-out and retired music-assistant-gate outright.

No group and no policy binding: any user who can authenticate gets in, as with
Immich, Audiobookshelf, Home Assistant and Music Assistant. The gates that pin
membership to a named user in git are admin interfaces — code-server over HA
secrets, esphome, zigbee2mqtt, prometheus — where the allowlist is the point. A
chat UI is not one of those, and its membership belongs in the user directory
rather than in a manifest.

The e2e checks move to the Service. They used to drive the public hostname,
which now answers with an Authentik redirect; going in-cluster is also how every
real consumer reaches it. A new check asserts the public hostname bounces an
anonymous request, so a regression that let Traefik reach llama-server directly
would fail rather than pass quietly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 21:41:08 +02:00
claude-bot 2f3d952656 refactor(secrets): drop the unused barman-cloud-creds
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Three ExternalSecrets synced a Secret that nothing reads. Established rather
than assumed, from three directions: every live ObjectStore names
hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore
(all seven run the barman-cloud plugin against an ObjectStore CR), and no pod
in the cluster mounts barman-cloud-creds.

They are almost certainly left from before the plugin migration, when the
inline configuration read a Secret of that name.

Worth stating plainly because it is not merely untidy: seed/{authentik,immich,
paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in
use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with
the three copies identical to each other. A valid key with no consumer is
exposure without benefit, so if it is a distinct S3 key it wants revoking at
Hetzner, which only a human can do — deleting our copies does not.

Removing the manifests does NOT remove the objects. All three carry
kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so
Flux skips them during garbage collection by design. The ExternalSecrets, their
Secrets and the three OpenBao paths have to be deleted explicitly after this
merges; git alone leaves them live and orphaned, which is worse than leaving
them declared.

The OpenBao paths deliberately stay until then. Deleting them first would leave
three ExternalSecrets failing to sync against a key that no longer exists, for
no gain.
2026-08-27 21:18:08 +02:00
claude-botandClaude Opus 5 b784ce3d38 feat(wordpress): host www.renner.uno in the cluster behind an Authentik gate
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Migrates the Hetzner VM 'wordpress-hochzeit' into the cluster. The site is not
the frozen wedding archive it looks like: it carries an unpublished, fully
composed birth announcement from 2026-05-14, so wp-admin has to keep working
for a human author rather than being replaced by a Git workflow.

Split routing on one host: '/' goes straight to WordPress so the blog stays
publicly readable, while /wp-admin and /wp-login.php route through a dedicated
wordpress-gate outpost. The inverse -- gate the host and skip_path_regex the
public paths, as paperless-gate does -- would need a negative lookahead, and
the outpost's RE2 has none.

Gating all of /wp-admin is only safe because the dead wpgiftregistry plugin is
dropped in the migration: it was the sole emitter of ajaxurl on this site, and
of 121 admin-ajax hits in the current log window not one returned 200.

Core lives in the pinned image, not on the volume, so upgrading is a tag bump
in Git. Only wp-content is persisted.

Backups are split the same way CNPG/Barman and Velero are elsewhere: the
operator takes a logical dump to Hetzner Object Storage at 02:45, Velero takes
the wp-content PVC at 08:45 and skips the database volume -- a crash-consistent
snapshot of a running database is not a backup.

Group membership for the gate is owned by Git, as with the Home Assistant,
Shelly and zone-configurator gates. s.renner is the Authentik account for the
site's WordPress Editor, whose login there is still the pre-2019 s.birzer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
2026-08-16 21:48:49 +02:00
claude-botandClaude Opus 4.8 91f5baf661 refactor(authentik): fold seed-authentik into the authentik KS — retire the seed layer (stage 2)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The last app in the seed->apps migration. Folds authentik's ak-pg CNPG backend
into the existing dedicated authentik KS and, since authentik was the final
holdout, fully retires kubernetes/seed/.

- git mv seed/authentik/{cluster,objectstore,scheduled-backup,barman-creds-
  externalsecret,hetzner-creds-externalsecret}.yaml -> apps/authentik/, add to
  apps/authentik/kustomization.yaml (authentik has no data PVCs — all state is
  in ak-pg).
- authentik KS: drop 'dependsOn: seed-authentik', add
  'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs.
- Delete clusters/homelab/seed-authentik.yaml and the now-empty seed/authentik/.

Seed-layer retirement cleanup (kubernetes/seed/ is now empty):
- git mv seed/RECOVERY.md -> kubernetes/RECOVERY.md (sibling to the MIGRATION-*
  docs); fix its stale 'flux reconcile kustomization seed-authentik' example to
  'authentik'.
- e2e-test.sh: drop the last seed-* readiness check (seed-authentik).
- bootstrap.sh: rewrite the obsolete 'per-app seed-<app> KSes' gating comment to
  reflect that app KSes now own their CNPG DBs directly, and de-'seed' the
  wait-description labels (the one-shot rsync/pg_restore seed Jobs are long gone).
- opentofu/object_storage.tf: fix the CNPG ObjectStore path comment
  (kubernetes/seed/* -> kubernetes/apps/*).

Safe ownership transfer (nothing pruned): all 5 seed/authentik objects carry
prune: disabled (from stage 1 #407), so deleting seed-authentik skips them in GC
and the authentik KS adopts them in place. No CNPG restart, no app restart.
Restore-test passed first: ak-pg backup restored cleanly (25 users, 368 tables,
1225 MB) using the corrected RECOVERY.md recipe, with a clean no-leak teardown.

Verified: kubectl kustomize apps/authentik builds with Cluster/ak-pg + exactly
5 prune-protected; bash -n bootstrap.sh + e2e-test.sh OK; kubernetes/seed/ fully
removed; no stale seed file-path references remain.

Left intentionally: barman/hetzner ExternalSecrets still read OpenBao keys under
seed/authentik/... (vault paths, not repo paths).

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-26 08:00:32 +02:00
claude-botandClaude Opus 4.8 de402ba531 Expose Zigbee2MQTT frontend via Authentik gate
ci/woodpecker/pr/pr-review Pipeline was successful
Gate zigbee2mqtt.home.renner.uno behind the same passkey-step-up Authentik
proxy-outpost pattern used for esphome/shelly-manager/vscode. The Z2M frontend
has no auth of its own (anyone on :8080 controls the whole Zigbee network), so
Traefik only ever routes the hostname to a dedicated outpost that validates the
session before forwarding.

New:
  - apps/home-assistant/zigbee2mqtt-gate-service.yaml — ExternalName mirror to
    ak-outpost-zigbee2mqtt-gate in the authentik ns (:9000).
  - apps/home-assistant/zigbee2mqtt-ingress.yaml — zigbee2mqtt.home.renner.uno →
    the outpost; cert-manager ingress-shim issues zigbee2mqtt-tls.
  - apps/authentik/zigbee2mqtt-gate-egress.yaml — CNP letting the outpost egress
    to the zigbee2mqtt pod on :8080.

Edited:
  - apps/authentik/blueprints-configmap.yaml — 5-entry blueprint block (group /
    proxyprovider / application / policybinding / outpost) named zigbee2mqtt-gate;
    internal_host carries the explicit :8080 (Service listens on 8080, unlike
    esphome's :80). Authentik spawns the ak-outpost-zigbee2mqtt-gate Deployment
    +Service from the outpost entry.
  - infrastructure/network-policies/home-assistant-ns.yaml — fromEndpoints rule
    admitting the zigbee2mqtt-gate outpost to :8080 (shared 8080 with vscode/
    shelly-manager; the fromEndpoints selector + the authentik-side egress scope
    it to the zigbee2mqtt pod).
  - the two kustomizations registering the new files.

DNS (k8s-gateway watches Ingress) and TLS (cert-manager) are automatic. Access
limited to the "Zigbee2MQTT Managers" group (k.renner). Verified: kustomize
builds, dry-run OK, blueprint YAML parses with the new block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 15:03:37 +02:00
claude-botandClaude Opus 4.8 a9cb28949f feat(zone-configurator): Authentik-gated ingress an zone-configurator.home.renner.uno
Der Everything-Presence mmWave Zone-Configurator lief bereits im Cluster
(Deployment in ns home-assistant, UI :42069), war aber nur ClusterIP-intern
erreichbar. Er hält ein HA Long-Lived Token und kann die ESPHome-Zonen-Entities
aller Präsenzmelder umschreiben, hat aber keine eigene Auth — daher wird er
(wie vscode/shelly-manager/esphome) hinter ein Authentik Passkey-Gate gelegt,
nicht nackt exponiert.

Neu:
- Authentik-Blueprint zone-configurator-gate: Group "Zone Configurators"
  (k.renner), ProxyProvider (passkey-stepup, internal_host …:42069,
  external_host zone-configurator.home.renner.uno), Application, PolicyBinding,
  dedizierter Outpost.
- ExternalName-Service ak-outpost-zone-configurator-gate + Traefik-Ingress
  (cert-manager letsencrypt-cluster) im app-Verzeichnis.
- Cilium-Egress (authentik) + Ingress-Allow (home-assistant-ns) für
  outpost -> zone-configurator:42069. Port 42069 ist eindeutig (wie esphome
  :6052), daher kein Port-Overlap mit anderen Apps.

Traffic: browser -> traefik -> ak-outpost-zone-configurator-gate -> Outpost
(Passkey) -> zone-configurator.svc:42069.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 08:12:15 +02:00
claude-botandClaude Fable 5 fa437a608c feat(secrets): migrate authentik secrets to OpenBao (final batch)
authentik-secrets + samba-bridge config: values in OpenBao KV, both
round-trip-verified text. The identity provider was deliberately last —
every other consumer proved the pipeline first. ZERO sops files remain
under kubernetes/; authentik KS decryption dropped. Phase 4 (age-key
retirement) is now unblocked pending soak.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
2026-07-14 01:34:04 +02:00
claude-botandClaude Opus 4.8 948852d83f feat(esphome): deploy ESPHome Device Builder behind an Authentik gate
Adds the ESPHome Device Builder (ghcr.io/esphome/esphome:2026.6.2, :6052) to the
home-assistant namespace so ESPHome devices — starting with an M5 Atom Echo BLE
proxy for shelly-manager's BLE provisioning — can be built, browser/USB-flashed
(ESP Web Tools / Web Serial), and config-managed in-cluster.

v1 deliberately runs with NO macvlan. Unlike shelly-manager it needs no L2
presence on the IoT VLAN: initial flashing is browser-side over USB and builds
are offline. That also sidesteps the macvlan-on-0.0.0.0 hole (a gated pod with a
macvlan bypasses Cilium policy and exposes its unauthenticated UI on the IoT VLAN
— confirmed against shelly-manager/matter-server/otbr). Live status / logs / OTA
would need cluster→v30iot reachability (a separate, still-open item); a macvlan
can be added later if wanted.

Mirrors the shelly-manager gate pattern exactly:
- home-assistant: esphome deployment (Recreate, RWO 10Gi ceph-block PVC for the
  PlatformIO build cache), service, gate ExternalName, and a Traefik ingress that
  backends the Authentik esphome-gate outpost (not esphome directly).
- authentik: an esphome-gate proxy provider + application + dedicated outpost
  (passkey step-up, "ESPHome Managers" group, git-owned membership) in
  blueprints-configmap.yaml, plus esphome-gate-egress CNP (outpost → :6052).
- network-policies: the home-assistant default-deny-ingress grants ONLY the
  esphome-gate outpost → :6052. :6052 is unique (no collision with the shared
  :8080 the vscode/shelly gates ride on).

Validated: kubectl kustomize builds cleanly for the home-assistant, authentik,
and network-policies kustomizations, all rendering the new resources.

Post-deploy checks: confirm the dashboard has no credentials:'omit' service
worker/manifest that 302-loops at the gate (add skip_path_regex if so); confirm
the image serves the dashboard on :6052 with the default entrypoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 19:08:59 +02:00
k.rennerandClaude Opus 4.8 ce0c78b503 Gate Prometheus behind Authentik
Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).

- Blueprint: prometheus-gate proxy provider (internal_host
  http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
  https://prometheus.renner.uno), application, outpost. REUSES the existing
  git-owned "Monitoring Access" group — created generically in #146 for
  exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
  frequently-checked monitoring UI; an authenticated session already closes
  the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
  index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
  false); standalone prometheus-gate-ingress.yaml backends the outpost
  ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
  matching egress in the authentik namespace.

Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.

Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.

Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.

Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:41:28 +02:00
k.rennerandClaude Opus 4.8 6ec3725371 Gate Alertmanager behind Authentik
Alertmanager (alerts.renner.uno, monitoring ns, :9093) had no auth of its
own, and its v2 API is writable: an unauthenticated LAN client could POST
/api/v2/silences and suppress ALL alerting — a tamper capability, not just
disclosure (2026-07-10 audit, highest-teeth of the deferred unauth
findings).

Gate it with the same Authentik proxy-outpost pattern as
paperless/vscode/shelly-manager:

- New proxy provider + application + dedicated outpost
  (ak-outpost-alertmanager-gate), mode proxy, forwarding to the ClusterIP
  Service (kube-prometheus-stack-alertmanager.monitoring.svc:9093).
- New git-owned "Monitoring Access" group (k.renner) bound to the app.
  Named generically so it can also gate the still-deferred Prometheus UI.
- The Helm-chart-generated Alertmanager Ingress is disabled
  (alertmanager.ingress.enabled=false); a standalone gate Ingress
  (alertmanager-gate-ingress.yaml) backends the outpost ExternalName —
  two Ingresses for the same host would conflict. Mirrors loki-ingress.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9093;
  matching egress rule in the authentik namespace.

implicit-consent flow, NOT passkey step-up: Alertmanager is a
frequently-checked monitoring UI, and requiring an authenticated Authentik
session already closes the unauthenticated-LAN exposure. Passkey re-auth on
every visit is disproportionate friction for a dashboard (paperless-gate
uses implicit-consent for the same reason). Bump to
passkey-stepup-authorization if the write API warrants it.

No skip_path_regex: the Alertmanager UI is a plain Elm SPA with no service
worker and no manifest (verified 2026-07-10 — index.html has no
serviceWorker/manifest/webmanifest), so none of the credentials:'omit' PWA
fetches that forced skip carve-outs elsewhere.

Prometheus's alerting path is UNAFFECTED: it posts to the ClusterIP Service
(verified: spec.alerting.alertmanagers targets
monitoring/kube-prometheus-stack-alertmanager:http-web), not this ingress.
Prometheus's own ingress is untouched (still enabled) — its unauth exposure
is a separate deferred finding, and it is read-only.

Verified: kubectl kustomize builds for authentik, monitoring and
network-policies; the embedded outpost survived the blueprint insert; the
rendered ingress backends ak-outpost-alertmanager-gate:9000, egress is
scoped to alertmanager:9093, and alertmanager.ingress.enabled=false while
prometheus.ingress.enabled=true.

Post-merge checks to run: only the gate Ingress exists for alerts.renner.uno
(the Helm one is gone); unauthenticated GET -> 302 to auth.renner.uno;
POST /api/v2/silences unauthenticated is refused; the outpost forwards to
:9093 after login; Prometheus is still delivering alerts to Alertmanager.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:27:45 +02:00
k.rennerandClaude Opus 4.8 2efd54ef82 Gate shelly-manager behind Authentik passkey step-up
shelly-manager has no authentication of its own. Anything that can reach
:8080 can rewrite Shelly device config, push scripts, and read the stored
per-device AdminPassword values. Until now the only control on it was
network reachability of shelly-manager.home.renner.uno.

Put it behind Authentik using the same proxy-outpost pattern as the
vscode gate (760e187) and paperless before it:

- New proxy provider + application + dedicated outpost
  (ak-outpost-shelly-manager-gate) in mode: proxy. authorization_flow is
  passkey-stepup-authorization, the same flow vscode-gate uses, so every
  authorize requires a passkey.
- New git-owned "Shelly Managers" group (membership in the blueprint,
  k.renner only) bound to the application as the explicit allowlist.
- shelly-manager Ingress now backends the gate outpost via an ExternalName
  mirror instead of the app Service directly.
- home-assistant CNP gains a scoped ingress rule: only the
  shelly-manager-gate outpost pod may reach :8080. Matching scoped egress
  CNP added in the authentik namespace.

No skip_path_regex, deliberately. The music-assistant gate died on PWA
assets fetched with credentials:'omit' (manifest, service worker), and
paperless/vscode carve those out. shelly-manager ships neither -- wwwroot
is app.css, favicon.png and lib/ only -- so every asset is a same-origin
credentialed fetch. It is a Blazor Web app whose SignalR circuit at
/_blazor upgrades to a websocket carrying the session cookie, which the
proxy outpost forwards.

This also restores the app. It has been returning 504 through Traefik
since 760e187 merged at 01:20 today: `default-deny-ingress` has
endpointSelector {}, so its traefik -> :8080 grant, written for the
code-server sidecar, was namespace-wide and was in fact the only rule
letting Traefik reach shelly-manager, which also listens on 8080. Removing
that grant to gate vscode took shelly-manager's ingress down as collateral.
Gating it properly is the correct fix rather than re-widening the rule --
the app should not have been reachable unauthenticated in the first place.

Verified: kubectl kustomize builds clean for authentik, home-assistant and
network-policies; the rendered Ingress backends ak-outpost-shelly-manager-
gate:9000, the egress CNP is scoped to shelly-manager:8080, and the two
gate ingress rules cannot reach each other's target.

The Authentik proxy flow, the passkey login, and the SignalR circuit
through the outpost all require a browser test after apply.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 13:42:51 +02:00
k.rennerandClaude Opus 4.8 760e187ba1 feat(home-assistant): gate vscode/code-server behind Authentik passkey step-up
code-server (the HA /config editor sidecar) runs with --auth=none, as
root, mounting the entire /config incl. plaintext secrets.yaml, .storage,
and the git PAT in the PVC remote URL. Until now the only control on it
was network reachability of vscode.home.renner.uno — any LAN client could
read every HA secret and edit config.

Put it behind Authentik using the existing paperless-gate proxy-outpost
pattern:

- New Authentik proxy provider + application + dedicated outpost
  (ak-outpost-vscode-gate) in mode: proxy. authorization_flow is the
  passkey-stepup-authorization flow (same one Homeassistant/Immich/
  Audiobookshelf/Netbird use), so every authorize requires a passkey.
- New "Home Assistant Config Editors" group (git-owned membership,
  k.renner only) bound to the app — explicit allowlist for who can read
  HA secrets via the editor.
- vscode Ingress now backends the gate outpost (ExternalName mirror)
  instead of code-server directly.
- home-assistant CNP tightened: traefik no longer allowed to :8080; only
  the vscode-gate outpost pod may reach code-server. Matching scoped
  egress CNP added in the authentik namespace.
- code-server is a service-worker app; skip_path_regex carves out its
  static/manifest/SW tree to avoid the credentials:'omit' redirect loop
  that forced retirement of the music-assistant gate. Exact skip set to
  be confirmed post-deploy (see PR checklist).

Verified: kubectl kustomize builds clean for home-assistant, authentik,
and network-policies. The Authentik proxy flow + code-server PWA
behaviour require a browser passkey login test after apply.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 01:17:46 +02:00
k.rennerandClaude Opus 4.7 47163f204c authentik: wire LDAP Unifi on sibling DN + LDAPS via cert-manager
Activates the dormant `LDAP Unifi` provider for UniFi Network admin
auth: rebases onto sibling `DC=unifi,DC=renner,DC=uno` to dodge the
existing Jellyfin/Unifi base-DN dispatch collision (same fix Synology
got in 90906e2), adds Unifi Admins group / unifi Application /
PolicyBinding. Exposes LDAPS on 636 using an LE cert from
cert-manager (DNS-01 via letsencrypt-cluster), imported into Authentik
as a CertificateKeyPair from env vars on the server + worker pods.
Cert is also attached to LDAP Synology so DSM's eventual 389→636
cutover only needs a DSM-side reconfigure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:27:02 +02:00
k.rennerandClaude Opus 4.7 ac011223ec authentik: wire samba-bridge for DSM SMB authentication
Deploys the authentik-samba-bridge stack:
gitea.renner.uno/k.renner/authentik-samba-bridge

Three pieces:

1. Server image override (helmrelease.yaml). global.image points at
   the bridge-server image (Authentik server + our Django app that
   subscribes to password_changed and writes Samba schema attrs to
   user.attributes). global.envFrom adds SAMBA_BRIDGE_DOMAIN_SID,
   SAMBA_BRIDGE_ENABLED_GROUP, SAMBA_BRIDGE_UID_OFFSET from a new
   SOPS secret. Both server AND worker pods inherit the bridge —
   the signal receiver registers in any Authentik server process
   that calls User.set_password().

2. LDAP outpost image override (blueprints-configmap.yaml).
   container_image on the LDAP Outpost record swaps the upstream
   outpost for our patched build that adds "sambaSamAccount" to
   GetUserOCs(). Without this, DSM's smbd filter
   (&(uid=...)(objectClass=sambaSamAccount)) short-circuits to zero
   entries inside the upstream outpost regardless of what's in
   user.attributes.

3. samba-bridge-secret.sops.yaml — the bridge's env contract. The
   domain SID was generated once from /dev/urandom and is now
   frozen forever (must match the SID typed into DSM's "Configure
   Samba Authentication" wizard).

Post-deploy: existing Synology Users members need to change their
password once via auth.renner.uno for the bridge to capture the
plaintext and synthesise the NT hash. Argon2 hashes already on
file can't be reverse-engineered into NT hashes; password reset
is the only path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 11:37:42 +02:00
k.rennerandClaude Opus 4.7 3ededebfdd authentik: declare LDAP Synology provider + expose outpost on ldap.renner.uno
Blueprint-manages a new LDAP integration for nas-1 DSM: dedicated
LDAPProvider, Application, Synology Users/Admins groups, and PolicyBindings.
Standalone LoadBalancer Service on the BGP pool (pinned 10.2.99.88/fd99::88,
LAN hostname ldap.renner.uno via external-dns-unifi) sidesteps the operator-
owned ClusterIP; CNP scopes :3389 ingress to the mgmt LAN so the cleartext
bind stays on a trusted segment until LDAPS is wired.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 20:09:04 +02:00
k.rennerandClaude Opus 4.7 a956529a2d authentik: standalone ServiceMonitors for outposts (ldap + paperless-gate)
The goauthentik.io operator auto-creates ServiceMonitors per outpost
but does not stamp `release: kube-prometheus-stack`, so this cluster's
serviceMonitorSelector silently ignores them. Declare standalone SMs
with the correct label, following the Velero precedent.

Radius is omitted on purpose: its container exposes only UDP 1812,
no /metrics endpoint. The operator-created SM for radius is dead
regardless of label.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 07:30:28 +02:00
k.rennerandClaude Opus 4.7 3a14aab4b3 paperless: convert trust-boundary policy to CiliumNetworkPolicy + add gate egress
The previous standard NetworkPolicy in paperless allowed ingress from
the gate outpost on paper, but the gate outpost still timed out at
134s when forwarding to paperless. Two reasons:

1. The cluster baseline CiliumNetworkPolicy in the paperless namespace
   (`default-deny-ingress`, sets ingress allow-list paperless / ingress-
   nginx / database / monitoring + host/remote-node) doesn't include
   authentik. Standard NetworkPolicy allow rules don't compose
   reliably with Cilium baseline CNPs for cross-namespace ingress —
   directly verified: an in-namespace curl from ingress-nginx to
   paperless succeeds (302 in 3 ms, allowed by the baseline CNP),
   while the same curl from the gate outpost times out (no policy
   path admits it).

2. The cluster baseline CNP in the authentik namespace
   (`default-deny-ingress`, egress to authentik / rook-ceph / kube-dns
   / world) doesn't include paperless either, so even on the source
   side the connection is blocked.

Replace the standard NP with two CiliumNetworkPolicies:

- `paperless/paperless-gate-ingress` (paperless namespace): selects
  the paperless app pod and allows ingress from endpoints labeled
  `goauthentik.io/outpost-name=paperless-gate` in the authentik
  namespace. CNPs union additively with the baseline CNP so the gate
  outpost is added to the allow set without broadening it.
- `authentik/paperless-gate-egress` (authentik namespace): selects
  pods labeled `goauthentik.io/outpost-name=paperless-gate` and
  allows egress to the paperless app pod specifically. Scoped to the
  gate outpost's label, not the whole authentik namespace, so other
  authentik workloads can't reach paperless.

Egress for paperless itself is also expressed as a CNP
(`paperless/paperless-egress`): in-namespace, kube-dns, and external
HTTPS only. Public-internet HTTPS uses Cilium's `world` entity which
already excludes cluster-internal IPs.

Together: the gate outpost is the only ingress path to paperless; the
gate outpost is the only thing in the authentik namespace that's even
permitted to talk to paperless; paperless's egress is restricted to
its known dependencies. Trust boundary lives on the gate outpost.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 09:38:55 +02:00
k.rennerandClaude Opus 4.7 00d0c19f9c authentik: route auth.renner.uno/outpost.goauthentik.io to the gate outpost
Each Authentik proxy outpost has its own session store. Sharing the
provider between the embedded outpost and the dedicated gate outpost
means the OAuth callback at /outpost.goauthentik.io/callback hits the
embedded outpost (which has no in-cluster /auth/nginx exposure), the
session is created in the embedded outpost's store, and the dedicated
gate outpost — which is what nginx auth_request actually queries —
never sees it. Result: an infinite redirect between /authorize and
/callback after every OAuth round-trip, surfaced as
ERR_TOO_MANY_REDIRECTS in the browser.

Add an Ingress in the authentik namespace that routes
auth.renner.uno/outpost.goauthentik.io/* to the dedicated outpost's
ClusterIP service, and detach the provider from the embedded outpost.
The dedicated outpost now serves /start, /callback, AND /auth/nginx
in the same process. The Authentik UI flow paths (/if, /application/o,
/api, /static, /flows, /media) continue to resolve via the chart's
authentik-server Ingress at the broader `/` rule, so 2FA still happens
on auth.renner.uno where credentials are bound.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 22:36:36 +02:00
k.rennerandClaude Opus 4.7 1a090ba3a3 authentik: switch outposts from Local Docker to in-cluster Kubernetes
The Authentik DB carried over a `Local Docker connection`
ServiceConnection from the Proxmox deployment, and all four outposts
(Caddy proxy, LDAP, Local Radius, embedded) still pointed at it. The
worker's `outpost_service_connection_monitor` task crashed every cycle
in-pod with `DockerException: ('Connection aborted.',
FileNotFoundError(...))` because the pod has no /var/run/docker.sock.

Ship a Blueprint via `blueprints.configMaps` that reassigns every
Outpost to the auto-created `Local Kubernetes Cluster` connection and
deletes the Docker connection. Authentik's chart-rendered Role already
grants the deploy/svc/ingress/secret/configmap perms needed to spawn
outposts via the K8s API.

While here:
- Grafana OIDC token_url/api_url were still hitting
  `http://authentik-server.authentik.svc.cluster.local`, violating the
  public-hostname-everywhere rule (split iss claim). Point them at
  https://auth.renner.uno/ alongside auth_url.
- seed-authentik comment still described the dropped pg_dump→pg_restore
  Job; replace with the current initdb + Garage CNPG recovery model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 14:16:10 +02:00
k.rennerandClaude Opus 4.7 5f25acae92 ingress: enforce dual-hostname convention across apps
Add the missing .k.renner.uno or .renner.uno host to loki, gatus,
and home-assistant Ingresses so every app is reachable on both the
internal LAN-side name and the public name. Drop the authentik test
Ingress; auth stays on auth.renner.uno only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 22:25:06 +02:00
k.rennerandClaude Opus 4.7 3063b54ab4 feat(seed): dedicated migration phase between infra and apps
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Replaces the per-app race-y CNPG bootstrap.recovery + velero-bootstrap-restore
pattern with a single seed Flux KS that populates databases and PVCs from
legacy hosts before any app KS reconciles.

Changes:

- New kubernetes/seed/<app>/ tree owning, per app:
  - CNPG Cluster CRs with bootstrap.initdb (was: bootstrap.recovery) +
    enableSuperuserAccess so seed Jobs can connect as postgres
  - DB seed Jobs: ssh legacy 'pg_dump -Fc' | pg_restore -d <db> against
    the running cluster (was: ephemeral PG + barman-cloud-backup → S3)
  - PVC seed Jobs: rsync/ssh+tar from legacy → app PVCs as standalone
    Jobs (was: in-pod init containers, blocking the app Deployment's
    progressDeadlineSeconds for hours on a 250 GiB rsync)
  - All ObjectStores, ScheduledBackups, barman-creds, PVCs, seed-ssh
    Secrets relocated here

- New top-level kubernetes/clusters/homelab/seed.yaml Flux KS:
  dependsOn infrastructure-cnpg-barman, timeout 4h, healthCheckExprs
  cover CNPG Cluster phase + native Job succeeded.

- Stripped from kubernetes/apps/<app>/: Cluster CRs, seed Jobs, init
  container rsync blocks, ssh volumes, seed ServiceAccount references.
  ~454 lines deleted, ~33 added.

- App KSs (immich, authentik, paperless, audiobookshelf,
  music-assistant, influxdb): dependsOn velero-bootstrap-restore →
  dependsOn seed.

- bootstrap.sh wait-apps: new wait_kustomization seed (4h) before
  per-app waits; per-app timeouts dropped to 60s since they're
  near-instant once seed is Ready.

- Deleted kubernetes/clusters/homelab/velero-bootstrap-restore.yaml;
  manifests stay in kubernetes/infrastructure/velero-bootstrap-restore/
  for post-cutover re-add (it remains the cold-rebuild path once
  legacy hosts are decommissioned and seed phase is removed).

Eliminates the documented race failure modes:
- cnpg-bootstrap-recovery-no-initdb-fallback (recovery Job pounds empty
  bucket until seed lands → multiple Error pods)
- velero-bootstrap-restore-bsl-sync-race (Job iterates schedules in an
  empty bucket for ~15 min)
- immich-server progressDeadlineSeconds blowup (250 GiB rsync init
  exceeds the 600s default and Flux marks the Deployment Failed)

Seed is migration-only: removed entirely once all services have cut
over from legacy. At that point the Cluster + ObjectStore +
ScheduledBackup + PVC manifests move back into kubernetes/apps/<app>/
and velero-bootstrap-restore.yaml is re-added at the cluster level.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 13:21:44 +02:00
k.rennerandClaude Opus 4.7 2e6f7dbb3f feat(seed): per-app cold-start Seed Job populates Garage for CNPG bootstrap.recovery
ci/woodpecker/manual/rebuild-k8s Pipeline failed
Adds kubernetes/apps/{authentik,paperless,immich}/seed-job.yaml — a
single k8s Job per app that:

  1. Idempotency-gates on `aws s3 ls s3://k8s-cnpg/<cluster>/` being
     empty (any object means a prior seed or live cluster archived).
  2. legacy-dump init container: ssh + pg_dump (or docker exec
     pg_dump for paperless/immich which run their legacy DB in
     docker-compose) → emptyDir.
  3. main container: ephemeral Postgres (initdb + WAL archive_command
     wired to barman-cloud-wal-archive) + pg_restore the dump +
     barman-cloud-backup writes a base backup to s3://k8s-cnpg/<cluster>/.

The Cluster's bootstrap.recovery now has a producer: the seed Job
populates Garage; CNPG's recovery Job retries on empty bucket, then
succeeds once the seed lands the base backup + end_wal. No more
chicken-and-egg where the in-pod init container needed the cluster
already up.

In-pod init containers stripped of DB seed:
  - authentik: removed entirely (no PVC seed needed). HelmRelease loses
    serviceAccountName, initContainers, ssh volume.
  - paperless / immich: rewritten to PVC rsync only, sentinel-guarded
    (`/data/.seed-done`) so subsequent restarts skip the rsync. No
    kubectl, no Backup CR creation, no serviceAccountName.

seed-rbac.yaml deleted from authentik / paperless / immich (the only
RBAC consumer was the in-pod init container's kubectl calls). influxdb
keeps its seed-rbac (its init container still creates Velero Backup CRs).

bootstrap.sh wait-apps timeouts bumped for cold-start reality:
  apps      5m → 15m  (Ceph mon/mgr/OSD bring-up on fresh cluster)
  authentik 5m → 20m  (seed Job + CNPG recovery from the seed)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 11:28:55 +02:00
k.rennerandClaude Opus 4.7 23764eca2d refactor(apps): merge *-db Flux KS into the app KS
authentik-db, paperless-db, immich-db are no longer separate Flux
Kustomizations — their CNPG Cluster, ObjectStore, ObjectBucketClaim,
Barman creds, and ScheduledBackup move into the app's Kustomization
(kubernetes/apps/<app>/) and the *-db Flux KS in
kubernetes/clusters/homelab/<app>.yaml is dropped.

Why: the prior split made the app KS dep-block on the *-db KS reaching
Ready, which kstatus evaluated against the CNPG Cluster's Ready
condition. On a cold-start with empty Garage, the Cluster never
reaches Ready (bootstrap.recovery has nothing to recover), so the app
KS never deploys and the seed initContainer that's supposed to
populate the DB never runs — chicken-and-egg. Merging removes the
inter-KS gate; the app Pod (with seed initContainer) and the Cluster
come up in the same KS unit.

Bootstrap mode change to make the seed actually viable lands in the
follow-on commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 11:17:05 +02:00
k.rennerandClaude Opus 4.7 176d763a73 refactor(migrations): seed becomes initContainer for the remaining five apps
Replicates the paperless pattern (49a1e75) for immich, audiobookshelf,
music-assistant, authentik, and influxdb:

- Seed work moves into an initContainer in each app's Pod template
  (Deployment for immich/audiobookshelf/music-assistant; HelmRelease
  values for authentik server + influxdb StatefulSet).
- Same Pod owns the PVC → no RWO/RWX contention regardless of access
  mode. Main container only starts after seed completes (initContainer
  ordering enforces "app starts after seed").
- Idempotency gate: completed CNPG Backup (DB apps) or completed
  Velero Backup (PVC apps); skip-fast on warm starts.
- SOPS-encrypted SSH secrets and seed-RBAC moved into kubernetes/apps/
  <app>/. PVC-only apps' RBAC is cross-namespace (Role+Binding in
  velero ns; SA in app ns).
- Drops kubernetes/migrations/ entirely and the five
  kubernetes/clusters/homelab/<app>-seed.yaml Flux Kustomizations.
- Removes <app>-seed from each app Kustomization's dependsOn.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:54:25 +02:00
k.rennerandClaude Opus 4.6 d9fa102b01 feat: Split Authentik into DB and app kustomizations
ci/woodpecker/manual/deploy-infrastructure Pipeline was successful
authentik-db deploys first: CNPG cluster, restore Job, S3 credentials.
FluxCD waits for the restore Job to complete before deploying authentik
(HelmRelease). This ensures the DB is migrated before Authentik starts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:25:50 +02:00
k.rennerandClaude Opus 4.6 4f32b07ef5 feat: Add restore Job to Authentik kustomization for FluxCD
The restore Job runs once via FluxCD, downloads the DB dump from S3,
and restores it into the CNPG cluster. Remove from kustomization
after migration is complete.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:21:20 +02:00
k.rennerandClaude Opus 4.6 bd4bc58a2f feat: Add migration S3 credentials secret for DB restore Job
ci/woodpecker/manual/migrate-authentik-db Pipeline was successful
SOPS-encrypted secret with MinIO migration user credentials.
Used by the restore-job.yaml to download the DB dump from S3.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:04:46 +02:00
k.rennerandClaude Opus 4.6 ca64f1be4e feat: Add auth.k.renner.uno test ingress for Authentik
Internal-only test ingress (no Cloudflare proxy) for validating
K8s Authentik before DNS cutover.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 07:58:48 +02:00
k.rennerandClaude Opus 4.6 d4652ba931 fix: Temporarily remove test ingress to allow HelmRelease upgrade
The chart ingress still has auth.k.renner.uno from a prior deploy.
Need to let the HelmRelease upgrade remove it first, then add the
standalone test ingress in a follow-up commit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 07:57:17 +02:00
k.rennerandClaude Opus 4.6 8401d55968 fix: Use internal-only test ingress for auth.k.renner.uno
Revert Cloudflared wildcard route - k.renner.uno is internal only.
Create separate ingress for auth.k.renner.uno without cloudflare-proxied
annotation so only UniFi ExternalDNS picks it up (A record to VIP).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 07:50:04 +02:00
k.rennerandClaude Opus 4.6 739a67e900 feat: Deploy Authentik via official Helm chart
CloudNativePG PostgreSQL (2 replicas), Authentik Helm chart with
2 server + 2 worker replicas. Same secret key as existing deployment.
SOPS secrets, Cloudflared tunnel route, ingress with TLS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 22:18:15 +02:00