Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5.5 a412ae4fb7 fix(paperclip): memory limit 32Gi
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Headroom for many concurrent agents in the one container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 21:49:57 +02:00
claude-botandClaude Opus 5.5 9319943f64 fix(paperclip): raise memory limit; agents run inside the container
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled
The container was OOMKilled (exit 137) at 19:43 under the 4Gi limit.
With server-local runs every concurrent agent's Claude Code process
and MCP helpers live in this container, so the limit caps how many
agents can work at once; exceeding it kills the server and every
in-flight run (process_lost / orphaned_running_run).

Limit 16Gi, request 2Gi (idle server ~1Gi). Nodes have ~93Gi
allocatable each at 12-49% use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 21:49:09 +02:00
argus-bot (Paperclip service account)andClaude Opus 5 0a82d61ff8 Merge PR #977: in-cluster app-health probe for paperless (REN-21)
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
Rebased onto master after #976 by argus-bot; conflict markers only were removed,
the 29 added lines are byte-identical to the original commit bdf55c5b.
blackbox-http 19 -> 20 targets; other two Probe documents unchanged.
Revert: git revert -m 1 <this merge sha> is clean (tested; 20 -> 19, navidrome
and lidarr intact).
Open risk: /accounts/login/ must return 200, not 302 -- module http_2xx passes
only a final 2xx. If it is a 302, revert this PR rather than widening
valid_status_codes. Unverified at merge time: no Prometheus read path. See REN-22.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-25 18:52:17 +00:00
svc-atlasandPaperclip cb9e73bd94 feat(monitoring): in-cluster app-health probe for paperless
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
paperless.renner.uno is served entirely through the paperless-gate
Authentik outpost. gatus already records the consequence at
apps/gatus/configmap.yaml:176-185: every path including /api/ answers
302 to an unauthenticated caller, so its check of that hostname — and
the UptimeRobot monitor REN-2 adds — is reachability only and stays
green while paperless is dead behind a live gate. This adds the app
health half of the pair, from the in-cluster vantage.

The target is /accounts/login/ because paperless-ngx has no
unauthenticated health endpoint. It is the app's own established
"the webserver answers" signal — the path its startup, liveness and
readiness probes already use — and it renders a Django view rather
than a static file. It does not cover the consumer/celery workers,
and the comment says so.

No CiliumNetworkPolicy change, and no edit to the paperless namespace.
The gatus comment above asserts an in-cluster probe needs one first;
that was written for gatus, which runs in `apps`. paperless-ns.yaml:24-26
already admits the whole monitoring namespace, where blackbox-exporter
runs, and Cilium unions the allow rules of every CNP selecting an
endpoint — the same mechanism apps/paperless/network-policy.yaml:8-12
relies on.

No PrometheusRule change: the target is covered by the existing
BlackboxProbeFailing catch-all and is correctly absent from its
exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds only
instances that have a dedicated rule elsewhere.

Refs REN-21.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 18:51:33 +00:00
argus-bot (Paperclip service account)andClaude Opus 5 16238d39ce Merge PR #976: in-cluster app-health probes for navidrome and lidarr (REN-21)
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
One file, kubernetes/apps/monitoring-rules/probes-blackbox.yaml, additive only.
blackbox-http 17 -> 19 targets; other two Probe documents unchanged.
Revert: delete the added target lines and comment block (tested clean).
Do NOT use git revert -m 1 on this merge once #977 has landed -- the natural
conflict resolution silently removes the paperless probe too. See REN-22.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-25 18:50:22 +00:00
svc-atlasandPaperclip 2219c928e0 feat(monitoring): in-cluster app-health probes for navidrome and lidarr
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
navidrome.renner.uno and lidarr.renner.uno are served entirely through
Authentik outposts. An external probe of either name therefore measures
DNS, TLS, Traefik and the gate outpost — and stays green while the app
behind a live gate is dead. This adds the second signal the
monitoring-vantage-point-rule asks for: an in-cluster probe of the
ClusterIP, which goes red on the app dying regardless of the gate.

Targets are /ping on both, the same path the pods' own kubelet
liveness/readiness probes already use. That it is ungated is not an
assumption: lidarr's reconcile script curls /ping in a loop before it
has read the API key out of config.xml (lidarr.yaml:29).

No CiliumNetworkPolicy change, and no edit to the navidrome namespace at
all. The wiki gotcha
cnp-cross-namespace-allow-apps-gatus-to-monitored-services is about
`apps`, where gatus runs; the influxdb allow-list it describes already
contained `monitoring`. navidrome-ns.yaml:25-27 likewise already admits
the whole monitoring namespace, which is where blackbox-exporter runs.

No PrometheusRule change either: both targets are covered by the
existing BlackboxProbeFailing catch-all and are correctly absent from
its exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds
only instances that have a dedicated rule elsewhere.

Refs REN-21.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 18:25:58 +00:00
k.renner d36ffc80c2 Merge pull request 'feat(wiki): API key and network path for Paperclip agents' (#974) from feat/paperclip-wiki-key into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#974
2026-09-25 17:42:31 +00:00
k.renner 8b6e93d412 Merge pull request 'feat(lidarr): missing-album view over the Navidrome library' (#973) from feat/lidarr into master
Reviewed-on: k.renner/homelab#973
2026-09-25 17:42:06 +00:00
claude-botandClaude Opus 5.5 157b102519 feat(wiki): API key and network path for Paperclip agents
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Paperclip agents get their own wiki machine identity: OpenBao
apps/ai/wiki-api-keys field `paperclip` (already written, version 5),
mapped to Auth__ApiKeys__paperclip. Reloader rolls the wiki on the new
key.

Agents reach the service in-cluster at wiki.ai.svc.cluster.local:8080,
like the CI agents, so only the app-level API key applies (no Traefik
BasicAuth user needed). The wiki CNP admits the paperclip pod on 8080
and the paperclip egress CNP allows exactly that pod and port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 19:35:29 +02:00
k.renner cf362548f1 Merge pull request 'feat(wiki): API key "paperclip" for the Paperclip agents' (#972) from feat/wiki-api-key-paperclip into master
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#972
2026-09-25 17:35:27 +00:00
claude-botandClaude Opus 5.5 378e433086 feat(lidarr): missing-album view over the Navidrome library
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Lidarr in the navidrome namespace as a "what is missing" UI: artist
discographies (Wanted -> Missing) and Spotify followed artists / saved
albums / playlists as import lists.

- Mounts navidrome-music-bulk read-write (Lidarr has no read-only root
  folder mode); config on ceph-block.
- A reconcile sidecar (same image, curl+jq) reads the API key from
  /config/config.xml and on every start sets renameTracks=false,
  writeAudioTags=no, no recycle bin, no empty-folder deletes, no
  permission changes -- then adds /music as root folder only if absent,
  so the automatic library import never runs with file-mutating
  settings.
- No download client or indexer.
- AuthenticationMethod=External behind a dedicated lidarr-gate outpost
  restricted to a new "Music Librarians" group (k.renner).
- Image automation on the linuxserver build number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 19:33:54 +02:00
claude-botandClaude Fable 5.1 f732528551 feat(wiki): API key "paperclip" for the Paperclip agents
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Second named key in apps/ai/wiki-api-keys (OpenBao version 4), mapped to
Auth__ApiKeys__paperclip; writes through the API are attributed to it.
Reloader rolls the pod when the Secret changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-25 19:33:38 +02:00
svc-argus (Paperclip service account) 890eab7dc3 Merge pull request 'test(access): remove the REN-3 branch protection probe file' (#970) from test/ren-3-access-cleanup into master
ci/woodpecker/cron/issue-fix Pipeline failed
ci/woodpecker/cron/alert-monitor Pipeline was successful
2026-09-25 13:17:37 +00:00
svc-atlasandPaperclip 5343b65562 test(access): remove the REN-3 branch protection probe file
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Leaves master's tree byte-identical to before the test.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 13:17:36 +00:00
svc-argus (Paperclip service account) f56391d88f Merge pull request 'test(access): REN-3 branch protection probe' (#969) from test/ren-3-access-check into master 2026-09-25 13:17:22 +00:00
svc-atlasandPaperclip d32bf7a4ab test(access): REN-3 branch protection probe
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 13:17:00 +00:00
k.renner e82303c314 Merge pull request 'feat(paperclip): disable sign-up; document the Claude connection' (#967) from feat/paperclip-signup-off into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#967
2026-09-25 11:57:08 +00:00
claude-botandClaude Opus 5.5 56e30bf065 fix(paperclip): agents use the Service name for the API
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The cluster-internal name keeps working when runs move to sandbox pods
(option C), where loopback would not reach the server. The name is
added to PAPERCLIP_ALLOWED_HOSTNAMES because the private-hostname guard
otherwise returns 403 for it. Same-namespace traffic is already allowed
by the baseline CNP in both directions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:52:48 +02:00
claude-botandClaude Opus 5.5 161d0561e7 fix(paperclip): agents call the API on loopback, not through the gate
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled
Upstream derives the agents' PAPERCLIP_API_URL from the public URL, so
agent API calls went out through Traefik to the paperclip-gate outpost.
With intercept_header_auth the outpost treats the agent's Authorization
header as an Authentik credential and returns 401; agents could not
create tasks, post cards or update status.

Agents run in the server pod (option B), so PAPERCLIP_API_URL is set to
http://localhost:3100 (upstream keeps an explicit value and passes it to
every run). Loopback passes the private-hostname guard and agent JWTs
still authenticate each call; the gate stays fully enforced for browser
traffic instead of carving out /api/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:51:43 +02:00
claude-botandClaude Opus 5.5 30931d3f50 feat(paperclip): disable sign-up; document the Claude connection
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled
The first admin is claimed, so PAPERCLIP_AUTH_DISABLE_SIGN_UP flips to
true.

Spec: Claude auth is a Paperclip-managed subscription connection created
once via the local sign-in. The server-env CLAUDE_CODE_OAUTH_TOKEN was
never used (stripAiAuthBindings blanks it for managed runs; upstream
accepts no pasted setup-token), so it is dropped from the spec. The
login must run as the node user: exec sessions start as root and
root-owned 0600 credentials are unreadable to the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:48:29 +02:00
security-bot bd14b3004c chore(image-automation): bump ghcr.io/paperclipai/paperclip:nightly@sha256:9222efbf11863160669e13d5559e6a1be881886dd1758650b74c3993dc3ef2a6
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-25 11:25:18 +00:00
k.renner 2c25acbfdd Merge pull request 'feat(paperclip): deploy Paperclip (server-local runs, trial)' (#966) from feat/paperclip into master
Reviewed-on: k.renner/homelab#966
2026-09-25 11:24:23 +00:00
claude-botandClaude Opus 5.5 50cf3e977a docs(paperclip): align spec with implementation and review fixes
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:22:43 +02:00
claude-botandClaude Opus 5.5 bb3a923c43 fix(paperclip): probes pass the hostname guard; tighten egress and SA token
- Paperclip's private-hostname guard (authenticated+private) returns 403
  for any Host but loopback and the public URL. Kubelet probes send the
  pod IP, so the pod could never become Ready. Probes now send
  Host: localhost.
- Liveness is a TCP check: /api/health runs SELECT 1, so a Postgres
  outage would have restarted the server and killed in-flight runs.
- Baseline CNP allowed world on every port; Cilium unions allows across
  CNPs, so the app-level :443 restriction was not enforced. world is
  now :443-only in the baseline as well.
- automountServiceAccountToken: false. Every agent shares the pod
  filesystem; bots bring their own cluster credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:22:17 +02:00
claude-botandClaude Opus 5.5 78cac12367 feat(paperclip): Flux Kustomization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:08:47 +02:00
claude-botandClaude Opus 5.5 5a22ea8df3 feat(paperclip): server Deployment, PVC, gated Ingress and CNPs
- Sign-up stays enabled: upstream's first-admin claim requires an
  existing signed-in Better Auth account (routes/access.ts
  /bootstrap/claim), so disabling it would lock out the claim. A
  follow-up flips PAPERCLIP_AUTH_DISABLE_SIGN_UP after the claim.
- Telemetry is off via PAPERCLIP_TELEMETRY_DISABLED=1, which
  resolveTelemetryConfig honours; config.json can't carry it alone
  because the schema requires a complete file.
- Container starts as root: docker-entrypoint.sh remaps the node user
  and chowns /paperclip before dropping privileges with gosu.
- Image pinned to the amd64 manifest digest of nightly; Flux digest
  reflection takes over from there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:08:32 +02:00
claude-botandClaude Opus 5.5 cb81157216 feat(authentik): paperclip-gate outpost and Paperclip Users group
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:07:14 +02:00
claude-botandClaude Opus 5.5 d3257c56ac feat(paperclip): CNPG pclp-pg with barman backups and OpenBao secrets
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:06:22 +02:00
claude-botandClaude Opus 5.5 2d56534581 feat(paperclip): Flux image automation tracks the nightly digest
digestReflectionPolicy: Always requires spec.interval on the ImagePolicy
(admission rejects it otherwise); set to 1h to match the ImageRepository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:05:43 +02:00
claude-botandClaude Opus 5.5 6fa4c340b8 feat(paperclip): namespace, baseline CNP and OpenBao allowlist
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:04:29 +02:00
claude-botandClaude Opus 5.5 c3c8662ca7 docs(paperclip): implementation plan
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 12:59:16 +02:00
claude-botandClaude Opus 5.5 c2a2481313 docs(paperclip): design spec for running Paperclip in the cluster
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 12:44:47 +02:00
k.renner 48bbe511b8 Merge pull request 'feat(netbird): retire the reverse proxy on the management host' (#965) from feat/netbird-retire-vps-proxy into master
ci/woodpecker/push/deploy-netbird-server Pipeline was successful
ci/woodpecker/push/deploy-netbird-proxy Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#965
2026-09-25 09:26:20 +00:00
k.renner 8d98acb3a4 Merge pull request 'feat(navidrome): self-hosted music streaming on the bulk pool' (#964) from feat/navidrome into master
Reviewed-on: k.renner/homelab#964
2026-09-25 09:19:19 +00:00
claude-botandClaude Opus 5.5 adbfd3bfc6 feat(netbird): retire the reverse proxy on the management host
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The public edge is proxy-1/proxy-2; proxy.renner.uno has not pointed at
netbird.renner.uno since the netcup fleet took over. The proxy that kept
running on the management host has served nothing since the controller
moved every service to TLS passthrough (2026-09-07): passthrough routes
live on the proxy's own :443 listener, while the host's Traefik forwards
HostSNI(*) to its :8443 main listener, which answers "unknown domain".

Its watchdog therefore failed every probe round and ran `docker restart`
every ~3 minutes since 2026-09-07 12:53 (last healthy round). Each
restart registered a fresh proxy peer and pushed a network-map update to
every peer in the mesh — ~10 stale proxy-* identities per ingress peer
and a map update every 1-2 minutes.

- netbird-server: drop the proxy service, its Traefik TCP passthrough,
  proxy.env and the proxy vars; `compose up --remove-orphans` removes
  the container, and one-off tasks stop/remove both watchdogs and delete
  the proxy volumes, config and state.
- install-netbird-server.yml no longer applies netbird-proxy-watchdog;
  its pipeline no longer triggers on that role.
- Comments that described the management host as a cluster member now
  describe the proxy hosts only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 11:17:46 +02:00
44 changed files with 2306 additions and 94 deletions
-3
View File
@@ -4,9 +4,6 @@ when:
branch: master
path:
- "ansible/roles/netbird-server/**"
# install-netbird-server.yml also applies the netbird-proxy-watchdog role,
# so changes to it must redeploy this host too.
- "ansible/roles/netbird-proxy-watchdog/**"
- "ansible/plays/install-netbird-server.yml"
- ".woodpecker/deploy-netbird-server.yml"
- event: deployment
+6 -7
View File
@@ -1,5 +1,5 @@
---
# Deploy a NetBird reverse-proxy HA cluster member (a second proxy host).
# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
# Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
#
# Prerequisites:
@@ -7,12 +7,11 @@
# listed under [netbird_proxy] in inventory/pve_hosts.yml. Docker is
# installed by the (Arch-aware) docker-install role.
# - DNS A/AAAA for the host's own name (e.g. proxy-1.renner.uno) -> its IP.
# - Uses the existing vault_netbird_proxy_token (shared with VPS-1) — no new
# token to generate; a shared token + same NB_PROXY_DOMAIN forms the cluster.
# - AFTER deploy + verify: add this host's IP to the proxy domain so traffic
# round-robins across the cluster. Repoint proxy.renner.uno away from its
# CNAME to netbird.renner.uno (management host) to A/AAAA records listing
# BOTH proxy IPs (VPS-1 + this host).
# - Uses the existing vault_netbird_proxy_token (shared by every member) — no
# new token to generate; a shared token + same NB_PROXY_DOMAIN forms the
# cluster.
# - AFTER deploy + verify: add this host's A/AAAA to proxy.renner.uno so
# traffic round-robins across the cluster.
- name: Deploy NetBird reverse-proxy cluster member
hosts: netbird_proxy
-1
View File
@@ -18,4 +18,3 @@
- common
- docker-install
- netbird-server
- netbird-proxy-watchdog
+4 -4
View File
@@ -43,10 +43,10 @@
#
# WHAT IS NOT, and must not be added:
#
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on both
# the server and the proxies, all Ansible
# templates — but the management server does not
# validate it from any of them. It is registered
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on the
# proxies, an Ansible template — but the
# management server does not validate it from
# any file. It is registered
# in the management DATABASE. Rotating it changes
# only what the proxies present, so every proxy
# is locked out: "proxy auth failed: too many
@@ -1,5 +1,5 @@
---
# Self-healing watchdog for the NetBird reverse proxy (VPS-1 and proxy-1).
# Self-healing watchdog for the NetBird reverse proxy on the proxy hosts.
#
# Why it exists: the proxy's embedded NetBird client pins each proxied
# domain's route to a specific routing-peer identity. The netbird-ingress
@@ -20,9 +20,7 @@ netbird_watchdog_probe_hosts:
- home.renner.uno
- immich.renner.uno
# Address the probe connects to. 127.0.0.1 reaches the local proxy on both
# members: VPS-1 routes it via Traefik's HostSNI(*) TCP passthrough, proxy-1
# binds :443 directly.
# Address the probe connects to. The proxy binds :443 on the host directly.
netbird_watchdog_probe_address: 127.0.0.1
netbird_watchdog_container: netbird-proxy
@@ -1,7 +1,7 @@
---
# Deploys the reverse-proxy watchdog (see defaults/main.yml for why).
- name: Ensure curl is installed (probe dependency; proxy-1 is Arch, VPS-1 Debian)
- name: Ensure curl is installed (probe dependency)
ansible.builtin.package:
name: curl
state: present
@@ -1,8 +1,8 @@
---
# NetBird reverse-proxy HA cluster member (proxy-only host, e.g. proxy-1).
# Joins the same proxy cluster as the VPS-1 proxy: identical NB_PROXY_DOMAIN
# + the same token. NetBird load-balances/fails the cluster over via DNS
# round-robin on the proxy domain (see plays/install-netbird-proxy.yml).
# Every member uses the same NB_PROXY_DOMAIN + token. NetBird load-balances/
# fails the cluster over via DNS round-robin on the proxy domain (see
# plays/install-netbird-proxy.yml).
# MUST match the netbird-server role's netbird_version — the reverse-proxy
# release line has to track the management server it enrolls against.
@@ -11,11 +11,13 @@ netbird_version: "0.73.2"
# Management server this proxy enrolls against (public API, TLS).
netbird_server_host: netbird.renner.uno
# Proxy domain — MUST be identical across every cluster member.
# Proxy domain — MUST be identical across every cluster member, and must NOT
# be a subdomain of the management domain (netbird.renner.uno) per NetBird
# docs (TLS/routing conflicts).
netbird_proxy_domain: "proxy.renner.uno"
# Reuse VPS-1's proxy token — a shared NB_PROXY_TOKEN across members with the
# same NB_PROXY_DOMAIN is a valid HA cluster, so no separate token is needed.
# A shared NB_PROXY_TOKEN across members with the same NB_PROXY_DOMAIN is a
# valid HA cluster, so one token serves every member.
netbird_proxy_token: "{{ vault_netbird_proxy_token }}"
# Relay auth secret — MUST be identical across every relay instance and the
@@ -1,8 +1,7 @@
# NetBird reverse-proxy — HA cluster member on a proxy-only host.
# Just the reverse-proxy container, bound directly to :443. No Traefik:
# unlike VPS-1 (where the proxy shares :443 with management/signal/relay via
# SNI routing), this host runs nothing else, so the proxy owns the port and
# terminates TLS + answers its own tls-alpn-01 ACME challenges directly.
# Just the reverse-proxy container, bound directly to :443. No Traefik: the
# proxy owns the port and terminates TLS + answers its own tls-alpn-01 ACME
# challenges directly.
services:
proxy:
image: netbirdio/reverse-proxy:{{ netbird_version }}
@@ -23,9 +23,3 @@ netbird_datastore_key: "{{ vault_netbird_datastore_key }}"
# Relay secret for WebSocket relay authentication
netbird_relay_secret: "{{ vault_netbird_relay_secret }}"
# Reverse proxy configuration
# IMPORTANT: Proxy domain must NOT be a subdomain of the management domain
# (netbird.renner.uno) per NetBird docs - causes TLS/routing conflicts
netbird_proxy_domain: "proxy.renner.uno"
netbird_proxy_token: "{{ vault_netbird_proxy_token }}"
@@ -16,7 +16,3 @@
- name: Restart netbird coturn
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d --force-recreate coturn
- name: Restart netbird proxy
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d --force-recreate proxy
+68 -9
View File
@@ -36,7 +36,6 @@
- /opt/netbird/management
- /opt/netbird/signal
- /opt/netbird/letsencrypt
- /opt/netbird/proxy/certs
- name: Deploy docker-compose.yml
ansible.builtin.template:
@@ -59,23 +58,83 @@
mode: "0644"
notify: Restart netbird coturn
- name: Deploy proxy.env
ansible.builtin.template:
src: proxy.env.j2
dest: /opt/netbird/proxy.env
mode: "0644"
notify: Restart netbird proxy
# The public edge is the proxy hosts (plays/install-netbird-proxy.yml); this
# host runs no reverse proxy. The tasks below remove the one it used to run,
# its watchdogs first so they cannot restart it mid-removal, and can go once
# the role has run against every host that ever carried it.
- name: Check for reverse-proxy watchdog units
ansible.builtin.stat:
path: "/etc/systemd/system/{{ item }}"
loop:
- netbird-proxy-watchdog.timer
- netbird-proxy-flood-watchdog.service
register: _proxy_watchdog_units
- name: Stop and disable reverse-proxy watchdogs
ansible.builtin.systemd:
name: "{{ item.item }}"
state: stopped
enabled: false
loop: "{{ _proxy_watchdog_units.results }}"
loop_control:
label: "{{ item.item }}"
when: item.stat.exists
- name: Remove reverse-proxy watchdog units and scripts
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /etc/systemd/system/netbird-proxy-watchdog.timer
- /etc/systemd/system/netbird-proxy-watchdog.service
- /etc/systemd/system/netbird-proxy-flood-watchdog.service
- /usr/local/bin/netbird-proxy-watchdog
- /usr/local/bin/netbird-proxy-flood-watchdog
- /run/netbird-proxy-watchdog
register: _proxy_watchdog_files
- name: Reload systemd after removing the watchdog units
ansible.builtin.systemd:
daemon_reload: true
when: _proxy_watchdog_files.changed
- name: Pull NetBird container images
ansible.builtin.shell: |
cd /opt/netbird && docker compose pull
changed_when: false
# --remove-orphans removes containers of services no longer in the compose
# file (the reverse proxy).
- name: Start NetBird containers
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d
cd /opt/netbird && docker compose up -d --remove-orphans
register: docker_compose_result
changed_when: "'Started' in docker_compose_result.stdout or 'Creating' in docker_compose_result.stdout"
changed_when: >-
'Started' in docker_compose_result.stdout or
'Creating' in docker_compose_result.stdout or
'Removed' in docker_compose_result.stdout or
'Removed' in docker_compose_result.stderr
- name: Remove reverse-proxy volumes
ansible.builtin.shell: |
set -eu
for v in netbird_netbird_proxy_certs netbird_netbird_proxy_data netbird_netbird_proxy_state; do
if [ -n "$(docker volume ls -q --filter "name=^${v}$")" ]; then
docker volume rm "$v"
echo "removed $v"
fi
done
register: _proxy_volumes
changed_when: "'removed' in _proxy_volumes.stdout"
- name: Remove reverse-proxy config and state
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /opt/netbird/proxy.env
- /opt/netbird/proxy
- /opt/netbird/proxy-state
# The management store has no cluster-side backup. Pulling one put a root
# key on this host that the cluster held, and this server's own state is
@@ -187,10 +187,9 @@ services:
- "traefik.http.routers.netbird-management.service=netbird-management"
- "traefik.http.services.netbird-management.loadbalancer.server.port=33073"
- "traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c"
# gRPC reverse-proxy service — remote reverse-proxy instances (e.g.
# proxy-1) connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443.
# The local proxy uses management:33073 directly, so it never needed
# this route; remote ones get a 404 without it. Same backend (h2c 33073).
# gRPC reverse-proxy service — the proxy hosts (proxy-1, proxy-2)
# connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443 and get
# a 404 without it. Same backend (h2c 33073).
- "traefik.http.routers.netbird-proxyservice.rule=Host(`{{ netbird_server_host }}`) && PathPrefix(`/management.ProxyService/`)"
- "traefik.http.routers.netbird-proxyservice.entrypoints=websecure"
- "traefik.http.routers.netbird-proxyservice.tls=true"
@@ -221,37 +220,6 @@ services:
max-size: "500m"
max-file: "2"
# Reverse Proxy Service (per docs.netbird.io/selfhosted/migration/enable-reverse-proxy)
proxy:
image: netbirdio/reverse-proxy:{{ netbird_version }}
container_name: netbird-proxy
restart: unless-stopped
networks:
- netbird
depends_on:
- management
env_file:
- ./proxy.env
volumes:
- netbird_proxy_certs:/certs
- /opt/netbird/proxy-state:/var/lib/netbird
labels:
- "traefik.enable=true"
- "traefik.tcp.routers.proxy-passthrough.entrypoints=websecure"
- "traefik.tcp.routers.proxy-passthrough.rule=HostSNI(`*`)"
- "traefik.tcp.routers.proxy-passthrough.tls.passthrough=true"
- "traefik.tcp.routers.proxy-passthrough.service=proxy-tls"
- "traefik.tcp.routers.proxy-passthrough.priority=1"
- "traefik.tcp.services.proxy-tls.loadbalancer.server.port=8443"
logging:
driver: "json-file"
options:
max-size: "500m"
max-file: "2"
networks:
netbird:
driver: bridge
volumes:
netbird_proxy_certs:
@@ -1,8 +0,0 @@
NB_PROXY_DOMAIN={{ netbird_proxy_domain }}
NB_PROXY_TOKEN={{ netbird_proxy_token }}
NB_PROXY_MANAGEMENT_ADDRESS=http://management:33073
NB_PROXY_ALLOW_INSECURE=true
NB_PROXY_ADDRESS=:8443
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
@@ -0,0 +1,976 @@
# Paperclip Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Deploy Paperclip (server-local agent runs, option B) into the homelab cluster behind an Authentik gate, with CNPG Postgres and OpenBao-sourced secrets.
**Architecture:** One Flux `Kustomization` (`paperclip`) applies `kubernetes/apps/paperclip/`: a single-replica Deployment on a ceph-block PVC, CNPG `pclp-pg` (PG17) with barman backups to Hetzner, and a Traefik Ingress that backends only the `paperclip-gate` Authentik outpost. Namespace, baseline CNP, ClusterSecretStore allowlist and image automation live in the existing `infrastructure` tree.
**Tech Stack:** Flux v2 (kustomize-controller, image-reflector v1.2.5), CNPG + barman-cloud plugin, External Secrets → OpenBao, Cilium CNP, Traefik, Authentik blueprints.
**Spec:** `docs/superpowers/specs/2026-09-25-paperclip-design.md`
## Global Constraints
- Worktree: `/home/krenner/source/homelab-worktrees/paperclip`, branch `feat/paperclip`. Never edit the main checkout.
- Commits as claude-bot: `git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit …`, message ends with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
- Push: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push …`. PR via `tea pr create --login claude-bot`. Never self-merge.
- One Bash command per call: no `&&`, `;`, `|` chaining, no `2>/dev/null`, no `python -c`.
- Namespace `paperclip`; hostname `paperclip.renner.uno`; LAN-only (no `dns.renner.uno/public` label).
- Image `ghcr.io/paperclipai/paperclip`, tag `nightly`, digest-pinned, `digestReflectionPolicy: Always`.
- Postgres: CNPG `pclp-pg`, `ghcr.io/cloudnative-pg/postgresql:17`, 2 instances, `ceph-block`.
- PVC `paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip`.
- OpenBao key `apps/paperclip` with fields `BETTER_AUTH_SECRET`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`, `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex), `CLAUDE_CODE_OAUTH_TOKEN`.
- Env: `PAPERCLIP_DEPLOYMENT_MODE=authenticated`, `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`, `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`, `HOST=0.0.0.0`, `PORT=3100`.
- Manifest comments: one terse factual line; rationale goes in commit messages/PR.
- No `kubectl apply`/`exec`/`rollout restart` against the live cluster; Flux applies after merge. Server-side `--dry-run=server` is allowed.
## Review Focus
1. **Pod replaced mid agent run** (image bump, secret rotation via Reloader, node drain) — `Recreate` kills in-flight runs; expected: Paperclip marks them failed/recoverable on restart, not wedged. Covered in Task 9 Step 6.
2. **Shared OAuth token expired or missing** — expected: agent runs fail with a visible auth error in Paperclip, server stays healthy. Covered in Task 9 Step 7.
3. **Direct access bypassing the gate** from another namespace pod — expected: connection refused/dropped by CNP. Covered in Task 9 Step 4.
4. **Unauthenticated LAN request** to `paperclip.renner.uno` — expected: redirect to Authentik, never Paperclip's UI. Covered in Task 9 Step 3.
5. **Nightly image with a breaking DB migration** — expected: a base backup exists before the first bump lands. Covered in Task 9 Step 5.
---
### Task 1: Upstream facts that decide two manifest values
No repo changes; produces two decisions consumed by Task 6.
**Interfaces:**
- Produces: `SIGNUP_DISABLED` (`"true"` or `"false"`), `CONFIG_PARTIAL_OK` (yes/no), `ENTRYPOINT_NEEDS_ROOT` (yes/no).
- [ ] **Step 1: Sign-up vs instance claim.** Read upstream `server/src/auth/better-auth.ts`, `server/src/routes/auth.ts`, and the claim flow for `authenticated`+`private` (search `claim` under `server/src`):
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/auth.ts -H "Accept: application/vnd.github.raw"`
Decide: if the first-admin claim creates the account through a path that ignores `disableSignUp` (e.g. invite/claim token handler calling the adapter directly), `SIGNUP_DISABLED="true"`. Otherwise `SIGNUP_DISABLED="false"` — the Authentik gate is the access control, and a follow-up PR flips it to `"true"` after the admin is claimed.
- [ ] **Step 2: Partial config.json.** Read `server/src/config.ts` and the config-file loader it calls (search `PAPERCLIP_CONFIG` / `readConfigFile`).
Run: `gh api repos/paperclipai/paperclip/contents/server/src/config.ts -H "Accept: application/vnd.github.raw"`
Decide `CONFIG_PARTIAL_OK=yes` if a file containing only `{"telemetry":{"enabled":false}}` parses (every other section optional/defaulted). If not, stop and report to the user — do not invent config fields.
- [ ] **Step 3: Entrypoint privilege.**
Run: `gh api repos/paperclipai/paperclip/contents/docker-entrypoint.sh -H "Accept: application/vnd.github.raw"`
`ENTRYPOINT_NEEDS_ROOT=yes` if it calls `usermod`/`groupmod`/`chown`/`gosu`. Record the result for Task 6.
- [ ] **Step 4: Anonymous health response.**
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/health.ts -H "Accept: application/vnd.github.raw"`
Confirm `GET /api/health` returns HTTP 200 to an anonymous caller when healthy (probes use it). If it returns 401 anonymously, use a `tcpSocket` probe on 3100 in Task 6 instead.
---
### Task 2: Namespace, baseline CNP, ClusterSecretStore allowlist
**Files:**
- Modify: `kubernetes/infrastructure/namespaces/namespaces.yaml` (append)
- Create: `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
- Modify: `kubernetes/infrastructure/network-policies/kustomization.yaml`
- Modify: `kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml` (sorted list)
**Interfaces:**
- Produces: namespace `paperclip`; ESO may reference `openbao` from it.
- [ ] **Step 1: Append the namespace** to the end of `namespaces.yaml`:
```yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: paperclip
```
- [ ] **Step 2: Create `paperclip-ns.yaml`** (copy of `paperless-ns.yaml`'s `default-deny-ingress`, ns renamed, paperless-only CNPs dropped):
```yaml
# Traefik is not admitted: the only entry is the paperclip-gate outpost (apps/paperclip/network-policy.yaml).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: default-deny-ingress
namespace: paperclip
spec:
endpointSelector: {}
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": database
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
- fromEntities:
- host
- remote-node
- fromEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": rook-ceph
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- toEntities:
- host
- remote-node
- world
- toEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
```
- [ ] **Step 3: Register it** — in `network-policies/kustomization.yaml` add ` - paperclip-ns.yaml` directly after ` - paperless-ns.yaml`.
- [ ] **Step 4: Allowlist the namespace** — in `openbao-clustersecretstore.yaml` insert ` - paperclip` between ` - monitoring` and ` - paperless` (list is sorted).
- [ ] **Step 5: Render check.**
Run: `kubectl kustomize kubernetes/infrastructure/network-policies`
Expected: output contains `namespace: paperclip`, exit 0.
Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml`
Expected: `clustersecretstore.external-secrets.io/openbao configured (server dry run)`.
- [ ] **Step 6: Commit** `feat(paperclip): namespace, baseline CNP and OpenBao allowlist`.
---
### Task 3: OpenBao secret `apps/paperclip`
Out-of-repo; must exist before merge or the ExternalSecret never becomes Ready.
**Interfaces:**
- Produces: OpenBao KV `apps/paperclip` with the four fields in Global Constraints.
- [ ] **Step 1:** Ask wiki-synth for the claude-bot OpenBao write procedure (note `claude-bot-openbao-write-only-identity`): login method, KV mount, exact `bao kv put`/`patch` syntax for an `apps/<name>` key. Use exactly what it returns.
- [ ] **Step 2: Write the generated fields** as claude-bot (write-only identity — never read values back). Values generated inline by the command:
`BETTER_AUTH_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_SECRETS_MASTER_KEY` = `openssl rand -hex 32` (64 hex chars = 32 bytes, accepted format).
Generate each with a separate `openssl rand -hex 32` call and pass it directly into the `bao kv put` from Step 1; do not echo or log them.
- [ ] **Step 3: Ask the user** to add the shared token themselves (never handle it): "Run `claude setup-token`, then `bao kv patch <mount>/apps/paperclip CLAUDE_CODE_OAUTH_TOKEN=<token>`." Wait for confirmation.
- [ ] **Step 4: Verify field names only** (no values): `bao kv metadata get <mount>/apps/paperclip` shows a current version ≥2. If claude-bot cannot read metadata, ask the user to confirm the four keys exist.
---
### Task 4: Image automation + initial digest
**Files:**
- Create: `kubernetes/infrastructure/image-automation/paperclip.yaml`
- Modify: `kubernetes/infrastructure/image-automation/kustomization.yaml`
**Interfaces:**
- Produces: ImagePolicy `flux-system:paperclip`; `INITIAL_DIGEST` (`sha256:…`) for Task 6.
- [ ] **Step 1: Create `paperclip.yaml`:**
```yaml
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
name: paperclip
namespace: flux-system
spec:
image: ghcr.io/paperclipai/paperclip
interval: 1h
---
# Upstream publishes no semver tags; track the nightly tag by digest.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
name: paperclip
namespace: flux-system
spec:
imageRepositoryRef:
name: paperclip
digestReflectionPolicy: Always
filterTags:
pattern: '^nightly$'
policy:
alphabetical:
order: asc
```
- [ ] **Step 2: Register** — add ` - paperclip.yaml` after ` - paperless-ngx.yaml` in `image-automation/kustomization.yaml`.
- [ ] **Step 3: Server dry-run.**
Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/image-automation/paperclip.yaml`
Expected: both objects `created (server dry run)`; no `unknown field "digestReflectionPolicy"` error.
- [ ] **Step 4: Resolve the current nightly digest.**
The workstation has `docker` only (no skopeo/crane/buildx, checked 2026-09-25); `docker manifest inspect` lists per-platform digests but not the index digest, so pull once:
Run: `docker pull ghcr.io/paperclipai/paperclip:nightly`
Run: `docker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/paperclipai/paperclip:nightly`
Expected: `ghcr.io/paperclipai/paperclip@sha256:<64 hex>`. Record the `sha256:…` part as `INITIAL_DIGEST`.
Run: `docker image rm ghcr.io/paperclipai/paperclip:nightly`
- [ ] **Step 5: Commit** `feat(paperclip): Flux image automation tracks the nightly digest`.
---
### Task 5: Postgres (CNPG) and secrets in the app directory
**Files:**
- Create: `kubernetes/apps/paperclip/cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`, `hetzner-creds-externalsecret.yaml`, `externalsecret.yaml`
**Interfaces:**
- Produces: Secret `pclp-pg-app` (CNPG, key `uri`), Service `pclp-pg-rw`, Secret `paperclip-secrets`.
- [ ] **Step 1: `cluster.yaml`:**
```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pclp-pg
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
# PG17: upstream's external Postgres target (docker/docker-compose.yml).
imageName: ghcr.io/cloudnative-pg/postgresql:17
storage:
storageClass: ceph-block
size: 10Gi
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: pclp-pg-store
bootstrap:
initdb:
database: paperclip
owner: paperclip
postgresql:
parameters:
shared_buffers: "256MB"
max_connections: "200"
checkpoint_timeout: "15min"
max_wal_size: "2GB"
wal_compression: "zstd"
archive_timeout: "30min"
synchronous:
method: any
number: 1
dataDurability: preferred
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 1Gi
```
- [ ] **Step 2: `objectstore.yaml`:**
```yaml
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pclp-pg-store
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup target
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/pclp-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
```
- [ ] **Step 3: `scheduled-backup.yaml`:**
```yaml
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: pclp-pg-daily
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup schedule
spec:
schedule: "0 45 2 * * *"
immediate: true
backupOwnerReference: self
cluster:
name: pclp-pg
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
```
- [ ] **Step 4: `hetzner-creds-externalsecret.yaml`:**
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hetzner-s3-creds
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # barman S3 creds
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: hetzner-s3-creds
dataFrom:
- extract:
key: infrastructure/hetzner-s3
```
- [ ] **Step 5: `externalsecret.yaml`:**
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: paperclip-secrets
namespace: paperclip
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: paperclip-secrets
dataFrom:
- extract:
key: apps/paperclip
```
- [ ] **Step 6: Commit** `feat(paperclip): CNPG pclp-pg with barman backups and OpenBao secrets`. (Rendered together with Task 6's kustomization.)
---
### Task 6: Server workload, storage, ingress, network policy
**Files:**
- Create: `kubernetes/apps/paperclip/pvc.yaml`, `paperclip.yaml`, `gate-service.yaml`, `ingress.yaml`, `network-policy.yaml`, `kustomization.yaml`
**Interfaces:**
- Consumes: `pclp-pg-app` (`uri`), `paperclip-secrets`, ImagePolicy `flux-system:paperclip`, `INITIAL_DIGEST` (Task 4), `SIGNUP_DISABLED` / `ENTRYPOINT_NEEDS_ROOT` / probe choice (Task 1).
- Produces: pods labelled `app.kubernetes.io/name: paperclip` listening on 3100; Service `paperclip:3100`; ExternalName `ak-outpost-paperclip-gate:9000`.
- [ ] **Step 1: `pvc.yaml`:**
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperclip-data
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
accessModes: [ReadWriteOnce]
storageClassName: ceph-block
resources:
requests:
storage: 20Gi
```
- [ ] **Step 2: `paperclip.yaml`** — replace `<INITIAL_DIGEST>` with the Task 4 value and `<SIGNUP_DISABLED>` with the Task 1 value. If Task 1 found `ENTRYPOINT_NEEDS_ROOT=no`, replace the container `securityContext` with `runAsNonRoot: true`, `runAsUser: 1000`, `runAsGroup: 1000`, `allowPrivilegeEscalation: false`, `capabilities: {drop: ["ALL"]}`. If Task 1 chose `tcpSocket`, replace each `httpGet` block with `tcpSocket: {port: http}`.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperclip
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
secret.reloader.stakater.com/reload: paperclip-secrets
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: paperclip
template:
metadata:
labels:
app.kubernetes.io/name: paperclip
spec:
enableServiceLinks: false
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
initContainers:
# Telemetry is only configurable in config.json.
- name: disable-telemetry
image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
command:
- sh
- -c
- |
set -eu
f=/paperclip/instances/default/config.json
mkdir -p "$(dirname "$f")"
if [ -s "$f" ]; then
jq '.telemetry.enabled = false' "$f" > "$f.tmp"
else
printf '%s\n' '{"telemetry":{"enabled":false}}' > "$f.tmp"
fi
mv "$f.tmp" "$f"
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: data
mountPath: /paperclip
containers:
- name: paperclip
image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
ports:
- name: http
containerPort: 3100
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: pclp-pg-app
key: uri
- name: HOST
value: "0.0.0.0"
- name: PORT
value: "3100"
- name: PAPERCLIP_DEPLOYMENT_MODE
value: authenticated
- name: PAPERCLIP_DEPLOYMENT_EXPOSURE
value: private
- name: PAPERCLIP_PUBLIC_URL
value: https://paperclip.renner.uno
- name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
value: "<SIGNUP_DISABLED>"
envFrom:
- secretRef:
name: paperclip-secrets
volumeMounts:
- name: data
mountPath: /paperclip
startupProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 60
livenessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 30
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 10
timeoutSeconds: 5
resources:
requests:
cpu: 250m
memory: 1Gi
limits:
memory: 4Gi
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
volumes:
- name: data
persistentVolumeClaim:
claimName: paperclip-data
---
apiVersion: v1
kind: Service
metadata:
name: paperclip
namespace: paperclip
spec:
selector:
app.kubernetes.io/name: paperclip
ports:
- name: http
port: 3100
targetPort: http
```
- [ ] **Step 3: `gate-service.yaml`:**
```yaml
# ExternalName mirror so the Ingress can backend the paperclip-gate outpost in the authentik namespace.
apiVersion: v1
kind: Service
metadata:
name: ak-outpost-paperclip-gate
namespace: paperclip
spec:
type: ExternalName
externalName: ak-outpost-paperclip-gate.authentik.svc.cluster.local
ports:
- port: 9000
targetPort: 9000
```
- [ ] **Step 4: `ingress.yaml`:**
```yaml
# LAN-only (no dns.renner.uno/public label); backends the gate outpost, never the Service.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: paperclip
namespace: paperclip
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
spec:
ingressClassName: traefik
tls:
- hosts:
- paperclip.renner.uno
secretName: paperclip-tls
rules:
- host: paperclip.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ak-outpost-paperclip-gate
port:
number: 9000
```
- [ ] **Step 5: `network-policy.yaml`** (Traefik pod-side port is 8443, verified via `kubectl get pods -n ingress -l app.kubernetes.io/name=traefik`):
```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-ingress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
goauthentik.io/outpost-name: paperclip-gate
toPorts:
- ports:
- port: "3100"
protocol: TCP
---
# Covers the server and every agent it spawns (same pod).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-egress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
cnpg.io/cluster: pclp-pg
toPorts:
- ports:
- port: "5432"
protocol: TCP
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
# Gitea, Woodpecker, HA, Loki via their *.renner.uno names.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8443"
protocol: TCP
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
- toEntities:
- world
toPorts:
- ports:
- port: "443"
protocol: TCP
```
- [ ] **Step 6: `kustomization.yaml`:**
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- pvc.yaml
- paperclip.yaml
- gate-service.yaml
- ingress.yaml
- network-policy.yaml
```
- [ ] **Step 7: Render check.**
Run: `kubectl kustomize kubernetes/apps/paperclip`
Expected: exit 0; output contains `kind: Cluster`, `kind: Deployment`, `claimName: paperclip-data`, and no literal `<INITIAL_DIGEST>` or `<SIGNUP_DISABLED>`.
Run: `grep -rn "<INITIAL_DIGEST>\|<SIGNUP_DISABLED>" kubernetes/apps/paperclip`
Expected: no output, exit 1.
- [ ] **Step 8: Commit** `feat(paperclip): server Deployment, PVC, gated Ingress and CNPs`.
---
### Task 7: Authentik gate
**Files:**
- Modify: `kubernetes/apps/authentik/blueprints-configmap.yaml` (insert before the line ` # navidrome-gate: the Navidrome web UI.`)
- Create: `kubernetes/apps/authentik/paperclip-gate-egress.yaml`
- Modify: `kubernetes/apps/authentik/kustomization.yaml`
**Interfaces:**
- Produces: outpost pods labelled `goauthentik.io/outpost-name: paperclip-gate`, Service `ak-outpost-paperclip-gate.authentik:9000`.
- [ ] **Step 1: Insert blueprint entries** (same indentation as the neighbouring `wiki-gate` block):
```yaml
# paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
- id: paperclip-users-group
model: authentik_core.group
identifiers:
name: Paperclip Users
state: present
attrs:
is_superuser: false
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: paperclip-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: paperclip-gate
state: present
attrs:
authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://paperclip.paperclip.svc.cluster.local:3100
internal_host_ssl_validation: false
external_host: https://paperclip.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: paperclip-gate
state: present
attrs:
name: Paperclip
slug: paperclip-gate
provider: !KeyOf paperclip-gate-provider
meta_launch_url: https://paperclip.renner.uno
meta_description: "Paperclip agent orchestration"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, paperclip-gate]]
group: !KeyOf paperclip-users-group
order: 0
state: present
attrs:
enabled: true
negate: false
- model: authentik_outposts.outpost
identifiers:
name: paperclip-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf paperclip-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
```
- [ ] **Step 2: `paperclip-gate-egress.yaml`:**
```yaml
# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: paperclip-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
app.kubernetes.io/name: paperclip
toPorts:
- ports:
- port: "3100"
protocol: TCP
```
- [ ] **Step 3: Register** — add ` - paperclip-gate-egress.yaml` after ` - paperless-gate-egress.yaml` in `apps/authentik/kustomization.yaml`.
- [ ] **Step 4: Render + dry-run.**
Run: `kubectl kustomize kubernetes/apps/authentik`
Expected: exit 0 (valid YAML incl. the blueprint ConfigMap).
Run: `kubectl apply --dry-run=server -f kubernetes/apps/authentik/paperclip-gate-egress.yaml`
Expected: `ciliumnetworkpolicy.cilium.io/paperclip-gate-egress created (server dry run)`.
- [ ] **Step 5: Commit** `feat(authentik): paperclip-gate outpost and Paperclip Users group`.
---
### Task 8: Flux Kustomization
**Files:**
- Create: `kubernetes/clusters/homelab/paperclip.yaml`
- [ ] **Step 1:**
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: paperclip
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 20m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/paperclip
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
failed: "has(status.phase) && status.phase.startsWith('Failed')"
current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
```
- [ ] **Step 2: Server dry-run.**
Run: `kubectl apply --dry-run=server -f kubernetes/clusters/homelab/paperclip.yaml`
Expected: `kustomization.kustomize.toolkit.fluxcd.io/paperclip created (server dry run)`.
- [ ] **Step 3: Commit** `feat(paperclip): Flux Kustomization`.
---
### Task 9: PR, merge by user, live verification
- [ ] **Step 1: Push and open the PR** (Task 3 must be complete).
Run: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push -u origin feat/paperclip`
Run: `tea pr create --login claude-bot --base master --head feat/paperclip --title "feat(paperclip): deploy Paperclip (server-local runs, trial)" --description "<summary of spec decisions + link to spec + bootstrap steps>\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)"`
Then wait for the user to merge. Do not merge.
- [ ] **Step 2: Reconciliation.** After merge:
Run: `flux get kustomizations paperclip`
Expected: `Ready True`. If False, read `kubectl describe kustomization paperclip -n flux-system` and the failing object's events; diagnose, don't poke.
Run: `kubectl get cluster -n paperclip pclp-pg`
Expected: `Cluster in healthy state`, 2 ready instances.
Run: `kubectl get pods -n paperclip -l app.kubernetes.io/name=paperclip`
Expected: `1/1 Running`.
- [ ] **Step 3: Gate enforced (Review Focus 4).**
Run: `curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https://paperclip.renner.uno/`
Expected: `302` redirecting to `https://paperclip.renner.uno/outpost.goauthentik.io/…` or `auth.renner.uno`, never `200`.
- [ ] **Step 4: Gate not bypassable (Review Focus 3).**
`default` has no CNPs (verified 2026-09-25), so its egress is open and only paperclip's ingress policy can block the probe.
Run: `kubectl get ciliumnetworkpolicies -n default`
Expected: `No resources found` (else pick another CNP-free namespace).
Run: `kubectl run cnp-probe -n default --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -sS -m 5 http://paperclip.paperclip.svc.cluster.local:3100/api/health`
Expected: timeout (exit 28), no HTTP response.
- [ ] **Step 5: First backup (Review Focus 5).**
Run: `kubectl get backup -n paperclip`
Expected: one `pclp-pg-daily-…` with phase `completed`.
- [ ] **Step 6: Image policy.**
Run: `kubectl get imagepolicy -n flux-system paperclip -o jsonpath='{.status.latestRef}'`
Expected: `nightly` tag with a `sha256:` digest.
- [ ] **Step 7: Hand-over to the user (browser steps, theirs):**
1. Open `https://paperclip.renner.uno`, pass Authentik, claim the instance as admin.
2. Create a test agent with the `claude-local` adapter and a trivial task ("reply with OK"); it must complete with the shared token (Review Focus 2: if it fails, report the run's error text — an expired token must show as an auth error, not a hang).
3. If `SIGNUP_DISABLED` was `"false"`, open the follow-up PR flipping `PAPERCLIP_AUTH_DISABLE_SIGN_UP` to `"true"` once the admin is claimed.
4. Review Focus 1 is observed on the first nightly bump: check the interrupted-run status in the UI.
- [ ] **Step 8: Wiki.** Route the verified Paperclip facts (execution topology, env vars, image tags, digest-reflection usage) through wiki-ingest → wiki-distill once the deployment has proven out.
@@ -0,0 +1,181 @@
# Paperclip in the homelab cluster — design
Date: 2026-09-25
Status: draft, awaiting review
## Goal
Run [Paperclip](https://github.com/paperclipai/paperclip) (MIT, agent-orchestration
server + UI) in the homelab cluster for real use: its agents do actual work against
homelab systems. This spec covers the **platform** only. Bot identities and their
credentials are out of scope (see "Deferred").
## Decisions
| Topic | Decision |
|---|---|
| Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
| Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`); Paperclip stores and refreshes it. Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
| Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
| Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
### Known trade-off of B (accepted for the trial)
Upstream `runChildProcess()` spawns every local adapter with
`{...sanitizeInheritedPaperclipEnv(process.env), ...opts.env}`; the sanitizer strips only
`PAPERCLIP_*` keys. Every agent therefore sees `DATABASE_URL`, `BETTER_AUTH_SECRET`,
the shared filesystem (including every other agent's workspace and the
`ai-local-logins` credentials) and every agent-scoped secret it can read from disk. Under B, no bot may hold
a credential we would not tolerate any other bot using. Per-bot isolation is C's job.
## Architecture
```
LAN client ─▶ Traefik ─▶ ExternalName svc ─▶ ak-outpost-paperclip-gate (authentik ns)
│ forward-auth OK
▼
paperclip Deployment (ns paperclip, 1 replica)
├─ server :3100 (+ agent CLIs as child processes)
├─ PVC /paperclip (ceph-block)
└─▶ pclp-pg-rw (CNPG, PG17, 2 instances) ─▶ barman ▶ Hetzner S3
```
## Components
All modeled on `kubernetes/apps/paperless/` unless stated.
### Namespace and baseline policy
- `Namespace paperclip` in `kubernetes/infrastructure/namespaces/namespaces.yaml`.
- Baseline default-deny CNP `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
(copy of `paperless-ns.yaml`, renamed).
- Add `paperclip` to the `openbao` ClusterSecretStore namespace allowlist.
### Flux
- `kubernetes/clusters/homelab/paperclip.yaml`: Flux `Kustomization`,
`path: ./kubernetes/apps/paperclip`, `prune: true`, `wait: true`, same `dependsOn` and
CNPG `healthCheckExprs` as `paperless.yaml`.
- `kubernetes/apps/paperclip/kustomization.yaml`: plain resource list.
### Server Deployment (`paperclip.yaml`)
- Image `ghcr.io/paperclipai/paperclip:nightly@sha256:<digest>` with the Flux
image-automation marker for digest updates.
- `replicas: 1`, `strategy: Recreate` (single writer on an RWO PVC).
- Starts as root: upstream `docker-entrypoint.sh` remaps the `node` user, chowns
`/paperclip`, then drops to uid 1000 via `gosu`. Caps limited to CHOWN,
DAC_OVERRIDE, FOWNER, SETGID, SETUID; `fsGroup: 1000`.
- `automountServiceAccountToken: false` (all agents share the pod filesystem).
- Port 3100; `Service paperclip` ClusterIP :3100.
- Probes: startup and readiness on `GET /api/health` with `Host: localhost` (the
private-hostname guard rejects the pod IP); liveness is a TCP check, since
`/api/health` runs `SELECT 1` and a DB outage must not restart the server.
- Env:
- `DATABASE_URL` assembled from CNPG `pclp-pg-app` Secret (`uri` key).
- `HOST=0.0.0.0`, `PORT=3100`
- `PAPERCLIP_DEPLOYMENT_MODE=authenticated`
- `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`
- `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`
- `PAPERCLIP_AUTH_DISABLE_SIGN_UP=false` until the first admin is claimed: the
claim (`/bootstrap/claim`) requires an already signed-in Better Auth account.
A follow-up flips it to `true`.
- `PAPERCLIP_TELEMETRY_DISABLED=1` (read by `resolveTelemetryConfig`).
- `PAPERCLIP_API_URL=http://paperclip.paperclip.svc.cluster.local:3100` plus
`PAPERCLIP_ALLOWED_HOSTNAMES=paperclip.paperclip.svc.cluster.local`: upstream
otherwise hands agents the public URL, and the gate (`intercept_header_auth`)
rejects their bearer tokens. The Service name also works for sandbox pods under
option C; agent JWTs still authenticate each call.
- `envFrom: secretRef paperclip-secrets`
- Reloader: `secret.reloader.stakater.com/reload: paperclip-secrets`.
### Storage
- `PVC paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip` (`PAPERCLIP_HOME`
and `HOME`): instance config, workspaces, repo checkouts, per-agent `claude-runtime`
dirs.
### Postgres
- CNPG `Cluster pclp-pg`: 2 instances, `imageName: ghcr.io/cloudnative-pg/postgresql:17`,
`ceph-block` 10Gi, `bootstrap.initdb {database: paperclip, owner: paperclip}`,
barman-cloud plugin as WAL archiver.
- `ObjectStore pclp-pg-store` → `s3://renner-uno-k8s-cnpg/pclp-pg`, creds via the
existing `hetzner-s3-creds` ExternalSecret pattern.
- `ScheduledBackup` nightly, `method: plugin`, `backupOwnerReference: self`.
- `kustomize.toolkit.fluxcd.io/prune: disabled` on Cluster, ObjectStore, ScheduledBackup.
- PG17 deviates from the cluster-wide PG16 convention because upstream's external
Postgres target is `postgres:17` (`docker/docker-compose.yml`) and its embedded default
is PG18.
### Secrets
One ExternalSecret `paperclip-secrets` ← OpenBao `apps/paperclip` (whole-object
extract):
- `BETTER_AUTH_SECRET`
- `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`
- `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex). Held in OpenBao, not on the PVC, so
encrypted agent secrets in Postgres survive PVC loss.
### Ingress and SSO
- Hostname `paperclip.renner.uno`, Traefik `Ingress`, `cert-manager.io/cluster-issuer:
letsencrypt-cluster`, **no** `dns.renner.uno/public` label (LAN-only).
- Authentik gate, per the paperless pattern:
- blueprint entries in `kubernetes/apps/authentik/blueprints-configmap.yaml`:
proxy provider `paperclip-gate` (`internal_host:
http://paperclip.paperclip.svc.cluster.local:3100`, `external_host:
https://paperclip.renner.uno`), application, group "Paperclip Users", policy binding;
- outpost `ak-outpost-paperclip-gate`;
- `kubernetes/apps/authentik/paperclip-gate-egress.yaml` (no `auth.renner.uno`
outpost Ingress: like wiki/shelly gates, the outpost serves its own callback on
`paperclip.renner.uno`);
- `ExternalName` Service in `paperclip` ns pointing at the outpost; the app Ingress
targets it on :9000.
- Paperclip's own email/password login stays on (no header/OIDC login upstream).
### Network policy (`kubernetes/apps/paperclip/network-policy.yaml`)
Covers the server **and every agent** (same pod under B).
- Ingress to the app pod: only from the `paperclip-gate` outpost (by label) on 3100.
- Egress from the app pod:
- `pclp-pg` pods :5432
- kube-dns :53
- `ingress` namespace (Traefik) :8443 (pod-side websecure port) — reaches Gitea, Woodpecker, HA, Loki by their
`*.renner.uno` names
- `kube-apiserver` entity :6443
- `world` TCP :443 (Anthropic API, GitHub, npm)
- No SSH egress; git over HTTPS only. The namespace baseline
(`paperclip-ns.yaml`) also restricts `world` to TCP 443 — Cilium unions allows
across CNPs, so an unrestricted baseline would override this list.
- Known gap (house baseline): pods in `monitoring`/`database` and other pods in
`paperclip` can reach :3100 without the gate; Paperclip's own login still applies.
### Image automation
`kubernetes/infrastructure/image-automation/paperclip.yaml`: `ImageRepository`
(`ghcr.io/paperclipai/paperclip`) + `ImagePolicy` filtering tag `^nightly$` with
`digestReflectionPolicy: Always` (image-reflector-controller v1.2.5 in cluster; field
available since v0.35.0). First use of digest reflection in this repo.
## Guardrails
- No bot receives cluster-admin or any binding in `flux-system`, `authentik`, `openbao`.
- Git work goes through PRs; bots get no merge rights on protected branches.
- Nothing is granted under B that we would not tolerate any agent misusing.
## Bootstrap (one-time)
1. Place `apps/paperclip` values in OpenBao (the three generated secrets).
2. Merge; Flux deploys with sign-up enabled.
3. Sign up through the gate and claim the instance as admin; then set
`PAPERCLIP_AUTH_DISABLE_SIGN_UP=true`.
4. In onboarding "Connect a model", choose Claude subscription and run the shown login
**as the node user** (the entrypoint makes exec sessions root, and root-owned 0600
credentials are unreadable to the server):
`kubectl exec -it -n paperclip deploy/paperclip -- gosu node sh -c 'export CLAUDE_CONFIG_DIR=<shown path> && mkdir -p "$CLAUDE_CONFIG_DIR" && claude auth login'`
Repeat after loss of the PVC or database.
## Verification
- Flux Kustomization `paperclip` Ready; CNPG `pclp-pg` healthy; first ScheduledBackup
completes.
- `curl https://paperclip.renner.uno/api/health` through the gate returns healthy after
Authentik login; unauthenticated request redirects to `auth.renner.uno`.
- Direct pod access from another namespace is denied (CNP).
- A test agent (claude-local) completes a trivial task using the shared Claude connection.
- Flux ImagePolicy reports a `nightly@sha256:` digest.
## Deferred
- **Option C** (sandboxed runs) — required before any per-bot credential isolation.
- **AIR-Manager** role and all bot identities/credentials.
- Public exposure.
@@ -1036,6 +1036,83 @@ data:
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
- id: paperclip-users-group
model: authentik_core.group
identifiers:
name: Paperclip Users
state: present
attrs:
is_superuser: false
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: paperclip-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: paperclip-gate
state: present
attrs:
authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://paperclip.paperclip.svc.cluster.local:3100
internal_host_ssl_validation: false
external_host: https://paperclip.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: paperclip-gate
state: present
attrs:
name: Paperclip
slug: paperclip-gate
provider: !KeyOf paperclip-gate-provider
meta_launch_url: https://paperclip.renner.uno
meta_description: "Paperclip agent orchestration"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, paperclip-gate]]
group: !KeyOf paperclip-users-group
order: 0
state: present
attrs:
enabled: true
negate: false
- model: authentik_outposts.outpost
identifiers:
name: paperclip-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf paperclip-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# navidrome-gate: the Navidrome web UI. Navidrome auto-creates the
# X-Authentik-Username user on first login; the first user is admin.
# /rest and /share bypass the gate at the Traefik router.
@@ -1094,6 +1171,85 @@ data:
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# lidarr-gate: Lidarr runs with AuthenticationMethod=External, so the gate
# is its only access control.
- id: music-librarians-group
model: authentik_core.group
identifiers:
name: Music Librarians
state: present
attrs:
is_superuser: false
# Membership is owned here; blueprint apply reverts UI edits.
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: lidarr-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: lidarr-gate
state: present
attrs:
authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://lidarr.navidrome.svc.cluster.local
internal_host_ssl_validation: false
external_host: https://lidarr.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: lidarr-gate
state: present
attrs:
name: Lidarr
slug: lidarr-gate
provider: !KeyOf lidarr-gate-provider
meta_launch_url: https://lidarr.renner.uno
meta_description: "Music library gaps: discographies, Spotify artists"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, lidarr-gate]]
group: !KeyOf music-librarians-group
order: 0
state: present
attrs:
enabled: true
negate: false
failure_result: false
- model: authentik_outposts.outpost
identifiers:
name: lidarr-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf lidarr-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
- model: authentik_outposts.outpost
identifiers:
name: authentik Embedded Outpost
@@ -7,6 +7,7 @@ resources:
- helmrelease.yaml
- paperless-gate-ingress.yaml
- paperless-gate-egress.yaml
- paperclip-gate-egress.yaml
- vscode-gate-egress.yaml
- shelly-manager-gate-egress.yaml
- esphome-gate-egress.yaml
@@ -18,6 +19,7 @@ resources:
- llama-gate-egress.yaml
- wiki-gate-egress.yaml
- navidrome-gate-egress.yaml
- lidarr-gate-egress.yaml
- ldap-outpost-cert.yaml
- ldap-outpost-lb.yaml
- ldap-outpost-cnp.yaml
@@ -0,0 +1,20 @@
# Lets the lidarr-gate outpost reach Lidarr; the matching ingress rule is in
# network-policies/navidrome-ns.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: lidarr-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: lidarr-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": navidrome
app.kubernetes.io/name: lidarr
toPorts:
- ports:
- port: "8686"
protocol: TCP
@@ -0,0 +1,19 @@
# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: paperclip-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
app.kubernetes.io/name: paperclip
toPorts:
- ports:
- port: "3100"
protocol: TCP
@@ -88,6 +88,64 @@ spec:
# /ready answers 200 unauthenticated and is in this namespace, so the
# same intra-namespace allowance the two above rely on covers it.
- http://loki.monitoring.svc.cluster.local:3100/ready
# APP-HEALTH probes for the two gated navidrome-namespace apps. Same
# rule as the block above (monitoring-vantage-point-rule), but these
# are the first targets here that leave the monitoring namespace.
#
# Both sit entirely behind an Authentik outpost — navidrome.renner.uno
# on ak-outpost-navidrome-gate (navidrome/ingress.yaml:21),
# lidarr.renner.uno on ak-outpost-lidarr-gate
# (navidrome/lidarr.yaml:218) — so an EXTERNAL probe of either name
# measures the gate, not the app, and stays green while the app behind
# a live gate is dead. That external signal is still wanted, and REN-2
# adds it from UptimeRobot; these are the other half of the pair.
#
# NO CiliumNetworkPolicy change is needed, unlike what a reading of
# cnp-cross-namespace-allow-apps-gatus-to-monitored-services suggests.
# That gotcha is about `apps`, where GATUS runs; the influxdb
# allow-list it describes already contained `monitoring`. The
# navidrome baseline CNP likewise already admits the whole monitoring
# namespace on every port
# (infrastructure/network-policies/navidrome-ns.yaml:25-27), which is
# where blackbox-exporter runs. Nothing in the navidrome namespace is
# edited by this change.
#
# /ping on both: unauthenticated, and already the pods' own kubelet
# probe path (navidrome/deployment.yaml:61-74,
# navidrome/lidarr.yaml:114-127). For lidarr that it is ungated is
# proven independently of the gate — lidarr.yaml:29 curls /ping in a
# loop BEFORE it has read the API key out of config.xml.
- http://navidrome.navidrome.svc.cluster.local:80/ping
- http://lidarr.navidrome.svc.cluster.local:80/ping
# APP-HEALTH probe for paperless. Same rule as the block above
# (monitoring-vantage-point-rule); the blind spot it closes is the one
# gatus already documents at apps/gatus/configmap.yaml:176-185 —
# through the paperless-gate outpost EVERY path, /api/ included,
# answers 302 to an unauthenticated caller, so no external probe of
# paperless.renner.uno can tell a live app from a live gate. That
# external signal is still wanted, and REN-2 adds it from UptimeRobot;
# this is the other half of the pair.
#
# That gatus comment says the in-cluster probe "needs a
# CiliumNetworkPolicy first". It does not, for THIS prober — the
# sentence was written for gatus, which runs in `apps`.
# paperless-ns.yaml:24-26 already admits the whole monitoring
# namespace, which is where blackbox-exporter runs, and Cilium unions
# the allow rules of every CNP selecting an endpoint (the mechanism
# apps/paperless/network-policy.yaml:8-12 relies on). Nothing in the
# paperless namespace is edited by this change.
#
# /accounts/login/ because paperless-ngx has no unauthenticated health
# endpoint at all. It is the app's own established "the webserver
# answers" signal — the same path its startup, liveness and readiness
# probes use (paperless.yaml:252-272) — and it renders a Django view
# rather than a static file. It does NOT cover the consumer/celery
# workers: see process-liveness-vs-protocol-liveness.
#
# The cluster-local Host header is accepted only because
# PAPERLESS_ALLOWED_HOSTS is "*" (paperless.yaml:27). Narrow that and
# Django answers 400 here and this probe goes permanently red.
- http://paperless.paperless.svc.cluster.local:80/accounts/login/
# WAN-path probes of the public edge BY PROXY IP. The name-based
# home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
# and never leaves the LAN — the vantage-point blind spot that let the
@@ -5,3 +5,4 @@ resources:
- deployment.yaml
- gate-service.yaml
- ingress.yaml
- lidarr.yaml
+224
View File
@@ -0,0 +1,224 @@
# Lidarr as a read-mostly "what is missing" view over the Navidrome library.
# No download client or indexer is configured.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lidarr-config
namespace: navidrome
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-block
resources:
requests:
storage: 5Gi
---
# Applied on every pod start, before the root folder exists: no renames, no tag
# writes, no deletes. Adding /music as root folder starts Lidarr's library import.
apiVersion: v1
kind: ConfigMap
metadata:
name: lidarr-reconcile
namespace: navidrome
data:
reconcile.sh: |
set -eu
API=http://localhost:8686/api/v1
until curl -sf http://localhost:8686/ping >/dev/null; do sleep 5; done
KEY=$(sed -n 's:.*<ApiKey>\(.*\)</ApiKey>.*:\1:p' /config/config.xml)
if [ -z "$KEY" ]; then
echo "FATAL: no ApiKey in /config/config.xml" >&2
exit 1
fi
H="X-Api-Key: $KEY"
put() {
cur=$(curl -sf -H "$H" "$API/$1")
id=$(printf '%s' "$cur" | jq -r '.id')
printf '%s' "$cur" | jq -c "$2" \
| curl -sf -X PUT -H "$H" -H 'Content-Type: application/json' -d @- "$API/$1/$id" >/dev/null
echo "reconciled $1"
}
put config/naming '.renameTracks = false'
put config/metadataprovider '.writeAudioTags = "no" | .scrubAudioTags = false | .embedCoverArt = false'
put config/mediamanagement '.deleteEmptyFolders = false | .createEmptyArtistFolders = false
| .recycleBin = "" | .setPermissionsLinux = false | .importExtraFiles = false
| .skipFreeSpaceCheckWhenImporting = true'
roots=$(curl -sf -H "$H" "$API/rootfolder" | jq '[.[] | select(.path == "/music")] | length')
if [ "$roots" = "0" ]; then
qp=$(curl -sf -H "$H" "$API/qualityprofile" | jq '.[0].id')
mp=$(curl -sf -H "$H" "$API/metadataprofile" | jq '([.[] | select(.name == "Standard")][0].id) // .[0].id')
jq -nc --argjson qp "$qp" --argjson mp "$mp" '{
name: "Music", path: "/music",
defaultQualityProfileId: $qp, defaultMetadataProfileId: $mp,
defaultMonitorOption: "all", defaultNewItemMonitorOption: "all",
defaultTags: []
}' \
| curl -sf -X POST -H "$H" -H 'Content-Type: application/json' -d @- "$API/rootfolder" >/dev/null
echo "added root folder /music"
fi
echo "reconcile done"
while true; do sleep 3600; done
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: lidarr
namespace: navidrome
labels:
app.kubernetes.io/name: lidarr
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: lidarr
template:
metadata:
labels:
app.kubernetes.io/name: lidarr
spec:
enableServiceLinks: false
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containers:
- name: lidarr
image: lscr.io/linuxserver/lidarr:3.1.0.4875-ls42 # {"$imagepolicy": "flux-system:lidarr"}
env:
- name: PUID
value: "1000"
- name: PGID
value: "1000"
- name: TZ
value: Europe/Berlin
# Identity is enforced by the lidarr-gate outpost.
- name: LIDARR__AUTH__METHOD
value: External
ports:
- name: http
containerPort: 8686
volumeMounts:
- name: config
mountPath: /config
- name: music
mountPath: /music
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 4Gi
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
- name: reconcile
image: lscr.io/linuxserver/lidarr:3.1.0.4875-ls42 # {"$imagepolicy": "flux-system:lidarr"}
command: ["sh", "/scripts/reconcile.sh"]
volumeMounts:
- name: config
mountPath: /config
readOnly: true
- name: reconcile
mountPath: /scripts
readOnly: true
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 64Mi
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumes:
- name: config
persistentVolumeClaim:
claimName: lidarr-config
- name: music
persistentVolumeClaim:
claimName: navidrome-music-bulk
- name: reconcile
configMap:
name: lidarr-reconcile
---
apiVersion: v1
kind: Service
metadata:
name: lidarr
namespace: navidrome
spec:
selector:
app.kubernetes.io/name: lidarr
ports:
- name: http
port: 80
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: ak-outpost-lidarr-gate
namespace: navidrome
spec:
type: ExternalName
externalName: ak-outpost-lidarr-gate.authentik.svc.cluster.local
ports:
- port: 9000
targetPort: 9000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lidarr
namespace: navidrome
labels:
dns.renner.uno/public: "true"
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
spec:
ingressClassName: traefik
rules:
- host: lidarr.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ak-outpost-lidarr-gate
port:
number: 9000
tls:
- secretName: lidarr-tls
hosts:
- lidarr.renner.uno
+43
View File
@@ -0,0 +1,43 @@
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pclp-pg
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
# PG17: upstream's external Postgres target (docker/docker-compose.yml).
imageName: ghcr.io/cloudnative-pg/postgresql:17
storage:
storageClass: ceph-block
size: 10Gi
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: pclp-pg-store
bootstrap:
initdb:
database: paperclip
owner: paperclip
postgresql:
parameters:
shared_buffers: "256MB"
max_connections: "200"
checkpoint_timeout: "15min"
max_wal_size: "2GB"
wal_compression: "zstd"
archive_timeout: "30min"
synchronous:
method: any
number: 1
dataDurability: preferred
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 1Gi
@@ -0,0 +1,15 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: paperclip-secrets
namespace: paperclip
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: paperclip-secrets
dataFrom:
- extract:
key: apps/paperclip
@@ -0,0 +1,12 @@
# ExternalName mirror so the Ingress can backend the paperclip-gate outpost in the authentik namespace.
apiVersion: v1
kind: Service
metadata:
name: ak-outpost-paperclip-gate
namespace: paperclip
spec:
type: ExternalName
externalName: ak-outpost-paperclip-gate.authentik.svc.cluster.local
ports:
- port: 9000
targetPort: 9000
@@ -0,0 +1,17 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hetzner-s3-creds
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # barman S3 creds
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: hetzner-s3-creds
dataFrom:
- extract:
key: infrastructure/hetzner-s3
+25
View File
@@ -0,0 +1,25 @@
# LAN-only (no dns.renner.uno/public label); backends the gate outpost, never the Service.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: paperclip
namespace: paperclip
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
spec:
ingressClassName: traefik
tls:
- hosts:
- paperclip.renner.uno
secretName: paperclip-tls
rules:
- host: paperclip.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ak-outpost-paperclip-gate
port:
number: 9000
@@ -0,0 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- pvc.yaml
- paperclip.yaml
- gate-service.yaml
- ingress.yaml
- network-policy.yaml
@@ -0,0 +1,78 @@
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-ingress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
goauthentik.io/outpost-name: paperclip-gate
toPorts:
- ports:
- port: "3100"
protocol: TCP
---
# Covers the server and every agent it spawns (same pod).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-egress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
cnpg.io/cluster: pclp-pg
toPorts:
- ports:
- port: "5432"
protocol: TCP
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
# Gitea, Woodpecker, HA, Loki via their *.renner.uno names.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8443"
protocol: TCP
# Notes wiki API, authenticated by the paperclip API key.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ai
app.kubernetes.io/name: wiki
toPorts:
- ports:
- port: "8080"
protocol: TCP
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
- toEntities:
- world
toPorts:
- ports:
- port: "443"
protocol: TCP
@@ -0,0 +1,26 @@
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pclp-pg-store
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup target
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/pclp-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
+123
View File
@@ -0,0 +1,123 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperclip
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
secret.reloader.stakater.com/reload: paperclip-secrets
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: paperclip
template:
metadata:
labels:
app.kubernetes.io/name: paperclip
spec:
enableServiceLinks: false
# Every agent shares this filesystem; bots bring their own cluster credentials.
automountServiceAccountToken: false
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containers:
- name: paperclip
image: ghcr.io/paperclipai/paperclip:nightly@sha256:9222efbf11863160669e13d5559e6a1be881886dd1758650b74c3993dc3ef2a6 # {"$imagepolicy": "flux-system:paperclip"}
ports:
- name: http
containerPort: 3100
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: pclp-pg-app
key: uri
- name: HOST
value: "0.0.0.0"
- name: PORT
value: "3100"
- name: PAPERCLIP_DEPLOYMENT_MODE
value: authenticated
- name: PAPERCLIP_DEPLOYMENT_EXPOSURE
value: private
- name: PAPERCLIP_PUBLIC_URL
value: https://paperclip.renner.uno
# Agents reach the API via the Service, not through the Authentik gate.
- name: PAPERCLIP_API_URL
value: http://paperclip.paperclip.svc.cluster.local:3100
- name: PAPERCLIP_ALLOWED_HOSTNAMES
value: paperclip.paperclip.svc.cluster.local
- name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
value: "true"
- name: PAPERCLIP_TELEMETRY_DISABLED
value: "1"
envFrom:
- secretRef:
name: paperclip-secrets
volumeMounts:
- name: data
mountPath: /paperclip
# The private-hostname guard 403s any Host but loopback and the public URL; the kubelet sends the pod IP.
startupProbe:
httpGet:
path: /api/health
port: http
httpHeaders:
- name: Host
value: localhost
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 60
# /api/health runs SELECT 1; liveness stays DB-independent so a Postgres outage doesn't restart the server.
livenessProbe:
tcpSocket:
port: http
periodSeconds: 30
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /api/health
port: http
httpHeaders:
- name: Host
value: localhost
periodSeconds: 10
timeoutSeconds: 5
# Every concurrent agent (Claude Code + MCP helpers) runs inside this container.
resources:
requests:
cpu: 250m
memory: 2Gi
limits:
memory: 32Gi
# Entrypoint remaps the node user and chowns /paperclip as root, then drops to it via gosu.
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
volumes:
- name: data
persistentVolumeClaim:
claimName: paperclip-data
---
apiVersion: v1
kind: Service
metadata:
name: paperclip
namespace: paperclip
spec:
selector:
app.kubernetes.io/name: paperclip
ports:
- name: http
port: 3100
targetPort: http
+13
View File
@@ -0,0 +1,13 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperclip-data
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
accessModes: [ReadWriteOnce]
storageClassName: ceph-block
resources:
requests:
storage: 20Gi
@@ -0,0 +1,16 @@
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: pclp-pg-daily
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup schedule
spec:
schedule: "0 45 2 * * *"
immediate: true
backupOwnerReference: self
cluster:
name: pclp-pg
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
@@ -42,6 +42,10 @@ spec:
remoteRef:
key: apps/ai/wiki-api-keys
property: workstation
- secretKey: Auth__ApiKeys__paperclip
remoteRef:
key: apps/ai/wiki-api-keys
property: paperclip
---
# TypeSafe System One key for the judge adapter (Wiki:Judges:Models:typesafe).
apiVersion: external-secrets.io/v1
@@ -0,0 +1,25 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: paperclip
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 20m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/paperclip
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
failed: "has(status.phase) && status.phase.startsWith('Failed')"
current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
@@ -9,12 +9,14 @@ resources:
- gotenberg.yaml
- home-assistant.yaml
- immich.yaml
- lidarr.yaml
- matter-server.yaml
- mmwave-configurator.yaml
- music-assistant.yaml
- navidrome.yaml
- netbird-ingress.yaml
- paperless-ngx.yaml
- paperclip.yaml
- protonmail-bridge.yaml
- mosquitto.yaml
- redis.yaml
@@ -0,0 +1,24 @@
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
name: lidarr
namespace: flux-system
spec:
image: lscr.io/linuxserver/lidarr
interval: 1h
---
# Tags are <major>.<minor>.<patch>.<build>-ls<n>; the build number is monotonic.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
name: lidarr
namespace: flux-system
spec:
imageRepositoryRef:
name: lidarr
filterTags:
pattern: '^\d+\.\d+\.\d+\.(?P<build>\d+)-ls\d+$'
extract: '$build'
policy:
numerical:
order: asc
@@ -0,0 +1,25 @@
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
name: paperclip
namespace: flux-system
spec:
image: ghcr.io/paperclipai/paperclip
interval: 1h
---
# Upstream publishes no semver tags; track the nightly tag by digest.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
name: paperclip
namespace: flux-system
spec:
imageRepositoryRef:
name: paperclip
digestReflectionPolicy: Always
interval: 1h
filterTags:
pattern: '^nightly$'
policy:
alphabetical:
order: asc
@@ -239,3 +239,8 @@ metadata:
labels:
app.kubernetes.io/name: woodpecker
pod-security.kubernetes.io/enforce: privileged
---
apiVersion: v1
kind: Namespace
metadata:
name: paperclip
@@ -150,8 +150,8 @@ spec:
protocol: TCP
---
# wiki: the wiki-gate outpost (humans, after Authentik), Traefik (the machine
# door wiki-index.renner.uno, BasicAuth with X-Authentik-Username stripped) and
# Gitea (push webhook) may reach it. Gitea and wiki-embed are reached through
# door wiki-index.renner.uno, BasicAuth with X-Authentik-Username stripped),
# Gitea (push webhook) and Paperclip agents (API key) may reach it. Gitea and wiki-embed are reached through
# the namespace egress above; no OpenBao access.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
@@ -173,6 +173,9 @@ spec:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": gitea
app.kubernetes.io/name: gitea
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
app.kubernetes.io/name: paperclip
toPorts:
- ports:
- port: "8080"
@@ -6,6 +6,7 @@ resources:
- ingress-isolation.yaml
- authentik-ns.yaml
- paperless-ns.yaml
- paperclip-ns.yaml
- audiobookshelf-ns.yaml
- home-assistant-ns.yaml
- spegel-ns.yaml
@@ -18,6 +18,10 @@ spec:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
goauthentik.io/outpost-name: navidrome-gate
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
goauthentik.io/outpost-name: lidarr-gate
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
@@ -0,0 +1,97 @@
# Traefik is not admitted: the only entry is the paperclip-gate outpost (apps/paperclip/network-policy.yaml).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: default-deny-ingress
namespace: paperclip
spec:
endpointSelector: {}
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": database
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
- fromEntities:
- host
- remote-node
- fromEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": rook-ceph
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- toEntities:
- host
- remote-node
# Cilium unions allows across CNPs, so world stays :443-only here too.
- toEntities:
- world
toPorts:
- ports:
- port: "443"
protocol: TCP
- toEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
@@ -27,6 +27,7 @@ spec:
- influxdb
- ingress
- monitoring
- paperclip
- paperless
- trmnl
- velero