Compare commits

..
Author SHA1 Message Date
claude-botandClaude Fable 5.1 7a6cbc4e2e feat(talos): move the node IP and etcd to pod0 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pod east-west has to leave the management VLAN, and the node IP is the
only thing that controls where it goes. Three tests established that,
rather than argument:

  - --ipv4-node / --ipv6-node, applied via CiliumNodeConfig: the agent
    logged receiving both and still published its mgmt0 address.
  - directRoutingDevice=pod0: the agent logged "Direct routing device
    detected direct-routing-device=pod0" and still installed
    10.244.2.0/24 via 10.2.40.138 dev mgmt0.
  - Both routing modes, both address families, always keyed to the
    peer's Kubernetes InternalIP.

etcd is pinned to the same segment. Left unset it advertises "the first
routable address of the node", which is not deterministic once the node
is re-addressed, so it has to be stated either way. pod0 is a closed 10G
L2 between the three nodes; the management VLAN is about to become a
single 1G NIC shared with everything else management does, which is the
wrong link for quorum traffic. listenSubnets defaults to this, so peer
and client listeners move together.

This does move the etcd peer URLs off 10.2.40.x, so unlike the node IP
it is not a no-op: one node at a time, waiting for a healthy quorum
between each.

The API VIP stays on mgmt0; it is a Layer2VIPConfig and does not follow
the node IP.

Apply one node at a time. Each node re-IPs, so per the note in
k8s-1.yaml (#635) kube-apiserver keeps advertising the old address until
its static pod restarts: after each node, check that
`kubectl get endpoints kubernetes -n default` agrees with
`kubectl get nodes -o wide` and reboot the node if it does not. The
cilium agent reads the node IP only at startup, so it needs recreating
on each node as well.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:54:57 +02:00
341 changed files with 1063 additions and 20044 deletions
+1 -1
View File
@@ -101,7 +101,7 @@ steps:
bash scripts/ci/make-diag-kubeconfig.sh
- name: alert-triage-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
-66
View File
@@ -1,66 +0,0 @@
# Paperclip agent runtime images (REN-284 / REN-286): the stock
# agent-runtime-claude plus a tool set, one image per sandbox environment.
# Built with kaniko like build-ci-images.yml and pushed with the same
# ci_images_token (package read+write only, see the header there).
#
# Own workflow, not a step in build-ci-images.yml. A change to that file
# rebuilds all of its images at once and fills the 24-CPU woodpecker-ci-quota;
# on the REN-286 merge (pipeline 11306) this step then got no pod
# ("exceeded quota") and the go image was never built. Here it runs only when
# its own sources change, and it can be re-run alone with a deployment to
# target `build-agent-runtime`, without rebuilding the other images.
#
# The name looks odd on purpose. The kubernetes sandbox plugin keeps the image
# name and tag `agent-runtime-claude:v1` and swaps only the registry prefix
# (`imageRegistry`), replacing host + org. So each variant needs its own
# prefix with the same name under it:
# imageRegistry gitea.renner.uno/konairius/agent-runtime-go
# -> gitea.renner.uno/konairius/agent-runtime-go/agent-runtime-claude:v1
# `v1` is what the plugin pulls and moves on every build; the date tag is the
# rollback point. konairius/ because the ci_images_token can push there and
# it is the target of the gitea_admin/* migration (#330).
when:
- event: push
branch: master
path:
- "ci/images/agent-runtime-*/**"
- ".woodpecker/build-agent-runtime.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-agent-runtime"'
steps:
- name: build-agent-runtime
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
directory: ci/images
# Context is ci/images (not the variant dir) so both Containerfiles can
# COPY the shared agent-runtime-common/ files.
# One pod, both images one after the other (kaniko --cleanup between
# builds), at 1 CPU so it still fits if it overlaps another pipeline.
backend_options:
kubernetes:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 4Gi
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
for VARIANT in go ops; do
/kaniko/executor \
--cleanup \
--context . \
--dockerfile "agent-runtime-$VARIANT/Containerfile" \
--destination "gitea.renner.uno/konairius/agent-runtime-$VARIANT/agent-runtime-claude:$TAG" \
--destination "gitea.renner.uno/konairius/agent-runtime-$VARIANT/agent-runtime-claude:v1" \
|| exit 1
done
+16 -48
View File
@@ -26,14 +26,8 @@
# 38-reference change tracked separately in #330 rather than bundled here.
# Note this pipeline runs no LLM agent, so the exfil path that motivated the
# other cutovers does not apply to it.
#
# Push builds run on master only. Without the branch filter a feature branch
# that touches this file started all steps at once: it pushed unreviewed
# images and used the whole woodpecker-ci-quota, so the PR checks could not
# get a pod (REN-286).
when:
- event: push
branch: master
path:
- "ci/images/**"
- "secret-rotator/**"
@@ -43,7 +37,7 @@ when:
steps:
- name: build-ansible
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/ansible
depends_on: []
# Only rebuild when THIS image's source (or the pipeline file) changed, or
@@ -71,7 +65,7 @@ steps:
--destination gitea.renner.uno/gitea_admin/ci-ansible:latest
- name: build-k8s
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/k8s
depends_on: []
when:
@@ -95,7 +89,7 @@ steps:
--destination gitea.renner.uno/gitea_admin/ci-k8s:latest
- name: build-vmtest
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/vmtest
depends_on: []
when:
@@ -119,7 +113,7 @@ steps:
--destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
- name: build-tofu
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/tofu
depends_on: []
when:
@@ -150,7 +144,7 @@ steps:
# Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
# used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
- name: build-secret-rotator
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: secret-rotator
depends_on: []
when:
@@ -201,7 +195,7 @@ steps:
#
# The gitea_admin/ copy stays until nothing references it any more.
- name: build-alpine-jq
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/alpine-jq
depends_on: []
when:
@@ -254,7 +248,7 @@ steps:
# note on build-alpine-jq. Anything an app needs while the cluster is still
# being built cannot live only in that cluster's own registry.
- name: build-cni-plugins
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/cni-plugins
depends_on: []
when:
@@ -296,7 +290,7 @@ steps:
# gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
# paperless in Init:ErrImagePull on exactly this image.
- name: build-alpine-openssl
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/alpine-openssl
depends_on: []
when:
@@ -333,7 +327,7 @@ steps:
# Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
# builds the same way and lands in the same registry so the audit sees it.
- name: build-heating-can
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/heating-can
depends_on: []
when:
@@ -356,17 +350,17 @@ steps:
--destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
--destination gitea.renner.uno/gitea_admin/heating-can:latest
# netbird-service-sync: the controller that reconciles NetBird reverse-proxy
# netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
# services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
# Runtime artifact like heating-can; same registry so the audit tracks it.
- name: build-netbird-service-sync
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
directory: ci/images/netbird-service-sync
- name: build-netbird-proxy-services
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/netbird-proxy-services
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/netbird-service-sync/**", ".woodpecker/build-ci-images.yml"]
path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
@@ -380,31 +374,5 @@ steps:
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/netbird-service-sync:$TAG \
--destination gitea.renner.uno/gitea_admin/netbird-service-sync:latest
# bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
# artifact like heating-can; same registry so the audit tracks it.
- name: build-bird
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
directory: ci/images/bird
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/bird:$TAG \
--destination gitea.renner.uno/gitea_admin/bird:latest
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
-69
View File
@@ -1,69 +0,0 @@
# Paperclip server image (REN-401): upstream paperclipai/paperclip at a pinned
# master commit plus the patches in ci/images/paperclip/. Needed while the k8s
# sandbox fix (upstream PR #13733) is not merged; see the README there for how
# to go back to upstream nightly.
#
# Own workflow for the reason given in build-agent-runtime.yml: a step in
# build-ci-images.yml runs on every change to that file and competes for the
# woodpecker-ci-quota. Re-run alone with a deployment to target
# `build-paperclip`.
#
# Tag `build.<unix-ts>` is what the Flux ImagePolicy `paperclip` sorts on
# (image-automation/paperclip.yaml); image automation then rolls it out.
# konairius/ and ci_images_token as for the agent-runtime images.
when:
- event: push
branch: master
path:
- "ci/images/paperclip/**"
- ".woodpecker/build-paperclip.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-paperclip"'
steps:
- name: prepare-source
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- sh ci/images/paperclip/prepare.sh build/paperclip-src
- name: build-paperclip
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
directory: build/paperclip-src
depends_on: [prepare-source]
# Rust runner + UI + server build in one pod. The server build alone sets
# a 4 GiB Node heap, so the memory limit is the LimitRange max (8Gi).
backend_options:
kubernetes:
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "4"
memory: 8Gi
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- . ../../ci/images/paperclip/upstream.env
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=build.$(date +%s)
# Same target and build args as upstream's docker.yml for nightly.
# --compressed-caching=false: pipeline 11717 was killed at the 8Gi limit
# (exit 137) in the server build; kaniko otherwise holds compressed
# layer tarballs in memory next to the 4 GiB Node heap.
- |
/kaniko/executor \
--context . \
--dockerfile Dockerfile.kaniko \
--target production \
--skip-unused-stages \
--snapshot-mode=redo \
--compressed-caching=false \
--build-arg PAPERCLIP_BUILD_COMMIT=$PAPERCLIP_COMMIT \
--build-arg PAPERCLIP_BUILD_VERSION=$PAPERCLIP_VERSION \
--build-arg CLI_TOOLS_CACHE_EPOCH=$(date -u +%G-W%V) \
--destination gitea.renner.uno/konairius/paperclip:$TAG
+22 -18
View File
@@ -14,8 +14,8 @@
# and ci-claude:latest. The Containerfile makes this layer
# byte-reproducible, so rebuilding an unchanged version dedups
# in the registry instead of orphaning a fresh ~260 MB copy.
# 3. bump-pr — (only when any pin is off the latest version) rewrite every
# consumer pin to ci-claude:<version> and open a PR. A human
# 3. bump-pr — (only when the version actually changed) rewrite the 5
# consumer pins to ci-claude:<version> and open a PR. A human
# reviews and merges it; nothing auto-merges.
#
# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
@@ -107,22 +107,31 @@ steps:
[0-9]*.[0-9]*.[0-9]*) : ;;
*) echo "npm returned an implausible version '$$NEW' — aborting."; exit 1 ;;
esac
# Distinct tags the consumers currently reference. Divergent pins are
# normal (a pipeline bumped on its own); the bump rewrites all of them.
CUR=$$(grep -hoE 'ci-claude:[0-9][0-9A-Za-z._-]*' $$CONSUMERS | sed 's/^ci-claude://' | sort -u | tr '\n' ' ' | sed 's/ $$//')
[ -n "$$CUR" ] || { echo "No ci-claude pin found in the consumers — aborting."; exit 1; }
echo "latest=$$NEW current=[$$CUR] event=$$CI_PIPELINE_EVENT"
# The tag every consumer currently references. They are kept identical,
# so this must resolve to EXACTLY ONE non-empty tag; anything else
# (no match, or a partial/disagreeing set) aborts rather than risk a
# corrupting sed later. Counting with `grep -c .` (non-empty lines)
# instead of `wc -l` is deliberate: an empty result yields 0 here but
# 1 under wc -l, which would slip past the guard.
CUR=$$(grep -hoE 'ci-claude:[0-9A-Za-z._-]+' $$CONSUMERS | sed 's/^ci-claude://' | sort -u)
NTAGS=$$(printf '%s' "$$CUR" | grep -c . || true)
if [ -z "$$CUR" ] || [ "$$NTAGS" -ne 1 ]; then
echo "Expected exactly one ci-claude tag across the consumers, got [$$CUR]."
echo "Refusing to guess — fix the pins by hand."; exit 1
fi
echo "latest=$$NEW current=$$CUR event=$$CI_PIPELINE_EVENT"
printf '%s' "$$NEW" > "$$CI_WORKSPACE/.ci-refresh/version"
printf '%s' "$$CUR" > "$$CI_WORKSPACE/.ci-refresh/current"
# Build when any pin is off latest, or on a push (Containerfile edit must
# ship even if the version is unchanged). Bump when any pin is off latest.
# Build when the version changed, or on a push (Containerfile edit must
# ship even if the version is unchanged). Bump the pins only when the
# version the consumers reference is not the latest.
if [ "$$CUR" != "$$NEW" ] || [ "$$CI_PIPELINE_EVENT" = "push" ]; then
touch "$$CI_WORKSPACE/.ci-refresh/do-build"; echo "will build $$NEW"
else
echo "already on $$NEW — nothing to build"
fi
if [ "$$CUR" != "$$NEW" ]; then
touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR [$$CUR] -> $$NEW"
touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR $$CUR -> $$NEW"
else
echo "consumers already pin $$NEW — no PR"
fi
@@ -131,7 +140,7 @@ steps:
# --build-arg to the resolved version (not "latest") guarantees the tag and
# the image content agree even if a new release lands mid-run.
- name: build
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/claude
depends_on: [decide]
environment:
@@ -166,9 +175,6 @@ steps:
- |
[ -f "$$CI_WORKSPACE/.ci-refresh/do-bump" ] || { echo "no version change — no PR"; exit 0; }
set -eu
# Secret values can carry a trailing newline; it breaks the push URL and the auth header.
PUSH_TOKEN=$$(printf '%s' "$$PUSH_TOKEN" | tr -d '\r\n')
[ -n "$$PUSH_TOKEN" ] || { echo "PUSH_TOKEN is empty — aborting"; exit 1; }
NEW=$$(cat "$$CI_WORKSPACE/.ci-refresh/version")
CUR=$$(cat "$$CI_WORKSPACE/.ci-refresh/current")
# Both were validated non-empty by the decide step; re-assert before a
@@ -184,11 +190,9 @@ steps:
# is not a maintained list.
CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
[ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
# NEW goes into a sed replacement: allow only a plain x.y.z version.
printf '%s' "$$NEW" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$$' || { echo "refusing non-x.y.z version '$$NEW'"; exit 1; }
# Every version-pinned tag becomes NEW, whatever it was; consumers only.
# Exact-string swap of the current tag for the new one, consumers only.
for f in $$CONSUMERS; do
sed -i "s#ci-claude:[0-9][0-9A-Za-z._-]*#ci-claude:$$NEW#g" "$$f"
sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
done
# Author is claude-bot (the PAT's account); the commit subject makes
# the automated-bump nature clear.
+17 -100
View File
@@ -17,44 +17,30 @@
# here is a static check that either passes or fails on the checkout alone, so it
# can gate PRs without becoming a bottleneck or a flake source.
when:
# Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is two
# steps on already-cached images, so running it everywhere is free; this one
# pulls four images and builds every kustomization in the tree, which is real
# Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
# step on one already-cached image, so running it everywhere is free; this one
# pulls three images and builds every kustomization in the tree, which is real
# cost to put on a docs-only PR. The filter covers everything the steps actually read:
# kubernetes/** for the manifests, scripts and promtool suites, .woodpecker/**
# for the lint (and so an edit to this file re-runs its own checks), and the
# promtool driver script by name.
#
# That last entry is NOT covered by kubernetes/**: promtool-rules.sh lives
# under scripts/, so without it a PR that only edits the script would never run
# the check it drives. It is listed as one file rather than scripts/ci/**
# because the rest of that directory is CI-agent helpers with nothing to do
# with this pipeline, and pulling four images for them is the cost this filter
# exists to avoid. Their own tests run in ci-scripts-test.yml on every PR.
# kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
# so an edit to this file re-runs its own checks).
- event: pull_request
path:
- "kubernetes/**"
- ".woodpecker/**"
- "scripts/ci/promtool-rules.sh"
- "scripts/ci/yaml-citations.sh"
# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
# both hand-roll the same shape — `find`, a floor guard against empty discovery,
# then a file-fed `while read`. Two copies was deliberate: the two traps being
# commented around (BusyBox find lacking `-printf`, and a piped loop running in
# a subshell where `fail` cannot propagate) are different, and both are worth
# spelling out where they bite. An earlier revision of this comment said a third
# instance would be the point to stop copying and factor it into `scripts/ci/`,
# where it would gain unit tests — and that is what `promtool` below does. Its
# discovery, floors and guards live in scripts/ci/promtool-rules.sh and are
# tested by scripts/ci/test-promtool-rules.sh on every PR via
# ci-scripts-test.yml. A FOURTH inline copy should go the same way; do not add
# one here.
# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
# the same shape — `find`, a floor guard against empty discovery, then a
# file-fed `while read`. Two copies is deliberate: the two traps being commented
# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
# where `fail` cannot propagate) are different, and both are worth spelling out
# where they bite. A third instance is the point to stop copying and factor it
# into a shared script under `scripts/ci/` — which is already covered by
# ci-scripts-test.yml, so it would gain unit tests in the process.
#
# Every step carries `depends_on: []` so all six run in PARALLEL and every
# Every step carries `depends_on: []` so all four run in PARALLEL and every
# failure shows in one run. Woodpecker's default is sequential, which would stop
# at the first failure and make a contributor fix-and-repush to discover the
# next — and these six are independent, cheap, offline checks with no ordering
# next — and these four are independent, cheap, offline checks with no ordering
# between them. Keep it on every step: `depends_on` is per-step, so dropping it
# from one silently re-serialises that step behind the others.
steps:
@@ -172,7 +158,7 @@ steps:
# whenever it trails the deployed server. If you bump the server, bump here.
- name: woodpecker-lint
depends_on: [] # see note above `steps:`
image: woodpeckerci/woodpecker-cli:v3.18.1-alpine@sha256:7eac3c4254befc1ba1e65269c3304c370b6032530b7386fc0c7648378fdd0516
image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
commands:
- |
set -eu
@@ -199,7 +185,7 @@ steps:
# separately rather than one this step quietly implies is covered.
- name: shellcheck
depends_on: [] # see note above `steps:`
image: koalaman/shellcheck-alpine:v0.11.0@sha256:9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002
image: koalaman/shellcheck-alpine:v0.11.0
commands:
- |
set -eu
@@ -272,72 +258,3 @@ steps:
exit 1
fi
echo "ok no digest pins under openbao/rotation/"
# Machine-check the alert rules: parse every PrometheusRule, run the committed
# promtool unit suites, then damage the rules on purpose and require the suites
# to notice. Everything it does lives in scripts/ci/promtool-rules.sh; read
# that file and kubernetes/apps/monitoring-rules/tests/README.md for the why.
#
# The gap it closes: the other four steps prove a PrometheusRule assembles into
# valid YAML and nothing more — not that an expression parses, not that a label
# join matches anything, not that a `for` window does what its comment claims.
# On REN-129 the CNPG surge rule was written `for: 2m`, which fires on six of
# seven measured nights and would have mailed the owner twice a night at 04:16
# and 04:22 local for a backlog that clears itself. Nothing here would have
# caught it, and the arithmetic that did catch it was hand-written from scratch
# twice, in two runs, and thrown away both times because there was nowhere in
# the repo to commit it. REN-152 retired that rule; its `for` margin is now
# guarded by mutations.txt:backlog-for-12m.
#
# Offline, like the rest of this pipeline: no cluster, no registry, no secrets,
# no network. promtool reads files and exits.
#
# IMAGE — prom/prometheus, not a registry-local ci-* image, and pinned to an
# exact upstream version rather than a dated tag. The dated-tag convention
# guarded by no-digest-pins above is about OUR images, whose digests are
# garbage-collected out from under a pin; this is upstream and immutable.
#
# The version must be 3.x, and that is not a preference. The cluster runs
# whatever kube-prometheus-stack >= 72.0.0 ships, which is Prometheus 3, and
# alert `for` evaluation is version-sensitive — a 2.x promtool would be
# checking a different evaluator than production, so the margins the suites
# assert would be claims about the wrong program. promtool-rules.sh asserts the
# major version it was handed and fails on 2.x, so a careless bump here is a
# red step rather than a quietly wrong check. Keep this at the latest stable
# 3.x, per the standing rule that any software uses the latest stable version.
#
# This image is NOT covered by image-automation (same reason as
# woodpecker-lint above: a Woodpecker pipeline file has no apiVersion/kind, so
# an $imagepolicy marker here is silently ignored — fluxcd/flux2#2826). The
# bump is manual.
#
# `prom/prometheus` is busybox-based and ships /bin/sh, so it can run a
# `commands:` block — unlike the plain woodpecker-cli tag, which cannot. Its
# find/grep/awk/sed are BusyBox, which is why promtool-rules.sh avoids
# `find -printf` and `grep -r --include`; it was verified against BusyBox
# 1.35.0 as well as GNU. The image also declares `USER nobody`, so the script
# writes its stripped mirror under mktemp and never into the workspace.
- name: promtool
depends_on: [] # see note above `steps:`
image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
commands:
- sh scripts/ci/promtool-rules.sh
# Comments cite other YAML blocks by anchor name, never by line number
# (REN-183; the exact syntax is in CONTRIBUTING.md, not repeated here because
# an example written in this comment would itself be checked). Line numbers drift on any edit
# above them and nothing noticed: three drift fixes were merged by hand
# before this step existed. This fails on a new line-number citation and on
# a citation whose file or anchor marker does not exist.
# Existence only — it never judges whether the anchored block still says
# what the citing comment claims, so ordinary edits cannot turn it red.
#
# It scans every YAML file in the checkout, but this pipeline only runs on
# the paths in `when:` above, so an edit confined to ansible/ or esphome/ is
# checked on the next PR that does touch kubernetes/ or .woodpecker/.
# Self-test: scripts/ci/test-yaml-citations.sh, in ci-scripts-test.yml.
- name: yaml-citations
depends_on: [] # see note above `steps:`
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- sh scripts/ci/yaml-citations.sh
+4 -34
View File
@@ -8,45 +8,15 @@
# against adversarial PR input. Running it here fails the offending PR's checks
# before it can merge.
#
# scripts/ci/test-promtool-rules.sh is the same idea for the alert-rule gate.
# What it tests is not whether promtool evaluates PromQL — ci-manifests-test.yml
# does that with the real binary against the real rules — but whether
# promtool-rules.sh REFUSES TO REPORT SUCCESS when it has checked nothing: zero
# manifests discovered, zero suites discovered, a `rule_files` path that does not
# resolve (promtool warns, loads zero rules, and passes every assertion), a
# mutation the suites no longer catch. Those are the states a green pipeline
# cannot tell apart from real coverage, so they need a test that asserts the
# failure. It stubs promtool, so it needs neither the binary nor the heavy image.
#
# Both are unfiltered by path on purpose: this pipeline is two steps on one
# already-cached image, so running it everywhere is free, and a guard whose test
# only runs when someone remembers to touch the right directory is not a guard.
#
# Only in homelab (scripts/ci/ lives here); the file is not deployed to the
# mirrored repos, so it does not run on their PRs. All tests here are offline (no
# token, no network, no cluster) — the ci-claude image just provides sh + jq, and
# it is already pulled by pr-review.yml on the same event, so the runner has it
# cached.
# mirrored repos, so it does not run on their PRs. The test is offline (no token,
# no network, no cluster) — the ci-claude image just provides sh + jq, and it is
# already pulled by pr-review.yml on the same event, so the runner has it cached.
when:
- event: pull_request
# `depends_on: []` on every step so they run in parallel and one failure does not hide
# the other. Same reasoning as the note above `steps:` in ci-manifests-test.yml.
steps:
- name: model-selection-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
commands:
- sh scripts/ci/test-model-selection.sh
- name: promtool-rules-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
commands:
- sh scripts/ci/test-promtool-rules.sh
- name: yaml-citations-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
commands:
- sh scripts/ci/test-yaml-citations.sh
+1 -1
View File
@@ -13,7 +13,7 @@ when:
steps:
- name: validate-secrets
image: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.24.1
environment:
TF_VAR_proxmox_api_url:
from_secret: proxmox_api_url
+3
View File
@@ -4,6 +4,9 @@ when:
branch: master
path:
- "ansible/roles/netbird-server/**"
# install-netbird-server.yml also applies the netbird-proxy-watchdog role,
# so changes to it must redeploy this host too.
- "ansible/roles/netbird-proxy-watchdog/**"
- "ansible/plays/install-netbird-server.yml"
- ".woodpecker/deploy-netbird-server.yml"
- event: deployment
+1 -1
View File
@@ -26,7 +26,7 @@ steps:
# fetch-models.sh response mapping. No cluster, no secrets, no network — the
# ci-claude image just provides sh + jq. Keeps the flag-injection regression
# test from rotting (it exercises the exact injection shapes).
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
commands:
- sh scripts/ci/test-model-selection.sh
+1 -1
View File
@@ -62,7 +62,7 @@ steps:
- bash scripts/ci/make-diag-kubeconfig.sh
- name: issue-fix-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -65,7 +65,7 @@ when:
steps:
- name: pr-review-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+3 -3
View File
@@ -18,12 +18,12 @@ when:
steps:
- name: iac-scan
image: aquasec/trivy:latest@sha256:af6acf9a6b85dfe389a1941505c0ce9efef52a4719635e1a962f022a3d855daa
image: aquasec/trivy:latest
commands:
- trivy fs --format json -o trivy-iac.json --severity CRITICAL,HIGH --ignore-unfixed .
- name: image-scans
image: aquasec/trivy:latest@sha256:af6acf9a6b85dfe389a1941505c0ce9efef52a4719635e1a962f022a3d855daa
image: aquasec/trivy:latest
commands:
- trivy image --download-db-only
- |
@@ -70,7 +70,7 @@ steps:
# this step is skipped — otherwise we'd fork-bomb PRs.
when:
- branch: master
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -30,7 +30,7 @@ when:
steps:
- name: update-audit-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
-198
View File
@@ -1,198 +0,0 @@
# Nightly wiki maintenance for the Notes vault (k.renner/notes), replacing the
# workstation's wiki-maintenance and wiki-consolidate timers: distill the raw/
# backlog into wiki/ (wiki-distill agent), audit wiki/ with the vault's own
# script (structural checks, no model), then consolidate (wiki-consolidate
# agent: verifies a cohort of notes against sources, repos and the live
# cluster and edits them). The pipeline commits and pushes as Claude Bot; the
# push webhook then refreshes the wiki service.
#
# The agents address the vault as ~/Notes and the repos as ~/source/<repo>, so
# each agent step links those into the workspace instead of rewriting them.
when:
- event: cron
cron: wiki-maintenance
# Hand-started runs: `woodpecker pipeline create k.renner/homelab --branch master`.
# The typesafe_api_key secret is scoped to cron and manual, not deployment.
- event: manual
steps:
- name: checkout-notes
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
pull: true
environment:
WIKI_BOT_TOKEN:
from_secret: wiki_bot_token
commands:
- |
set -eu
# Token only on the wire, never in .git/config. Secrets set from files carry
# a trailing newline (run 10220: "could not read Username"); strip CR/LF.
TOKEN=$$(printf '%s' "$$WIKI_BOT_TOKEN" | tr -d '\r\n')
git -c http.extraheader="AUTHORIZATION: token $$TOKEN" \
clone --depth 50 --branch main https://gitea.renner.uno/k.renner/notes.git notes
echo "raw backlog: $$(ls notes/raw/*.md | grep -v _index.md | wc -l) file(s)"
# Full history for the consolidate agent's `git log` checks; the
# workspace clone is shallow and tree-filtered.
mkdir -p sources
git clone --quiet https://gitea.renner.uno/k.renner/homelab.git sources/homelab
git -c http.extraheader="AUTHORIZATION: token $$TOKEN" \
clone --quiet https://gitea.renner.uno/k.renner/homeassistant.git sources/homeassistant
- name: fetch-diag
# Read-only live-cluster access for the consolidate agent, same mechanism
# as issue-fix.yml: ./.ci-diag/kubectl plus a 1h ci-diagnostics token.
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
S3_ACCESS_KEY_ID:
from_secret: hetzner_s3_access_key
S3_SECRET_ACCESS_KEY:
from_secret: hetzner_s3_secret_key
commands:
- bash scripts/ci/fetch-kubeconfig.sh
- bash scripts/ci/make-diag-kubeconfig.sh
- name: distill
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
pull: true
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
TYPESAFE_API_KEY:
from_secret: typesafe_api_key
WIKI_INDEX_BASIC_AUTH:
from_secret: wiki_index_basic_auth
commands:
- |
set -eu
# An empty queue must not start the agent: its startup context alone
# costs more than the whole audit.
BACKLOG=$$(ls notes/raw/*.md | grep -v _index.md | wc -l)
if [ "$$BACKLOG" -eq 0 ]; then
echo "distill: raw/ is empty, skipping the agent"
exit 0
fi
ln -sfn "$$PWD/notes" "$$HOME/Notes"
mkdir -p "$$HOME/.config/wiki-index"
printf '%s' "$$WIKI_INDEX_BASIC_AUTH" | tr -d '\r\n' > "$$HOME/.config/wiki-index/basic-auth"
chmod 600 "$$HOME/.config/wiki-index/basic-auth"
"$$HOME/Notes/bin/wiki_rank.py" --probe-index
MODEL=$$(sh scripts/ci/pick-model.sh "distill raw source notes into atomic wiki notes: read sources, check the wiki for existing coverage, write or merge notes; careful writing, moderate reasoning")
if [ -n "$$MODEL" ]; then echo "distill: selected model $$MODEL"; else echo "distill: using the CLI default model"; fi
cd notes
cat ../ci/agents/wiki-maintenance/distill.md | claude $${MODEL:+--model $$MODEL} \
--agent wiki-distill \
--permission-mode acceptEdits \
--allowedTools "Read,Edit,Write,Glob,Grep,Bash(/root/Notes/bin/wiki_rank.py:*),Bash(/root/Notes/bin/wiki_ask.py:*),Bash(mv /root/Notes/raw/:*),Bash(mkdir -p /root/Notes/:*),Bash(~/Notes/bin/wiki_rank.py:*),Bash(~/Notes/bin/wiki_ask.py:*),Bash(mv ~/Notes/raw/:*),Bash(mkdir -p ~/Notes/:*)" \
--settings ../ci/agents/wiki-maintenance/settings.json \
-p
- name: audit
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
pull: true
commands:
# Findings are report lines in output/, not a pipeline failure.
- python3 -B notes/bin/wiki_audit.py --vault "$$PWD/notes"
- name: consolidate
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
pull: true
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
TYPESAFE_API_KEY:
from_secret: typesafe_api_key
WIKI_INDEX_BASIC_AUTH:
from_secret: wiki_index_basic_auth
commands:
- |
set -eu
ln -sfn "$$PWD/notes" "$$HOME/Notes"
ln -sfn "$$PWD/sources" "$$HOME/source"
mkdir -p "$$HOME/.config/wiki-index"
printf '%s' "$$WIKI_INDEX_BASIC_AUTH" | tr -d '\r\n' > "$$HOME/.config/wiki-index/basic-auth"
chmod 600 "$$HOME/.config/wiki-index/basic-auth"
if [ -f .ci-diag/kubeconfig ]; then
ln -sf "$$PWD/.ci-diag/kubectl" /usr/local/bin/kubectl
export KUBECONFIG="$$PWD/.ci-diag/kubeconfig"
else
echo "consolidate: no ci-diag kubeconfig, live cluster checks unavailable"
fi
cd notes
# JSON output: the agent's report and the run's cost come out of one
# record at the end instead of a 45-minute silent log.
# One source for the Bash allowlist: the settings file (run 10608: the
# agent writes ~/Notes, the rules carried only /root/Notes).
BASH_RULES=$$(jq -r '[.permissions.allow[] | select(startswith("Bash("))] | join(",")' ../ci/agents/wiki-maintenance/consolidate-settings.json)
rc=0
cat ../ci/agents/wiki-maintenance/consolidate.md | claude \
--agent wiki-consolidate \
--permission-mode acceptEdits \
--allowedTools "Read,Edit,Write,Glob,Grep,Agent,$$BASH_RULES" \
--settings ../ci/agents/wiki-maintenance/consolidate-settings.json \
--output-format json \
-p > /tmp/consolidate.json || rc=$$?
# A killed or crashed claude leaves no valid JSON; report rc, not a jq error.
if [ -s /tmp/consolidate.json ] && jq -e . /tmp/consolidate.json > /dev/null; then
jq -r '.result // empty' /tmp/consolidate.json
jq -c '{duration_ms, num_turns, total_cost_usd, is_error}' /tmp/consolidate.json
else
echo "consolidate: no result record (exit $$rc)"; head -c 2000 /tmp/consolidate.json || true
fi
echo "consolidate: git log --oneline of the run:"
git log --oneline origin/main..HEAD
exit $$rc
- name: push-notes
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
pull: true
environment:
WIKI_BOT_TOKEN:
from_secret: wiki_bot_token
commands:
- |
set -eu
cd notes
git config user.email "claude-bot@renner.uno"
git config user.name "Claude Bot"
# Only the vault content is committed; anything an agent leaves elsewhere
# in the checkout (run 10362: scratch scripts in the root) stays behind.
# The consolidate agent commits its own snapshots, so HEAD may already be
# ahead of origin/main with a clean tree.
git add -A -- wiki raw output .wiki-consolidation-state.json
git diff --cached --quiet || git commit -q -m "wiki maintenance $$(date -u +%F): distill + audit + consolidate (CI run $$CI_PIPELINE_NUMBER)"
if [ "$$(git rev-list --count origin/main..HEAD)" -eq 0 ]; then
echo "wiki-maintenance: no changes"
exit 0
fi
# The workstation pushes main on its own (notes-push.path), so main moves
# while the agents run (run 10241: non-fast-forward). The run's work goes
# to a branch and a PR that this step merges through Gitea; a conflict
# leaves the PR open for a human and fails the run.
BRANCH="wiki-maintenance/$$(date -u +%F)-$$CI_PIPELINE_NUMBER"
git checkout -q -b "$$BRANCH"
TOKEN=$$(printf '%s' "$$WIKI_BOT_TOKEN" | tr -d '\r\n')
git -c http.extraheader="AUTHORIZATION: token $$TOKEN" push origin "HEAD:$$BRANCH"
API="https://gitea.renner.uno/api/v1/repos/k.renner/notes"
jq -n --arg title "wiki maintenance $$(date -u +%F) (CI run $$CI_PIPELINE_NUMBER)" \
--arg body "Nightly distill + audit + consolidate. Merged by the pipeline; open only if the merge failed." \
--arg head "$$BRANCH" '{title: $$title, body: $$body, head: $$head, base: "main"}' > /tmp/pr.json
code=$$(curl -s -o /tmp/pr-resp.json -w '%{http_code}' -X POST \
-H "Authorization: token $$TOKEN" -H "Content-Type: application/json" --data @/tmp/pr.json "$$API/pulls")
echo "POST pull -> HTTP $$code"
[ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/pr-resp.json; exit 1; }
NUM=$$(jq -r '.number' /tmp/pr-resp.json)
# Merge without delete_branch_after_merge: run 10254 got HTTP 500 from that
# call although the merge itself had landed. Verify the PR state instead of
# trusting the status code, then drop the branch as a separate, best-effort step.
code=$$(curl -s -o /tmp/merge-resp.json -w '%{http_code}' -X POST \
-H "Authorization: token $$TOKEN" -H "Content-Type: application/json" \
--data '{"Do":"merge"}' "$$API/pulls/$$NUM/merge")
echo "POST merge #$$NUM -> HTTP $$code"
MERGED=$$(curl -s -H "Authorization: token $$TOKEN" "$$API/pulls/$$NUM" | jq -r '.merged')
echo "PR #$$NUM merged: $$MERGED"
[ "$$MERGED" = "true" ] || { cat /tmp/merge-resp.json; echo "PR #$$NUM left open"; exit 1; }
code=$$(curl -s -o /tmp/del-resp.json -w '%{http_code}' -X DELETE -H "Authorization: token $$TOKEN" \
"$$API/branches/$$(printf '%s' "$$BRANCH" | sed 's|/|%2F|g')")
echo "DELETE branch $$BRANCH -> HTTP $$code (best effort)"
-7
View File
@@ -1,7 +0,0 @@
# Contributing
## Citing other YAML from a comment
- Mark the block you cite with a comment line `# anchor: <name>` (kebab-case, unique in that file). If the block is inside a `|` block scalar, put the marker just above the key, never inside the scalar.
- Cite it as `path.yaml#<name>`, with the path relative to the citing file's directory or to the repo root. Never cite `path.yaml:<line>`, because line numbers go stale on any edit above them.
- `scripts/ci/yaml-citations.sh` enforces this in `ci-manifests-test.yml`: it fails on line-number citations and on citations whose file or anchor does not exist.
-70
View File
@@ -1,70 +0,0 @@
# homelab
GitOps source for the Renner homelab cluster. Flux applies `master` to
production, so every merge is a production change. Changes go in by pull
request only; see [CONTRIBUTING.md](CONTRIBUTING.md) for YAML conventions.
## Change rules that protect Home Assistant
Owner decision (REN-240, REN-243). Home Assistant runs the house, so it
gets two rules on top of normal review.
### 1. Disruptive work runs only 02:00–05:00 Europe/Berlin
Planned work that can restart Home Assistant or disrupt the cluster is
merged and applied only between **02:00 and 05:00 Europe/Berlin**. That
covers:
- node drain, node reboot, Talos upgrade
- CNI (Cilium, Multus) and cluster-wide network policy
- storage (Rook/Ceph, CSI, snapshot controller, CNPG operator)
- Flux core (`kubernetes/clusters/*/flux-system/`, Flux controllers)
- Authentik
- OpenBao, including secret rotation that restarts consumers
- anything in `kubernetes/apps/home-assistant/`
A PR in this list states in its description that it waits for the window.
Flux applies on merge, so "wait for the window" means **merge** in the
window, not just apply. If unsure whether a change is disruptive, treat it
as disruptive.
Not covered: Home Assistant image bumps by Flux image automation. They stay
fully automatic by owner decision. If one breaks Home Assistant, roll it
back with [RUNBOOK-home-assistant-image-rollback.md](kubernetes/RUNBOOK-home-assistant-image-rollback.md).
An incident rollback is not planned work; the owner decides its timing.
### 2. HA-impact review before merge
Every PR that touches `kubernetes/apps/home-assistant/`, or makes a
cluster-wide disruptive change from the list above, needs an **HA-impact
review** before merge: a named reviewer states in the PR what Home
Assistant will notice (restart, lost integration, lost history, none) and
approves.
- Reviewer: Hestia, the Home Assistant maintainer.
- Until Hestia exists: the owner does the HA-impact review.
This is in addition to the normal review and merge by Argus. Argus does
not merge such a PR without the HA-impact approval.
## Agent runtime images
Pod images for the Paperclip Kubernetes sandbox (REN-284). Each is the stock
`ghcr.io/paperclipai/agent-runtime-claude` (pinned by digest) plus a tool
set. Base user (1000, non-root), entrypoint and cmd are unchanged; the
images run with a read-only root FS. Source in `ci/images/agent-runtime-*`,
built by `.woodpecker/build-agent-runtime.yml` (re-run alone: deployment to
target `build-agent-runtime`), versions bumped by Renovate. Both also
carry `claude-agent-acp` for the ACP engine, pinned and patched like
Paperclip ships it, SDK override included
(`ci/images/agent-runtime-common/`); bump it by hand
together with Paperclip. All tools are on the default `PATH`.
| Image | `imageRegistry` | Adds |
|---|---|---|
| `gitea.renner.uno/konairius/agent-runtime-go/agent-runtime-claude:v1` | `gitea.renner.uno/konairius/agent-runtime-go` | Go toolchain, golangci-lint, govulncheck |
| `gitea.renner.uno/konairius/agent-runtime-ops/agent-runtime-claude:v1` | `gitea.renner.uno/konairius/agent-runtime-ops` | kubectl, flux, kustomize, helm |
The plugin keeps the name `agent-runtime-claude:v1` and swaps only host and
org, hence the nested path. Every build also pushes a date tag
(`:YYYY-MM-DD`) as the rollback point. `konairius` is public: no pull secret.
+7 -6
View File
@@ -1,5 +1,5 @@
---
# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
# Deploy a NetBird reverse-proxy HA cluster member (a second proxy host).
# Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
#
# Prerequisites:
@@ -7,11 +7,12 @@
# listed under [netbird_proxy] in inventory/pve_hosts.yml. Docker is
# installed by the (Arch-aware) docker-install role.
# - DNS A/AAAA for the host's own name (e.g. proxy-1.renner.uno) -> its IP.
# - Uses the existing vault_netbird_proxy_token (shared by every member) — no
# new token to generate; a shared token + same NB_PROXY_DOMAIN forms the
# cluster.
# - AFTER deploy + verify: add this host's A/AAAA to proxy.renner.uno so
# traffic round-robins across the cluster.
# - Uses the existing vault_netbird_proxy_token (shared with VPS-1) — no new
# token to generate; a shared token + same NB_PROXY_DOMAIN forms the cluster.
# - AFTER deploy + verify: add this host's IP to the proxy domain so traffic
# round-robins across the cluster. Repoint proxy.renner.uno away from its
# CNAME to netbird.renner.uno (management host) to A/AAAA records listing
# BOTH proxy IPs (VPS-1 + this host).
- name: Deploy NetBird reverse-proxy cluster member
hosts: netbird_proxy
+1
View File
@@ -18,3 +18,4 @@
- common
- docker-install
- netbird-server
- netbird-proxy-watchdog
+4 -4
View File
@@ -43,10 +43,10 @@
#
# WHAT IS NOT, and must not be added:
#
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on the
# proxies, an Ansible template — but the
# management server does not validate it from
# any file. It is registered
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on both
# the server and the proxies, all Ansible
# templates — but the management server does not
# validate it from any of them. It is registered
# in the management DATABASE. Rotating it changes
# only what the proxies present, so every proxy
# is locked out: "proxy auth failed: too many
+1 -12
View File
@@ -26,16 +26,8 @@ edge_open3e_poll_seconds: 30
# entries when added), so the multi-frame read is cheap at a 30 s poll.
# Records faults/info events only (NormalOperation, LagDeviceDtcReported,
# LegionellaProtectionTemperatureReached) — NOT setpoint or program changes.
# onebase.*: the 29 paired ZigBee slots on the One Base gateway (ECU 0x6c3) --
# 8 radiator valves, 9 climate sensors, 9 underfloor actuators, 3 floor
# thermostats. Per-device room temperature, active setpoint, humidity, valve
# opening/demand, battery and RSSI; the local replacement for the ViCare cloud
# room sensors. Needs the onebase overlay (files/open3e-onebase.py) -- the stock
# codec declares 57 bytes and a paired slot answers with 68.
# Only CurrentValues are polled. The room and device registries (1884.., 2084..)
# are static configuration and are read on demand, not every 30 s.
edge_open3e_datapoints: >-
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791,onebase.2086,onebase.2089,onebase.2092,onebase.2095,onebase.2098,onebase.2101,onebase.2104,onebase.2107,onebase.2110,onebase.2113,onebase.2116,onebase.2119,onebase.2122,onebase.2125,onebase.2128,onebase.2131,onebase.2134,onebase.2137,onebase.2140,onebase.2143,onebase.2262,onebase.2265,onebase.2268,onebase.2271,onebase.2274,onebase.2277,onebase.2280,onebase.2283,onebase.2286
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791
# ECU addresses; -dev selects codec set, tx selects the UDS address.
# vcmu (refrigerant-circuit controller) has no dedicated codec set —
# omitting dpList makes open3e fall back to the generic datapoint list,
@@ -46,9 +38,6 @@ edge_open3e_devices:
# files/open3e-vx3plus.py) — die Stock-vx3-Liste filtert sie sonst weg.
vx3: { tx: "0x6a1", dpList: "/config/vx3plus.py" }
vcmu: { tx: "0x68c" }
# One Base ZigBee gateway. Overlay re-declares the paired ZigBee slots at
# their real 68-byte length (see files/open3e-onebase.py).
onebase: { tx: "0x6c3", dpList: "/config/onebase.py" }
edge_e3oncan_gap_seconds: 10
edge_e3oncan_mqtt_format: "{device}_{didNumber:04d}_{didName}"
@@ -1,54 +0,0 @@
# Managed by ansible (roles/heating-can). Datapoint overlay for the One Base
# ZigBee gateway on ECU 0x6c3 (BACKENDGATEWAY).
#
# Upstream declares ZigBee<N>DeviceCurrentValues as 57 bytes. A PAIRED device
# slot on this gateway answers with 68: the 57 known bytes plus the parent
# device serial and three trailing bytes. udsoncan reads only the declared
# length and then treats the next two bytes -- the first half of that serial --
# as a foreign data identifier, so every paired slot fails to decode with
# "Server returned values for data identifier 0x.... that was not requested".
# The bogus identifier differs per device because the serial does.
#
# This overlay re-declares the 29 paired slots at 68 bytes, reusing upstream's
# sub-type list so the per-device-type O3ESwitch (climate sensor / TRV / floor
# thermostat / actuator, incl. DeviceTemperatureSetpoint) is inherited rather
# than copied.
#
# EMPTY slots answer with 57 and decode against the stock codec, so they are
# left alone and are not polled. A slot that is later unpaired therefore starts
# reporting a decode error on its topic instead of a value.
#
# Verified on ECU 0x6c3, 2026-09-17: 29 paired slots, all 68 bytes.
from open3e.Open3Ecodecs import O3EComplexType, RawCodec
from open3e.Open3Edatapoints import dataIdentifiers as _master
# ZigBee<N>DeviceCurrentValues of the paired slots: devices 1-20 run 2086..2143
# in steps of 3, devices 21-29 continue at 2262..2286.
_PAIRED_CURRENT_VALUES = [2086 + 3 * n for n in range(20)] + [
2262 + 3 * n for n in range(9)
]
def _with_parent_serial(did):
stock = _master["dids"][did]
return O3EComplexType(
68,
stock.id,
list(stock.subTypes)
+ [
RawCodec(8, "ParentDeviceSerialnumber"),
RawCodec(2, "Unknown_65_66"),
RawCodec(1, "Unknown_67"),
],
desc=stock.desc,
info=stock.info,
acc=stock.acc,
)
dataIdentifiers = {
"dids": {
**_master["dids"],
**{did: _with_parent_serial(did) for did in _PAIRED_CURRENT_VALUES},
}
}
-7
View File
@@ -129,13 +129,6 @@
mode: "0644"
notify: Restart heating-can
- name: Deploy open3e One Base datapoint overlay (68-byte ZigBee slots)
ansible.builtin.copy:
src: open3e-onebase.py
dest: "{{ heating_can_base_dir }}/onebase.py"
mode: "0644"
notify: Restart heating-can
# --- Compose project ---------------------------------------------------------
- name: Deploy the broker credential env file
ansible.builtin.template:
@@ -17,7 +17,6 @@ services:
volumes:
- ./devices.json:/config/devices.json:ro
- ./vx3plus.py:/config/vx3plus.py:ro
- ./onebase.py:/config/onebase.py:ro
env_file:
# Broker credential, 0600 -- see heating-can.env.j2.
- heating-can.env
@@ -1,5 +1,5 @@
---
# Self-healing watchdog for the NetBird reverse proxy on the proxy hosts.
# Self-healing watchdog for the NetBird reverse proxy (VPS-1 and proxy-1).
#
# Why it exists: the proxy's embedded NetBird client pins each proxied
# domain's route to a specific routing-peer identity. The netbird-ingress
@@ -20,7 +20,9 @@ netbird_watchdog_probe_hosts:
- home.renner.uno
- immich.renner.uno
# Address the probe connects to. The proxy binds :443 on the host directly.
# Address the probe connects to. 127.0.0.1 reaches the local proxy on both
# members: VPS-1 routes it via Traefik's HostSNI(*) TCP passthrough, proxy-1
# binds :443 directly.
netbird_watchdog_probe_address: 127.0.0.1
netbird_watchdog_container: netbird-proxy
@@ -1,7 +1,7 @@
---
# Deploys the reverse-proxy watchdog (see defaults/main.yml for why).
- name: Ensure curl is installed (probe dependency)
- name: Ensure curl is installed (probe dependency; proxy-1 is Arch, VPS-1 Debian)
ansible.builtin.package:
name: curl
state: present
@@ -1,8 +1,8 @@
---
# NetBird reverse-proxy HA cluster member (proxy-only host, e.g. proxy-1).
# Every member uses the same NB_PROXY_DOMAIN + token. NetBird load-balances/
# fails the cluster over via DNS round-robin on the proxy domain (see
# plays/install-netbird-proxy.yml).
# Joins the same proxy cluster as the VPS-1 proxy: identical NB_PROXY_DOMAIN
# + the same token. NetBird load-balances/fails the cluster over via DNS
# round-robin on the proxy domain (see plays/install-netbird-proxy.yml).
# MUST match the netbird-server role's netbird_version — the reverse-proxy
# release line has to track the management server it enrolls against.
@@ -11,13 +11,11 @@ netbird_version: "0.73.2"
# Management server this proxy enrolls against (public API, TLS).
netbird_server_host: netbird.renner.uno
# Proxy domain — MUST be identical across every cluster member, and must NOT
# be a subdomain of the management domain (netbird.renner.uno) per NetBird
# docs (TLS/routing conflicts).
# Proxy domain — MUST be identical across every cluster member.
netbird_proxy_domain: "proxy.renner.uno"
# A shared NB_PROXY_TOKEN across members with the same NB_PROXY_DOMAIN is a
# valid HA cluster, so one token serves every member.
# Reuse VPS-1's proxy token — a shared NB_PROXY_TOKEN across members with the
# same NB_PROXY_DOMAIN is a valid HA cluster, so no separate token is needed.
netbird_proxy_token: "{{ vault_netbird_proxy_token }}"
# Relay auth secret — MUST be identical across every relay instance and the
@@ -1,7 +1,8 @@
# NetBird reverse-proxy — HA cluster member on a proxy-only host.
# Just the reverse-proxy container, bound directly to :443. No Traefik: the
# proxy owns the port and terminates TLS + answers its own tls-alpn-01 ACME
# challenges directly.
# Just the reverse-proxy container, bound directly to :443. No Traefik:
# unlike VPS-1 (where the proxy shares :443 with management/signal/relay via
# SNI routing), this host runs nothing else, so the proxy owns the port and
# terminates TLS + answers its own tls-alpn-01 ACME challenges directly.
services:
proxy:
image: netbirdio/reverse-proxy:{{ netbird_version }}
@@ -23,3 +23,9 @@ netbird_datastore_key: "{{ vault_netbird_datastore_key }}"
# Relay secret for WebSocket relay authentication
netbird_relay_secret: "{{ vault_netbird_relay_secret }}"
# Reverse proxy configuration
# IMPORTANT: Proxy domain must NOT be a subdomain of the management domain
# (netbird.renner.uno) per NetBird docs - causes TLS/routing conflicts
netbird_proxy_domain: "proxy.renner.uno"
netbird_proxy_token: "{{ vault_netbird_proxy_token }}"
@@ -16,3 +16,7 @@
- name: Restart netbird coturn
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d --force-recreate coturn
- name: Restart netbird proxy
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d --force-recreate proxy
+9 -68
View File
@@ -36,6 +36,7 @@
- /opt/netbird/management
- /opt/netbird/signal
- /opt/netbird/letsencrypt
- /opt/netbird/proxy/certs
- name: Deploy docker-compose.yml
ansible.builtin.template:
@@ -58,83 +59,23 @@
mode: "0644"
notify: Restart netbird coturn
# The public edge is the proxy hosts (plays/install-netbird-proxy.yml); this
# host runs no reverse proxy. The tasks below remove the one it used to run,
# its watchdogs first so they cannot restart it mid-removal, and can go once
# the role has run against every host that ever carried it.
- name: Check for reverse-proxy watchdog units
ansible.builtin.stat:
path: "/etc/systemd/system/{{ item }}"
loop:
- netbird-proxy-watchdog.timer
- netbird-proxy-flood-watchdog.service
register: _proxy_watchdog_units
- name: Stop and disable reverse-proxy watchdogs
ansible.builtin.systemd:
name: "{{ item.item }}"
state: stopped
enabled: false
loop: "{{ _proxy_watchdog_units.results }}"
loop_control:
label: "{{ item.item }}"
when: item.stat.exists
- name: Remove reverse-proxy watchdog units and scripts
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /etc/systemd/system/netbird-proxy-watchdog.timer
- /etc/systemd/system/netbird-proxy-watchdog.service
- /etc/systemd/system/netbird-proxy-flood-watchdog.service
- /usr/local/bin/netbird-proxy-watchdog
- /usr/local/bin/netbird-proxy-flood-watchdog
- /run/netbird-proxy-watchdog
register: _proxy_watchdog_files
- name: Reload systemd after removing the watchdog units
ansible.builtin.systemd:
daemon_reload: true
when: _proxy_watchdog_files.changed
- name: Deploy proxy.env
ansible.builtin.template:
src: proxy.env.j2
dest: /opt/netbird/proxy.env
mode: "0644"
notify: Restart netbird proxy
- name: Pull NetBird container images
ansible.builtin.shell: |
cd /opt/netbird && docker compose pull
changed_when: false
# --remove-orphans removes containers of services no longer in the compose
# file (the reverse proxy).
- name: Start NetBird containers
ansible.builtin.shell: |
cd /opt/netbird && docker compose up -d --remove-orphans
cd /opt/netbird && docker compose up -d
register: docker_compose_result
changed_when: >-
'Started' in docker_compose_result.stdout or
'Creating' in docker_compose_result.stdout or
'Removed' in docker_compose_result.stdout or
'Removed' in docker_compose_result.stderr
- name: Remove reverse-proxy volumes
ansible.builtin.shell: |
set -eu
for v in netbird_netbird_proxy_certs netbird_netbird_proxy_data netbird_netbird_proxy_state; do
if [ -n "$(docker volume ls -q --filter "name=^${v}$")" ]; then
docker volume rm "$v"
echo "removed $v"
fi
done
register: _proxy_volumes
changed_when: "'removed' in _proxy_volumes.stdout"
- name: Remove reverse-proxy config and state
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /opt/netbird/proxy.env
- /opt/netbird/proxy
- /opt/netbird/proxy-state
changed_when: "'Started' in docker_compose_result.stdout or 'Creating' in docker_compose_result.stdout"
# The management store has no cluster-side backup. Pulling one put a root
# key on this host that the cluster held, and this server's own state is
@@ -187,9 +187,10 @@ services:
- "traefik.http.routers.netbird-management.service=netbird-management"
- "traefik.http.services.netbird-management.loadbalancer.server.port=33073"
- "traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c"
# gRPC reverse-proxy service — the proxy hosts (proxy-1, proxy-2)
# connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443 and get
# a 404 without it. Same backend (h2c 33073).
# gRPC reverse-proxy service — remote reverse-proxy instances (e.g.
# proxy-1) connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443.
# The local proxy uses management:33073 directly, so it never needed
# this route; remote ones get a 404 without it. Same backend (h2c 33073).
- "traefik.http.routers.netbird-proxyservice.rule=Host(`{{ netbird_server_host }}`) && PathPrefix(`/management.ProxyService/`)"
- "traefik.http.routers.netbird-proxyservice.entrypoints=websecure"
- "traefik.http.routers.netbird-proxyservice.tls=true"
@@ -220,6 +221,37 @@ services:
max-size: "500m"
max-file: "2"
# Reverse Proxy Service (per docs.netbird.io/selfhosted/migration/enable-reverse-proxy)
proxy:
image: netbirdio/reverse-proxy:{{ netbird_version }}
container_name: netbird-proxy
restart: unless-stopped
networks:
- netbird
depends_on:
- management
env_file:
- ./proxy.env
volumes:
- netbird_proxy_certs:/certs
- /opt/netbird/proxy-state:/var/lib/netbird
labels:
- "traefik.enable=true"
- "traefik.tcp.routers.proxy-passthrough.entrypoints=websecure"
- "traefik.tcp.routers.proxy-passthrough.rule=HostSNI(`*`)"
- "traefik.tcp.routers.proxy-passthrough.tls.passthrough=true"
- "traefik.tcp.routers.proxy-passthrough.service=proxy-tls"
- "traefik.tcp.routers.proxy-passthrough.priority=1"
- "traefik.tcp.services.proxy-tls.loadbalancer.server.port=8443"
logging:
driver: "json-file"
options:
max-size: "500m"
max-file: "2"
networks:
netbird:
driver: bridge
volumes:
netbird_proxy_certs:
@@ -0,0 +1,8 @@
NB_PROXY_DOMAIN={{ netbird_proxy_domain }}
NB_PROXY_TOKEN={{ netbird_proxy_token }}
NB_PROXY_MANAGEMENT_ADDRESS=http://management:33073
NB_PROXY_ALLOW_INSECURE=true
NB_PROXY_ADDRESS=:8443
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
@@ -1,120 +0,0 @@
{
"//": "Permission scope for the wiki-consolidate agent in CI: read the vault and the",
"//2": "source repos cloned under /root/source, read the live cluster through the",
"//3": "ci-diagnostics kubeconfig, edit wiki/, output/ and the rotation state only.",
"//4": "Deny rules win; git and kubectl are read-only subcommands, no network tools.",
"permissions": {
"allow": [
"Read(//root/Notes/**)",
"Read(//root/source/**)",
"Edit(//root/Notes/wiki/**)",
"Edit(//root/Notes/output/**)",
"Edit(//root/Notes/.wiki-consolidation-state.json)",
"Bash(/root/Notes/bin/wiki-snapshot.sh:*)",
"Bash(/root/Notes/bin/wiki_rank.py:*)",
"Bash(/root/Notes/bin/wiki_ask.py:*)",
"Bash(~/Notes/bin/wiki-snapshot.sh:*)",
"Bash(~/Notes/bin/wiki_rank.py:*)",
"Bash(~/Notes/bin/wiki_ask.py:*)",
"Bash(~/Notes/bin/wiki-delete.sh:*)",
"Bash(git -C /root/Notes log:*)",
"Bash(git -C /root/Notes show:*)",
"Bash(git -C /root/Notes diff:*)",
"Bash(git -C /root/Notes status:*)",
"Bash(git -C /root/Notes rev-parse:*)",
"Bash(git -C /root/Notes ls-files:*)",
"Bash(git -C ~/Notes log:*)",
"Bash(git -C ~/Notes show:*)",
"Bash(git -C ~/Notes diff:*)",
"Bash(git -C ~/Notes status:*)",
"Bash(git -C ~/Notes rev-parse:*)",
"Bash(git -C ~/Notes ls-files:*)",
"Bash(kubectl get:*)",
"Bash(kubectl describe:*)",
"Bash(kubectl version:*)",
"Bash(kubectl api-resources:*)",
"Bash(kubectl explain:*)",
"Bash(git log:*)",
"Bash(git show:*)",
"Bash(git status:*)",
"Bash(git diff:*)",
"Bash(git rev-parse:*)",
"Bash(git ls-files:*)",
"Bash(git -C /root/source/homelab log:*)",
"Bash(git -C /root/source/homelab show:*)",
"Bash(git -C /root/source/homelab diff:*)",
"Bash(git -C /root/source/homelab status:*)",
"Bash(git -C /root/source/homelab rev-parse:*)",
"Bash(git -C /root/source/homelab ls-files:*)",
"Bash(git -C /root/source/homeassistant log:*)",
"Bash(git -C /root/source/homeassistant show:*)",
"Bash(git -C /root/source/homeassistant diff:*)",
"Bash(git -C /root/source/homeassistant status:*)",
"Bash(git -C /root/source/homeassistant rev-parse:*)",
"Bash(git -C /root/source/homeassistant ls-files:*)",
"Bash(git -C ~/source/homelab log:*)",
"Bash(git -C ~/source/homelab show:*)",
"Bash(git -C ~/source/homelab diff:*)",
"Bash(git -C ~/source/homelab status:*)",
"Bash(git -C ~/source/homelab rev-parse:*)",
"Bash(git -C ~/source/homelab ls-files:*)",
"Bash(git -C ~/source/homeassistant log:*)",
"Bash(git -C ~/source/homeassistant show:*)",
"Bash(git -C ~/source/homeassistant diff:*)",
"Bash(git -C ~/source/homeassistant status:*)",
"Bash(git -C ~/source/homeassistant rev-parse:*)",
"Bash(git -C ~/source/homeassistant ls-files:*)",
"Bash(/root/Notes/bin/wiki-delete.sh:*)",
"Bash(ls:*)",
"Bash(date:*)",
"Bash(wc:*)",
"Bash(stat:*)",
"Bash(head:*)",
"Bash(tail:*)"
],
"deny": [
"Read(//proc/**)",
"Read(//etc/**)",
"Read(//root/.config/**)",
"Read(//root/.kube/**)",
"Read(//woodpecker/**/.ci-diag/**)",
"Read(//root/Notes/.git/**)",
"Read(//root/source/*/.git/**)",
"Read(//root/Notes/_archive/**)",
"Edit(//root/Notes/raw/**)",
"Edit(//root/Notes/_archive/**)",
"Edit(//root/source/**)",
"Bash(kubectl apply:*)",
"Bash(kubectl delete:*)",
"Bash(kubectl patch:*)",
"Bash(kubectl edit:*)",
"Bash(kubectl scale:*)",
"Bash(kubectl rollout:*)",
"Bash(kubectl exec:*)",
"Bash(kubectl cp:*)",
"Bash(kubectl drain:*)",
"Bash(kubectl cordon:*)",
"Bash(kubectl label:*)",
"Bash(kubectl annotate:*)",
"Bash(git push:*)",
"Bash(git reset:*)",
"Bash(git checkout:*)",
"Bash(git clean:*)",
"Bash(git rebase:*)",
"Bash(git config:*)",
"Bash(git -C /root/Notes config:*)",
"Bash(git -C ~/Notes config:*)",
"Bash(git -C /root/source/homelab config:*)",
"Bash(git -C /root/source/homeassistant config:*)",
"Bash(git -C ~/source/homelab config:*)",
"Bash(git -C ~/source/homeassistant config:*)",
"Bash(rm:*)",
"Bash(rm -rf:*)",
"Bash(curl:*)",
"Bash(wget:*)",
"Bash(ssh:*)",
"Bash(tea:*)",
"Bash(woodpecker:*)"
]
}
}
@@ -1,5 +0,0 @@
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes); the homelab and homeassistant repositories are checked out under ~/source/. `kubectl` reads the live cluster with a read-only identity.
Run the wiki-consolidate agent now for today's scheduled consolidation pass. Follow its instructions exactly: snapshot the vault to git first, abort if that fails, select today's rotating cohort of 20 from the state file, verify claims against both their sources and the live systems, act on the verdicts, write the dated report to ~/Notes/output/, update the state file, and snapshot again at the end. Do not push; the pipeline pushes the run's commits.
Report the counts and the coverage progress when you are done.
-7
View File
@@ -1,7 +0,0 @@
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes). Do exactly this, nothing else:
Glob ~/Notes/raw/*.md (top level only, never ~/Notes/raw/_processed/). Distill every file you find into ~/Notes/wiki/ following your process, including the ranker check for every concept. Move each fully processed source to ~/Notes/raw/_processed/. If raw/ has no top-level files besides _index.md, say so and stop.
Do not commit or push; the pipeline does that after you finish. Do not touch anything outside ~/Notes/wiki/ and ~/Notes/raw/.
End with one line per file: which wiki notes it produced or merged into, or "nothing distillable".
-41
View File
@@ -1,41 +0,0 @@
{
"//": "Permission scope for the wiki-maintenance agent (wiki-distill) in CI.",
"//2": "The step's environment holds a Gitea push token, a TypeSafe key and the wiki-index",
"//3": "client credential, so the Read tool is kept away from /proc (env dumps), /root outside",
"//4": "the vault symlink target, /etc and the checkout's .git/. The vault itself is the work.",
"//5": "Mirrors ci/agents/issue-fix/settings.json; absolute globs because the vault is reached",
"//6": "through the ~/Notes symlink the pipeline creates.",
"permissions": {
"allow": [
"Read(//root/Notes/**)",
"Edit(//root/Notes/wiki/**)",
"Edit(//root/Notes/output/**)",
"Edit(//root/Notes/raw/**)",
"Bash(/root/Notes/bin/wiki_rank.py:*)",
"Bash(/root/Notes/bin/wiki_ask.py:*)",
"Bash(mv /root/Notes/raw/:*)",
"Bash(mkdir -p /root/Notes/:*)",
"Bash(~/Notes/bin/wiki_rank.py:*)",
"Bash(~/Notes/bin/wiki_ask.py:*)",
"Bash(mv ~/Notes/raw/:*)",
"Bash(mkdir -p ~/Notes/:*)",
"Bash(ls:*)",
"Bash(date:*)",
"Bash(wc:*)",
"Bash(head:*)",
"Bash(tail:*)"
],
"deny": [
"Read(//proc/**)",
"Read(//etc/**)",
"Read(//root/.config/**)",
"Read(//root/Notes/.git/**)",
"Read(//root/Notes/_archive/**)",
"Edit(//root/Notes/_archive/**)",
"Bash(rm:*)",
"Bash(git:*)",
"Bash(curl:*)",
"Bash(wget:*)"
]
}
}
@@ -1,66 +0,0 @@
diff --git a/dist/acp-agent.js b/dist/acp-agent.js
index 0214a509eb5e7b777a69e4e34637941c275d27b1..403be02df66e25eee29e7d5bef14627011e02b30 100644
--- a/dist/acp-agent.js
+++ b/dist/acp-agent.js
@@ -2922,9 +2922,19 @@ export class ClaudeAcpAgent {
amount: message.total_cost_usd,
currency: "USD",
},
- ...(message.origin && {
- _meta: { "_claude/origin": message.origin },
- }),
+ _meta: {
+ ...(message.origin && { "_claude/origin": message.origin }),
+ // ACP's core Usage shape exposes context occupancy and
+ // cost, but not the billable input/output split. Keep the
+ // SDK's bounded aggregate in extension metadata so ACPX can
+ // normalize it without persisting a raw provider message.
+ usage: {
+ input_tokens: message.usage.input_tokens,
+ output_tokens: message.usage.output_tokens,
+ cache_read_input_tokens: message.usage.cache_read_input_tokens,
+ cache_creation_input_tokens: message.usage.cache_creation_input_tokens,
+ },
+ },
},
});
}
@@ -5311,6 +5321,10 @@ export class ClaudeAcpAgent {
settingSources: ["user", "project", "local"],
...(thinking !== undefined && { thinking }),
...userProvidedOptions,
+ // Paperclip Runner owns the complete session context. Its isolated
+ // user root contains only assigned skills; project/local settings
+ // would reintroduce host prompts, plugins, skills, and MCP servers.
+ ...(process.env.PAPERCLIP_ACPX_ISOLATED_CONTEXT === "1" && { settingSources: ["user"] }),
...(settings && { settings }),
env,
// Override certain fields that must be controlled by ACP
@@ -5318,7 +5332,9 @@ export class ClaudeAcpAgent {
includePartialMessages: true,
forwardSubagentText,
mcpServers: {
- ...(userProvidedOptions?.mcpServers || {}),
+ ...(process.env.PAPERCLIP_ACPX_ISOLATED_CONTEXT === "1"
+ ? {}
+ : (userProvidedOptions?.mcpServers || {})),
...mcpServers,
...(fileChangeAuditSupport
? { [FILE_CHANGE_AUDIT_SERVER_NAME]: fileChangeAuditSupport.mcpServer }
@@ -5329,6 +5345,16 @@ export class ClaudeAcpAgent {
allowDangerouslySkipPermissions: ALLOW_BYPASS,
permissionMode: initialPermissionMode,
canUseTool: this.canUseTool(sessionId),
+ // The runner pins this actual MCP endpoint before provider launch.
+ // Exempt only task delivery/control tools at the SDK dispatch boundary,
+ // not arbitrary provider permission metadata or MCP name prefixes.
+ ...(process.env.PAPERCLIP_ACPX_ISOLATED_CONTEXT === "1" && {
+ allowedTools: mcpServers.paperclip?.type === "http"
+ && mcpServers.paperclip.url === process.env.PAPERCLIP_ACPX_TASK_TOOL_BRIDGE_URL
+ ? ["paperclip_finish", "paperclip_block", "read_current_wake_comments", "request_human_input"]
+ .map((tool) => `mcp__paperclip__${tool}`)
+ : [],
+ }),
// Forward MCP elicitation requests onto ACP elicitation. Only attached
// when the client advertised support, so non-supporting clients keep the
// SDK's default (auto-decline) behavior. (AskUserQuestion is handled in
@@ -1,9 +0,0 @@
{
"private": true,
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "0.73.0"
},
"overrides": {
"@anthropic-ai/claude-agent-sdk": "0.3.280"
}
}
-68
View File
@@ -1,68 +0,0 @@
# Paperclip agent runtime + Go toolchain (REN-284 / REN-286). Runs as the pod
# image of the `k8s-go` sandbox environment.
#
# Every tool comes from its upstream image, pinned by tag AND digest, so the
# dockerfile manager in Renovate bumps version and digest together. A curl +
# sha256 pin (as in ci/images/k8s) would break on every Renovate bump.
#
# Both binaries are static; they run on the Ubuntu 22.04 base unchanged.
FROM golang:1.27.1-bookworm@sha256:69a7b9788769bec032d238959b61854e9ae87f57be9029ec04e9885fabf99195 AS go
# govulncheck (REN-488) has no upstream image: build it in the go stage, static.
# The version pins the module; go verifies it against sum.golang.org. Renovate
# does not track it (no regex manager here): bump by hand.
FROM go AS govulncheck
RUN CGO_ENABLED=0 GOBIN=/out go install golang.org/x/vuln/cmd/govulncheck@v1.8.0
FROM golangci/golangci-lint:v2.14.0@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS golangci-lint
# upstream publishes no `v1` tag, only git-<sha> tags. Renovate keeps the
# digest current but cannot order commit tags; a newer base is a manual bump.
FROM ghcr.io/paperclipai/agent-runtime-claude:git-38d8f371722b315d2fb3bbaa512518742e33ce2f@sha256:33ad285f9476b8637d6d940c08d577797030125675bdf4f497a8ed1f79261c8b
COPY --from=go /usr/local/go /usr/local/go
COPY --from=golangci-lint /usr/bin/golangci-lint /usr/local/bin/golangci-lint
COPY --from=govulncheck /out/govulncheck /usr/local/bin/govulncheck
# go and gofmt also in /usr/local/bin: a process started with the default PATH
# (not the image ENV) still finds them.
USER root
RUN ln -s /usr/local/go/bin/go /usr/local/bin/go \
&& ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt
USER 1000:1000
# ACP server the Paperclip ACP engine starts in the pod (REN-302). Version and
# patch match what Paperclip ships: packages/adapters/claude-local/package.json
# pins 0.73.0 and applies patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch
# (isolated context, task-tool allow-list, usage metadata). Not bumped by
# Renovate on purpose: bump only together with Paperclip, and replace the patch.
# Paperclip also overrides the SDK (package.json pnpm overrides:
# claude-agent-sdk 0.3.280); a plain `npm install -g` ignores that and pulls an
# older SDK. So install from a package.json with the same override into a
# fixed prefix, and link the binary into /usr/local/bin.
COPY agent-runtime-common/claude-agent-acp.package.json /opt/claude-agent-acp/package.json
COPY agent-runtime-common/claude-agent-acp-0.73.0.patch /tmp/claude-agent-acp.patch
USER root
RUN cd /opt/claude-agent-acp \
&& npm install --omit=dev --no-audit --no-fund \
&& git -C node_modules/@agentclientprotocol/claude-agent-acp apply /tmp/claude-agent-acp.patch \
&& npm ls @anthropic-ai/claude-agent-sdk | grep -q 'claude-agent-sdk@0.3.280' \
&& ln -s /opt/claude-agent-acp/node_modules/.bin/claude-agent-acp /usr/local/bin/claude-agent-acp \
&& rm /tmp/claude-agent-acp.patch \
&& npm cache clean --force
USER 1000:1000
# The sandbox pod has a read-only root FS. Writable mounts: $HOME
# (/home/paperclip, 1Gi), $HOME/.cache (1Gi), /tmp (2Gi), /workspace.
# - GOPATH (module cache) goes to /tmp, the biggest one.
# - GOCACHE and the golangci-lint cache keep their default under $HOME/.cache.
# - GOTOOLCHAIN=local: a go.mod asking for a newer Go fails clearly instead of
# downloading a toolchain through the egress allow-list.
ENV PATH=/usr/local/go/bin:$PATH \
GOPATH=/tmp/go \
GOTOOLCHAIN=local
# Base user (1000:1000), entrypoint (tini) and cmd (agent shim) are inherited.
# The PATH check uses the base image's default PATH, not the ENV above.
RUN go version && golangci-lint version \
&& env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
sh -ec 'for t in go gofmt golangci-lint govulncheck claude-agent-acp; do command -v "$t"; done'
-51
View File
@@ -1,51 +0,0 @@
# Paperclip agent runtime + cluster tools (REN-284 / REN-286). Runs as the pod
# image of the `k8s-ops` sandbox environment.
#
# Every tool comes from its upstream image, pinned by tag AND digest, so the
# dockerfile manager in Renovate bumps version and digest together. A curl +
# sha256 pin (as in ci/images/k8s) would break on every Renovate bump.
#
# kubectl and flux match the ci-k8s image. All four binaries are static; they
# run on the Ubuntu 22.04 base unchanged.
FROM registry.k8s.io/kubectl:v1.37.1@sha256:b7cab618e281b1ee7484e7b706a96e2135fbb6e072c2a573a7dab4e87d7f2385 AS kubectl
FROM ghcr.io/fluxcd/flux-cli:v2.9.6@sha256:b1ac18156f227af9a524b842a96f2c20986c7a779ef721df3ba4e4540f449d76 AS flux
FROM registry.k8s.io/kustomize/kustomize:v5.8.1@sha256:899fcd3bc898160e62bcaf82932b0cb29ba38d16272353db2e7acbba82129429 AS kustomize
FROM alpine/helm:4.3.0@sha256:a6cf54599ccb99d90cf0712b30f03fdb3cab062e6b94e0418cc4db7e8a1464b2 AS helm
# upstream publishes no `v1` tag, only git-<sha> tags. Renovate keeps the
# digest current but cannot order commit tags; a newer base is a manual bump.
FROM ghcr.io/paperclipai/agent-runtime-claude:git-38d8f371722b315d2fb3bbaa512518742e33ce2f@sha256:33ad285f9476b8637d6d940c08d577797030125675bdf4f497a8ed1f79261c8b
COPY --from=kubectl /bin/kubectl /usr/local/bin/kubectl
COPY --from=flux /usr/local/bin/flux /usr/local/bin/flux
COPY --from=kustomize /app/kustomize /usr/local/bin/kustomize
COPY --from=helm /usr/bin/helm /usr/local/bin/helm
# ACP server the Paperclip ACP engine starts in the pod (REN-302). Version and
# patch match what Paperclip ships: packages/adapters/claude-local/package.json
# pins 0.73.0 and applies patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch
# (isolated context, task-tool allow-list, usage metadata). Not bumped by
# Renovate on purpose: bump only together with Paperclip, and replace the patch.
# Paperclip also overrides the SDK (package.json pnpm overrides:
# claude-agent-sdk 0.3.280); a plain `npm install -g` ignores that and pulls an
# older SDK. So install from a package.json with the same override into a
# fixed prefix, and link the binary into /usr/local/bin.
COPY agent-runtime-common/claude-agent-acp.package.json /opt/claude-agent-acp/package.json
COPY agent-runtime-common/claude-agent-acp-0.73.0.patch /tmp/claude-agent-acp.patch
USER root
RUN cd /opt/claude-agent-acp \
&& npm install --omit=dev --no-audit --no-fund \
&& git -C node_modules/@agentclientprotocol/claude-agent-acp apply /tmp/claude-agent-acp.patch \
&& npm ls @anthropic-ai/claude-agent-sdk | grep -q 'claude-agent-sdk@0.3.280' \
&& ln -s /opt/claude-agent-acp/node_modules/.bin/claude-agent-acp /usr/local/bin/claude-agent-acp \
&& rm /tmp/claude-agent-acp.patch \
&& npm cache clean --force
USER 1000:1000
# Read-only root FS: kubectl (~/.kube/cache) and helm (~/.cache, ~/.config)
# write only under $HOME, which the sandbox pod mounts writable. No ENV needed.
# Base user (1000:1000), entrypoint (tini) and cmd (agent shim) are inherited.
RUN kubectl version --client && flux version --client && kustomize version && helm version \
&& env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
sh -ec 'for t in kubectl flux kustomize helm claude-agent-acp; do command -v "$t"; done'
+1 -1
View File
@@ -1,4 +1,4 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Alpine plus jq. Nothing application-specific — a general-purpose base for any
# workload that needs to read JSON from a shell script.
+1 -1
View File
@@ -1,4 +1,4 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Alpine plus the openssl CLI. Nothing application-specific — named for what it
# is, so anything needing `openssl s_client` / `x509` from a shell script can use
+1 -1
View File
@@ -1,4 +1,4 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
RUN apk add --no-cache \
ansible \
-22
View File
@@ -1,22 +0,0 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
# step — the image is the binary plus its runtime.
#
# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
# 3.22/community.
#
# Config is read from /etc/bird/bird.conf, not the package default
# /etc/bird.conf, so the mount cannot shadow what the package ships.
#
# kubectl renders the init container's per-node config: go-template-file for
# the templating, a field selector for excluding the node itself, in-cluster
# config for the auth. From the distro, so there is no version literal to drift
# and no download to trust. It is only ever used to read core/v1 Nodes.
RUN apk add --no-cache bird2 kubectl
# Fails the build rather than the first pod if a binary or its path is wrong.
RUN /usr/sbin/bird --version \
&& kubectl version --client > /dev/null
ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]
+2 -4
View File
@@ -1,4 +1,4 @@
FROM node:24-alpine@sha256:ebfe2f90462722a7a4de65e91990e97fe0d401c70e0e762c5b53302f905ec1c1
FROM node:22-alpine@sha256:16e22a550f3863206a3f701448c45f7912c6896a62de43add43bb9c86130c3e2
# Default to the latest Claude Code. Pinning a fixed version is the wrong
# default here: old releases eventually stop working against the API, so a
@@ -13,9 +13,7 @@ RUN apk add --no-cache \
git \
jq \
openssh-client \
openssl \
python3 \
py3-yaml
openssl
# Install Claude Code, then make the resulting layer byte-stable so that a
# rebuild of the SAME version DEDUPS in the Gitea registry instead of orphaning
+4 -4
View File
@@ -28,10 +28,10 @@
# amd64 only, matching the ARCH the installer already hardcoded and the cluster's
# node architecture. Add a second stage if a different arch is ever needed.
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 AS fetch
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d AS fetch
ARG CNI_PLUGINS_VERSION=v1.9.1
ARG CNI_PLUGINS_SHA256=b98f74a0f8522f0a83867178729c1aa70f2158f90c45a2ca8fa791db1c76b303
ARG CNI_PLUGINS_VERSION=v1.5.1
ARG CNI_PLUGINS_SHA256=77baa2f669980a82255ffa2f2717de823992480271ee778aa51a9c60ae89ff9b
# busybox wget and tar are already in the base; nothing is installed here
# either. sha256sum -c fails the BUILD if the artifact does not match, which is
@@ -61,7 +61,7 @@ RUN set -eu; \
[ -x "/plugins/$p" ] || { echo "FATAL: /plugins/$p missing or not executable" >&2; exit 1; }; \
done
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Alpine rather than scratch: the installer needs a shell to copy the binaries
# onto the host. busybox covers cp/mv/chown/ls, so nothing is added.
+2 -2
View File
@@ -23,7 +23,7 @@
# namespace, so can0 is only reachable with network_mode: host. That is also
# how the local MQTT broker on 127.0.0.1:1883 is reached. CAN_RAW sockets need
# no capabilities, so both run unprivileged.
FROM python:3.14-slim@sha256:c3e521df8b2b498a7a682e7e18676771cb80c6b75b8699af886b2d554ce40151 AS build
FROM python:3.13-slim AS build
# Pins live here, in the artifact, rather than in Ansible variables.
#
@@ -56,7 +56,7 @@ RUN git clone --quiet https://github.com/MyHomeMyData/E3onCAN.git /opt/e3oncan \
&& rm -rf /opt/e3oncan/.git
FROM python:3.14-slim@sha256:c3e521df8b2b498a7a682e7e18676771cb80c6b75b8699af886b2d554ce40151
FROM python:3.13-slim
ARG OPEN3E_REF=6553c10453a6bae1210908f38a5af2481a46a3e5
ARG E3ONCAN_REF=461dcbdf97e8d72d2edb9929c63b39382af6c3a1
+3 -3
View File
@@ -1,6 +1,6 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TALOS_VERSION=v1.13.10
ARG TALOS_VERSION=v1.13.9
ARG VELERO_VERSION=v1.16.0
ARG KUBECTL_VERSION=v1.36.3
ARG FLUX_VERSION=2.9.5
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -1,8 +1,8 @@
# netbird-service-sync: reconciles NetBird reverse-proxy services (public
# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
# public label. Standard library only, so the image is the interpreter plus one
# file; runs unprivileged as its own user.
FROM python:3.14-slim@sha256:c3e521df8b2b498a7a682e7e18676771cb80c6b75b8699af886b2d554ce40151
FROM python:3.13-slim
RUN useradd --system --no-create-home --shell /usr/sbin/nologin controller
COPY controller.py /opt/controller.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""netbird-service-sync: reconcile NetBird reverse-proxy services from Ingresses.
"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
For each host on such an Ingress the NetBird management gets one reverse-proxy
@@ -30,11 +30,11 @@ import urllib.error
import urllib.parse
import urllib.request
LOG = logging.getLogger("netbird-service-sync")
LOG = logging.getLogger("netbird-proxy-services")
NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-service-sync")
BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
@@ -156,7 +156,7 @@ def ensure_network(nb):
if n["name"] == NETWORK_NAME:
return n["id"]
n = nb.post("networks", {"name": NETWORK_NAME,
"description": "managed by netbird-service-sync; public ingress via TLS passthrough"})
"description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
return n["id"]
@@ -166,7 +166,7 @@ def ensure_resource(nb, net):
if r.get("address") == SERVICE_CIDR:
return r["id"]
r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
"description": "managed by netbird-service-sync",
"description": "managed by netbird-proxy-services",
"enabled": True, "groups": []})
LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
return r["id"]
-87
View File
@@ -1,87 +0,0 @@
# Paperclip server image (REN-401)
Upstream `paperclipai/paperclip` at the commit in `upstream.env`, plus every
`*.patch` in this directory. Built by `.woodpecker/build-paperclip.yml`,
pushed as `gitea.renner.uno/konairius/paperclip:build.<unix-ts>`, rolled out
by Flux image automation (`kubernetes/infrastructure/image-automation/paperclip.yaml`).
## Why
k8s sandbox runs fail: the run path never sets the lease `remoteCwd`, so file
sync into the pod fails (paperclipai/paperclip#13864). The fix is
paperclipai/paperclip#13733, open and not merged.
`pr-13733-lease-remote-cwd.patch` is that PR's diff (head `5f320bf6`).
#13733 covers only the run path. Adapter environment tests (binding an agent
to a k8s environment) sync managed config before any workspace is realized,
so the lease still has no `remoteCwd` and the Claude test fails with
`claude_managed_config_dir_failed` (REN-409). The fix is
paperclipai/paperclip#14769, also open: the plugin sets `remoteCwd:
/workspace` when it acquires or resumes a lease.
`pr-14769-probe-remote-cwd.patch` is that PR's diff (head `10dc7371`).
Claude auto memory is lost after every run (REN-541): it lives under
`CLAUDE_CONFIG_DIR`, which Paperclip makes fresh per run and deletes.
`ren-541-claude-auto-memory.patch` makes the ACPX engine write
`autoMemoryDirectory: <AGENT_HOME>/memory` into the per-run
`.claude/settings.local.json`. `AGENT_HOME` is restored before and saved after
every run, local and in k8s pods. The value is not part of the session
fingerprint, so resume is unchanged. No upstream PR yet; rename the patch to
`pr-<n>-…` once there is one.
Claude sessions never resume with a managed AI connection (REN-563): the
server gives every run a fresh temp home (`HOME`, `XDG_*`,
`CLAUDE_CONFIG_DIR`, …) and deletes it after the run.
`ren-563-acpx-claude-session-resume.patch` fixes both halves for local runs:
the ACPX engine hashes a fixed placeholder for those paths in the session
fingerprint (`adapterEnvHash`), as the server's own fingerprint already does,
and links `CLAUDE_CONFIG_DIR/projects` (the Claude transcripts) to
`<stateDir>/claude-projects`, which persists. Both must ship together: a
stable fingerprint without the transcript makes the resumed turn fail with
"Resource not found" (homelab#1111, reverted in #1113). Remote runs are
unchanged. No upstream PR yet.
A cancelled or failed k8s pod run blocks its card for good (REN-841). The
owner's next comment waits on `remote_cleanup`, because the server continues
only when every remote lease carries a provider termination receipt
(`explicit-native-continuation.ts`), and the Kubernetes plugin's
`onEnvironmentReleaseLease` / `onEnvironmentDestroyLease` returned `void`.
`ren-841-k8s-termination-receipt.patch` makes both return
`{ providerLeaseId, state: "destroyed" }`, but only after the Sandbox CR or
Job (and, on destroy, the pod) reads 404, polling for up to 20 s, below the
server's 30 s plugin RPC timeout. If that check times out or fails, the hook
returns no receipt, as before. No upstream PR yet.
## Files
- `upstream.env` — pinned commit and the version string stamped into the
image. Bump both together.
- `prepare.sh` — fetches the commit, applies the patches, writes
`Dockerfile.kaniko` (upstream's `COPY --parents` line split into plain
`COPY` lines; kaniko does not support `--parents`), and adds the build of
the Kubernetes sandbox provider plugin to the `build` stage (REN-405):
upstream's `production` target ships only its source, so the plugin fails
with "no longer exposes a Paperclip manifest". The build fails if
`dist/manifest.js` is missing.
- `*.patch` — applied in name order with `git apply`.
## Moving to a newer upstream commit
Set `PAPERCLIP_COMMIT` to the new master commit and `PAPERCLIP_VERSION` to
`git describe --tags --match 'v*' --long <commit>`, then run
`sh prepare.sh /tmp/src`: it fails if a patch no longer applies.
## Exit: when every patch here is merged upstream
Renovate cannot watch an upstream PR, so this is a manual step. Once
upstream nightly contains all of them:
1. `kubernetes/infrastructure/image-automation/paperclip.yaml`: back to
`ghcr.io/paperclipai/paperclip` with `filterTags.pattern: '^nightly$'`
(the version before REN-401 in git history).
2. `renovate.json`: swap `gitea.renner.uno/konairius/paperclip` back to
`ghcr.io/paperclipai/paperclip` in the image-automation exclusion list.
3. Delete this directory and `.woodpecker/build-paperclip.yml`.
Image automation then rewrites `kubernetes/apps/paperclip/paperclip.yaml`
to the nightly digest by itself.
@@ -1,93 +0,0 @@
diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts
index f3d1fc659..812cae357 100644
--- a/server/src/__tests__/environment-run-orchestrator.test.ts
+++ b/server/src/__tests__/environment-run-orchestrator.test.ts
@@ -430,6 +430,51 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
});
+ it("persists the realized cwd as remoteCwd when realization returns no workspaceRealization payload", async () => {
+ const runtime = makeMockRuntime({
+ realizeWorkspace: vi.fn().mockResolvedValue({
+ cwd: "/workspace",
+ metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+ }),
+ } as Partial<EnvironmentRuntimeService>);
+ mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+ const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+ await orchestrator.realizeForRun(
+ makeRealizeInput({ environment: makeEnvironment("sandbox") }),
+ );
+
+ expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
+ "lease-1",
+ expect.objectContaining({ remoteCwd: "/workspace" }),
+ );
+ // Nothing to record about the realization shape, so the execution workspace
+ // row is left untouched.
+ expect(mockUpdateExecutionWorkspace).not.toHaveBeenCalled();
+ });
+
+ it("does not rewrite lease metadata when the realized cwd already matches the lease", async () => {
+ const runtime = makeMockRuntime({
+ realizeWorkspace: vi.fn().mockResolvedValue({
+ cwd: "/workspace",
+ metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+ }),
+ } as Partial<EnvironmentRuntimeService>);
+ mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+ const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+ await orchestrator.realizeForRun(
+ makeRealizeInput({
+ environment: makeEnvironment("sandbox"),
+ lease: makeLease({ metadata: { remoteCwd: "/workspace" } }),
+ }),
+ );
+
+ expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+ });
+
it("runs a remote provision command after workspace realization when configured", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts
index 97cca667e..e76f64913 100644
--- a/server/src/services/environment-run-orchestrator.ts
+++ b/server/src/services/environment-run-orchestrator.ts
@@ -485,16 +485,29 @@ export function environmentRunOrchestrator(
}
// Step 3: Persist realization metadata on lease and execution workspace
- if (Object.keys(workspaceRealization).length > 0) {
+ const hasWorkspaceRealization = Object.keys(workspaceRealization).length > 0;
+ // A driver that realizes the workspace reports the directory it prepared
+ // through the realization `cwd`. Persist it on the lease as `remoteCwd`, so
+ // every later step that receives only the lease — native file sync, a
+ // resumed lease, a second run against the same lease — confines itself to
+ // the same root. Without it a driver whose realization returns no
+ // `workspaceRealization` payload leaves the lease without a root at all,
+ // and the sync step fails even though realization succeeded.
+ const currentRemoteCwd =
+ typeof lease.metadata?.remoteCwd === "string" ? lease.metadata.remoteCwd.trim() : "";
+ const realizedRemoteCwd =
+ realizedWorkspaceCwd && realizedWorkspaceCwd !== currentRemoteCwd ? realizedWorkspaceCwd : null;
+ if (hasWorkspaceRealization || realizedRemoteCwd) {
const nextLeaseMetadata = {
...(lease.metadata ?? {}),
- workspaceRealization,
+ ...(hasWorkspaceRealization ? { workspaceRealization } : {}),
+ ...(realizedRemoteCwd ? { remoteCwd: realizedRemoteCwd } : {}),
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
if (updatedLease) {
lease = updatedLease;
}
- if (persistedExecutionWorkspace) {
+ if (hasWorkspaceRealization && persistedExecutionWorkspace) {
const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
metadata: {
...(persistedExecutionWorkspace.metadata ?? {}),
@@ -1,86 +0,0 @@
diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md
index a51c6ecee60e..72618cb11296 100644
--- a/packages/plugins/sandbox-providers/kubernetes/README.md
+++ b/packages/plugins/sandbox-providers/kubernetes/README.md
@@ -140,6 +140,8 @@ Every agent pod is:
Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs).
+The lease starts with `/workspace` as the native file-sync confinement root. Adapter environment tests can upload managed config before the server realizes a workspace. When resuming a lease, the provider keeps a previously recorded `remoteCwd` so a custom realized workspace is not replaced with the default.
+
The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Job, so a single `kubectl delete job …` cascades cleanly to the Pod and Secret.
## Optional Kata-FC microVM isolation
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
index 8ae47de8ecac..98016385da05 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
@@ -454,6 +454,9 @@ const plugin = definePlugin({
secretName,
phase: "Pending",
backend: config.backend,
+ // Adapter environment probes sync managed assets before workspace
+ // realization. Give native sync the pod's mounted confinement root now.
+ remoteCwd: "/workspace",
scopedNetworkPolicyName,
scopedNetworkEgress,
// Native file sync streams over a pod exec; only the sandbox-cr backend
@@ -529,6 +532,10 @@ const plugin = definePlugin({
secretName,
phase: check.phase,
backend: leaseBackend,
+ remoteCwd:
+ typeof params.leaseMetadata?.remoteCwd === "string" && params.leaseMetadata.remoteCwd.trim()
+ ? params.leaseMetadata.remoteCwd.trim()
+ : "/workspace",
scopedNetworkPolicyName:
typeof params.leaseMetadata?.scopedNetworkPolicyName === "string"
? params.leaseMetadata.scopedNetworkPolicyName
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/types.ts b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
index 5626daa93846..ef3d0d889a4c 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
@@ -90,6 +90,8 @@ export interface KubernetesLeaseMetadata {
phase: "Pending" | "Running" | "Succeeded" | "Failed";
/** Which backend provisioned this lease. */
backend: "sandbox-cr" | "job";
+ /** Native sync confinement root; present before workspace realization for probes. */
+ remoteCwd: string;
scopedNetworkPolicyName: string | null;
scopedNetworkEgress: {
allowFqdns: string[];
diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
index 7ad0ca4807e5..4c23410d4d15 100644
--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
@@ -80,6 +80,7 @@ describe("onEnvironmentResumeLease", () => {
secretName: "pc-abc-env",
phase: "Running",
backend: "sandbox-cr",
+ remoteCwd: "/workspace",
resumedLease: true,
// sandbox-cr has a pod-exec channel, so native file sync stays enabled.
nativeFileSyncUnsupported: false,
@@ -87,6 +88,22 @@ describe("onEnvironmentResumeLease", () => {
);
});
+ it("keeps a custom realized cwd when resuming a sandbox lease", async () => {
+ h.clients = {
+ custom: { getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")) },
+ core: { readNamespacedPod: vi.fn().mockResolvedValue({ status: { phase: "Running" } }) },
+ };
+ const lease = await plugin.definition.onEnvironmentResumeLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata({ remoteCwd: "/workspace/custom" }),
+ });
+ expect(lease.metadata?.remoteCwd).toBe("/workspace/custom");
+ });
+
it("flags a resumed job-backend lease as native-sync-unsupported so the server keeps the base64 fallback", async () => {
h.clients = {
batch: {
-59
View File
@@ -1,59 +0,0 @@
#!/bin/sh
# Fetch upstream Paperclip at the pinned commit, apply our patches and make
# the Dockerfile buildable with kaniko. Usage: prepare.sh <target-dir>
set -eu
HERE=$(cd "$(dirname "$0")" && pwd)
. "$HERE/upstream.env"
OUT=$1
rm -rf "$OUT"
git init -q "$OUT"
cd "$OUT"
git fetch -q --depth 1 https://github.com/paperclipai/paperclip.git "$PAPERCLIP_COMMIT"
git -c advice.detachedHead=false checkout -q FETCH_HEAD
test "$(git rev-parse HEAD)" = "$PAPERCLIP_COMMIT"
for p in "$HERE"/*.patch; do
echo "applying $(basename "$p")"
git apply "$p"
done
# kaniko does not know `COPY --parents` (BuildKit only). Upstream uses it for
# one line: every sandbox provider's package.json. Replace it with one plain
# COPY per provider directory, which copies the same files to the same place.
grep -c -- '^COPY --parents packages/plugins/sandbox-providers/\./\*/package.json packages/plugins/sandbox-providers/$' Dockerfile | grep -qx 1
awk '
/^COPY --parents packages\/plugins\/sandbox-providers\// {
while (("ls -d packages/plugins/sandbox-providers/*/package.json" | getline f) > 0) {
d = f; sub(/package.json$/, "", d)
print "COPY " f " " d
}
next
}
{ print }
' Dockerfile > Dockerfile.kaniko
if grep -q -- '--parents' Dockerfile.kaniko; then
echo "Dockerfile.kaniko still uses COPY --parents" >&2
exit 1
fi
# Build the Kubernetes sandbox provider into the image (REN-405). Upstream's
# `production` target ships only its source; the plugin loader needs
# dist/manifest.js. Same commands as upstream's `cloud-plugins` stage, run in
# the `build` stage right after the server build, so production's
# `COPY --from=build /app /app` carries dist/ and node_modules along. No extra
# stage: kaniko would snapshot the whole build stage again for it.
export ANCHOR='RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)'
grep -c -F -x -- "$ANCHOR" Dockerfile.kaniko | grep -qx 1
K=packages/plugins/sandbox-providers/kubernetes
awk -v k="$K" '
{ print }
$0 == ENVIRON["ANCHOR"] {
print "RUN pnpm -C " k " install --ignore-workspace --no-lockfile \\"
print " && pnpm -C " k " build \\"
print " && test -f " k "/dist/manifest.js"
}
' Dockerfile.kaniko > Dockerfile.kaniko.new
mv Dockerfile.kaniko.new Dockerfile.kaniko
grep -q -- "test -f $K/dist/manifest.js" Dockerfile.kaniko
rm -rf .git
@@ -1,118 +0,0 @@
diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts
index a26f980..112e2eb 100644
--- a/packages/adapter-utils/src/acpx-engine/execute.test.ts
+++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts
@@ -2544,6 +2544,37 @@ describe("shared ACPX engine runtime behavior", () => {
expect(note).toBeTruthy();
});
+ it("points Claude auto memory at the run's AGENT_HOME without changing the session fingerprint", async () => {
+ const root = await makeTempRoot();
+ const stateDir = path.join(root, "state");
+ const cwd = path.join(root, "worktree");
+ await fs.mkdir(cwd, { recursive: true });
+ const stableHome = path.join(root, "agent-home");
+ const runAgentHome = (runId: string) => path.join(root, "file-sync", "runs", runId, "live");
+ const readSettings = async () =>
+ JSON.parse(await fs.readFile(path.join(cwd, ".claude", "settings.local.json"), "utf8")) as {
+ autoMemoryDirectory?: string;
+ permissions?: { additionalDirectories?: string[] };
+ };
+
+ const first = await runExecutor(
+ { agent: "claude", stateDir, cwd },
+ { context: { paperclipWorkspace: { cwd, agentHome: runAgentHome("a"), agentHomeForPermissions: stableHome } } },
+ );
+ expect((await readSettings()).autoMemoryDirectory).toBe(path.join(runAgentHome("a"), "memory"));
+
+ const second = await runExecutor(
+ { agent: "claude", stateDir, cwd },
+ { context: { paperclipWorkspace: { cwd, agentHome: runAgentHome("b"), agentHomeForPermissions: stableHome } } },
+ );
+ const written = await readSettings();
+ expect(written.autoMemoryDirectory).toBe(path.join(runAgentHome("b"), "memory"));
+ expect(written.permissions?.additionalDirectories).toContain(stableHome);
+ expect(written.permissions?.additionalDirectories).not.toContain(runAgentHome("b"));
+ expect(second.result.sessionParams?.configFingerprint).toBeTypeOf("string");
+ expect(second.result.sessionParams?.configFingerprint).toBe(first.result.sessionParams?.configFingerprint);
+ });
+
it("merges Paperclip allowlist into an existing .claude/settings.local.json without losing user entries", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts
index b1dc569..e5681b8 100644
--- a/packages/adapter-utils/src/acpx-engine/execute.ts
+++ b/packages/adapter-utils/src/acpx-engine/execute.ts
@@ -1558,6 +1558,7 @@ interface PaperclipClaudeSettingsResult {
additionalDirectories: string[];
defaultMode: string;
overrodeDontAsk: boolean;
+ autoMemoryDirectory: string | null;
}
function uniqueSorted(values: Array<string | null | undefined>): string[] {
@@ -1571,10 +1572,19 @@ function uniqueSorted(values: Array<string | null | undefined>): string[] {
// denies every non-allowlisted tool and never reaches `canUseTool`), and we
// widen the SDK's Read sandbox to include the Paperclip state dirs the agent
// needs to talk to its own control plane.
+//
+// It also points Claude's auto memory at `<AGENT_HOME>/memory`. The default
+// location lives under CLAUDE_CONFIG_DIR, which is a per-run temp home, so
+// memories were lost after every run. AGENT_HOME is restored before and saved
+// after each run, locally and in remote sandboxes. Claude ignores
+// `autoMemoryDirectory` only in checked-in project settings, not local ones.
async function writePaperclipClaudeSettings(input: {
cwd: string;
stateDir: string;
agentHome: string;
+ // The run's actual AGENT_HOME (the in-sandbox path for remote runs). It may
+ // differ from `agentHome`, which stays stable for session identity.
+ runAgentHome: string;
companyId: string;
}): Promise<PaperclipClaudeSettingsResult> {
const filePath = path.join(input.cwd, ".claude", "settings.local.json");
@@ -1630,7 +1640,14 @@ async function writePaperclipClaudeSettings(input: {
additionalDirectories: mergedAdditionalDirectories,
defaultMode,
};
- const next: Record<string, unknown> = { ...existing, permissions: nextPermissions };
+ const autoMemoryDirectory = input.runAgentHome
+ ? path.posix.join(input.runAgentHome, "memory")
+ : null;
+ const next: Record<string, unknown> = {
+ ...existing,
+ permissions: nextPermissions,
+ ...(autoMemoryDirectory ? { autoMemoryDirectory } : {}),
+ };
await writeFileAtomically({
target: filePath,
contents: `${JSON.stringify(next, null, 2)}\n`,
@@ -1642,6 +1659,7 @@ async function writePaperclipClaudeSettings(input: {
additionalDirectories: mergedAdditionalDirectories,
defaultMode,
overrodeDontAsk,
+ autoMemoryDirectory,
};
}
@@ -2064,12 +2082,18 @@ async function buildRuntime(input: {
// by the company root already granted in these settings; the process env
// still receives this run's actual AGENT_HOME.
agentHome: asString(workspaceContext.agentHomeForPermissions, agentHome),
+ // Auto memory follows the actual AGENT_HOME so it is saved with the run.
+ // It is not part of the session fingerprint: a moved memory directory
+ // must not invalidate a resumable conversation.
+ runAgentHome: agentHome,
companyId: agent.companyId,
});
skillCommandNotes.push(
`Wrote Paperclip-managed Claude settings to ${paperclipClaudeSettings.filePath} (defaultMode=${paperclipClaudeSettings.defaultMode}${
paperclipClaudeSettings.overrodeDontAsk ? "; overrode user dontAsk" : ""
- }, +${paperclipClaudeSettings.additionalDirectories.length} read root(s), +${paperclipClaudeSettings.allow.length} allow rule(s)).`,
+ }, +${paperclipClaudeSettings.additionalDirectories.length} read root(s), +${paperclipClaudeSettings.allow.length} allow rule(s)${
+ paperclipClaudeSettings.autoMemoryDirectory ? `, autoMemoryDirectory=${paperclipClaudeSettings.autoMemoryDirectory}` : ""
+ }).`,
);
} else if (acpxAgent === "codex") {
// Step 2 — codex-home.seed: the codex managed-home + skills preparation.
@@ -1,128 +0,0 @@
diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts
index 112e2eb..004baf0 100644
--- a/packages/adapter-utils/src/acpx-engine/execute.test.ts
+++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts
@@ -1857,6 +1857,45 @@ describe("shared ACPX engine runtime behavior", () => {
expect(changed.result.sessionParams?.configFingerprint).not.toBe(first.result.sessionParams?.configFingerprint);
});
+ it("keeps the per-run managed AI home out of session identity and keeps Claude transcripts", async () => {
+ const root = await makeTempRoot();
+ const stateDir = path.join(root, "state");
+ const config = { agentCommand: "node ./fake-acp.js", stateDir };
+ async function withManagedHome(home: string, claudeConfigDir = path.join(home, "provider")) {
+ return runExecutor({
+ ...config,
+ managedAiConnection: { identity: "grant-1:shared:abc" },
+ env: {
+ HOME: home,
+ XDG_CONFIG_HOME: path.join(home, "config"),
+ XDG_DATA_HOME: path.join(home, "data"),
+ CODEX_HOME: path.join(home, "provider"),
+ GROK_HOME: path.join(home, "provider"),
+ CLAUDE_CONFIG_DIR: claudeConfigDir,
+ },
+ }, { context: { taskId: "issue-1" } });
+ }
+ const fp = (run: Awaited<ReturnType<typeof runExecutor>>) => run.result.sessionParams?.configFingerprint;
+ const first = await withManagedHome(path.join(root, "paperclip-ai-a"));
+ const second = await withManagedHome(path.join(root, "paperclip-ai-b"));
+ const overridden = await withManagedHome(path.join(root, "paperclip-ai-c"), path.join(root, "custom-claude"));
+ expect(fp(first)).toBeTypeOf("string");
+ expect(fp(second)).toBe(fp(first));
+ expect(fp(overridden)).not.toBe(fp(first));
+
+ // The transcript dir of each managed home points at one dir in the state
+ // dir, so a resumed session finds its transcript after the home is deleted.
+ const transcripts = path.join(stateDir, "claude-projects");
+ expect(await fs.readlink(path.join(root, "paperclip-ai-a", "provider", "projects"))).toBe(transcripts);
+ expect(await fs.readlink(path.join(root, "paperclip-ai-b", "provider", "projects"))).toBe(transcripts);
+ await expect(fs.lstat(path.join(root, "custom-claude", "projects"))).rejects.toThrow();
+
+ // Without a managed AI connection the same paths are user config and still count.
+ const plainA = await runExecutor({ ...config, env: { HOME: path.join(root, "home-a") } }, { context: { taskId: "issue-1" } });
+ const plainB = await runExecutor({ ...config, env: { HOME: path.join(root, "home-b") } }, { context: { taskId: "issue-1" } });
+ expect(fp(plainB)).not.toBe(fp(plainA));
+ });
+
it("busts the session fingerprint when a stable configured PAPERCLIP_* value rotates", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts
index e5681b8..0fd486c 100644
--- a/packages/adapter-utils/src/acpx-engine/execute.ts
+++ b/packages/adapter-utils/src/acpx-engine/execute.ts
@@ -1663,6 +1663,26 @@ async function writePaperclipClaudeSettings(input: {
};
}
+// Claude keeps conversation transcripts in CLAUDE_CONFIG_DIR/projects. A
+// managed AI home is a per-run temp dir that the server deletes after the run,
+// so the next run could not resume ("Resource not found"). Point `projects` at
+// a directory in the agent's state dir, like ~/.claude/projects on a normal
+// install. The server's recursive cleanup removes the link, not its target.
+async function linkManagedClaudeTranscripts(input: {
+ claudeConfigDir: string;
+ stateDir: string;
+}): Promise<string> {
+ const target = path.join(input.stateDir, "claude-projects");
+ await fs.mkdir(input.claudeConfigDir, { recursive: true, mode: 0o700 });
+ await fs.mkdir(target, { recursive: true, mode: 0o700 });
+ const link = path.join(input.claudeConfigDir, "projects");
+ const current = await fs.readlink(link).catch(() => null);
+ if (current === target) return target;
+ await fs.rm(link, { recursive: true, force: true });
+ await fs.symlink(target, link, "dir");
+ return target;
+}
+
// Cross the CLI's staging seam for a runner-backed remote sandbox: ship the
// workspace (and, in PR 2, the per-adapter managed-home `assets`) into the
// sandbox and obtain the in-sandbox `workspaceRemoteDir` plus the non-null
@@ -2027,6 +2047,30 @@ async function buildRuntime(input: {
// are absent from tempKeysApplied and keep their compatibility protection.
if (!scratchKeys.has(key) || value !== scratch.dir) resolvedAdapterEnv[key] = value;
}
+ // A managed AI connection gets a fresh server-made temp home on every run
+ // (HOME, XDG_*, and the provider home under it). Hash a fixed placeholder for
+ // those paths so the session can resume; the server's own fingerprint does
+ // the same (managedAiSessionFingerprintConfig). Values that differ from the
+ // managed layout are real overrides and stay in the hash. A Claude session
+ // can only resume if its transcript survives the run, so this pairs with
+ // linkManagedClaudeTranscripts below. Remote runs keep their transcript in
+ // the sandbox and are left as they are.
+ const managedHome = config.managedAiConnection && !executionTargetIsRemote ? resolvedAdapterEnv.HOME : undefined;
+ if (managedHome) {
+ const managedLayout: Record<string, string> = {
+ HOME: "",
+ XDG_CONFIG_HOME: "config",
+ XDG_DATA_HOME: "data",
+ CODEX_HOME: "provider",
+ GROK_HOME: "provider",
+ CLAUDE_CONFIG_DIR: "provider",
+ };
+ for (const [key, sub] of Object.entries(managedLayout)) {
+ if (resolvedAdapterEnv[key] === (sub ? path.join(managedHome, sub) : managedHome)) {
+ resolvedAdapterEnv[key] = sub ? `<managed-ai-home>/${sub}` : "<managed-ai-home>";
+ }
+ }
+ }
if (authToken) env.PAPERCLIP_API_KEY = authToken;
// For the claude agent, set model via ANTHROPIC_MODEL at startup rather than
// via session/set_config_option — the ACP server's set_config_option handler
@@ -2088,6 +2132,15 @@ async function buildRuntime(input: {
runAgentHome: agentHome,
companyId: agent.companyId,
});
+ // Same condition as the fingerprint placeholder above: a resumable session
+ // must find its transcript in the next run's managed home.
+ if (resolvedAdapterEnv.CLAUDE_CONFIG_DIR === "<managed-ai-home>/provider" && env.CLAUDE_CONFIG_DIR) {
+ const transcriptsDir = await linkManagedClaudeTranscripts({
+ claudeConfigDir: env.CLAUDE_CONFIG_DIR,
+ stateDir,
+ });
+ skillCommandNotes.push(`Linked Claude transcripts to ${transcriptsDir}.`);
+ }
skillCommandNotes.push(
`Wrote Paperclip-managed Claude settings to ${paperclipClaudeSettings.filePath} (defaultMode=${paperclipClaudeSettings.defaultMode}${
paperclipClaudeSettings.overrodeDontAsk ? "; overrode user dontAsk" : ""
@@ -1,384 +0,0 @@
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts b/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts
index 87e18ae..2224e60 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts
@@ -21,6 +21,7 @@ import { deleteJob, findPodForJob, getJobStatus } from "./job-orchestrator.js";
import {
deleteSandboxCr,
findPodForSandbox,
+ getSandboxCrStatus,
waitForSandboxReady,
} from "./sandbox-cr-orchestrator.js";
@@ -191,3 +192,60 @@ export async function destroyLeaseResources(
);
}
}
+
+export interface WorkloadGoneInput {
+ namespace: string;
+ /** Workload resource name (Sandbox CR name or Job name) == providerLeaseId. */
+ name: string;
+ backend: "sandbox-cr" | "job";
+ podName?: string | null;
+ /** Stays below the server's 30 s default plugin RPC timeout. */
+ timeoutMs?: number;
+ pollMs?: number;
+}
+
+/**
+ * Poll until the workload (and the named pod, if any) returns 404. Deletes use
+ * Foreground propagation, so the workload disappears only after its pod is
+ * gone. True is the evidence behind a `destroyed` termination receipt; false
+ * (still present at the deadline, or a read failed) means the caller must not
+ * claim one. Never throws: the delete itself already succeeded.
+ */
+export async function waitForWorkloadGone(
+ clients: KubeClients,
+ input: WorkloadGoneInput,
+): Promise<boolean> {
+ const deadline = Date.now() + (input.timeoutMs ?? 20_000);
+ const pollMs = input.pollMs ?? 500;
+ const gone = async (read: () => Promise<unknown>): Promise<boolean> => {
+ try {
+ await read();
+ return false;
+ } catch (err) {
+ if (isKubeNotFoundError(err)) return true;
+ throw err;
+ }
+ };
+ try {
+ return await pollUntilGone();
+ } catch {
+ return false;
+ }
+
+ async function pollUntilGone(): Promise<boolean> {
+ for (;;) {
+ const workloadGone = await gone(() =>
+ input.backend === "sandbox-cr"
+ ? getSandboxCrStatus(clients, input.namespace, input.name)
+ : getJobStatus(clients, input.namespace, input.name),
+ );
+ const podName = input.podName;
+ const podGone = !podName || await gone(() =>
+ clients.core.readNamespacedPod({ namespace: input.namespace, name: podName }),
+ );
+ if (workloadGone && podGone) return true;
+ if (Date.now() >= deadline) return false;
+ await new Promise((resolve) => setTimeout(resolve, pollMs));
+ }
+ }
+}
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
index 9801638..bf5ce0a 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
@@ -15,6 +15,7 @@ import type {
PluginEnvironmentSyncInParams,
PluginEnvironmentSyncOutParams,
PluginEnvironmentSyncResult,
+ PluginEnvironmentTerminationReceipt,
PluginEnvironmentValidateConfigParams,
PluginEnvironmentValidationResult,
} from "@paperclipai/plugin-sdk";
@@ -38,7 +39,11 @@ import {
} from "./sandbox-cr-orchestrator.js";
import { execInPod, execInPodStreaming, wrapCommandWithEnv } from "./pod-exec.js";
import { performSyncIn, performSyncOut, type PodStreamExec } from "./file-sync.js";
-import { checkLeaseResumable, destroyLeaseResources } from "./lease-lifecycle.js";
+import {
+ checkLeaseResumable,
+ destroyLeaseResources,
+ waitForWorkloadGone,
+} from "./lease-lifecycle.js";
import {
appendNetworkEgressDenyHint,
createScopedNetworkEgressPolicyOrReleaseWorkload,
@@ -578,7 +583,7 @@ const plugin = definePlugin({
async onEnvironmentReleaseLease(
params: PluginEnvironmentReleaseLeaseParams,
- ): Promise<void> {
+ ): Promise<PluginEnvironmentTerminationReceipt | void> {
if (!params.providerLeaseId) return;
const config = kubernetesProviderConfigSchema.parse(params.config);
const namespace =
@@ -613,11 +618,22 @@ const plugin = definePlugin({
?? (err as { code?: number; statusCode?: number }).statusCode;
if (code !== 404) throw err;
}
+
+ // The server grants continuation after a stopped run only with a
+ // termination receipt; without one a cancelled run's queued comment
+ // waits on `remote_cleanup` forever (REN-841). Claim `destroyed` only
+ // once the workload is confirmed gone.
+ const gone = await waitForWorkloadGone(clients, {
+ namespace,
+ name: params.providerLeaseId,
+ backend: leaseBackend,
+ });
+ if (gone) return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
},
async onEnvironmentDestroyLease(
params: PluginEnvironmentDestroyLeaseParams,
- ): Promise<void> {
+ ): Promise<PluginEnvironmentTerminationReceipt | void> {
if (!params.providerLeaseId) return;
const config = kubernetesProviderConfigSchema.parse(params.config);
const namespace =
@@ -658,6 +674,13 @@ const plugin = definePlugin({
podName,
secretName,
});
+ const gone = await waitForWorkloadGone(clients, {
+ namespace,
+ name: params.providerLeaseId,
+ backend: leaseBackend,
+ podName,
+ });
+ if (gone) return { providerLeaseId: params.providerLeaseId, state: "destroyed" };
},
async onEnvironmentExecute(
diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/lease-lifecycle.test.ts
index af0875f..d775799 100644
--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/lease-lifecycle.test.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/lease-lifecycle.test.ts
@@ -3,6 +3,7 @@ import {
checkLeaseResumable,
destroyLeaseResources,
isKubeNotFoundError,
+ waitForWorkloadGone,
} from "../../src/lease-lifecycle.js";
const SANDBOX_GROUP = "agents.x-k8s.io";
@@ -346,3 +347,64 @@ describe("destroyLeaseResources", () => {
).rejects.toThrow("forbidden");
});
});
+
+describe("waitForWorkloadGone", () => {
+ const input = { namespace: "ns", name: "pc-abc", backend: "sandbox-cr" as const, pollMs: 1 };
+
+ it("is true once the Sandbox CR and the pod return 404", async () => {
+ const getCr = vi.fn()
+ .mockResolvedValueOnce(readySandboxCr("pc-abc-pod"))
+ .mockRejectedValue(notFound());
+ const readPod = vi.fn().mockRejectedValue(notFound());
+ const clients = { custom: { getNamespacedCustomObject: getCr }, core: { readNamespacedPod: readPod } };
+
+ await expect(
+ waitForWorkloadGone(clients as never, { ...input, podName: "pc-abc-pod" }),
+ ).resolves.toBe(true);
+ expect(getCr).toHaveBeenCalledTimes(2);
+ expect(getCr).toHaveBeenCalledWith(expect.objectContaining({
+ group: SANDBOX_GROUP, version: SANDBOX_VERSION, plural: SANDBOX_PLURAL, namespace: "ns", name: "pc-abc",
+ }));
+ });
+
+ it("is false while the workload still exists at the deadline", async () => {
+ const clients = {
+ custom: { getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr()) },
+ };
+
+ await expect(
+ waitForWorkloadGone(clients as never, { ...input, timeoutMs: 5 }),
+ ).resolves.toBe(false);
+ });
+
+ it("is false while the pod still exists even if the workload is gone", async () => {
+ const clients = {
+ custom: { getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()) },
+ core: { readNamespacedPod: vi.fn().mockResolvedValue({ status: { phase: "Running" } }) },
+ };
+
+ await expect(
+ waitForWorkloadGone(clients as never, { ...input, podName: "pc-abc-pod", timeoutMs: 5 }),
+ ).resolves.toBe(false);
+ });
+
+ it("checks the Job for job-backend leases", async () => {
+ const readJob = vi.fn().mockRejectedValue(notFound());
+ const clients = { batch: { readNamespacedJobStatus: readJob } };
+
+ await expect(
+ waitForWorkloadGone(clients as never, { ...input, backend: "job" }),
+ ).resolves.toBe(true);
+ expect(readJob).toHaveBeenCalledWith({ namespace: "ns", name: "pc-abc" });
+ });
+
+ it("is false (never throws) when a read fails with another error", async () => {
+ const clients = {
+ custom: {
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(Object.assign(new Error("x"), { code: 403 })),
+ },
+ };
+
+ await expect(waitForWorkloadGone(clients as never, input)).resolves.toBe(false);
+ });
+});
diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
index 4c23410..f4fd073 100644
--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
@@ -184,12 +184,19 @@ describe("onEnvironmentDestroyLease", () => {
const deletePod = vi.fn().mockResolvedValue({});
const deleteSecret = vi.fn().mockResolvedValue({});
h.clients = {
- custom: { deleteNamespacedCustomObject: deleteCr },
- core: { deleteNamespacedPod: deletePod, deleteNamespacedSecret: deleteSecret },
+ custom: {
+ deleteNamespacedCustomObject: deleteCr,
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ },
+ core: {
+ deleteNamespacedPod: deletePod,
+ deleteNamespacedSecret: deleteSecret,
+ readNamespacedPod: vi.fn().mockRejectedValue(notFound()),
+ },
batch: { deleteNamespacedJob: vi.fn() },
};
- await plugin.definition.onEnvironmentDestroyLease!({
+ const receipt = await plugin.definition.onEnvironmentDestroyLease!({
driverKey: "kubernetes",
companyId: "acme",
environmentId: "env-1",
@@ -209,14 +216,47 @@ describe("onEnvironmentDestroyLease", () => {
namespace: "paperclip-acme",
name: "pc-abc-env",
});
+ expect(receipt).toEqual({ providerLeaseId: "pc-abc", state: "destroyed" });
});
it("is idempotent: resolves cleanly when every resource is already gone (404)", async () => {
h.clients = {
- custom: { deleteNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()) },
+ custom: {
+ deleteNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ },
core: {
deleteNamespacedPod: vi.fn().mockRejectedValue(notFound()),
deleteNamespacedSecret: vi.fn().mockRejectedValue(notFound()),
+ readNamespacedPod: vi.fn().mockRejectedValue(notFound()),
+ },
+ batch: { deleteNamespacedJob: vi.fn() },
+ };
+
+ await expect(
+ plugin.definition.onEnvironmentDestroyLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata(),
+ }),
+ ).resolves.toEqual({ providerLeaseId: "pc-abc", state: "destroyed" });
+ });
+
+ it("returns no receipt when the gone check cannot read the cluster", async () => {
+ h.clients = {
+ custom: {
+ deleteNamespacedCustomObject: vi.fn().mockResolvedValue({}),
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(
+ Object.assign(new Error("forbidden"), { code: 403 }),
+ ),
+ },
+ core: {
+ deleteNamespacedPod: vi.fn().mockResolvedValue({}),
+ deleteNamespacedSecret: vi.fn().mockResolvedValue({}),
+ readNamespacedPod: vi.fn().mockRejectedValue(notFound()),
},
batch: { deleteNamespacedJob: vi.fn() },
};
@@ -261,8 +301,12 @@ describe("onEnvironmentDestroyLease", () => {
core: {
deleteNamespacedPod: vi.fn().mockResolvedValue({}),
deleteNamespacedSecret: vi.fn().mockResolvedValue({}),
+ readNamespacedPod: vi.fn().mockRejectedValue(notFound()),
+ },
+ batch: {
+ deleteNamespacedJob: deleteJob,
+ readNamespacedJobStatus: vi.fn().mockRejectedValue(notFound()),
},
- batch: { deleteNamespacedJob: deleteJob },
};
await plugin.definition.onEnvironmentDestroyLease!({
@@ -280,3 +324,71 @@ describe("onEnvironmentDestroyLease", () => {
expect(deleteCr).not.toHaveBeenCalled();
});
});
+
+describe("onEnvironmentReleaseLease", () => {
+ it("returns a destroyed receipt once the Sandbox CR is gone (REN-841)", async () => {
+ const deleteCr = vi.fn().mockResolvedValue({});
+ h.clients = {
+ custom: {
+ deleteNamespacedCustomObject: deleteCr,
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ },
+ };
+
+ const receipt = await plugin.definition.onEnvironmentReleaseLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata(),
+ });
+
+ expect(deleteCr).toHaveBeenCalledWith(
+ expect.objectContaining({ namespace: "paperclip-acme", name: "pc-abc" }),
+ );
+ expect(receipt).toEqual({ providerLeaseId: "pc-abc", state: "destroyed" });
+ });
+
+ it("returns a destroyed receipt when the Sandbox CR was already deleted", async () => {
+ h.clients = {
+ custom: {
+ deleteNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()),
+ },
+ };
+
+ await expect(
+ plugin.definition.onEnvironmentReleaseLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata(),
+ }),
+ ).resolves.toEqual({ providerLeaseId: "pc-abc", state: "destroyed" });
+ });
+
+ it("returns no receipt when the gone check fails", async () => {
+ h.clients = {
+ custom: {
+ deleteNamespacedCustomObject: vi.fn().mockResolvedValue({}),
+ getNamespacedCustomObject: vi.fn().mockRejectedValue(
+ Object.assign(new Error("server error"), { code: 500 }),
+ ),
+ },
+ };
+
+ await expect(
+ plugin.definition.onEnvironmentReleaseLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata(),
+ }),
+ ).resolves.toBeUndefined();
+ });
+});
-7
View File
@@ -1,7 +0,0 @@
# Upstream paperclipai/paperclip commit this image is built from (REN-401).
# a027f76 is the master commit the nightly image ran when this was pinned, so
# the only change against that image is the patch next to this file.
# PAPERCLIP_VERSION is `git describe --tags --match 'v*' --long` of that
# commit, the value upstream stamps into its own images. Bump both together.
PAPERCLIP_COMMIT=a027f76a726e8a556674eded807c2de0e55f5cc0
PAPERCLIP_VERSION=v2026.916.0-219-ga027f76a7
+1 -1
View File
@@ -1,4 +1,4 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TOFU_VERSION=1.11.0
+3 -3
View File
@@ -1,4 +1,4 @@
FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Everything `bootstrap.sh vmtest` needs, so the rebuild test can run as a Job in
# the cluster instead of on a workstation. The workstation has 31G and the test
@@ -14,7 +14,7 @@ FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cab
# kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
# file at run time and passes it to the provisioner; this pin only decides which
# client talks to it.
ARG TALOS_VERSION=v1.13.10
ARG TALOS_VERSION=v1.13.9
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KOPIA_VERSION=0.23.1
@@ -49,7 +49,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -1,976 +0,0 @@
# Paperclip Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Deploy Paperclip (server-local agent runs, option B) into the homelab cluster behind an Authentik gate, with CNPG Postgres and OpenBao-sourced secrets.
**Architecture:** One Flux `Kustomization` (`paperclip`) applies `kubernetes/apps/paperclip/`: a single-replica Deployment on a ceph-block PVC, CNPG `pclp-pg` (PG17) with barman backups to Hetzner, and a Traefik Ingress that backends only the `paperclip-gate` Authentik outpost. Namespace, baseline CNP, ClusterSecretStore allowlist and image automation live in the existing `infrastructure` tree.
**Tech Stack:** Flux v2 (kustomize-controller, image-reflector v1.2.5), CNPG + barman-cloud plugin, External Secrets → OpenBao, Cilium CNP, Traefik, Authentik blueprints.
**Spec:** `docs/superpowers/specs/2026-09-25-paperclip-design.md`
## Global Constraints
- Worktree: `/home/krenner/source/homelab-worktrees/paperclip`, branch `feat/paperclip`. Never edit the main checkout.
- Commits as claude-bot: `git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit …`, message ends with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
- Push: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push …`. PR via `tea pr create --login claude-bot`. Never self-merge.
- One Bash command per call: no `&&`, `;`, `|` chaining, no `2>/dev/null`, no `python -c`.
- Namespace `paperclip`; hostname `paperclip.renner.uno`; LAN-only (no `dns.renner.uno/public` label).
- Image `ghcr.io/paperclipai/paperclip`, tag `nightly`, digest-pinned, `digestReflectionPolicy: Always`.
- Postgres: CNPG `pclp-pg`, `ghcr.io/cloudnative-pg/postgresql:17`, 2 instances, `ceph-block`.
- PVC `paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip`.
- OpenBao key `apps/paperclip` with fields `BETTER_AUTH_SECRET`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`, `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex), `CLAUDE_CODE_OAUTH_TOKEN`.
- Env: `PAPERCLIP_DEPLOYMENT_MODE=authenticated`, `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`, `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`, `HOST=0.0.0.0`, `PORT=3100`.
- Manifest comments: one terse factual line; rationale goes in commit messages/PR.
- No `kubectl apply`/`exec`/`rollout restart` against the live cluster; Flux applies after merge. Server-side `--dry-run=server` is allowed.
## Review Focus
1. **Pod replaced mid agent run** (image bump, secret rotation via Reloader, node drain) — `Recreate` kills in-flight runs; expected: Paperclip marks them failed/recoverable on restart, not wedged. Covered in Task 9 Step 6.
2. **Shared OAuth token expired or missing** — expected: agent runs fail with a visible auth error in Paperclip, server stays healthy. Covered in Task 9 Step 7.
3. **Direct access bypassing the gate** from another namespace pod — expected: connection refused/dropped by CNP. Covered in Task 9 Step 4.
4. **Unauthenticated LAN request** to `paperclip.renner.uno` — expected: redirect to Authentik, never Paperclip's UI. Covered in Task 9 Step 3.
5. **Nightly image with a breaking DB migration** — expected: a base backup exists before the first bump lands. Covered in Task 9 Step 5.
---
### Task 1: Upstream facts that decide two manifest values
No repo changes; produces two decisions consumed by Task 6.
**Interfaces:**
- Produces: `SIGNUP_DISABLED` (`"true"` or `"false"`), `CONFIG_PARTIAL_OK` (yes/no), `ENTRYPOINT_NEEDS_ROOT` (yes/no).
- [ ] **Step 1: Sign-up vs instance claim.** Read upstream `server/src/auth/better-auth.ts`, `server/src/routes/auth.ts`, and the claim flow for `authenticated`+`private` (search `claim` under `server/src`):
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/auth.ts -H "Accept: application/vnd.github.raw"`
Decide: if the first-admin claim creates the account through a path that ignores `disableSignUp` (e.g. invite/claim token handler calling the adapter directly), `SIGNUP_DISABLED="true"`. Otherwise `SIGNUP_DISABLED="false"` — the Authentik gate is the access control, and a follow-up PR flips it to `"true"` after the admin is claimed.
- [ ] **Step 2: Partial config.json.** Read `server/src/config.ts` and the config-file loader it calls (search `PAPERCLIP_CONFIG` / `readConfigFile`).
Run: `gh api repos/paperclipai/paperclip/contents/server/src/config.ts -H "Accept: application/vnd.github.raw"`
Decide `CONFIG_PARTIAL_OK=yes` if a file containing only `{"telemetry":{"enabled":false}}` parses (every other section optional/defaulted). If not, stop and report to the user — do not invent config fields.
- [ ] **Step 3: Entrypoint privilege.**
Run: `gh api repos/paperclipai/paperclip/contents/docker-entrypoint.sh -H "Accept: application/vnd.github.raw"`
`ENTRYPOINT_NEEDS_ROOT=yes` if it calls `usermod`/`groupmod`/`chown`/`gosu`. Record the result for Task 6.
- [ ] **Step 4: Anonymous health response.**
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/health.ts -H "Accept: application/vnd.github.raw"`
Confirm `GET /api/health` returns HTTP 200 to an anonymous caller when healthy (probes use it). If it returns 401 anonymously, use a `tcpSocket` probe on 3100 in Task 6 instead.
---
### Task 2: Namespace, baseline CNP, ClusterSecretStore allowlist
**Files:**
- Modify: `kubernetes/infrastructure/namespaces/namespaces.yaml` (append)
- Create: `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
- Modify: `kubernetes/infrastructure/network-policies/kustomization.yaml`
- Modify: `kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml` (sorted list)
**Interfaces:**
- Produces: namespace `paperclip`; ESO may reference `openbao` from it.
- [ ] **Step 1: Append the namespace** to the end of `namespaces.yaml`:
```yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: paperclip
```
- [ ] **Step 2: Create `paperclip-ns.yaml`** (copy of `paperless-ns.yaml`'s `default-deny-ingress`, ns renamed, paperless-only CNPs dropped):
```yaml
# Traefik is not admitted: the only entry is the paperclip-gate outpost (apps/paperclip/network-policy.yaml).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: default-deny-ingress
namespace: paperclip
spec:
endpointSelector: {}
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": database
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
- fromEntities:
- host
- remote-node
- fromEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": rook-ceph
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- toEntities:
- host
- remote-node
- world
- toEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
```
- [ ] **Step 3: Register it** — in `network-policies/kustomization.yaml` add ` - paperclip-ns.yaml` directly after ` - paperless-ns.yaml`.
- [ ] **Step 4: Allowlist the namespace** — in `openbao-clustersecretstore.yaml` insert ` - paperclip` between ` - monitoring` and ` - paperless` (list is sorted).
- [ ] **Step 5: Render check.**
Run: `kubectl kustomize kubernetes/infrastructure/network-policies`
Expected: output contains `namespace: paperclip`, exit 0.
Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml`
Expected: `clustersecretstore.external-secrets.io/openbao configured (server dry run)`.
- [ ] **Step 6: Commit** `feat(paperclip): namespace, baseline CNP and OpenBao allowlist`.
---
### Task 3: OpenBao secret `apps/paperclip`
Out-of-repo; must exist before merge or the ExternalSecret never becomes Ready.
**Interfaces:**
- Produces: OpenBao KV `apps/paperclip` with the four fields in Global Constraints.
- [ ] **Step 1:** Ask wiki-synth for the claude-bot OpenBao write procedure (note `claude-bot-openbao-write-only-identity`): login method, KV mount, exact `bao kv put`/`patch` syntax for an `apps/<name>` key. Use exactly what it returns.
- [ ] **Step 2: Write the generated fields** as claude-bot (write-only identity — never read values back). Values generated inline by the command:
`BETTER_AUTH_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_SECRETS_MASTER_KEY` = `openssl rand -hex 32` (64 hex chars = 32 bytes, accepted format).
Generate each with a separate `openssl rand -hex 32` call and pass it directly into the `bao kv put` from Step 1; do not echo or log them.
- [ ] **Step 3: Ask the user** to add the shared token themselves (never handle it): "Run `claude setup-token`, then `bao kv patch <mount>/apps/paperclip CLAUDE_CODE_OAUTH_TOKEN=<token>`." Wait for confirmation.
- [ ] **Step 4: Verify field names only** (no values): `bao kv metadata get <mount>/apps/paperclip` shows a current version ≥2. If claude-bot cannot read metadata, ask the user to confirm the four keys exist.
---
### Task 4: Image automation + initial digest
**Files:**
- Create: `kubernetes/infrastructure/image-automation/paperclip.yaml`
- Modify: `kubernetes/infrastructure/image-automation/kustomization.yaml`
**Interfaces:**
- Produces: ImagePolicy `flux-system:paperclip`; `INITIAL_DIGEST` (`sha256:…`) for Task 6.
- [ ] **Step 1: Create `paperclip.yaml`:**
```yaml
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
name: paperclip
namespace: flux-system
spec:
image: ghcr.io/paperclipai/paperclip
interval: 1h
---
# Upstream publishes no semver tags; track the nightly tag by digest.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
name: paperclip
namespace: flux-system
spec:
imageRepositoryRef:
name: paperclip
digestReflectionPolicy: Always
filterTags:
pattern: '^nightly$'
policy:
alphabetical:
order: asc
```
- [ ] **Step 2: Register** — add ` - paperclip.yaml` after ` - paperless-ngx.yaml` in `image-automation/kustomization.yaml`.
- [ ] **Step 3: Server dry-run.**
Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/image-automation/paperclip.yaml`
Expected: both objects `created (server dry run)`; no `unknown field "digestReflectionPolicy"` error.
- [ ] **Step 4: Resolve the current nightly digest.**
The workstation has `docker` only (no skopeo/crane/buildx, checked 2026-09-25); `docker manifest inspect` lists per-platform digests but not the index digest, so pull once:
Run: `docker pull ghcr.io/paperclipai/paperclip:nightly`
Run: `docker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/paperclipai/paperclip:nightly`
Expected: `ghcr.io/paperclipai/paperclip@sha256:<64 hex>`. Record the `sha256:…` part as `INITIAL_DIGEST`.
Run: `docker image rm ghcr.io/paperclipai/paperclip:nightly`
- [ ] **Step 5: Commit** `feat(paperclip): Flux image automation tracks the nightly digest`.
---
### Task 5: Postgres (CNPG) and secrets in the app directory
**Files:**
- Create: `kubernetes/apps/paperclip/cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`, `hetzner-creds-externalsecret.yaml`, `externalsecret.yaml`
**Interfaces:**
- Produces: Secret `pclp-pg-app` (CNPG, key `uri`), Service `pclp-pg-rw`, Secret `paperclip-secrets`.
- [ ] **Step 1: `cluster.yaml`:**
```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pclp-pg
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
# PG17: upstream's external Postgres target (docker/docker-compose.yml).
imageName: ghcr.io/cloudnative-pg/postgresql:17
storage:
storageClass: ceph-block
size: 10Gi
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: pclp-pg-store
bootstrap:
initdb:
database: paperclip
owner: paperclip
postgresql:
parameters:
shared_buffers: "256MB"
max_connections: "200"
checkpoint_timeout: "15min"
max_wal_size: "2GB"
wal_compression: "zstd"
archive_timeout: "30min"
synchronous:
method: any
number: 1
dataDurability: preferred
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 1Gi
```
- [ ] **Step 2: `objectstore.yaml`:**
```yaml
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pclp-pg-store
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup target
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/pclp-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
```
- [ ] **Step 3: `scheduled-backup.yaml`:**
```yaml
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: pclp-pg-daily
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # backup schedule
spec:
schedule: "0 45 2 * * *"
immediate: true
backupOwnerReference: self
cluster:
name: pclp-pg
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
```
- [ ] **Step 4: `hetzner-creds-externalsecret.yaml`:**
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hetzner-s3-creds
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # barman S3 creds
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: hetzner-s3-creds
dataFrom:
- extract:
key: infrastructure/hetzner-s3
```
- [ ] **Step 5: `externalsecret.yaml`:**
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: paperclip-secrets
namespace: paperclip
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: paperclip-secrets
dataFrom:
- extract:
key: apps/paperclip
```
- [ ] **Step 6: Commit** `feat(paperclip): CNPG pclp-pg with barman backups and OpenBao secrets`. (Rendered together with Task 6's kustomization.)
---
### Task 6: Server workload, storage, ingress, network policy
**Files:**
- Create: `kubernetes/apps/paperclip/pvc.yaml`, `paperclip.yaml`, `gate-service.yaml`, `ingress.yaml`, `network-policy.yaml`, `kustomization.yaml`
**Interfaces:**
- Consumes: `pclp-pg-app` (`uri`), `paperclip-secrets`, ImagePolicy `flux-system:paperclip`, `INITIAL_DIGEST` (Task 4), `SIGNUP_DISABLED` / `ENTRYPOINT_NEEDS_ROOT` / probe choice (Task 1).
- Produces: pods labelled `app.kubernetes.io/name: paperclip` listening on 3100; Service `paperclip:3100`; ExternalName `ak-outpost-paperclip-gate:9000`.
- [ ] **Step 1: `pvc.yaml`:**
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperclip-data
namespace: paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled # data guard
spec:
accessModes: [ReadWriteOnce]
storageClassName: ceph-block
resources:
requests:
storage: 20Gi
```
- [ ] **Step 2: `paperclip.yaml`** — replace `<INITIAL_DIGEST>` with the Task 4 value and `<SIGNUP_DISABLED>` with the Task 1 value. If Task 1 found `ENTRYPOINT_NEEDS_ROOT=no`, replace the container `securityContext` with `runAsNonRoot: true`, `runAsUser: 1000`, `runAsGroup: 1000`, `allowPrivilegeEscalation: false`, `capabilities: {drop: ["ALL"]}`. If Task 1 chose `tcpSocket`, replace each `httpGet` block with `tcpSocket: {port: http}`.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperclip
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
secret.reloader.stakater.com/reload: paperclip-secrets
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: paperclip
template:
metadata:
labels:
app.kubernetes.io/name: paperclip
spec:
enableServiceLinks: false
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
initContainers:
# Telemetry is only configurable in config.json.
- name: disable-telemetry
image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
command:
- sh
- -c
- |
set -eu
f=/paperclip/instances/default/config.json
mkdir -p "$(dirname "$f")"
if [ -s "$f" ]; then
jq '.telemetry.enabled = false' "$f" > "$f.tmp"
else
printf '%s\n' '{"telemetry":{"enabled":false}}' > "$f.tmp"
fi
mv "$f.tmp" "$f"
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: data
mountPath: /paperclip
containers:
- name: paperclip
image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
ports:
- name: http
containerPort: 3100
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: pclp-pg-app
key: uri
- name: HOST
value: "0.0.0.0"
- name: PORT
value: "3100"
- name: PAPERCLIP_DEPLOYMENT_MODE
value: authenticated
- name: PAPERCLIP_DEPLOYMENT_EXPOSURE
value: private
- name: PAPERCLIP_PUBLIC_URL
value: https://paperclip.renner.uno
- name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
value: "<SIGNUP_DISABLED>"
envFrom:
- secretRef:
name: paperclip-secrets
volumeMounts:
- name: data
mountPath: /paperclip
startupProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 60
livenessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 30
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 10
timeoutSeconds: 5
resources:
requests:
cpu: 250m
memory: 1Gi
limits:
memory: 4Gi
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
capabilities:
drop: ["ALL"]
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
volumes:
- name: data
persistentVolumeClaim:
claimName: paperclip-data
---
apiVersion: v1
kind: Service
metadata:
name: paperclip
namespace: paperclip
spec:
selector:
app.kubernetes.io/name: paperclip
ports:
- name: http
port: 3100
targetPort: http
```
- [ ] **Step 3: `gate-service.yaml`:**
```yaml
# ExternalName mirror so the Ingress can backend the paperclip-gate outpost in the authentik namespace.
apiVersion: v1
kind: Service
metadata:
name: ak-outpost-paperclip-gate
namespace: paperclip
spec:
type: ExternalName
externalName: ak-outpost-paperclip-gate.authentik.svc.cluster.local
ports:
- port: 9000
targetPort: 9000
```
- [ ] **Step 4: `ingress.yaml`:**
```yaml
# LAN-only (no dns.renner.uno/public label); backends the gate outpost, never the Service.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: paperclip
namespace: paperclip
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
spec:
ingressClassName: traefik
tls:
- hosts:
- paperclip.renner.uno
secretName: paperclip-tls
rules:
- host: paperclip.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ak-outpost-paperclip-gate
port:
number: 9000
```
- [ ] **Step 5: `network-policy.yaml`** (Traefik pod-side port is 8443, verified via `kubectl get pods -n ingress -l app.kubernetes.io/name=traefik`):
```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-ingress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
goauthentik.io/outpost-name: paperclip-gate
toPorts:
- ports:
- port: "3100"
protocol: TCP
---
# Covers the server and every agent it spawns (same pod).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-egress
namespace: paperclip
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: paperclip
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
cnpg.io/cluster: pclp-pg
toPorts:
- ports:
- port: "5432"
protocol: TCP
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
# Gitea, Woodpecker, HA, Loki via their *.renner.uno names.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8443"
protocol: TCP
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
- toEntities:
- world
toPorts:
- ports:
- port: "443"
protocol: TCP
```
- [ ] **Step 6: `kustomization.yaml`:**
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- pvc.yaml
- paperclip.yaml
- gate-service.yaml
- ingress.yaml
- network-policy.yaml
```
- [ ] **Step 7: Render check.**
Run: `kubectl kustomize kubernetes/apps/paperclip`
Expected: exit 0; output contains `kind: Cluster`, `kind: Deployment`, `claimName: paperclip-data`, and no literal `<INITIAL_DIGEST>` or `<SIGNUP_DISABLED>`.
Run: `grep -rn "<INITIAL_DIGEST>\|<SIGNUP_DISABLED>" kubernetes/apps/paperclip`
Expected: no output, exit 1.
- [ ] **Step 8: Commit** `feat(paperclip): server Deployment, PVC, gated Ingress and CNPs`.
---
### Task 7: Authentik gate
**Files:**
- Modify: `kubernetes/apps/authentik/blueprints-configmap.yaml` (insert before the line ` # navidrome-gate: the Navidrome web UI.`)
- Create: `kubernetes/apps/authentik/paperclip-gate-egress.yaml`
- Modify: `kubernetes/apps/authentik/kustomization.yaml`
**Interfaces:**
- Produces: outpost pods labelled `goauthentik.io/outpost-name: paperclip-gate`, Service `ak-outpost-paperclip-gate.authentik:9000`.
- [ ] **Step 1: Insert blueprint entries** (same indentation as the neighbouring `wiki-gate` block):
```yaml
# paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
- id: paperclip-users-group
model: authentik_core.group
identifiers:
name: Paperclip Users
state: present
attrs:
is_superuser: false
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: paperclip-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: paperclip-gate
state: present
attrs:
authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://paperclip.paperclip.svc.cluster.local:3100
internal_host_ssl_validation: false
external_host: https://paperclip.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: paperclip-gate
state: present
attrs:
name: Paperclip
slug: paperclip-gate
provider: !KeyOf paperclip-gate-provider
meta_launch_url: https://paperclip.renner.uno
meta_description: "Paperclip agent orchestration"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, paperclip-gate]]
group: !KeyOf paperclip-users-group
order: 0
state: present
attrs:
enabled: true
negate: false
- model: authentik_outposts.outpost
identifiers:
name: paperclip-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf paperclip-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
```
- [ ] **Step 2: `paperclip-gate-egress.yaml`:**
```yaml
# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: paperclip-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
app.kubernetes.io/name: paperclip
toPorts:
- ports:
- port: "3100"
protocol: TCP
```
- [ ] **Step 3: Register** — add ` - paperclip-gate-egress.yaml` after ` - paperless-gate-egress.yaml` in `apps/authentik/kustomization.yaml`.
- [ ] **Step 4: Render + dry-run.**
Run: `kubectl kustomize kubernetes/apps/authentik`
Expected: exit 0 (valid YAML incl. the blueprint ConfigMap).
Run: `kubectl apply --dry-run=server -f kubernetes/apps/authentik/paperclip-gate-egress.yaml`
Expected: `ciliumnetworkpolicy.cilium.io/paperclip-gate-egress created (server dry run)`.
- [ ] **Step 5: Commit** `feat(authentik): paperclip-gate outpost and Paperclip Users group`.
---
### Task 8: Flux Kustomization
**Files:**
- Create: `kubernetes/clusters/homelab/paperclip.yaml`
- [ ] **Step 1:**
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: paperclip
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 20m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/paperclip
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
failed: "has(status.phase) && status.phase.startsWith('Failed')"
current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
```
- [ ] **Step 2: Server dry-run.**
Run: `kubectl apply --dry-run=server -f kubernetes/clusters/homelab/paperclip.yaml`
Expected: `kustomization.kustomize.toolkit.fluxcd.io/paperclip created (server dry run)`.
- [ ] **Step 3: Commit** `feat(paperclip): Flux Kustomization`.
---
### Task 9: PR, merge by user, live verification
- [ ] **Step 1: Push and open the PR** (Task 3 must be complete).
Run: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push -u origin feat/paperclip`
Run: `tea pr create --login claude-bot --base master --head feat/paperclip --title "feat(paperclip): deploy Paperclip (server-local runs, trial)" --description "<summary of spec decisions + link to spec + bootstrap steps>\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)"`
Then wait for the user to merge. Do not merge.
- [ ] **Step 2: Reconciliation.** After merge:
Run: `flux get kustomizations paperclip`
Expected: `Ready True`. If False, read `kubectl describe kustomization paperclip -n flux-system` and the failing object's events; diagnose, don't poke.
Run: `kubectl get cluster -n paperclip pclp-pg`
Expected: `Cluster in healthy state`, 2 ready instances.
Run: `kubectl get pods -n paperclip -l app.kubernetes.io/name=paperclip`
Expected: `1/1 Running`.
- [ ] **Step 3: Gate enforced (Review Focus 4).**
Run: `curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https://paperclip.renner.uno/`
Expected: `302` redirecting to `https://paperclip.renner.uno/outpost.goauthentik.io/…` or `auth.renner.uno`, never `200`.
- [ ] **Step 4: Gate not bypassable (Review Focus 3).**
`default` has no CNPs (verified 2026-09-25), so its egress is open and only paperclip's ingress policy can block the probe.
Run: `kubectl get ciliumnetworkpolicies -n default`
Expected: `No resources found` (else pick another CNP-free namespace).
Run: `kubectl run cnp-probe -n default --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -sS -m 5 http://paperclip.paperclip.svc.cluster.local:3100/api/health`
Expected: timeout (exit 28), no HTTP response.
- [ ] **Step 5: First backup (Review Focus 5).**
Run: `kubectl get backup -n paperclip`
Expected: one `pclp-pg-daily-…` with phase `completed`.
- [ ] **Step 6: Image policy.**
Run: `kubectl get imagepolicy -n flux-system paperclip -o jsonpath='{.status.latestRef}'`
Expected: `nightly` tag with a `sha256:` digest.
- [ ] **Step 7: Hand-over to the user (browser steps, theirs):**
1. Open `https://paperclip.renner.uno`, pass Authentik, claim the instance as admin.
2. Create a test agent with the `claude-local` adapter and a trivial task ("reply with OK"); it must complete with the shared token (Review Focus 2: if it fails, report the run's error text — an expired token must show as an auth error, not a hang).
3. If `SIGNUP_DISABLED` was `"false"`, open the follow-up PR flipping `PAPERCLIP_AUTH_DISABLE_SIGN_UP` to `"true"` once the admin is claimed.
4. Review Focus 1 is observed on the first nightly bump: check the interrupted-run status in the UI.
- [ ] **Step 8: Wiki.** Route the verified Paperclip facts (execution topology, env vars, image tags, digest-reflection usage) through wiki-ingest → wiki-distill once the deployment has proven out.
@@ -1,207 +0,0 @@
# Paperclip in the homelab cluster — design
Date: 2026-09-25
Status: draft, awaiting review
## Goal
Run [Paperclip](https://github.com/paperclipai/paperclip) (MIT, agent-orchestration
server + UI) in the homelab cluster for real use: its agents do actual work against
homelab systems. This spec covers the **platform** only. Bot identities and their
credentials are out of scope (see "Deferred").
## Decisions
| Topic | Decision |
|---|---|
| Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
| Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`), then its value replaced with a long-lived `claude setup-token` token. Paperclip stores the value as company secret "My Claude subscription" and injects it per run as `CLAUDE_CODE_OAUTH_TOKEN`; it never refreshes Anthropic credentials, so the login's ~8h access token expires (runs then fail with "ACP agent reported a terminal access failure", 0 tokens). Renew by rotating that secret with a new setup-token (~yearly). Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
| Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
| Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
### Known trade-off of B (accepted for the trial)
Upstream `runChildProcess()` spawns every local adapter with
`{...sanitizeInheritedPaperclipEnv(process.env), ...opts.env}`; the sanitizer strips only
`PAPERCLIP_*` keys. Every agent therefore sees `DATABASE_URL`, `BETTER_AUTH_SECRET`,
the shared filesystem (including every other agent's workspace and the
`ai-local-logins` credentials) and every agent-scoped secret it can read from disk. Under B, no bot may hold
a credential we would not tolerate any other bot using. Per-bot isolation is C's job.
## Architecture
```
LAN client ─▶ Traefik ─▶ ExternalName svc ─▶ ak-outpost-paperclip-gate (authentik ns)
│ forward-auth OK
▼
paperclip Deployment (ns paperclip, 1 replica)
├─ server :3100 (+ agent CLIs as child processes)
├─ PVC /paperclip (ceph-block)
└─▶ pclp-pg-rw (CNPG, PG17, 2 instances) ─▶ barman ▶ Hetzner S3
```
## Components
All modeled on `kubernetes/apps/paperless/` unless stated.
### Namespace and baseline policy
- `Namespace paperclip` in `kubernetes/infrastructure/namespaces/namespaces.yaml`.
- Baseline default-deny CNP `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
(copy of `paperless-ns.yaml`, renamed).
- Add `paperclip` to the `openbao` ClusterSecretStore namespace allowlist.
### Flux
- `kubernetes/clusters/homelab/paperclip.yaml`: Flux `Kustomization`,
`path: ./kubernetes/apps/paperclip`, `prune: true`, `wait: true`, same `dependsOn` and
CNPG `healthCheckExprs` as `paperless.yaml`.
- `kubernetes/apps/paperclip/kustomization.yaml`: plain resource list.
### Server Deployment (`paperclip.yaml`)
- Image `ghcr.io/paperclipai/paperclip:nightly@sha256:<digest>` with the Flux
image-automation marker for digest updates.
- `replicas: 1`, `strategy: Recreate` (single writer on an RWO PVC).
- Starts as root: upstream `docker-entrypoint.sh` remaps the `node` user, chowns
`/paperclip`, then drops to uid 1000 via `gosu`. Caps limited to CHOWN,
DAC_OVERRIDE, FOWNER, SETGID, SETUID; `fsGroup: 1000`.
- `serviceAccountName: paperclip-prometheus-read` with
`automountServiceAccountToken: true`. All agents share the pod filesystem, so all
of them can read that token — it is mounted anyway because the Role behind it
grants exactly `get` on `services/proxy` for
`kube-prometheus-stack-prometheus:9090` and
`kube-prometheus-stack-alertmanager:9093` in `monitoring`, and nothing else
(`kubernetes/apps/paperclip/prometheus-read-rbac.yaml`, REN-36 then REN-95).
It was `false` until then.
Three things about that grant are worth knowing before you use or debug it:
- **It scopes the HTTP method, not the endpoint.** The apiserver maps GET→`get`,
POST→`create`, PUT→`update` on a service-proxy request, so `get` alone refuses
`POST /-/reload` and `POST /-/quit` in the authorizer. It does *not* narrow
which GET endpoints are reachable: the whole read surface of both services is,
including `/api/v1/status/config` (scrape credentials there render as
`<secret>`) and the web UI.
- **Force GET in the caller.** Prometheus accepts both GET and POST on
`/api/v1/query`; under this Role only the GET form works, so a client library
that switches to POST for long queries will 403 for a reason that has nothing
to do with the query.
- **If reads start 403ing, check the Service name against `resourceNames`
first.** `resourceNames` is matched verbatim, and
`kubernetes/apps/monitoring/helmrelease-prometheus.yaml` floats the chart at
`>=72.0.0` — a release rename breaks this path silently,
with everything still looking correct in git. The port-qualified form is the
only one granted: `svc`, `svc:9090` and `http:svc:9090` are three different
strings to the authorizer.
- Port 3100; `Service paperclip` ClusterIP :3100.
- Probes: startup and readiness on `GET /api/health` with `Host: localhost` (the
private-hostname guard rejects the pod IP); liveness is a TCP check, since
`/api/health` runs `SELECT 1` and a DB outage must not restart the server.
- Env:
- `DATABASE_URL` assembled from CNPG `pclp-pg-app` Secret (`uri` key).
- `HOST=0.0.0.0`, `PORT=3100`
- `PAPERCLIP_DEPLOYMENT_MODE=authenticated`
- `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`
- `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`
- `PAPERCLIP_AUTH_DISABLE_SIGN_UP=false` until the first admin is claimed: the
claim (`/bootstrap/claim`) requires an already signed-in Better Auth account.
A follow-up flips it to `true`.
- `PAPERCLIP_TELEMETRY_DISABLED=1` (read by `resolveTelemetryConfig`).
- `PAPERCLIP_API_URL=http://paperclip.paperclip.svc.cluster.local:3100` plus
`PAPERCLIP_ALLOWED_HOSTNAMES=paperclip.paperclip.svc.cluster.local`: upstream
otherwise hands agents the public URL, and the gate (`intercept_header_auth`)
rejects their bearer tokens. The Service name also works for sandbox pods under
option C; agent JWTs still authenticate each call.
- `envFrom: secretRef paperclip-secrets`
- Reloader: `secret.reloader.stakater.com/reload: paperclip-secrets`.
### Storage
- `PVC paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip` (`PAPERCLIP_HOME`
and `HOME`): instance config, workspaces, repo checkouts, per-agent `claude-runtime`
dirs.
### Postgres
- CNPG `Cluster pclp-pg`: 2 instances, `imageName: ghcr.io/cloudnative-pg/postgresql:17`,
`ceph-block` 10Gi, `bootstrap.initdb {database: paperclip, owner: paperclip}`,
barman-cloud plugin as WAL archiver.
- `ObjectStore pclp-pg-store` → `s3://renner-uno-k8s-cnpg/pclp-pg`, creds via the
existing `hetzner-s3-creds` ExternalSecret pattern.
- `ScheduledBackup` nightly, `method: plugin`, `backupOwnerReference: self`.
- `kustomize.toolkit.fluxcd.io/prune: disabled` on Cluster, ObjectStore, ScheduledBackup.
- PG17 deviates from the cluster-wide PG16 convention because upstream's external
Postgres target is `postgres:17` (`docker/docker-compose.yml`) and its embedded default
is PG18.
### Secrets
One ExternalSecret `paperclip-secrets` ← OpenBao `apps/paperclip` (whole-object
extract):
- `BETTER_AUTH_SECRET`
- `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`
- `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex). Held in OpenBao, not on the PVC, so
encrypted agent secrets in Postgres survive PVC loss.
### Ingress and SSO
- Hostname `paperclip.renner.uno`, Traefik `Ingress`, `cert-manager.io/cluster-issuer:
letsencrypt-cluster`, **no** `dns.renner.uno/public` label (LAN-only).
- Authentik gate, per the paperless pattern:
- blueprint entries in `kubernetes/apps/authentik/blueprints-configmap.yaml`:
proxy provider `paperclip-gate` (`internal_host:
http://paperclip.paperclip.svc.cluster.local:3100`, `external_host:
https://paperclip.renner.uno`), application, group "Paperclip Users", policy binding;
- outpost `ak-outpost-paperclip-gate`;
- `kubernetes/apps/authentik/paperclip-gate-egress.yaml` (no `auth.renner.uno`
outpost Ingress: like wiki/shelly gates, the outpost serves its own callback on
`paperclip.renner.uno`);
- `ExternalName` Service in `paperclip` ns pointing at the outpost; the app Ingress
targets it on :9000.
- Paperclip's own email/password login stays on (no header/OIDC login upstream).
### Network policy (`kubernetes/apps/paperclip/network-policy.yaml`)
Covers the server **and every agent** (same pod under B).
- Ingress to the app pod: only from the `paperclip-gate` outpost (by label) on 3100.
- Egress from the app pod:
- `pclp-pg` pods :5432
- kube-dns :53
- `ingress` namespace (Traefik) :8443 (pod-side websecure port) — reaches Gitea, Woodpecker, HA, Loki by their
`*.renner.uno` names
- `kube-apiserver` entity :6443
- `world` TCP :443 (Anthropic API, GitHub, npm)
- No SSH egress; git over HTTPS only. The namespace baseline
(`paperclip-ns.yaml`) also restricts `world` to TCP 443 — Cilium unions allows
across CNPs, so an unrestricted baseline would override this list.
- Known gap (house baseline): pods in `monitoring`/`database` and other pods in
`paperclip` can reach :3100 without the gate; Paperclip's own login still applies.
### Image automation
`kubernetes/infrastructure/image-automation/paperclip.yaml`: `ImageRepository`
(`ghcr.io/paperclipai/paperclip`) + `ImagePolicy` filtering tag `^nightly$` with
`digestReflectionPolicy: Always` (image-reflector-controller v1.2.5 in cluster; field
available since v0.35.0). First use of digest reflection in this repo.
## Guardrails
- No bot receives cluster-admin or any binding in `flux-system`, `authentik`, `openbao`.
- Git work goes through PRs; bots get no merge rights on protected branches.
- Nothing is granted under B that we would not tolerate any agent misusing.
## Bootstrap (one-time)
1. Place `apps/paperclip` values in OpenBao (the three generated secrets).
2. Merge; Flux deploys with sign-up enabled.
3. Sign up through the gate and claim the instance as admin; then set
`PAPERCLIP_AUTH_DISABLE_SIGN_UP=true`.
4. In onboarding "Connect a model", choose Claude subscription and run the shown login
**as the node user** (the entrypoint makes exec sessions root, and root-owned 0600
credentials are unreadable to the server):
`kubectl exec -it -n paperclip deploy/paperclip -- gosu node sh -c 'export CLAUDE_CONFIG_DIR=<shown path> && mkdir -p "$CLAUDE_CONFIG_DIR" && claude auth login'`
Then Settings → Secrets → "My Claude subscription": replace the value with the
output of `claude setup-token`. Repeat both after loss of the database.
## Verification
- Flux Kustomization `paperclip` Ready; CNPG `pclp-pg` healthy; first ScheduledBackup
completes.
- `curl https://paperclip.renner.uno/api/health` through the gate returns healthy after
Authentik login; unauthenticated request redirects to `auth.renner.uno`.
- Direct pod access from another namespace is denied (CNP).
- A test agent (claude-local) completes a trivial task using the shared Claude connection.
- Flux ImagePolicy reports a `nightly@sha256:` digest.
## Deferred
- **Option C** (sandboxed runs) — required before any per-bot credential isolation.
- **AIR-Manager** role and all bot identities/credentials.
- Public exposure.
-99
View File
@@ -1,99 +0,0 @@
# Traffic classes on the k8s nodes
Issue #576. Five classes, each on its own VLAN.
## Where each class runs
| class | link | mechanism |
|---|---|---|
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
Hard constraint, set by the owner: inter-node and pod traffic never traverses
the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
## Pod traffic: one table
Two rules, one per family, send every packet with a pod source to table 47
(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
table decides by longest prefix; no rule carves out exceptions.
bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
taken from the interfaces;
- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
peer's `pod0` address;
- the node's own pod CIDR via `cilium_host`, originated rather than read back
from the kernel (reading it back made BIRD depend on the shape of Cilium's
route, and KRT rejects the IPv4 one);
- the egress default via the UDM on `ingress0`, from the node annotations
`net.renner.uno/egress-gw-v4/v6`.
`node0` has to be in the table: a pod reaching a node address — the
`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
address, or a hostNetwork pod — would otherwise take the default to the UDM.
The routing decision runs before Cilium's masquerade hook, so the rule sees the
pod address and the masquerade then rewrites it to the egress link's.
Node-originated traffic does not match the rules and uses main, where Cilium
routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
No address is written into this repo twice. Each node publishes its `pod0`
addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
pod CIDRs come from the Node objects. bird's init container renders both from
the API and ends with `bird -p`, so BIRD's own parser gates startup.
## Node↔node
`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
is IPv4, and the apiserver's advertise address must match it.
The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
kubelet, scheduler, controller-manager and Cilium go through KubePrism
(`localhost:7445`), which balances over every address of every control-plane
member; pods use the ClusterIP.
Talos never updates an existing etcd member's peer URLs
(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
and is promoted. EPHEMERAL does not need a wipe.
## Moving node IPs: the mixed state
While some nodes have their InternalIP on the new network and some on the old,
host→remote-pod traffic leaves with a node address that is not an InternalIP:
the kernel picks the source from the route to the peer's InternalIP. Cilium
classifies that as `world`, and every ingress policy that admits only
`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
lose the OSDs on the other side; RBD mounts hang.
Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
admitting the node subnets of both networks, both families, with
`enableDefaultDeny: false` so it restricts nothing else, and delete it once
every node has moved. Keep all OSD nodes schedulable while one is drained:
draining two takes Ceph below its data availability.
## Failure behaviour
If bird stops, table 47 empties, the rules find nothing and fall through to
main: pod traffic returns to the node networks, unseparated rather than
misrouted. The same holds during a cold rebuild until Gitea serves bird's
image. The default route lives in table 47, not in the Talos config, so a
default cannot outlive the specific routes and send pod traffic to the UDM.
## Sources
Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
`machine.nodeAnnotations`, configuration patching (lists are appended,
`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.
@@ -0,0 +1,194 @@
# Proposal — separate the four traffic classes onto their own VLANs
Status: draft. The question this started with is **answered** — from the Cilium
source and flag reference, not from a test — and the answer changes the shape
of the proposal rather than confirming it.
Supersedes the framing in issue #576, which described this as "give node
management its own NIC, and Ceph public its own VLAN". That was two of four
classes and one of them turned out already solved.
## The four classes
| class | what it is | where it goes today |
|---|---|---|
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
Three of four share one interface. That is not a design decision — until the
bare-metal conversion there was one NIC to work with.
Ceph client I/O living on the pod network is correct rather than a leftover:
that traffic *is* cluster-internal. Ceph is done.
## Principle
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
separate, changeable decision.**
Defining classes by VLAN means the separation survives recabling, port failures
and capacity changes, and it does not require one port per class. Defining them
by interface would freeze a capacity decision into the architecture.
## The finding that shapes everything else
The original open question was whether `nodePort.directRoutingDevice` moves the
next-hop of Cilium's direct routes, or only selects an interface. It is
answerable from the source, and the answer is: **only the interface.**
`installDirectRoute` takes the next-hop as a parameter:
```go
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
skipUnreachable bool) (routeSpec *netlink.Route, err error)
```
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
cluster there is no path by which Cilium's node IP differs from the kubelet's
`InternalIP`.
`--direct-routing-device` is documented as "Device name used to connect nodes in
direct routing mode" — a device, not an address.
**So cluster-internal traffic is welded to the node IP, and the node IP is the
kubelet InternalIP.** Two consequences:
- The class cannot be moved off whatever VLAN the node IP lives on.
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
from the node IP; native mode uses it as the direct-route next-hop. Native
routing buys nothing here, so the disruptive, non-rolling step this proposal
previously contained is simply gone.
## Consequence: draw the line the other way
If cluster-internal cannot leave the node IP, then separation has to come from
moving *management services* off it — and each one has to be checked
individually, because they do not share a mechanism.
| management service | can it leave the node IP? | lever |
|---|---|---|
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
| kubelet | **no** | it *is* the node IP, by definition |
So four VLANs are reachable, but the boundary does not fall where the class
names suggest: **kubelet stays with cluster-internal.** That is the honest
result, and it is worth stating plainly rather than filing kubelet under
management and hoping.
The remaining classes keep the levers established earlier:
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
For eBGP the session source is the next-hop, so pinning the session pins
where return traffic arrives. The session to the UDM has been running for
months.
- **cluster-external egress** — eBPF masquerading follows the output device, so
this is a default-route decision plus `enable-masquerade-to-route-source`
(today `false`).
- **Ceph replication** — Multus macvlan on VLAN 49, done.
## The general escape hatch
Ceph is separated because its OSDs got a second interface and stopped using the
pod network for replication. That is the *only* mechanism that fully detaches a
workload from the node IP, and it generalises: anything that must not ride the
node IP gets its own `NetworkAttachmentDefinition`.
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
change. Worth it for Ceph replication. Not worth it for east-west traffic in
general.
## Recommended mapping
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
and Talos API. Cleanest match to the class names, but it changes every node's
identity, which is the one address the whole cluster is administered through.
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
Identical end state, and no node ever changes its InternalIP.
Under B:
| VLAN | carries |
|---|---|
| 49 | Ceph replication |
| 40 (relabelled *cluster*) | pod east-west, kubelet |
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
| new *external* | LoadBalancer ingress, egress default route |
The property the user asked for — management carries only management traffic —
holds under B. The label moves; the property is what matters.
## Migration shape
Every step is rolling and reversible. The routing-mode change is gone.
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
Still unused; this only proves the L2 works.
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
waiting for a healthy quorum between each.
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
`k8s.renner.uno` follows.
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
Reversible by reverting the override; failure is visible immediately in
`cilium-dbg shell -- bgp/peers`.
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
Step 3 is the one that deserves care — etcd is the only step where a mistake
costs quorum rather than connectivity.
## What this does not buy
Management currently shares a 10 Gb link and is not contended. Nothing observed
today is slow because of this. The case for the split is that traffic classes
with different failure modes and security postures should not be
indistinguishable — not that something is broken.
## Sources and measurements
Source, Cilium v1.20.1 (the running version):
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
a next-hop that is not directly reachable.
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
Kubernetes Node.
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
to connect nodes in direct routing mode".
Talos:
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
defaults to it. kube-apiserver and Talos API listen addresses are not
covered there.
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
`10.2.40.60` with `link: mgmt0`.
Measured on the cluster:
- `apid` answered on a node address that is not the InternalIP, so the Talos
API is not bound to one address.
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
InternalIP had changed since the last session, which is why nothing here
names an address as fixed.
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
unset and no direct-routing device configured.
Not verified: whether the kube-apiserver's `--advertise-address` follows the
node IP independently of the VIP. It does not affect the recommendation, since
clients reach the API through the VIP either way.
-315
View File
@@ -1,315 +0,0 @@
# Wiki: local web interface and agent API, as a modular .NET solution
## Goal
A web interface for the Notes wiki that runs entirely in the cluster, and an
HTTP API through which agents ask the wiki typed questions and deposit facts.
Built as one C# solution with a MudBlazor front end, and built so that every
external dependency is replaceable: where the notes live, which model embeds
them, what judges a typed question, what runs an agent, how the vector index
is stored. The first adapters are what the homelab uses today. The point of
the design is that the next one is a project and a config line, not a
rewrite.
## Where we are
- `wiki-index` (Python, ns `ai`) syncs `wiki/` from `k.renner/notes` by blob
SHA, embeds through `wiki-embed` (bge-m3 on llama-server), answers
`POST /rank`. It keeps slug, title and vector, not the text. LAN-only behind
Traefik BasicAuth; the Gitea webhook hits the ClusterIP.
- `~/Notes/bin/wiki_rank.py` and `wiki_ask.py` rank through the index, build a
document context from a local vault checkout, send it with typed questions
to TypeSafe, gate on `covered`. The wiki-synth agent composes answers from
those judgements.
- New knowledge enters as a file in `raw/` (facts with provenance, never
hypotheses), distilled nightly in CI.
- The house C# project, shelly-manager, sets the conventions: net10.0, an
`.slnx` with `src/` and `test/`, xUnit, MudBlazor 9, Blazor Web App with
interactive server rendering, minimal APIs, an Authentik proxy-header
authentication handler for humans and an API-key handler for machines, a
multi-stage Dockerfile, `dotnet test` on every push and a kaniko release
pipeline to the Gitea registry. Human-facing apps sit behind an Authentik
outpost (`<app>-gate`) with a CNP that admits only that outpost.
## Architecture: ports in the core, adapters around it
```
┌──────────────────────────────────────────┐
│ Wiki.Host (ASP.NET Core + Blazor/Mud) │ composition root,
│ /api/v1, /rank, /webhook, UI pages │ config, auth
└───────────────┬──────────────────────────┘
│ uses
┌───────────────▼──────────────────────────┐
│ Wiki.Application (use cases) │ Sync, Rank, Ask,
│ │ DepositFact, Queue-
└───────────────┬──────────────────────────┘ Question, Evaluate
│ depends only on
┌───────────────▼──────────────────────────┐
│ Wiki.Core (domain + ports) │ Note, RawSource,
│ no package references │ Report, LinkGraph,
└──────────────────────────────────────────┘ the interfaces
implemented by ▲ ▲ ▲ ▲ ▲
Adapters.Gitea Adapters.LlamaCpp Adapters.TypeSafe Adapters.FileIndex Adapters.Anthropic
(later: Database, Ollama/Onnx, local Jev, pgvector/Qdrant, OpenAI-compatible answer models)
```
Rules that keep it modular, enforced by project references and a test:
- `Wiki.Core` references nothing. It holds the domain types and the port
interfaces. Adapters reference Core only; they never reference each other
or the host. `Wiki.Host` is the only project that knows which adapters
exist.
- Every port has a **contract test suite** in `test/Wiki.Contracts` that
runs against every adapter of that port. A new adapter is done when the
suite is green. That is what makes "swap the storage" a safe operation.
- Adapters are selected by configuration and registered by name (keyed
services), so several can be active at once. The UI's lab page and the
evaluator use that to compare, the API uses the one marked active.
- Nothing in Core or Application knows about HTTP, Gitea, TypeSafe, files or
Kubernetes.
### Ports
| port | what it abstracts | first adapter | examples of the next ones |
|---|---|---|---|
| `INoteStore` | read notes, raw sources and reports at a head; create a new document (a fact becomes a raw note in the application layer, the store only moves text) | Gitea repository API (`k.renner/notes`, blob SHA) | git checkout on a volume; a database (CNPG Postgres) with the vault imported; an Obsidian sync folder |
| `IEmbeddingModel` | text → vector, with a model id and dimension | llama-server `/v1/embeddings` (bge-m3 via wiki-embed, the model that won the German-document comparison on accuracy per cost) | the qwen3 embedding variants from that comparison; ONNX in-process; any OpenAI-compatible endpoint |
| `IVectorIndex` | store vectors per (model id, document), query top-k | in-memory cosine with a file snapshot per model, as today | pgvector in the database; Qdrant |
| `IChunker` | split a document into addressable units for typed questions | paragraph splitter (headings, fences), as in `wiki_ask.py` | sentence windows; semantic chunks |
| `IJudge` | typed questions over a state: Choice, Noul, Score, with probabilities | TypeSafe System One (`jev-latest`, cloud; the wiki records no locally hosted Jev today) | a locally hosted Jev with the same request shape, should TypeSafe offer one; a constrained-output local LLM adapter, flagged as uncalibrated |
| `IAnswerModel` | prose from a question plus evidence: the selected paragraphs with their notes and the judgements; must cite by slug; streaming | Anthropic Messages API (Claude, model from config, Sonnet by default) | any OpenAI-compatible endpoint, including the local llama-server; a different Claude model per question |
| `IEvaluator` | measure a retrieval or judge configuration against the labelled set (questions and cited sources in `output/`) | recall@k, hit rate, MRR, as `wiki_rank_eval.py` | judge accuracy against hand-labelled paragraphs |
| `IIdentity` | who is calling, for attribution of facts and saved answers | Authentik proxy headers (humans), API key (machines), as in shelly-manager | OIDC bearer tokens |
Two ports deserve a sentence of scope each. The judge returns judgements
with probabilities, never prose, and the UI never presents its output as an
answer, only as ranked, quoted evidence. The answer model turns that
evidence into words and is allowed nothing else: it sees the selected
paragraphs and the judgements, not the whole context, and its output is
shown next to the evidence it was given. Both are single model calls inside
the request. The service runs no agent: no tool loop, no vault edits. The
wiki-synth agent in a Claude session remains the path for artifacts and for
multi-question synthesis, and it can call `/questions` itself.
### Solution layout
```
Wiki.slnx
src/
Wiki.Core/ domain types, ports, frontmatter and link parsing
Wiki.Application/ use cases; depends on Core only
Wiki.Adapters.Gitea/ INoteStore (read at a head, create)
Wiki.Adapters.LlamaCpp/ IEmbeddingModel (OpenAI-compatible embeddings)
Wiki.Adapters.TypeSafe/ IJudge (System One client)
Wiki.Adapters.FileIndex/ IVectorIndex (memory + snapshot per model)
Wiki.Adapters.Anthropic/ IAnswerModel (Messages API, streaming)
Wiki.Host/ ASP.NET Core: API, webhook, MudBlazor UI, auth,
composition root, appsettings
test/
Wiki.Core.Tests/ parsing, link graph, chunker
Wiki.Contracts/ one suite per port, parameterised over adapters
Wiki.Application.Tests/ use cases with fake ports
Wiki.Host.Tests/ API surface, auth handlers
```
Own repository `k.renner/wiki`, same shape as shelly-manager: `Dockerfile`,
`.woodpecker/test.yml` on pull requests, `.woodpecker/release.yml` building
the image with kaniko into the Gitea registry on push to `main`, tagged
`build.<unix-ts>` plus the commit SHA so Flux image automation can follow
it. The homelab repo holds only the manifests. `claude-bot` needs write
access to the new repository.
Two things the shelly-manager history settles in advance: the TypeSafe
client is written from the documented field contract (endpoint, bearer key,
`jev-latest`, Noul/Choice/Score, 255-option cap, 32k-token state budget),
there is no prior C# client to copy; and the UI needs no login page, because
identity comes from the outpost's headers, which avoids the Blazor-circuit
sign-in constraint shelly-manager ran into.
### Configuration
Adapters are declared in `appsettings` and selected by name; secrets come
through environment variables, filled by ExternalSecret, never from the file.
```json
"Wiki": {
"NoteStore": { "Active": "gitea", "gitea": { "Repository": "k.renner/notes", "Branch": "main" } },
"Embeddings": {
"Active": "bge-m3",
"Models": {
"bge-m3": { "Adapter": "LlamaCpp", "Url": "http://wiki-embed.ai.svc:8080", "Model": "bge-m3" },
"e5-large": { "Adapter": "Ollama", "Url": "http://ollama.ai.svc:11434", "Model": "multilingual-e5-large" }
}
},
"Judges": {
"Active": "typesafe",
"Models": {
"typesafe": { "Adapter": "TypeSafe", "Url": "https://api.typesafe.ai/v1/systemone", "Model": "jev-latest" },
"jev-local": { "Adapter": "TypeSafe", "Url": "http://jev.ai.svc:8080/v1/systemone", "Model": "jev" } // illustration: same adapter, other endpoint
}
},
"Index": { "Active": "file", "file": { "Path": "/data/index" } },
"AnswerModels": {
"Active": "claude-sonnet",
"Models": {
"claude-sonnet": { "Adapter": "Anthropic", "Model": "claude-sonnet-5" },
"qwen-local": { "Adapter": "OpenAI", "Url": "http://llama-server.ai.svc:8080/v1", "Model": "qwen3.6-35b-a3b" }
}
}
}
```
Every embedding model gets its own index namespace. Switching the active
model is a config change after the new index has been built in the
background; until then both exist and can be compared. Nothing is
re-embedded in place.
## The web interface (MudBlazor)
Blazor Web App, interactive server rendering, hosted in `Wiki.Host`. Static
assets vendored into the image, no CDN. Pages:
1. **Search**: a query box, results ranked by the active index with score and
tags; for a phrased question, optionally the paragraph-level hits from the
judge with their probabilities, shown as quoted evidence.
2. **Note**: frontmatter, rendered body with `[[wikilinks]]` resolved,
backlinks, sources linking to their raw file.
3. **Browse**: by tag, by `updated`, recently changed, orphans.
4. **Answers**: a question box. The answer streams in within seconds: rank,
judge, compose, all in the request. Below it the evidence: the paragraphs
the judge selected with their notes and probabilities, and the `covered`
gate; a low gate is shown as "the wiki does not cover this" instead of an
answer. A save button writes the answer to `output/` with the wiki-synth
frontmatter (`question`, `sources`, `confidence`), so it joins the
labelled set. Underneath, the saved answers, newest first, including the
ones the wiki-synth agent wrote from a session.
5. **Maintenance**: the raw queue with ages, the last consolidation and audit
reports, index and sync status.
6. **Lab**: pick any registered embedding model and judge, run the same query
through them side by side, and run the evaluator on the labelled set to
get recall@k per configuration. This is the page for "play around with
different models".
7. **Settings**: read-only view of the active adapters and their health.
No editing of notes in the browser. The vault stays a git repository edited
by the consolidate agent and by hand.
## API v1
Reads are authenticated by the hostname's gate (humans) or API key
(machines); writes always need an API key so a fact carries its author.
| method and path | purpose |
|---|---|
| `GET /api/v1/status` | index commit and age, counts, active adapters |
| `GET /api/v1/notes?q=&tag=&limit=` | list; `q` ranks through the active index |
| `GET /api/v1/notes/{slug}` | frontmatter, body, links, backlinks, sources |
| `GET /api/v1/tags` | tags with counts |
| `POST /api/v1/rank` | `{question, k, model?}` → ranked slugs; also served at `/rank` for the existing scripts |
| `POST /api/v1/ask` | `{queries, questions, k, judge?, model?}` → context, typed judgements with the answering paragraphs verbatim, `covered`. For agents and code. |
| `POST /api/v1/questions` | `{question, k?, judge?, model?, answerModel?, save?}` → the answer text with `[[slug]]` citations, the evidence (paragraphs, probabilities), `covered`, model and token usage; streamed as server-sent events when requested; with `save`, the `output/` path and commit |
| `POST /api/v1/facts` | deposit a raw source: title, source, `kind: fact`, observed_at, content, tags → path and commit |
| `GET /api/v1/raw` | the distill queue with ages |
| `GET /api/v1/reports` | `output/` documents by type, newest first |
| `POST /api/v1/lab/eval` | run the evaluator for a (model, judge) pair |
| `POST /webhook` | Gitea push, HMAC as today |
`POST /api/v1/facts` is rejected without `source`, with a `kind` other than
`fact`, or with content under a minimum length. The raw rule is enforced by
shape: there is no field for a hypothesis. The file gets the vault's
provenance frontmatter with `captured_by` set from the API key's identity.
`POST /api/v1/ask` is `wiki_ask.py` moved server-side, so the scripts become
thin clients and agents no longer need a vault checkout. Its result is
judgements and quotations, never prose.
`POST /api/v1/questions` is `/ask` plus one answer-model call, in process:
rank the question, build the context, run the default typed questions
(answered, where, covered), hand the selected paragraphs and judgements to
the answer model, stream the prose back. Below the `covered` threshold it
returns the gap instead of an answer. Time is the sum of the parts, about
0.2 s rank, 1 to 4 s judge, a few seconds of generation. Cost is one model
call over a few thousand tokens, a few cents on Sonnet, against a quarter
dollar for a full agent run.
## Auth and network
- Humans: `wiki.renner.uno`, LAN-only (no `dns.renner.uno/public` label),
Ingress backend `ak-outpost-wiki-gate:9000`, a `wiki-gate-egress` CNP in
`authentik`, the proxy-header handler trusts `X-Authentik-*` only from that
outpost.
- Machines: `wiki-index.renner.uno` with the API-key handler; the workstation
key stays where it is. CI reaches the ClusterIP.
- Secrets (Gitea token, webhook secret, API keys, TypeSafe key, Anthropic
key) reach the
pod as environment variables from Kubernetes Secrets filled by
ExternalSecret from the `openbao` ClusterSecretStore. The pod itself has
no OpenBao access. The Python service fetched its secrets from OpenBao
directly through its own Kubernetes-auth role; that role, its policy and
the `ai` → `openbao` CNP entry are retired with it.
- Egress from the pod: Gitea, wiki-embed, `api.typesafe.ai:443`,
`api.anthropic.com:443`. Every
further adapter that talks to a new endpoint gets its own CNP line, which
is the one place adding an adapter touches the cluster.
## Health endpoints
Three endpoints with three different meanings, wired to the .NET health
checks with tags so that the probes and the humans see the right thing:
| endpoint | meaning | probe |
|---|---|---|
| `GET /health` | the process serves HTTP. Always 200 once Kestrel listens; no dependency checks. | liveness |
| `GET /ready` | the active note store has synced at least once and the active index is loaded; 503 while the first sync or index build runs. A failed later sync keeps the last good state and stays ready. | readiness, plus a startup probe with enough budget for the first sync (the Python service was killed mid-build when its `/health` returned 503 during the initial sync) |
| `GET /api/v1/status` | the full picture for humans and the Settings page: index commit and age, counts, queue lengths, and one entry per registered adapter with its last check result and latency. | none |
Adapter checks (Gitea reachable, embeddings endpoint answering, judge
endpoint answering, answer model answering) are reported in `/status` and
on the Settings page and are never part of `/ready`: a judge outage must not
take the reader down. Their failures show up as red entries, not as
restarts.
## Phases
One PR each, verified before the next. The Python `wiki-index` keeps running
until phase 3 proves parity.
0. **Skeleton.** Repository, solution, `Wiki.Core` with the ports and domain
types, `Wiki.Contracts` with the suites, `Wiki.Host` serving `/health`,
`/ready` and `/api/v1/status`. CI: test and release pipelines. Verify:
image in the registry, pipeline green, the Deployment's three probes
wired and the pod Ready.
1. **Parity with the Python service.** Gitea, LlamaCpp and FileIndex
adapters, the sync use case, `/rank` and `/webhook`. Verify: the 40-query
recall eval gives the same numbers as the Python service on the same
commit; `wiki_rank.py --probe-index` against the new service passes.
2. **Read API and UI.** Notes, tags, raw, reports; Search, Note, Browse,
Maintenance pages; `wiki-gate`. Verify: a note renders with resolved
links; an unauthenticated request is redirected to Authentik.
3. **Ask.** Chunker, TypeSafe adapter, `/api/v1/ask`, scripts as clients.
Verify: `test_wiki_ask.py` passes against the API; a wiki-synth run with
the definition unchanged. Then the Python service is retired and the
`wiki-index` hostname points at the new one.
4. **Facts.** API keys as an ExternalSecret, the deposit-fact use case, `/api/v1/facts`. Verify:
a fact posted from the workstation shows as a `claude-bot` commit and in
the next distill.
5. **Questions.** Anthropic adapter, `/api/v1/questions` with streaming,
the Answers page with save. Verify: a question answered in under ten
seconds with citations that resolve to the shown evidence; a question the
wiki does not cover returns the gap; a saved answer appears in `output/`
with the wiki-synth frontmatter and is picked up by the evaluator.
6. **Lab.** Evaluator, a second embedding model and a second answer model
registered, the Lab page. Verify: two models compared on the labelled set
from the UI.
## Not in scope
- Editing notes in the browser.
- Replacing Obsidian; the vault stays a git repository and `notes-push.path`
keeps pushing.
- Public exposure; LAN and VPN only.
- Running or triggering agents. The service makes single model calls inside
a request; it never runs a tool loop or edits the vault. Distill,
consolidate and multi-question synthesis stay Claude processes in CI or in
a session; the service shows their inputs and outputs and serves their API
calls.
-15
View File
@@ -165,21 +165,6 @@ permanent recovery.
---
## Velero restore-test: 80% OSD capacity gate
The monthly Velero restore-test restores into the production Ceph pools. Before
each schedule's restore it projects the restore's raw size (DataUpload
`totalBytes` x the pool's replication/EC factor, spread over all OSDs) onto
current OSD usage from Prometheus. If the fullest OSD would go above **80%**
(`OSD_MAX_PCT` in `restore-test-cronjob.yaml`), that schedule is not restored:
the Job log shows it as **SKIP** with the projected %, and the summary lists it
under "Skipped (backup NOT restore-tested)". A SKIP does not fail the Job, so
read the summary each month. If the projection cannot be made (Prometheus
down, unsized DataUpload), the schedule is not restored and counts as FAIL.
Background: REN-441, REN-442.
---
## CNPG restore-test (verify backups are restorable — non-destructive)
The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
+1 -1
View File
@@ -107,7 +107,7 @@ config, and the nodes are addressed by name.
## Step 1b — boot all three machines from install media
```
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.10/metal-amd64.iso
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
```
Write it to a USB stick and boot each MS-01 from it. Each comes up in
@@ -1,71 +0,0 @@
# Runbook — roll back a Home Assistant image bump
Use when an automatic Home Assistant update (a
`chore(image-automation): bump ghcr.io/home-assistant/home-assistant:<new>`
commit on `master`) breaks Home Assistant. The owner approves the rollback
and its timing. Atlas writes the PR; Argus merges it.
## Why a plain `git revert` is not enough
Image automation (`kubernetes/infrastructure/image-automation/`) re-scans
the registry every hour and rewrites the tag every 30 minutes to the
highest release matching the `home-assistant` ImagePolicy. A plain revert
of the bump commit would be bumped forward again within about an hour.
So the rollback commit does two things at once: it reverts the tag **and**
drops the `# {"$imagepolicy": ...}` marker from that image line. Without
the marker the Setters strategy has nothing to rewrite, so the old tag
stays pinned. It is still one commit, and reverting that commit restores
automation.
## Before you roll back
1. Find the bump commit:
```sh
git log --oneline -5 origin/master -- kubernetes/apps/home-assistant/deployment.yaml
```
2. Check the release notes of the broken version for a **database schema**
change. Home Assistant migrates the recorder schema (CNPG cluster
`ha-pg`) on start and cannot downgrade it. An older version started on a
newer schema runs without recorder history until the schema matches
again. If the bump crossed a schema change, tell the owner before going
on: the choice is roll back without history, or restore the recorder
database from backup as a separate step.
## Roll back (one commit, one PR)
```sh
git worktree add -b atlas/ha-rollback-<old> "$WT" origin/master && cd "$WT"
git revert --no-commit <bump-sha>
# In kubernetes/apps/home-assistant/deployment.yaml, delete the trailing
# # {"$imagepolicy": "flux-system:home-assistant"}
# from the home-assistant image line. Leave every other line alone.
git diff --cached --stat # expect: 1 file, 1 line changed
git commit -m "fix(home-assistant): pin <old>, pause image automation (bump to <new> broke HA)"
git push -u origin HEAD
```
The PR description states: broken version, pinned version, symptom, schema
check result, and "revert this commit to resume automatic updates".
HA-impact review and merge follow [README.md](../README.md#2-ha-impact-review-before-merge).
On merge Flux applies the Deployment (strategy `Recreate`): Home Assistant
restarts once, about one to two minutes of downtime.
## Verify
```sh
kubectl -n home-assistant get deploy home-assistant \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="home-assistant")].image}'
kubectl -n home-assistant rollout status deploy/home-assistant
```
Read-only commands only (see REN-240 protection rule 1). One hour after
merge, confirm no new `chore(image-automation)` commit touched
`deployment.yaml`.
## Resume automatic updates
When a fixed Home Assistant release is out, revert the rollback commit. The
marker comes back, and image automation bumps to the newest release on its
next pass. That bump is itself a Home Assistant restart.
@@ -48,7 +48,7 @@ spec:
# cert when doing server-side OIDC token exchange. Bundle both
# roots into one PEM since NODE_EXTRA_CA_CERTS accepts one file.
- name: fetch-le-staging-roots
image: curlimages/curl:8.22.0@sha256:58adaa4e8dca9c988bae2aba4ab3434a0bb2da16bbe3f92dec39ec7785166777
image: curlimages/curl:8.10.1
command:
- sh
- -c
@@ -70,7 +70,7 @@ spec:
mountPath: /ca
containers:
- name: audiobookshelf
image: ghcr.io/advplyr/audiobookshelf:2.37.1 # {"$imagepolicy": "flux-system:audiobookshelf"}
image: ghcr.io/advplyr/audiobookshelf:2.36.0 # {"$imagepolicy": "flux-system:audiobookshelf"}
ports:
- name: http
containerPort: 80
@@ -122,7 +122,7 @@ spec:
claimName: audiobookshelf-metadata
- name: books
persistentVolumeClaim:
claimName: audiobookshelf-books-bulk
claimName: audiobookshelf-books
- name: le-ca
emptyDir:
medium: Memory
+2 -2
View File
@@ -35,14 +35,14 @@ spec:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: audiobookshelf-books-bulk
name: audiobookshelf-books
namespace: audiobookshelf
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
accessModes:
- ReadWriteMany
storageClassName: ceph-filesystem-bulk
storageClassName: ceph-filesystem
resources:
requests:
storage: 300Gi
@@ -976,280 +976,6 @@ data:
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# wiki-gate: the wiki service (ns ai) behind the same passkey flows as
# llama-gate. The service trusts X-Authentik-Username only from pod
# addresses, and its CNP admits only this outpost, so the gate is the
# whole access control for wiki.renner.uno.
- id: wiki-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: wiki-gate
state: present
attrs:
authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://wiki.ai.svc.cluster.local:8080
internal_host_ssl_validation: false
external_host: https://wiki.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: wiki-gate
state: present
attrs:
name: Wiki
slug: wiki-gate
provider: !KeyOf wiki-gate-provider
meta_launch_url: https://wiki.renner.uno
meta_description: "Notes wiki: search, notes, answers, maintenance"
- model: authentik_outposts.outpost
identifiers:
name: wiki-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf wiki-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
- id: paperclip-users-group
model: authentik_core.group
identifiers:
name: Paperclip Users
state: present
attrs:
is_superuser: false
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: paperclip-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: paperclip-gate
state: present
attrs:
authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://paperclip.paperclip.svc.cluster.local:3100
internal_host_ssl_validation: false
external_host: https://paperclip.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: paperclip-gate
state: present
attrs:
name: Paperclip
slug: paperclip-gate
provider: !KeyOf paperclip-gate-provider
meta_launch_url: https://paperclip.renner.uno
meta_description: "Paperclip agent orchestration"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, paperclip-gate]]
group: !KeyOf paperclip-users-group
order: 0
state: present
attrs:
enabled: true
negate: false
- model: authentik_outposts.outpost
identifiers:
name: paperclip-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf paperclip-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# navidrome-gate: the Navidrome web UI. Navidrome auto-creates the
# X-Authentik-Username user on first login; the first user is admin.
# /rest and /share bypass the gate at the Traefik router.
- id: navidrome-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: navidrome-gate
state: present
attrs:
authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://navidrome.navidrome.svc.cluster.local
internal_host_ssl_validation: false
external_host: https://navidrome.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: navidrome-gate
state: present
attrs:
name: Navidrome
slug: navidrome-gate
provider: !KeyOf navidrome-gate-provider
meta_launch_url: https://navidrome.renner.uno
meta_description: "Music streaming (Subsonic)"
- model: authentik_outposts.outpost
identifiers:
name: navidrome-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf navidrome-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
# lidarr-gate: Lidarr runs with AuthenticationMethod=External, so the gate
# is its only access control.
- id: music-librarians-group
model: authentik_core.group
identifiers:
name: Music Librarians
state: present
attrs:
is_superuser: false
# Membership is owned here; blueprint apply reverts UI edits.
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: lidarr-gate-provider
model: authentik_providers_proxy.proxyprovider
identifiers:
name: lidarr-gate
state: present
attrs:
authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
mode: proxy
internal_host: http://lidarr.navidrome.svc.cluster.local
internal_host_ssl_validation: false
external_host: https://lidarr.renner.uno
cookie_domain: renner.uno
access_token_validity: hours=24
refresh_token_validity: days=30
intercept_header_auth: true
- model: authentik_core.application
identifiers:
slug: lidarr-gate
state: present
attrs:
name: Lidarr
slug: lidarr-gate
provider: !KeyOf lidarr-gate-provider
meta_launch_url: https://lidarr.renner.uno
meta_description: "Music library gaps: discographies, Spotify artists"
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, lidarr-gate]]
group: !KeyOf music-librarians-group
order: 0
state: present
attrs:
enabled: true
negate: false
failure_result: false
- model: authentik_outposts.outpost
identifiers:
name: lidarr-gate
state: present
attrs:
type: proxy
service_connection: !KeyOf k8s-conn
providers:
- !KeyOf lidarr-gate-provider
config:
log_level: info
authentik_host: "http://authentik-server.authentik.svc.cluster.local"
authentik_host_insecure: false
authentik_host_browser: "https://auth.renner.uno/"
refresh_interval: "minutes=5"
container_image: null
object_naming_template: "ak-outpost-%(name)s"
kubernetes_namespace: authentik
kubernetes_replicas: 1
kubernetes_service_type: ClusterIP
kubernetes_disabled_components: ["ingress"]
kubernetes_image_pull_secrets: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_class_name: null
kubernetes_ingress_secret_name: "authentik-outpost-tls"
kubernetes_json_patches: null
- model: authentik_outposts.outpost
identifiers:
name: authentik Embedded Outpost
@@ -2121,74 +1847,6 @@ data:
- url: https://dash.renner.uno/login/generic_oauth
matching_mode: strict
# Matrix (REN-567): upstream login for MAS (account.matrix.renner.uno).
# Git owns this provider, its application and the member list.
# Public client + PKCE, so there is no client secret to share.
# RS256 with the built-in self-signed keypair: a public client has no
# secret for HS256, and MAS verifies the ID token signature.
- id: matrix-users-group
model: authentik_core.group
identifiers:
name: Matrix Users
state: present
attrs:
is_superuser: false
# Every member can sign in to Matrix and gets an account named
# after their Authentik username. Add/remove members HERE.
users:
- !Find [authentik_core.user, [username, k.renner]]
- id: matrix-mas-provider
model: authentik_providers_oauth2.oauth2provider
identifiers:
name: Matrix (MAS)
state: present
attrs:
authentication_flow: !KeyOf passkey-passwordless-flow
authorization_flow: !KeyOf passkey-stepup-flow
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
client_type: public
client_id: matrix-mas
# Authentik 2026.5 checks the grant type against this list, and a
# provider created by a blueprint starts with it empty, so every
# authorize fails with invalid_request ("Invalid grant_type for
# provider"). Providers made in the UI get the defaults.
grant_types:
- authorization_code
- refresh_token
signing_key: !Find [authentik_crypto.certificatekeypair, [name, authentik Self-signed Certificate]]
property_mappings:
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-openid]]
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-profile]]
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-email]]
redirect_uris:
# The path ends in the provider ULID from apps/matrix/mas.yaml.
- url: https://account.matrix.renner.uno/upstream/callback/01M4480KWZGN9HBNB5SSNGT2M1
matching_mode: strict
- model: authentik_core.application
identifiers:
slug: matrix
state: present
attrs:
name: Matrix
slug: matrix
provider: !KeyOf matrix-mas-provider
meta_launch_url: https://element.renner.uno
meta_description: "Matrix chat (Element)"
# Only the Matrix Users group passes.
- model: authentik_policies.policybinding
identifiers:
target: !Find [authentik_core.application, [slug, matrix]]
group: !KeyOf matrix-users-group
order: 0
state: present
attrs:
enabled: true
negate: false
failure_result: false
# Superseded by the single passkey-webauthn-validation stage above.
# Their flowstagebindings are FK-CASCADEd away with them; the
# identification stage's webauthn_stage FK is SET_NULL and is
@@ -7,7 +7,6 @@ resources:
- helmrelease.yaml
- paperless-gate-ingress.yaml
- paperless-gate-egress.yaml
- paperclip-gate-egress.yaml
- vscode-gate-egress.yaml
- shelly-manager-gate-egress.yaml
- esphome-gate-egress.yaml
@@ -17,9 +16,6 @@ resources:
- prometheus-gate-egress.yaml
- wordpress-gate-egress.yaml
- llama-gate-egress.yaml
- wiki-gate-egress.yaml
- navidrome-gate-egress.yaml
- lidarr-gate-egress.yaml
- ldap-outpost-cert.yaml
- ldap-outpost-lb.yaml
- ldap-outpost-cnp.yaml
@@ -40,7 +40,7 @@ metadata:
# so this Service produces no public record — LDAP stays
# LAN-internal, which matches the threat model of the cleartext
# bind on 389.
external-dns.kubernetes.io/hostname: ldap.renner.uno
external-dns.alpha.kubernetes.io/hostname: ldap.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -1,20 +0,0 @@
# Lets the lidarr-gate outpost reach Lidarr; the matching ingress rule is in
# network-policies/navidrome-ns.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: lidarr-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: lidarr-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": navidrome
app.kubernetes.io/name: lidarr
toPorts:
- ports:
- port: "8686"
protocol: TCP
@@ -1,20 +0,0 @@
# Lets the navidrome-gate outpost reach Navidrome; the matching ingress rule
# is in network-policies/navidrome-ns.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: navidrome-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: navidrome-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": navidrome
app.kubernetes.io/name: navidrome
toPorts:
- ports:
- port: "4533"
protocol: TCP
@@ -1,19 +0,0 @@
# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: paperclip-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
app.kubernetes.io/name: paperclip
toPorts:
- ports:
- port: "3100"
protocol: TCP
@@ -1,21 +0,0 @@
# Lets the wiki-gate outpost reach the wiki service. The authentik namespace
# baseline CNP otherwise stops any authentik pod from entering ai. Scoped to
# this outpost's pod label; the matching ingress rule is in ai-ns.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: wiki-gate-egress
namespace: authentik
spec:
endpointSelector:
matchLabels:
goauthentik.io/outpost-name: wiki-gate
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ai
app.kubernetes.io/name: wiki
toPorts:
- ports:
- port: "8080"
protocol: TCP
+1 -29
View File
@@ -5,11 +5,6 @@ metadata:
namespace: evcc
labels:
app.kubernetes.io/name: evcc
annotations:
# The MQTT client certificate reaches evcc as env (below), which does not
# follow a renewal. cert-manager renews it every ~60 days; Reloader then
# restarts evcc so it picks up the new PEMs before the old ones expire.
secret.reloader.stakater.com/reload: mqtt-evcc-client-tls
spec:
replicas: 1
# RWO PVC (`evcc-data`, ceph-block) can't be mounted twice, so the
@@ -28,36 +23,13 @@ spec:
enableServiceLinks: false
containers:
- name: evcc
image: evcc/evcc:0.316.2@sha256:070c8a8912c5107e8424cef60bf8c3d3cac0eb32aa28d1b021d226545f0aac7b
image: evcc/evcc:0.310.1
# evcc auto-discovers /etc/evcc.yaml on its default search path;
# the file is delivered (with the MQTT password) via the
# sops-encrypted evcc-config Secret mounted below.
env:
- name: TZ
value: Europe/Berlin
# MQTT by client certificate (register account evcc-client,
# REN-953) on the mTLS listener. These override the mqtt block of
# evcc.yaml (viper AutomaticEnv, checked against 0.316.2); the
# file's user/password are ignored there, as the broker takes the
# username from the certificate CN. evcc wants the PEM text, not
# a path.
- name: EVCC_MQTT_BROKER
value: tls://mosquitto.home-assistant.svc.cluster.local:8884
- name: EVCC_MQTT_CACERT
valueFrom:
secretKeyRef:
name: mqtt-evcc-client-tls
key: ca.crt
- name: EVCC_MQTT_CLIENTCERT
valueFrom:
secretKeyRef:
name: mqtt-evcc-client-tls
key: tls.crt
- name: EVCC_MQTT_CLIENTKEY
valueFrom:
secretKeyRef:
name: mqtt-evcc-client-tls
key: tls.key
ports:
- name: http
containerPort: 7070
@@ -3,4 +3,3 @@ kind: Kustomization
resources:
- externalsecret.yaml
- helmrelease.yaml
- networkpolicy.yaml
@@ -1,22 +0,0 @@
# The apps namespace is default-deny-ingress, so the ServiceMonitor added in
# monitoring-rules/external-dns-servicemonitor.yaml scraped into a timeout.
# The selector matches both releases — the Cloudflare one deployed here and
# external-dns-unifi from infrastructure/ — since they share the chart's
# app.kubernetes.io/name and land in the same namespace.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: external-dns-allow-metrics-scrape
namespace: apps
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: external-dns
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: "7979"
protocol: TCP
-75
View File
@@ -30,42 +30,7 @@ metadata:
# Web Service 9 = Music Assistant
# Web Service 10 = TRMNL
# Web Service 11 = Paperless (gate-level only, see its entry)
# Web Service 12 = Element Web
# Web Service 13 = Matrix (Synapse client API)
# Web Service 14 = Matrix Authentication Service
#
# CORRECTION (REN-21, 2026-09-25) to the Web Service 11 comment further down,
# which claims that an in-cluster probe of the paperless ClusterIP "needs a
# CiliumNetworkPolicy first — the monitoring probes get away without one only
# because their targets share a namespace with the prober".
#
# That is wrong, and it was wrong when written. It is stated here rather than
# fixed in place because the sentence lives inside the `config.yaml: |` block
# scalar below: any edit to it changes ConfigMap DATA, and
# `configmap.reloader.stakater.com/reload: gatus-config`
# (deployment.yaml#reloader) would then roll the gatus Deployment — a
# production restart for a comment. This header is outside `data:`, so it is
# inert; the served config is byte-identical.
#
# * When written, the paperless baseline admitted the whole monitoring
# namespace on every port, and monitoring is where blackbox-exporter runs.
# That admit is gone (REN-28); the probe is now admitted by the dedicated
# ../../infrastructure/network-policies/paperless-ns.yaml#paperless-allow-blackbox,
# scoped to blackbox-exporter on the paperless pod's :8000. Cilium unions
# the allow rules of every CNP selecting an endpoint, so nothing else is
# required.
# * The in-cluster paperless probe therefore exists:
# ../monitoring-rules/probes-blackbox.yaml#paperless-app-health, target
# http://paperless.paperless.svc.cluster.local:80/accounts/login/
# * The claim does NOT generalise to gatus itself. Gatus runs in `apps`,
# which is a different source namespace; a gatus-sourced in-cluster check
# would need its own allow, exactly as
# cnp-cross-namespace-allow-apps-gatus-to-monitored-services describes.
#
# The Web Service 11 entry below stays a reachability-only check on purpose.
# The app-health half of the pair is the blackbox probe named above.
data:
# anchor: gatus-endpoints
config.yaml: |
storage:
type: sqlite
@@ -231,46 +196,6 @@ data:
conditions:
- "[STATUS] == any(200, 301, 302)"
# Matrix (REN-663): Element Web, the Synapse client API and MAS.
- name: Web Service 12
group: services
url: https://element.renner.uno/config.json
interval: 60s
ui:
hide-hostname: true
hide-url: true
hide-port: true
hide-conditions: true
hide-errors: true
conditions:
- "[STATUS] == 200"
- name: Web Service 13
group: services
url: https://matrix.renner.uno/_matrix/client/versions
interval: 60s
ui:
hide-hostname: true
hide-url: true
hide-port: true
hide-conditions: true
hide-errors: true
conditions:
- "[STATUS] == 200"
- name: Web Service 14
group: services
url: https://account.matrix.renner.uno/.well-known/openid-configuration
interval: 60s
ui:
hide-hostname: true
hide-url: true
hide-port: true
hide-conditions: true
hide-errors: true
conditions:
- "[STATUS] == 200"
# Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
# InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
# no longer deployed. Re-add when/if they come back.
+1 -2
View File
@@ -6,7 +6,6 @@ metadata:
labels:
app: gatus
annotations:
# anchor: reloader
# Gatus reads its config once at startup and never re-reads the file, so
# without this a ConfigMap change reconciles cleanly and then does nothing:
# git and the running checks drift apart silently, with no failed resource
@@ -47,7 +46,7 @@ spec:
type: RuntimeDefault
containers:
- name: gatus
image: twinproduction/gatus:v5.37.0 # {"$imagepolicy": "flux-system:gatus"}
image: twinproduction/gatus:v5.36.0 # {"$imagepolicy": "flux-system:gatus"}
securityContext:
allowPrivilegeEscalation: false
capabilities:
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gitea
namespace: gitea
namespace: apps
labels:
app.kubernetes.io/name: gitea
annotations:
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: gitea-ssh
namespace: gitea
namespace: apps
labels:
app.kubernetes.io/name: gitea
spec:
+1 -2
View File
@@ -6,8 +6,7 @@ kind: Kustomization
# and verified. See MIGRATION-gitea-woodpecker.md.
resources:
- statefulset.yaml
# The live size of the data volume; see the file (REN-583).
- pvc.yaml
- pdb.yaml
- service.yaml
- ingressroute-ssh.yaml
- ingress.yaml
+26 -6
View File
@@ -1,8 +1,28 @@
# Prometheus scrape and Traefik are admitted by the namespace baseline in
# infrastructure/network-policies/gitea-ns.yaml.
#
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-metrics-scrape
namespace: apps
spec:
# Prometheus (monitoring ns) scrapes gitea's /metrics on :3000. The apps
# default-deny-ingress only allows same-ns / home-assistant / host, so the
# scrape is dropped (context deadline exceeded). This became necessary only
# after the LXC->in-cluster cutover — the old static target was external, so
# no in-cluster policy applied. Scoped to the gitea pod + :3000.
endpointSelector:
matchLabels:
app.kubernetes.io/name: gitea
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
toPorts:
- ports:
- port: "3000"
protocol: TCP
---
# Secret rotation: the flux-git rotator (openbao ns) reaches gitea to mint a new
# flux PAT. It uses the in-cluster Service (gitea.gitea.svc:3000) — the public
# flux PAT. It uses the in-cluster Service (gitea.apps.svc:3000) — the public
# gitea.renner.uno name hairpins off the ingress LB from a pod (same failure as
# pve.renner.uno). Scoped to the gitea pod :3000 and the rotator label; the
# egress half is rotator-gitea-flux-egress in network-policies/openbao-ns.yaml.
@@ -10,7 +30,7 @@ apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-flux-rotator
namespace: gitea
namespace: apps
spec:
endpointSelector:
matchLabels:
@@ -34,7 +54,7 @@ apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-review-bot-rotator
namespace: gitea
namespace: apps
spec:
endpointSelector:
matchLabels:
+23
View File
@@ -0,0 +1,23 @@
---
# Eviction protection for the single-replica Gitea StatefulSet.
#
# Gitea runs on SQLite (single writer) and its repos + DB live on a single
# ReadWriteOnce PVC, so it is deliberately replicas:1 and cannot fail over.
# With no PDB, a node drain / maintenance evicts the pod abruptly, mid-write,
# with zero coordination. minAvailable:1 on a 1-replica workload makes the
# eviction API refuse *voluntary* evictions: `kubectl drain` blocks and
# surfaces this pod, forcing a conscious decision (drain with
# --disable-eviction, or scale down first) instead of yanking the live DB.
# It does NOT block involuntary disruptions (node crash, OOM).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: gitea
namespace: apps
labels:
app.kubernetes.io/name: gitea
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: gitea
-39
View File
@@ -1,39 +0,0 @@
# The live data volume of gitea-0, owned here so its size is set in git
# (REN-583). The StatefulSet controller created this claim from the
# volumeClaimTemplates in statefulset.yaml, but that template is immutable on
# an existing StatefulSet, so changing the size there cannot be applied. A
# PVC's resources.requests.storage CAN grow in place (ceph-block has
# allowVolumeExpansion: true; ceph-csi grows the RBD image and the filesystem
# online, no pod restart). Flux applies this over the existing claim with
# server-side apply and the resize follows.
#
# - Name, namespace, accessModes, storageClassName and volumeMode must match
# the live claim exactly: they are immutable, a mismatch fails the whole
# gitea Kustomization (no change to the live claim).
# - volumeName must stay: kustomize-controller has owned spec.volumeName
# since the namespace move pre-created this claim (eb23d3ac, #878).
# Without it, server-side apply tries to clear the field and the API server
# rejects the whole claim as an immutable-spec change.
# - Grow only. A smaller value is rejected by the API server.
# - prune: disabled, so removing this file from git never deletes the claim
# and its data. To hand ownership back, delete the file; the claim stays.
#
# Size history: 20Gi -> 40Gi 2026-07-22 (#336, live resize by hand);
# 40Gi -> 60Gi 2026-10-03 (REN-583, after the volume filled again on REN-580).
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-gitea-0
namespace: gitea
labels:
app.kubernetes.io/name: gitea
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: ceph-block
volumeMode: Filesystem
volumeName: pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be
resources:
requests:
storage: 60Gi
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: gitea
namespace: gitea
namespace: apps
labels:
app.kubernetes.io/name: gitea
spec:
+11 -7
View File
@@ -2,7 +2,7 @@ apiVersion: apps/v1
kind: StatefulSet
metadata:
name: gitea
namespace: gitea
namespace: apps
labels:
app.kubernetes.io/name: gitea
spec:
@@ -37,7 +37,7 @@ spec:
# both handled as explicit steps in MIGRATION-gitea-woodpecker.md. Pinned
# to match the LXC for a clean SQLite lift; no imagepolicy marker (an
# auto-bump across versions on live SQLite data must never be applied).
image: gitea/gitea:1.21.4-rootless@sha256:600f1f1e1d090c763866db16882b5eac5c6a80b7664ee53bec0806c656c0033f
image: gitea/gitea:1.21.4-rootless
env:
- name: GITEA_WORK_DIR
value: /var/lib/gitea
@@ -147,10 +147,14 @@ spec:
accessModes: ["ReadWriteOnce"]
storageClassName: ceph-block
resources:
# This template only applies when the claim does not exist yet. The
# live size of data-gitea-0 is set in pvc.yaml (REN-583), not here:
# volumeClaimTemplates is immutable on an existing StatefulSet, so a
# change to this value cannot be applied and Flux would reject the
# whole object. Leave it at 40Gi.
# Expanded 20Gi -> 40Gi on 2026-07-22 (ceph-block online resize) to
# clear a 100%-full outage (#336). The live PVC was resized directly.
# NOTE: volumeClaimTemplates is immutable on an existing StatefulSet,
# so bumping this value cannot be `kubectl apply`-ed in place — Flux
# will reject the whole object. Landing this size change requires a
# one-time `kubectl delete statefulset gitea -n apps --cascade=orphan`
# (pod + PVC keep running) so Flux re-creates the STS from this
# template and re-adopts them. After the cleanup_packages GC reclaims
# the ~10G of orphaned blobs, real usage is ~9G, so 40Gi is headroom.
requests:
storage: 40Gi
+1 -3
View File
@@ -4,9 +4,7 @@ metadata:
name: gf-pg
namespace: monitoring
spec:
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
@@ -0,0 +1,15 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: home-assistant-backup-password
namespace: home-assistant
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: home-assistant-backup-password # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: apps/home-assistant/backup-password
@@ -18,7 +18,7 @@ metadata:
lb-pool: bgp
annotations:
lbipam.cilium.io/ips: "10.2.99.87,fd99::87"
external-dns.kubernetes.io/hostname: ha.renner.uno
external-dns.alpha.kubernetes.io/hostname: ha.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -60,7 +60,7 @@ spec:
# repo's configuration.yaml reads this key, so HA does not start
# without it. Removing this init container takes the recorder down.
- name: render-db-url
image: busybox:1.38@sha256:fd7dc98638c8e305f4dc34e979f1c0fdfdcaeb0fbf8fcff77ae834b6da3d7e6e
image: busybox:1.36
command:
- sh
- -c
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.9.4 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
@@ -159,7 +159,7 @@ spec:
# substitution to POD_IP so nginx-runtime variables like $host
# / $http_x_forwarded_for in the template stay intact.
- name: proxy
image: nginxinc/nginx-unprivileged:1.31-alpine@sha256:26b0bf6fbf07297983cb341998d79c831508787de26627dd2a112321b9c3a4af
image: nginxinc/nginx-unprivileged:1.27-alpine
env:
- name: POD_IP
valueFrom:
@@ -193,7 +193,7 @@ spec:
# any LAN client can edit, matching the user's trust-the-network
# model. Runs as root so its writes match HA's root-owned files.
- name: code-server
image: codercom/code-server:4.140.0@sha256:1c9ad972a78450f430155144110da4c02d3627c72a598e9197e2781fb9f92e28
image: codercom/code-server:4.118.0
args:
- "--auth=none"
- "--bind-addr=0.0.0.0:8080"
@@ -253,7 +253,7 @@ spec:
# in the PVC's remote URL; reload auth uses a dedicated long-lived token.
# Runs as root to match HA's root-owned /config files. See config-sync.sh.
- name: config-sync
image: alpine/git:2.54.0@sha256:a4bb51f1a3553df194ce679fc1db721d8bfba2046fa1a88fe9d4ac551ffbce25
image: alpine/git:2.54.0
command: ["/bin/sh", "/scripts/sync.sh"]
env:
- name: SYNC_BRANCH
@@ -36,7 +36,7 @@ spec:
type: RuntimeDefault
containers:
- name: esphome
image: ghcr.io/esphome/esphome:2026.9.1@sha256:776a845d508909bf426d2ee7bdd81796361ebe6a7f091807ce1d6d0b48985413
image: ghcr.io/esphome/esphome:2026.6.2
imagePullPolicy: IfNotPresent
# Default entrypoint runs the dashboard (Device Builder) against the
# mounted /config — no command override needed.
@@ -30,71 +30,16 @@ metadata:
name: ha-pg
namespace: home-assistant
spec:
instances: 2
# switchover, not CNPG's default `restart`: any change that rolls the pods
# (walStorage below did) then promotes the already-rolled replica instead of
# restarting the primary in place, so the recorder loses seconds of writes
# rather than stopDelay plus a pg_wal move plus startup (REN-176).
primaryUpdateMethod: switchover
smartShutdownTimeout: 10
stopDelay: 60
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
# Sized for WAL headroom during an archiving stall, not for table growth.
# The data itself needs far less: migrated statistics 10.8M rows +
# statistics_short_term 1.9M (~1.8 GB with indexes in Postgres) and ~18M
# states rows for the purge_keep_days:7 window (~4 GB).
#
# 20Gi left only 6.63 GiB free, and that is the number that matters here.
# WAL archiving stalled for 10 minutes and the un-archived backlog reached
# 1.84 GiB — 28% of that free space (REN-129).
#
# That stall was first blamed on Velero repository maintenance. It is not
# the cause: REN-136 measured seven nights and the archiver's sustained
# drain (~14 segments/min, its serial ceiling) is flat whether Kopia is
# running or not, and Kopia's nearest batch drifts across 02:03-02:24Z
# while the burst stays pinned to 02:13Z. What is real is a nightly
# producer burst — 56-119 segments queued at 02:13-02:15 UTC on all seven
# nights, against a steady-state queue of 0. Postgres cannot recycle a WAL
# segment until the archiver has shipped it, so a burst that arrives while
# archiving is stalled walks the volume toward full, and a full PGDATA
# stops the recorder outright.
#
# 32Gi takes free space to ~18.6 GiB, which puts the same 1.84 GiB backlog
# at 10% of headroom rather than 28%. In REN-136's terms: at 6.36 GiB free
# one stalled night survives and two fill the volume; ~18.6 GiB carries
# about four. Approved by the owner on REN-135.
# ceph-block has allowVolumeExpansion: true, so this grows in place.
#
# Since REN-176 WAL lives on walStorage below, so the stall headroom above
# now belongs to that volume. 32Gi stays because a PVC cannot shrink.
size: 32Gi
# WAL on its own volume (ha-pg-N-wal), approved by the owner on REN-176.
#
# PERMANENT. The CNPG webhook rejects removing walStorage once set ("walStorage
# cannot be disabled once configured"), so reverting this block fails
# admission and leaves the home-assistant Kustomization stuck. Undoing it
# means a new Cluster recovered from ha-pg-store. Adding it on 1.30.1 needs
# no re-clone: the operator creates the PVC, rolls each pod, and the instance
# manager moves pg_wal onto it before Postgres starts.
#
# This does NOT keep the recorder up when WAL fills: a full WAL disk still
# stops Postgres. What it does is keep WAL writable if PGDATA fills (a purge
# that stops running), and make WAL-volume usage a direct archiver-backlog
# signal.
#
# Sized so stall survival is no worse than the shared 32Gi gives today.
# Today: ~18.6 GiB free on PGDATA / 4.66 GiB/day of WAL = ~4.0 days.
# 1.86 GiB normal pg_wal footprint (max_wal_size 2GB)
# + 18.64 GiB four days of full archiver outage at 4.66 GiB/day
# (the nightly 56-119 segment burst, REN-136, is inside that rate)
# = 20.5 GiB -> 21Gi: (21 - 1.86) / 4.66 = ~4.1 days. 18Gi would give ~3.5,
# worse than today. RBD is thin, so the ceiling costs nothing until a stall
# fills it. The webhook requires min_wal_size < max_wal_size < this size.
walStorage:
storageClass: ceph-block
size: 21Gi
# Sized for the migrated statistics plus 7 days of states at the current
# rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with
# indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window
# (~4 GB). 20Gi leaves room for autovacuum bloat and index churn.
# ceph-block has allowVolumeExpansion: true, so this can grow in place.
size: 20Gi
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
@@ -179,7 +124,8 @@ spec:
# padding rather than data.
#
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
# its volumes are lost outright.
# its volume are lost outright. This is a single-instance cluster, so the
# archive is the only off-node copy.
archive_timeout: "30min"
# Fewer checkpoints => fewer full-page writes, since a page is only
# fully-logged on its first modification after a checkpoint.
@@ -1,6 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- backup-password-externalsecret.yaml
- pvcs.yaml
- deployment.yaml
- service.yaml
@@ -12,13 +13,9 @@ resources:
- prometheus-token-externalsecret.yaml
- config-sync-token-externalsecret.yaml
- mosquitto-bootstrap.yaml
# Former `health` probe credential. The broker no longer reads it (the
# probes use the cert account mosquitto-probe, REN-953); removed with its
# rotator in the REN-802 cleanup step.
# health broker credential: OpenBao-backed + Class-A rotated (first of the
# eight users to migrate off mosquitto-bootstrap.yaml).
- mosquitto-health-credentials-externalsecret.yaml
# open3e-int broker credential: open3e-esp32 gateway on the Vitocal internal
# CAN; OpenBao-backed + Class-A rotated.
- mosquitto-open3e-int-credentials-externalsecret.yaml
- mosquitto-config.yaml
- mosquitto-certificate.yaml
- mosquitto-service.yaml
@@ -69,12 +66,6 @@ resources:
- ha-pg-objectstore.yaml
- ha-pg-cluster.yaml
- ha-pg-scheduled-backup.yaml
# MQTT credential management (REN-802): the account controller manages the
# production broker's Dynamic Security (REN-900). mosquitto-dynsec-test is
# the step-1 test broker, no longer managed; removal follows separately.
# Register: mqtt-accounts.yaml (configMapGenerator).
- mosquitto-dynsec-test.yaml
- mqtt-account-controller.yaml
configMapGenerator:
# nginx config for the reverse-proxy sidecar. Hash suffix disabled so
# the cm name stays stable; editing proxy.conf does not auto-roll the
@@ -110,28 +101,5 @@ configMapGenerator:
namespace: home-assistant
files:
- guard.sh=zigbee2mqtt-discovery-guard.sh
# Z2M external converters, copied into /app/data/external_converters by the
# seed-data init container on every start. Hash suffix disabled (name stays
# stable), so editing a converter does not auto-roll the pod — restart
# manually with `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`.
- name: zigbee2mqtt-external-converters
namespace: home-assistant
files:
- wetterstation-ws90.js=zigbee2mqtt-ws90-converter.js
# MQTT account register, read by mqtt-account-controller on every loop.
# Hash suffix disabled (name stays stable); the volume is not a subPath
# mount, so an edit reaches the running controller without a restart.
- name: mqtt-accounts
namespace: home-assistant
files:
- mqtt-accounts.yaml
# shelly-manager api-proxy sidecar config (REN-935). Hash suffix disabled
# (name stays stable), so editing shelly-manager-api-proxy.conf does not
# auto-roll the pod — restart manually with
# `kubectl rollout restart deploy/shelly-manager -n home-assistant`.
- name: shelly-manager-api-proxy
namespace: home-assistant
files:
- api.conf.template=shelly-manager-api-proxy.conf
generatorOptions:
disableNameSuffixHash: true
@@ -42,9 +42,6 @@ spec:
labels:
app.kubernetes.io/name: mdns-relay
annotations:
# Bump to restart the pod and re-attach net1 (DHCP via multus).
# 2026-10-06: first attach on containernetworking/plugins v1.9.1.
homelab.renner.uno/restarted-for: "2026-10-06 cni-plugins v1.9.1 attach check"
k8s.v1.cni.cncf.io/networks: mdns-relay-iot
spec:
enableServiceLinks: false
@@ -85,7 +82,7 @@ spec:
# deliver, and the ClusterIP has no backend while HA restarts. The
# trap forwards SIGTERM so the pod still terminates promptly.
- name: socat-ipv4
image: alpine/socat:1.8.1.3@sha256:82ad20f6f6e29b91ff33b6662d24063522b1378f08fd8569cd3ab412cac13f50
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
imagePullPolicy: IfNotPresent
command:
- /bin/sh
@@ -127,7 +124,7 @@ spec:
# depends on the kernel's bindv6only default rather than on anything
# stated here, so the roles are made explicit instead of inherited.
- name: socat-ipv6
image: alpine/socat:1.8.1.3@sha256:82ad20f6f6e29b91ff33b6662d24063522b1378f08fd8569cd3ab412cac13f50
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
imagePullPolicy: IfNotPresent
command:
- /bin/sh
@@ -44,10 +44,6 @@ kind: Job
metadata:
name: mosquitto-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -59,7 +55,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c
@@ -116,10 +112,6 @@ kind: Job
metadata:
name: mosquitto-admin-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -131,7 +123,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c
@@ -187,10 +179,6 @@ kind: Job
metadata:
name: mosquitto-shellies-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -202,7 +190,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c
@@ -257,10 +245,6 @@ kind: Job
metadata:
name: mosquitto-sml2mqtt-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -272,7 +256,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c
@@ -329,10 +313,6 @@ kind: Job
metadata:
name: mosquitto-open3e-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -344,7 +324,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c
@@ -403,10 +383,6 @@ kind: Job
metadata:
name: mosquitto-zigbee2mqtt-credentials-bootstrap
namespace: home-assistant
annotations:
# Job pod templates are immutable; force-recreate so image bumps
# apply on Flux re-reconcile. Safe: the Job is idempotent (see above).
kustomize.toolkit.fluxcd.io/force: enabled
spec:
backoffLimit: 4
template:
@@ -418,7 +394,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: bootstrap
image: alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
image: alpine:3.21
command:
- sh
- -c

Some files were not shown because too many files have changed in this diff Show More