fix: [alert] HomeAssistantDown (home-assistant) (#1081)

Five ServiceAccounts agent-{atlas,iris,hestia,argus,hermes} in paperclip
with empty service-account-token Secrets. The four readers get the same
rights they hold today via the shared pod SA: `view` cluster-wide and the
existing monitoring/paperclip-prometheus-read Role. Hermes and the shared
SA get `get` on the five named token Secrets only.

CiliumNetworkPolicy paperclip-renner/paperclip-agents-kube-apiserver lets
the five agents' pods reach the kube-apiserver entity on 6443.

Spec: REN-412. Nothing restarts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.woodpecker/build-paperclip.yml

@@ -0,0 +1,69 @@
+# Paperclip server image (REN-401): upstream paperclipai/paperclip at a pinned
+# master commit plus the patches in ci/images/paperclip/. Needed while the k8s
+# sandbox fix (upstream PR #13733) is not merged; see the README there for how
+# to go back to upstream nightly.
+#
+# Own workflow for the reason given in build-agent-runtime.yml: a step in
+# build-ci-images.yml runs on every change to that file and competes for the
+# woodpecker-ci-quota. Re-run alone with a deployment to target
+# `build-paperclip`.
+#
+# Tag `build.<unix-ts>` is what the Flux ImagePolicy `paperclip` sorts on
+# (image-automation/paperclip.yaml); image automation then rolls it out.
+# konairius/ and ci_images_token as for the agent-runtime images.
+when:
+  - event: push
+    branch: master
+    path:
+      - "ci/images/paperclip/**"
+      - ".woodpecker/build-paperclip.yml"
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-paperclip"'
+
+steps:
+  - name: prepare-source
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
+    commands:
+      - sh ci/images/paperclip/prepare.sh build/paperclip-src
+
+  - name: build-paperclip
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: build/paperclip-src
+    depends_on: [prepare-source]
+    # Rust runner + UI + server build in one pod. The server build alone sets
+    # a 4 GiB Node heap, so the memory limit is the LimitRange max (8Gi).
+    backend_options:
+      kubernetes:
+        resources:
+          requests:
+            cpu: "1"
+            memory: 2Gi
+          limits:
+            cpu: "4"
+            memory: 8Gi
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - . ../../ci/images/paperclip/upstream.env
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=build.$(date +%s)
+      # Same target and build args as upstream's docker.yml for nightly.
+      # --compressed-caching=false: pipeline 11717 was killed at the 8Gi limit
+      # (exit 137) in the server build; kaniko otherwise holds compressed
+      # layer tarballs in memory next to the 4 GiB Node heap.
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Dockerfile.kaniko \
+          --target production \
+          --skip-unused-stages \
+          --snapshot-mode=redo \
+          --compressed-caching=false \
+          --build-arg PAPERCLIP_BUILD_COMMIT=$PAPERCLIP_COMMIT \
+          --build-arg PAPERCLIP_BUILD_VERSION=$PAPERCLIP_VERSION \
+          --build-arg CLI_TOOLS_CACHE_EPOCH=$(date -u +%G-W%V) \
+          --destination gitea.renner.uno/konairius/paperclip:$TAG

ci/images/paperclip/README.md

@@ -0,0 +1,55 @@
+# Paperclip server image (REN-401)
+
+Upstream `paperclipai/paperclip` at the commit in `upstream.env`, plus every
+`*.patch` in this directory. Built by `.woodpecker/build-paperclip.yml`,
+pushed as `gitea.renner.uno/konairius/paperclip:build.<unix-ts>`, rolled out
+by Flux image automation (`kubernetes/infrastructure/image-automation/paperclip.yaml`).
+
+## Why
+
+k8s sandbox runs fail: the run path never sets the lease `remoteCwd`, so file
+sync into the pod fails (paperclipai/paperclip#13864). The fix is
+paperclipai/paperclip#13733, open and not merged.
+`pr-13733-lease-remote-cwd.patch` is that PR's diff (head `5f320bf6`).
+
+#13733 covers only the run path. Adapter environment tests (binding an agent
+to a k8s environment) sync managed config before any workspace is realized,
+so the lease still has no `remoteCwd` and the Claude test fails with
+`claude_managed_config_dir_failed` (REN-409). The fix is
+paperclipai/paperclip#14769, also open: the plugin sets `remoteCwd:
+/workspace` when it acquires or resumes a lease.
+`pr-14769-probe-remote-cwd.patch` is that PR's diff (head `10dc7371`).
+
+## Files
+
+- `upstream.env` — pinned commit and the version string stamped into the
+  image. Bump both together.
+- `prepare.sh` — fetches the commit, applies the patches, writes
+  `Dockerfile.kaniko` (upstream's `COPY --parents` line split into plain
+  `COPY` lines; kaniko does not support `--parents`), and adds the build of
+  the Kubernetes sandbox provider plugin to the `build` stage (REN-405):
+  upstream's `production` target ships only its source, so the plugin fails
+  with "no longer exposes a Paperclip manifest". The build fails if
+  `dist/manifest.js` is missing.
+- `*.patch` — applied in name order with `git apply`.
+
+## Moving to a newer upstream commit
+
+Set `PAPERCLIP_COMMIT` to the new master commit and `PAPERCLIP_VERSION` to
+`git describe --tags --match 'v*' --long <commit>`, then run
+`sh prepare.sh /tmp/src`: it fails if a patch no longer applies.
+
+## Exit: when #13733 and #14769 are merged upstream
+
+Renovate cannot watch an upstream PR, so this is a manual step. Once
+upstream nightly contains both fixes:
+
+1. `kubernetes/infrastructure/image-automation/paperclip.yaml`: back to
+   `ghcr.io/paperclipai/paperclip` with `filterTags.pattern: '^nightly$'`
+   (the version before REN-401 in git history).
+2. `renovate.json`: swap `gitea.renner.uno/konairius/paperclip` back to
+   `ghcr.io/paperclipai/paperclip` in the image-automation exclusion list.
+3. Delete this directory and `.woodpecker/build-paperclip.yml`.
+
+Image automation then rewrites `kubernetes/apps/paperclip/paperclip.yaml`
+to the nightly digest by itself.

ci/images/paperclip/pr-13733-lease-remote-cwd.patch

@@ -0,0 +1,93 @@
+diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts
+index f3d1fc659..812cae357 100644
+--- a/server/src/__tests__/environment-run-orchestrator.test.ts
++++ b/server/src/__tests__/environment-run-orchestrator.test.ts
+@@ -430,6 +430,51 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
+     expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
+   });
+ 
++  it("persists the realized cwd as remoteCwd when realization returns no workspaceRealization payload", async () => {
++    const runtime = makeMockRuntime({
++      realizeWorkspace: vi.fn().mockResolvedValue({
++        cwd: "/workspace",
++        metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
++      }),
++    } as Partial<EnvironmentRuntimeService>);
++    mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
++
++    const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
++
++    await orchestrator.realizeForRun(
++      makeRealizeInput({ environment: makeEnvironment("sandbox") }),
++    );
++
++    expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
++      "lease-1",
++      expect.objectContaining({ remoteCwd: "/workspace" }),
++    );
++    // Nothing to record about the realization shape, so the execution workspace
++    // row is left untouched.
++    expect(mockUpdateExecutionWorkspace).not.toHaveBeenCalled();
++  });
++
++  it("does not rewrite lease metadata when the realized cwd already matches the lease", async () => {
++    const runtime = makeMockRuntime({
++      realizeWorkspace: vi.fn().mockResolvedValue({
++        cwd: "/workspace",
++        metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
++      }),
++    } as Partial<EnvironmentRuntimeService>);
++    mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
++
++    const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
++
++    await orchestrator.realizeForRun(
++      makeRealizeInput({
++        environment: makeEnvironment("sandbox"),
++        lease: makeLease({ metadata: { remoteCwd: "/workspace" } }),
++      }),
++    );
++
++    expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
++  });
++
+   it("runs a remote provision command after workspace realization when configured", async () => {
+     mockBuildWorkspaceRealizationRequest.mockReturnValue({
+       version: 1,
+diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts
+index 97cca667e..e76f64913 100644
+--- a/server/src/services/environment-run-orchestrator.ts
++++ b/server/src/services/environment-run-orchestrator.ts
+@@ -485,16 +485,29 @@ export function environmentRunOrchestrator(
+     }
+ 
+     // Step 3: Persist realization metadata on lease and execution workspace
+-    if (Object.keys(workspaceRealization).length > 0) {
++    const hasWorkspaceRealization = Object.keys(workspaceRealization).length > 0;
++    // A driver that realizes the workspace reports the directory it prepared
++    // through the realization `cwd`. Persist it on the lease as `remoteCwd`, so
++    // every later step that receives only the lease — native file sync, a
++    // resumed lease, a second run against the same lease — confines itself to
++    // the same root. Without it a driver whose realization returns no
++    // `workspaceRealization` payload leaves the lease without a root at all,
++    // and the sync step fails even though realization succeeded.
++    const currentRemoteCwd =
++      typeof lease.metadata?.remoteCwd === "string" ? lease.metadata.remoteCwd.trim() : "";
++    const realizedRemoteCwd =
++      realizedWorkspaceCwd && realizedWorkspaceCwd !== currentRemoteCwd ? realizedWorkspaceCwd : null;
++    if (hasWorkspaceRealization || realizedRemoteCwd) {
+       const nextLeaseMetadata = {
+         ...(lease.metadata ?? {}),
+-        workspaceRealization,
++        ...(hasWorkspaceRealization ? { workspaceRealization } : {}),
++        ...(realizedRemoteCwd ? { remoteCwd: realizedRemoteCwd } : {}),
+       };
+       const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
+       if (updatedLease) {
+         lease = updatedLease;
+       }
+-      if (persistedExecutionWorkspace) {
++      if (hasWorkspaceRealization && persistedExecutionWorkspace) {
+         const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
+           metadata: {
+             ...(persistedExecutionWorkspace.metadata ?? {}),

ci/images/paperclip/pr-14769-probe-remote-cwd.patch

@@ -0,0 +1,86 @@
+diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md
+index a51c6ecee60e..72618cb11296 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/README.md
++++ b/packages/plugins/sandbox-providers/kubernetes/README.md
+@@ -140,6 +140,8 @@ Every agent pod is:
+ 
+ Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs).
+ 
++The lease starts with `/workspace` as the native file-sync confinement root. Adapter environment tests can upload managed config before the server realizes a workspace. When resuming a lease, the provider keeps a previously recorded `remoteCwd` so a custom realized workspace is not replaced with the default.
++
+ The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Job, so a single `kubectl delete job …` cascades cleanly to the Pod and Secret.
+ 
+ ## Optional Kata-FC microVM isolation
+diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+index 8ae47de8ecac..98016385da05 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
++++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+@@ -454,6 +454,9 @@ const plugin = definePlugin({
+       secretName,
+       phase: "Pending",
+       backend: config.backend,
++      // Adapter environment probes sync managed assets before workspace
++      // realization. Give native sync the pod's mounted confinement root now.
++      remoteCwd: "/workspace",
+       scopedNetworkPolicyName,
+       scopedNetworkEgress,
+       // Native file sync streams over a pod exec; only the sandbox-cr backend
+@@ -529,6 +532,10 @@ const plugin = definePlugin({
+       secretName,
+       phase: check.phase,
+       backend: leaseBackend,
++      remoteCwd:
++        typeof params.leaseMetadata?.remoteCwd === "string" && params.leaseMetadata.remoteCwd.trim()
++          ? params.leaseMetadata.remoteCwd.trim()
++          : "/workspace",
+       scopedNetworkPolicyName:
+         typeof params.leaseMetadata?.scopedNetworkPolicyName === "string"
+           ? params.leaseMetadata.scopedNetworkPolicyName
+diff --git a/packages/plugins/sandbox-providers/kubernetes/src/types.ts b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+index 5626daa93846..ef3d0d889a4c 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/src/types.ts
++++ b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+@@ -90,6 +90,8 @@ export interface KubernetesLeaseMetadata {
+   phase: "Pending" | "Running" | "Succeeded" | "Failed";
+   /** Which backend provisioned this lease. */
+   backend: "sandbox-cr" | "job";
++  /** Native sync confinement root; present before workspace realization for probes. */
++  remoteCwd: string;
+   scopedNetworkPolicyName: string | null;
+   scopedNetworkEgress: {
+     allowFqdns: string[];
+diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+index 7ad0ca4807e5..4c23410d4d15 100644
+--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
++++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+@@ -80,6 +80,7 @@ describe("onEnvironmentResumeLease", () => {
+         secretName: "pc-abc-env",
+         phase: "Running",
+         backend: "sandbox-cr",
++        remoteCwd: "/workspace",
+         resumedLease: true,
+         // sandbox-cr has a pod-exec channel, so native file sync stays enabled.
+         nativeFileSyncUnsupported: false,
+@@ -87,6 +88,22 @@ describe("onEnvironmentResumeLease", () => {
+     );
+   });
+ 
++  it("keeps a custom realized cwd when resuming a sandbox lease", async () => {
++    h.clients = {
++      custom: { getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")) },
++      core: { readNamespacedPod: vi.fn().mockResolvedValue({ status: { phase: "Running" } }) },
++    };
++    const lease = await plugin.definition.onEnvironmentResumeLease!({
++      driverKey: "kubernetes",
++      companyId: "acme",
++      environmentId: "env-1",
++      config: CONFIG,
++      providerLeaseId: "pc-abc",
++      leaseMetadata: leaseMetadata({ remoteCwd: "/workspace/custom" }),
++    });
++    expect(lease.metadata?.remoteCwd).toBe("/workspace/custom");
++  });
++
+   it("flags a resumed job-backend lease as native-sync-unsupported so the server keeps the base64 fallback", async () => {
+     h.clients = {
+       batch: {

ci/images/paperclip/prepare.sh

@@ -0,0 +1,59 @@
+#!/bin/sh
+# Fetch upstream Paperclip at the pinned commit, apply our patches and make
+# the Dockerfile buildable with kaniko. Usage: prepare.sh <target-dir>
+set -eu
+HERE=$(cd "$(dirname "$0")" && pwd)
+. "$HERE/upstream.env"
+OUT=$1
+
+rm -rf "$OUT"
+git init -q "$OUT"
+cd "$OUT"
+git fetch -q --depth 1 https://github.com/paperclipai/paperclip.git "$PAPERCLIP_COMMIT"
+git -c advice.detachedHead=false checkout -q FETCH_HEAD
+test "$(git rev-parse HEAD)" = "$PAPERCLIP_COMMIT"
+
+for p in "$HERE"/*.patch; do
+  echo "applying $(basename "$p")"
+  git apply "$p"
+done
+
+# kaniko does not know `COPY --parents` (BuildKit only). Upstream uses it for
+# one line: every sandbox provider's package.json. Replace it with one plain
+# COPY per provider directory, which copies the same files to the same place.
+grep -c -- '^COPY --parents packages/plugins/sandbox-providers/\./\*/package.json packages/plugins/sandbox-providers/$' Dockerfile | grep -qx 1
+awk '
+  /^COPY --parents packages\/plugins\/sandbox-providers\// {
+    while (("ls -d packages/plugins/sandbox-providers/*/package.json" | getline f) > 0) {
+      d = f; sub(/package.json$/, "", d)
+      print "COPY " f " " d
+    }
+    next
+  }
+  { print }
+' Dockerfile > Dockerfile.kaniko
+if grep -q -- '--parents' Dockerfile.kaniko; then
+  echo "Dockerfile.kaniko still uses COPY --parents" >&2
+  exit 1
+fi
+
+# Build the Kubernetes sandbox provider into the image (REN-405). Upstream's
+# `production` target ships only its source; the plugin loader needs
+# dist/manifest.js. Same commands as upstream's `cloud-plugins` stage, run in
+# the `build` stage right after the server build, so production's
+# `COPY --from=build /app /app` carries dist/ and node_modules along. No extra
+# stage: kaniko would snapshot the whole build stage again for it.
+export ANCHOR='RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)'
+grep -c -F -x -- "$ANCHOR" Dockerfile.kaniko | grep -qx 1
+K=packages/plugins/sandbox-providers/kubernetes
+awk -v k="$K" '
+  { print }
+  $0 == ENVIRON["ANCHOR"] {
+    print "RUN pnpm -C " k " install --ignore-workspace --no-lockfile \\"
+    print "  && pnpm -C " k " build \\"
+    print "  && test -f " k "/dist/manifest.js"
+  }
+' Dockerfile.kaniko > Dockerfile.kaniko.new
+mv Dockerfile.kaniko.new Dockerfile.kaniko
+grep -q -- "test -f $K/dist/manifest.js" Dockerfile.kaniko
+rm -rf .git

ci/images/paperclip/upstream.env

@@ -0,0 +1,7 @@
+# Upstream paperclipai/paperclip commit this image is built from (REN-401).
+# a027f76 is the master commit the nightly image ran when this was pinned, so
+# the only change against that image is the patch next to this file.
+# PAPERCLIP_VERSION is `git describe --tags --match 'v*' --long` of that
+# commit, the value upstream stamps into its own images. Bump both together.
+PAPERCLIP_COMMIT=a027f76a726e8a556674eded807c2de0e55f5cc0
+PAPERCLIP_VERSION=v2026.916.0-219-ga027f76a7

kubernetes/apps/home-assistant/config-sync-token-externalsecret.yaml

@@ -4,7 +4,7 @@ metadata:
   name: home-assistant-config-sync-token
   namespace: home-assistant
 spec:
-  refreshInterval: 1h
+  refreshInterval: 1m # rotator revokes the old LLAT immediately — see prometheus-token-externalsecret.yaml
   secretStoreRef:
     kind: ClusterSecretStore
     name: openbao

kubernetes/apps/home-assistant/prometheus-token-externalsecret.yaml

@@ -1,10 +1,14 @@
+# refreshInterval is 1m: the rotator (openbao/rotation/home-assistant.yaml)
+# revokes the old LLAT seconds after storing the new one, so until ESO re-reads,
+# this Secret holds a DEAD token. At 1h that gap fired HomeAssistantDown (401
+# scrapes, issue #1081); 1m keeps it under the alert's `for: 5m`.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: home-assistant-prometheus-token
   namespace: home-assistant
 spec:
-  refreshInterval: 1h
+  refreshInterval: 1m
   secretStoreRef:
     kind: ClusterSecretStore
     name: openbao

kubernetes/apps/home-assistant/shelly-manager-llat-externalsecret.yaml

@@ -4,7 +4,7 @@ metadata:
   name: shelly-manager-llat
   namespace: home-assistant
 spec:
-  refreshInterval: 1h
+  refreshInterval: 1m # rotator revokes the old LLAT immediately — see prometheus-token-externalsecret.yaml
   secretStoreRef:
     kind: ClusterSecretStore
     name: openbao

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -194,15 +194,12 @@ spec:
             periodSeconds: 30
             timeoutSeconds: 5
             failureThreshold: 4
-          # Node sizes the V8 heap from this limit: 512Mi gave a 259 MB heap,
-          # and Z2M died on "JavaScript heap out of memory" (exit 134) at
-          # 2026-09-28 17:56Z and 2026-09-29 07:36Z.
           resources:
             requests:
               cpu: 50m
-              memory: 256Mi
+              memory: 128Mi
             limits:
-              memory: 1Gi
+              memory: 512Mi
         # Fires a forced interview whenever a still-un-interviewed device
         # transmits — i.e. at the only moment a sleepy battery device can
         # actually complete one. Z2M itself only retries interviews on a ZDO

kubernetes/apps/paperclip/agent-kubectl-rbac.yaml

@@ -0,0 +1,214 @@
+---
+# Per-agent kubectl identities for agent pods (REN-412, parent REN-284).
+#
+# Each agent gets its own ServiceAccount instead of the shared pod token of
+# `paperclip-prometheus-read`. Atlas, Iris, Hestia and Argus get exactly what
+# they hold today through that shared account: built-in `view` cluster-wide
+# (no Secrets, no write verb) plus the `monitoring/paperclip-prometheus-read`
+# Role (see prometheus-read-rbac.yaml). Hermes gets only `get` on the five
+# token Secrets below, to issue and rotate the kubeconfigs.
+#
+# Token Secrets carry no data in git; kube-controller-manager fills
+# `data.token`. Flux applies server-side and does not own `data`, so it does
+# not wipe the token. Revoke one agent: delete its Secret (the token dies at
+# once; Flux recreates it with a fresh token on the next reconcile).
+#
+# The shared account may read the tokens too (Hermes still runs in the shared
+# pod). Each token is worth at most what the shared account already holds, so
+# that is no escalation. Drop that subject once Hermes runs in its own pod.
+#
+# PRODUCTION
+#
+# ServiceAccounts, bindings and empty token Secrets only. No workload changes,
+# nothing restarts. Revert: delete this file and its line in kustomization.yaml.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-atlas
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-iris
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-hestia
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-argus
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: agent-hermes
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+automountServiceAccountToken: false
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-atlas-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-atlas
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-iris-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-iris
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-hestia-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-hestia
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-argus-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-argus
+---
+apiVersion: v1
+kind: Secret
+type: kubernetes.io/service-account-token
+metadata:
+  name: agent-hermes-token
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    kubernetes.io/service-account.name: agent-hermes
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: paperclip-agents-view
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: agent-atlas
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-iris
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hestia
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-argus
+    namespace: paperclip
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: paperclip-agents-prometheus-read
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  # Existing Role in prometheus-read-rbac.yaml, unchanged.
+  name: paperclip-prometheus-read
+# Subjects live in `paperclip`, the binding in `monitoring`: `namespace:` is
+# load-bearing.
+subjects:
+  - kind: ServiceAccount
+    name: agent-atlas
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-iris
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hestia
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-argus
+    namespace: paperclip
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: agent-token-read
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get"]
+    resourceNames:
+      - agent-atlas-token
+      - agent-iris-token
+      - agent-hestia-token
+      - agent-argus-token
+      - agent-hermes-token
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: agent-token-read
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: agent-token-read
+subjects:
+  # Hermes still runs in the shared pod; drop this subject once Hermes runs in
+  # its own pod.
+  - kind: ServiceAccount
+    name: paperclip-prometheus-read
+    namespace: paperclip
+  - kind: ServiceAccount
+    name: agent-hermes
+    namespace: paperclip

kubernetes/apps/paperclip/kustomization.yaml

@@ -10,6 +10,7 @@ resources:
   - prometheus-read-rbac.yaml
   - cluster-view-rbac.yaml
   - sandbox-rbac.yaml
+  - agent-kubectl-rbac.yaml
   - paperclip.yaml
   - gate-service.yaml
   - ingress.yaml

kubernetes/apps/paperclip/paperclip.yaml

@@ -34,7 +34,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: paperclip
-          image: ghcr.io/paperclipai/paperclip:nightly@sha256:c02c7a8385fe5e3e9999ec03ecb45f29d60b8bf44dac3fedf032d7ef2046045d # {"$imagepolicy": "flux-system:paperclip"}
+          image: gitea.renner.uno/konairius/paperclip:build.1790806287@sha256:f0b4b4e09cc02fbc472faedf3edb6d39339ade7e2932c6a435f601209b92d646 # {"$imagepolicy": "flux-system:paperclip"}
           ports:
             - name: http
               containerPort: 3100

kubernetes/apps/speech-to-phrase/deployment.yaml

@@ -4,6 +4,9 @@ metadata:
   name: speech-to-phrase
   labels:
     app.kubernetes.io/name: speech-to-phrase
+  annotations:
+    # The token is read once at startup (--hass-token); restart on quarterly rotation.
+    secret.reloader.stakater.com/reload: home-assistant-prometheus-token
 spec:
   replicas: 1
   strategy:

kubernetes/apps/trmnl/terminus.yaml

@@ -134,7 +134,7 @@ spec:
         # is a device-bricking class of risk. Bumps are a reviewed, manual
         # act — see kubernetes/infrastructure/image-automation/README.md.
         - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
           ports:
             - name: http
               containerPort: 2345
@@ -197,7 +197,7 @@ spec:
             capabilities:
               drop: ["ALL"]
         - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
           command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
           envFrom:
             - configMapRef:

kubernetes/apps/zone-configurator/deployment.yaml

@@ -4,6 +4,9 @@ metadata:
   name: zone-configurator
   labels:
     app.kubernetes.io/name: zone-configurator
+  annotations:
+    # The token is read once at startup (env); restart on quarterly rotation.
+    secret.reloader.stakater.com/reload: home-assistant-prometheus-token
 spec:
   replicas: 1
   strategy:

kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml

@@ -937,7 +937,7 @@ subjects:
 # with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
 stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
   name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
             secretKeyRef:
               key: SIDECAR_IMAGE
               name: plugin-barman-cloud-kg5c2795f7
-        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
+        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
         name: barman-cloud
         ports:
         - containerPort: 9090

kubernetes/infrastructure/image-automation/paperclip.yaml

@@ -1,13 +1,18 @@
+# Our own build (REN-401): upstream master at a pinned commit plus the k8s
+# sandbox fix, built by .woodpecker/build-paperclip.yml. Back to
+# ghcr.io/paperclipai/paperclip:nightly once upstream PR #13733 is merged;
+# see ci/images/paperclip/README.md.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 metadata:
   name: paperclip
   namespace: flux-system
 spec:
-  image: ghcr.io/paperclipai/paperclip
-  interval: 1h
+  image: gitea.renner.uno/konairius/paperclip
+  interval: 10m
 ---
-# Upstream publishes no semver tags; track the nightly tag by digest.
+# The pipeline tags every build `build.<unix-ts>`; latest build wins. Pinned by
+# digest in the manifest like before.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -17,9 +22,10 @@ spec:
   imageRepositoryRef:
     name: paperclip
   digestReflectionPolicy: Always
-  interval: 1h
+  interval: 10m
   filterTags:
-    pattern: '^nightly$'
+    pattern: '^build\.(?P<ts>\d+)$'
+    extract: '$ts'
   policy:
-    alphabetical:
+    numerical:
       order: asc

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -7,6 +7,7 @@ resources:
   - authentik-ns.yaml
   - paperless-ns.yaml
   - paperclip-ns.yaml
+  - paperclip-renner-ns.yaml
   - audiobookshelf-ns.yaml
   - home-assistant-ns.yaml
   - spegel-ns.yaml

kubernetes/infrastructure/network-policies/paperclip-renner-ns.yaml

@@ -0,0 +1,70 @@
+# Agent pods in paperclip-renner (REN-284) reach Gitea through Traefik (REN-411).
+#
+# gitea.renner.uno resolves to the Traefik LB VIP (fd99::86 / 10.2.99.86).
+# Cilium translates the VIP to a Traefik pod before egress policy, so the
+# destination is a cluster endpoint, not `world`, and the plugin's `toFQDNs`
+# rule for gitea.renner.uno never matches it. Allow the Traefik pods by
+# label instead, on the post-DNAT port: service 443 -> targetPort websecure
+# = container port 8443.
+#
+# Not labelled `paperclip.io/managed-by`, so the Paperclip k8s plugin does not
+# own or delete it. Cilium unions allows, so this only adds to
+# paperclip-egress-fqdn and paperclip-deny-all. Traefik's side is open:
+# ingress-default-deny (ingress-isolation.yaml) admits `cluster`.
+#
+# Revert: delete this file and its line in kustomization.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: agent-egress-traefik
+  namespace: paperclip-renner
+spec:
+  endpointSelector:
+    matchLabels:
+      paperclip.io/role: agent
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+---
+# Agent pods reach the Kubernetes API server with their own kubeconfig
+# (REN-412, parent REN-284; RBAC in apps/paperclip/agent-kubectl-rbac.yaml).
+#
+# A `toCIDRSet` rule cannot open this path: `policy-cidr-match-mode` is unset
+# in cilium-config, so CIDR rules never match node IPs, and the API server
+# (10.2.46.11-13:6443, Service 10.96.0.1:443 translated by KPR) runs on the
+# nodes. The `kube-apiserver` entity matches it. Port is the post-DNAT 6443.
+#
+# Selects by agent id, so only the five agents with a kubectl identity get
+# the path; other agent pods stay closed. Not labelled
+# `paperclip.io/managed-by`, so the plugin does not own or delete it.
+#
+# Revert: delete this document.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-agents-kube-apiserver
+  namespace: paperclip-renner
+spec:
+  endpointSelector:
+    matchExpressions:
+      - key: paperclip.io/agent-id
+        operator: In
+        values:
+          - 27d813a4-ae75-4913-b25b-3516a893d614 # Atlas
+          - 6eb4d648-83b4-40cb-b16e-d05d3ef372cb # Iris
+          - 32b3c62a-a5dc-4499-b970-25f5c7f60e8c # Hestia
+          - b023e112-1466-46f1-9fc1-e87e5cd3f005 # Argus
+          - b1be203c-d260-4922-8b48-ab9e87e73fb0 # Hermes
+  egress:
+    - toEntities:
+        - kube-apiserver
+      toPorts:
+        - ports:
+            - port: "6443"
+              protocol: TCP

renovate.json

@@ -41,12 +41,12 @@
         "ghcr.io/immich-app/immich-server",
         "ghcr.io/matter-js/python-matter-server",
         "ghcr.io/music-assistant/server",
-        "ghcr.io/paperclipai/paperclip",
         "ghcr.io/paperless-ngx/paperless-ngx",
         "ghcr.io/usetrmnl/terminus",
         "ghcr.io/usetrmnl/trmnl-ha-amd64",
         "gitea.renner.uno/k.renner/shelly-manager",
         "gitea.renner.uno/k.renner/vsftpd",
+        "gitea.renner.uno/konairius/paperclip",
         "gitea.renner.uno/konairius/wiki",
         "gotenberg/gotenberg",
         "lscr.io/linuxserver/lidarr",