Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5.5 3e1599bcad zigbee2mqtt: raise memory limit to 1Gi
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Node derives the V8 heap from the container limit: 512Mi gave a
259 MB heap, and Z2M crashed with "JavaScript heap out of memory"
(exit 134) at 2026-09-28 17:56Z and 2026-09-29 07:36Z. Request
raised to 256Mi to cover the measured 232Mi working set.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 07:30:17 +02:00
23 changed files with 23 additions and 889 deletions
-69
View File
@@ -1,69 +0,0 @@
# Paperclip server image (REN-401): upstream paperclipai/paperclip at a pinned
# master commit plus the patches in ci/images/paperclip/. Needed while the k8s
# sandbox fix (upstream PR #13733) is not merged; see the README there for how
# to go back to upstream nightly.
#
# Own workflow for the reason given in build-agent-runtime.yml: a step in
# build-ci-images.yml runs on every change to that file and competes for the
# woodpecker-ci-quota. Re-run alone with a deployment to target
# `build-paperclip`.
#
# Tag `build.<unix-ts>` is what the Flux ImagePolicy `paperclip` sorts on
# (image-automation/paperclip.yaml); image automation then rolls it out.
# konairius/ and ci_images_token as for the agent-runtime images.
when:
- event: push
branch: master
path:
- "ci/images/paperclip/**"
- ".woodpecker/build-paperclip.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-paperclip"'
steps:
- name: prepare-source
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- sh ci/images/paperclip/prepare.sh build/paperclip-src
- name: build-paperclip
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: build/paperclip-src
depends_on: [prepare-source]
# Rust runner + UI + server build in one pod. The server build alone sets
# a 4 GiB Node heap, so the memory limit is the LimitRange max (8Gi).
backend_options:
kubernetes:
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "4"
memory: 8Gi
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- . ../../ci/images/paperclip/upstream.env
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=build.$(date +%s)
# Same target and build args as upstream's docker.yml for nightly.
# --compressed-caching=false: pipeline 11717 was killed at the 8Gi limit
# (exit 137) in the server build; kaniko otherwise holds compressed
# layer tarballs in memory next to the 4 GiB Node heap.
- |
/kaniko/executor \
--context . \
--dockerfile Dockerfile.kaniko \
--target production \
--skip-unused-stages \
--snapshot-mode=redo \
--compressed-caching=false \
--build-arg PAPERCLIP_BUILD_COMMIT=$PAPERCLIP_COMMIT \
--build-arg PAPERCLIP_BUILD_VERSION=$PAPERCLIP_VERSION \
--build-arg CLI_TOOLS_CACHE_EPOCH=$(date -u +%G-W%V) \
--destination gitea.renner.uno/konairius/paperclip:$TAG
-55
View File
@@ -1,55 +0,0 @@
# Paperclip server image (REN-401)
Upstream `paperclipai/paperclip` at the commit in `upstream.env`, plus every
`*.patch` in this directory. Built by `.woodpecker/build-paperclip.yml`,
pushed as `gitea.renner.uno/konairius/paperclip:build.<unix-ts>`, rolled out
by Flux image automation (`kubernetes/infrastructure/image-automation/paperclip.yaml`).
## Why
k8s sandbox runs fail: the run path never sets the lease `remoteCwd`, so file
sync into the pod fails (paperclipai/paperclip#13864). The fix is
paperclipai/paperclip#13733, open and not merged.
`pr-13733-lease-remote-cwd.patch` is that PR's diff (head `5f320bf6`).
#13733 covers only the run path. Adapter environment tests (binding an agent
to a k8s environment) sync managed config before any workspace is realized,
so the lease still has no `remoteCwd` and the Claude test fails with
`claude_managed_config_dir_failed` (REN-409). The fix is
paperclipai/paperclip#14769, also open: the plugin sets `remoteCwd:
/workspace` when it acquires or resumes a lease.
`pr-14769-probe-remote-cwd.patch` is that PR's diff (head `10dc7371`).
## Files
- `upstream.env` — pinned commit and the version string stamped into the
image. Bump both together.
- `prepare.sh` — fetches the commit, applies the patches, writes
`Dockerfile.kaniko` (upstream's `COPY --parents` line split into plain
`COPY` lines; kaniko does not support `--parents`), and adds the build of
the Kubernetes sandbox provider plugin to the `build` stage (REN-405):
upstream's `production` target ships only its source, so the plugin fails
with "no longer exposes a Paperclip manifest". The build fails if
`dist/manifest.js` is missing.
- `*.patch` — applied in name order with `git apply`.
## Moving to a newer upstream commit
Set `PAPERCLIP_COMMIT` to the new master commit and `PAPERCLIP_VERSION` to
`git describe --tags --match 'v*' --long <commit>`, then run
`sh prepare.sh /tmp/src`: it fails if a patch no longer applies.
## Exit: when #13733 and #14769 are merged upstream
Renovate cannot watch an upstream PR, so this is a manual step. Once
upstream nightly contains both fixes:
1. `kubernetes/infrastructure/image-automation/paperclip.yaml`: back to
`ghcr.io/paperclipai/paperclip` with `filterTags.pattern: '^nightly$'`
(the version before REN-401 in git history).
2. `renovate.json`: swap `gitea.renner.uno/konairius/paperclip` back to
`ghcr.io/paperclipai/paperclip` in the image-automation exclusion list.
3. Delete this directory and `.woodpecker/build-paperclip.yml`.
Image automation then rewrites `kubernetes/apps/paperclip/paperclip.yaml`
to the nightly digest by itself.
@@ -1,93 +0,0 @@
diff --git a/server/src/__tests__/environment-run-orchestrator.test.ts b/server/src/__tests__/environment-run-orchestrator.test.ts
index f3d1fc659..812cae357 100644
--- a/server/src/__tests__/environment-run-orchestrator.test.ts
+++ b/server/src/__tests__/environment-run-orchestrator.test.ts
@@ -430,6 +430,51 @@ describe("environmentRunOrchestrator — realizeForRun", () => {
expect(result.persistedExecutionWorkspace).toEqual(updatedEw);
});
+ it("persists the realized cwd as remoteCwd when realization returns no workspaceRealization payload", async () => {
+ const runtime = makeMockRuntime({
+ realizeWorkspace: vi.fn().mockResolvedValue({
+ cwd: "/workspace",
+ metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+ }),
+ } as Partial<EnvironmentRuntimeService>);
+ mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+ const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+ await orchestrator.realizeForRun(
+ makeRealizeInput({ environment: makeEnvironment("sandbox") }),
+ );
+
+ expect(mockUpdateLeaseMetadata).toHaveBeenCalledWith(
+ "lease-1",
+ expect.objectContaining({ remoteCwd: "/workspace" }),
+ );
+ // Nothing to record about the realization shape, so the execution workspace
+ // row is left untouched.
+ expect(mockUpdateExecutionWorkspace).not.toHaveBeenCalled();
+ });
+
+ it("does not rewrite lease metadata when the realized cwd already matches the lease", async () => {
+ const runtime = makeMockRuntime({
+ realizeWorkspace: vi.fn().mockResolvedValue({
+ cwd: "/workspace",
+ metadata: { provider: "kubernetes", remoteCwd: "/workspace" },
+ }),
+ } as Partial<EnvironmentRuntimeService>);
+ mockResolveEnvironmentExecutionTarget.mockResolvedValue({ kind: "local", environmentId: "env-1", leaseId: "lease-1" });
+
+ const orchestrator = environmentRunOrchestrator(mockDb, { environmentRuntime: runtime });
+
+ await orchestrator.realizeForRun(
+ makeRealizeInput({
+ environment: makeEnvironment("sandbox"),
+ lease: makeLease({ metadata: { remoteCwd: "/workspace" } }),
+ }),
+ );
+
+ expect(mockUpdateLeaseMetadata).not.toHaveBeenCalled();
+ });
+
it("runs a remote provision command after workspace realization when configured", async () => {
mockBuildWorkspaceRealizationRequest.mockReturnValue({
version: 1,
diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts
index 97cca667e..e76f64913 100644
--- a/server/src/services/environment-run-orchestrator.ts
+++ b/server/src/services/environment-run-orchestrator.ts
@@ -485,16 +485,29 @@ export function environmentRunOrchestrator(
}
// Step 3: Persist realization metadata on lease and execution workspace
- if (Object.keys(workspaceRealization).length > 0) {
+ const hasWorkspaceRealization = Object.keys(workspaceRealization).length > 0;
+ // A driver that realizes the workspace reports the directory it prepared
+ // through the realization `cwd`. Persist it on the lease as `remoteCwd`, so
+ // every later step that receives only the lease — native file sync, a
+ // resumed lease, a second run against the same lease — confines itself to
+ // the same root. Without it a driver whose realization returns no
+ // `workspaceRealization` payload leaves the lease without a root at all,
+ // and the sync step fails even though realization succeeded.
+ const currentRemoteCwd =
+ typeof lease.metadata?.remoteCwd === "string" ? lease.metadata.remoteCwd.trim() : "";
+ const realizedRemoteCwd =
+ realizedWorkspaceCwd && realizedWorkspaceCwd !== currentRemoteCwd ? realizedWorkspaceCwd : null;
+ if (hasWorkspaceRealization || realizedRemoteCwd) {
const nextLeaseMetadata = {
...(lease.metadata ?? {}),
- workspaceRealization,
+ ...(hasWorkspaceRealization ? { workspaceRealization } : {}),
+ ...(realizedRemoteCwd ? { remoteCwd: realizedRemoteCwd } : {}),
};
const updatedLease = await environmentsSvc.updateLeaseMetadata(lease.id, nextLeaseMetadata);
if (updatedLease) {
lease = updatedLease;
}
- if (persistedExecutionWorkspace) {
+ if (hasWorkspaceRealization && persistedExecutionWorkspace) {
const updatedEw = await executionWorkspacesSvc.update(persistedExecutionWorkspace.id, {
metadata: {
...(persistedExecutionWorkspace.metadata ?? {}),
@@ -1,86 +0,0 @@
diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md
index a51c6ecee60e..72618cb11296 100644
--- a/packages/plugins/sandbox-providers/kubernetes/README.md
+++ b/packages/plugins/sandbox-providers/kubernetes/README.md
@@ -140,6 +140,8 @@ Every agent pod is:
Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs).
+The lease starts with `/workspace` as the native file-sync confinement root. Adapter environment tests can upload managed config before the server realizes a workspace. When resuming a lease, the provider keeps a previously recorded `remoteCwd` so a custom realized workspace is not replaced with the default.
+
The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Job, so a single `kubectl delete job …` cascades cleanly to the Pod and Secret.
## Optional Kata-FC microVM isolation
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
index 8ae47de8ecac..98016385da05 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts
@@ -454,6 +454,9 @@ const plugin = definePlugin({
secretName,
phase: "Pending",
backend: config.backend,
+ // Adapter environment probes sync managed assets before workspace
+ // realization. Give native sync the pod's mounted confinement root now.
+ remoteCwd: "/workspace",
scopedNetworkPolicyName,
scopedNetworkEgress,
// Native file sync streams over a pod exec; only the sandbox-cr backend
@@ -529,6 +532,10 @@ const plugin = definePlugin({
secretName,
phase: check.phase,
backend: leaseBackend,
+ remoteCwd:
+ typeof params.leaseMetadata?.remoteCwd === "string" && params.leaseMetadata.remoteCwd.trim()
+ ? params.leaseMetadata.remoteCwd.trim()
+ : "/workspace",
scopedNetworkPolicyName:
typeof params.leaseMetadata?.scopedNetworkPolicyName === "string"
? params.leaseMetadata.scopedNetworkPolicyName
diff --git a/packages/plugins/sandbox-providers/kubernetes/src/types.ts b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
index 5626daa93846..ef3d0d889a4c 100644
--- a/packages/plugins/sandbox-providers/kubernetes/src/types.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/src/types.ts
@@ -90,6 +90,8 @@ export interface KubernetesLeaseMetadata {
phase: "Pending" | "Running" | "Succeeded" | "Failed";
/** Which backend provisioned this lease. */
backend: "sandbox-cr" | "job";
+ /** Native sync confinement root; present before workspace realization for probes. */
+ remoteCwd: string;
scopedNetworkPolicyName: string | null;
scopedNetworkEgress: {
allowFqdns: string[];
diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
index 7ad0ca4807e5..4c23410d4d15 100644
--- a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
+++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts
@@ -80,6 +80,7 @@ describe("onEnvironmentResumeLease", () => {
secretName: "pc-abc-env",
phase: "Running",
backend: "sandbox-cr",
+ remoteCwd: "/workspace",
resumedLease: true,
// sandbox-cr has a pod-exec channel, so native file sync stays enabled.
nativeFileSyncUnsupported: false,
@@ -87,6 +88,22 @@ describe("onEnvironmentResumeLease", () => {
);
});
+ it("keeps a custom realized cwd when resuming a sandbox lease", async () => {
+ h.clients = {
+ custom: { getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")) },
+ core: { readNamespacedPod: vi.fn().mockResolvedValue({ status: { phase: "Running" } }) },
+ };
+ const lease = await plugin.definition.onEnvironmentResumeLease!({
+ driverKey: "kubernetes",
+ companyId: "acme",
+ environmentId: "env-1",
+ config: CONFIG,
+ providerLeaseId: "pc-abc",
+ leaseMetadata: leaseMetadata({ remoteCwd: "/workspace/custom" }),
+ });
+ expect(lease.metadata?.remoteCwd).toBe("/workspace/custom");
+ });
+
it("flags a resumed job-backend lease as native-sync-unsupported so the server keeps the base64 fallback", async () => {
h.clients = {
batch: {
-59
View File
@@ -1,59 +0,0 @@
#!/bin/sh
# Fetch upstream Paperclip at the pinned commit, apply our patches and make
# the Dockerfile buildable with kaniko. Usage: prepare.sh <target-dir>
set -eu
HERE=$(cd "$(dirname "$0")" && pwd)
. "$HERE/upstream.env"
OUT=$1
rm -rf "$OUT"
git init -q "$OUT"
cd "$OUT"
git fetch -q --depth 1 https://github.com/paperclipai/paperclip.git "$PAPERCLIP_COMMIT"
git -c advice.detachedHead=false checkout -q FETCH_HEAD
test "$(git rev-parse HEAD)" = "$PAPERCLIP_COMMIT"
for p in "$HERE"/*.patch; do
echo "applying $(basename "$p")"
git apply "$p"
done
# kaniko does not know `COPY --parents` (BuildKit only). Upstream uses it for
# one line: every sandbox provider's package.json. Replace it with one plain
# COPY per provider directory, which copies the same files to the same place.
grep -c -- '^COPY --parents packages/plugins/sandbox-providers/\./\*/package.json packages/plugins/sandbox-providers/$' Dockerfile | grep -qx 1
awk '
/^COPY --parents packages\/plugins\/sandbox-providers\// {
while (("ls -d packages/plugins/sandbox-providers/*/package.json" | getline f) > 0) {
d = f; sub(/package.json$/, "", d)
print "COPY " f " " d
}
next
}
{ print }
' Dockerfile > Dockerfile.kaniko
if grep -q -- '--parents' Dockerfile.kaniko; then
echo "Dockerfile.kaniko still uses COPY --parents" >&2
exit 1
fi
# Build the Kubernetes sandbox provider into the image (REN-405). Upstream's
# `production` target ships only its source; the plugin loader needs
# dist/manifest.js. Same commands as upstream's `cloud-plugins` stage, run in
# the `build` stage right after the server build, so production's
# `COPY --from=build /app /app` carries dist/ and node_modules along. No extra
# stage: kaniko would snapshot the whole build stage again for it.
export ANCHOR='RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1)'
grep -c -F -x -- "$ANCHOR" Dockerfile.kaniko | grep -qx 1
K=packages/plugins/sandbox-providers/kubernetes
awk -v k="$K" '
{ print }
$0 == ENVIRON["ANCHOR"] {
print "RUN pnpm -C " k " install --ignore-workspace --no-lockfile \\"
print " && pnpm -C " k " build \\"
print " && test -f " k "/dist/manifest.js"
}
' Dockerfile.kaniko > Dockerfile.kaniko.new
mv Dockerfile.kaniko.new Dockerfile.kaniko
grep -q -- "test -f $K/dist/manifest.js" Dockerfile.kaniko
rm -rf .git
-7
View File
@@ -1,7 +0,0 @@
# Upstream paperclipai/paperclip commit this image is built from (REN-401).
# a027f76 is the master commit the nightly image ran when this was pinned, so
# the only change against that image is the patch next to this file.
# PAPERCLIP_VERSION is `git describe --tags --match 'v*' --long` of that
# commit, the value upstream stamps into its own images. Bump both together.
PAPERCLIP_COMMIT=a027f76a726e8a556674eded807c2de0e55f5cc0
PAPERCLIP_VERSION=v2026.916.0-219-ga027f76a7
@@ -194,12 +194,15 @@ spec:
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 4
# Node sizes the V8 heap from this limit: 512Mi gave a 259 MB heap,
# and Z2M died on "JavaScript heap out of memory" (exit 134) at
# 2026-09-28 17:56Z and 2026-09-29 07:36Z.
resources:
requests:
cpu: 50m
memory: 128Mi
memory: 256Mi
limits:
memory: 512Mi
memory: 1Gi
# Fires a forced interview whenever a still-un-interviewed device
# transmits — i.e. at the only moment a sleepy battery device can
# actually complete one. Z2M itself only retries interviews on a ZDO
-8
View File
@@ -18,14 +18,6 @@ kind: PersistentVolumeClaim
metadata:
name: immich-library
namespace: immich
labels:
# Pre-migration copy: no pod mounts it since the 2026-09-25 move to
# immich-library-bulk, and its backup size has been frozen since. Left
# in the daily backup, the monthly restore-test stages it (~194 GiB x3
# replicas) alongside library-bulk and overruns Ceph capacity (#1089).
# Existing backups keep it until their 720h TTL; delete the PVC/PV by
# hand once no longer wanted.
velero.io/exclude-from-backup: "true"
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
@@ -29,17 +29,6 @@
# been dropped and the metric would have stayed absent. The matching ingress rule
# is added to infrastructure/network-policies/external-secrets-ns.yaml in the
# same pull request. Without it this file is decoration.
#
# NO TLS AND NO AUTH ON THIS SCRAPE, and that is the chart's posture rather than
# an omission here. In 2.11.0 `metrics.listen.secure.enabled` and
# `metrics.listen.auth.enabled` are both false (values.yaml:365-369) and
# infrastructure/external-secrets/helmrelease.yaml overrides neither, so the
# controller serves /metrics over plain HTTP to anything that can reach :8080.
# Hence no `scheme`, no `tlsConfig` and no `bearerTokenSecret` below: adding
# them would break the scrape, not harden it. What limits exposure is the
# network policy — a single port from a single namespace — so turning on
# `secure.enabled` here would mean a Helm upgrade of ESO to close a hole the
# CiliumNetworkPolicy already closes.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
@@ -11,31 +11,6 @@ spec:
groups:
- name: disk-wear.rules
rules:
# The drives carrying Talos's EPHEMERAL volume (/var), as (node, device)
# in smartctl-exporter's labels. EPHEMERAL sits on retired Ceph drives
# whose contents rebuild, and the owner's call is that they run to
# failure rather than be replaced on wear (REN-419, REN-420). So the two
# wear alerts below skip them. The failure alerts — spare, critical
# warning, media errors — still cover them: run to failure still means
# knowing when it failed.
#
# Resolved from node-exporter's /var mount, not a list of serials, so it
# follows a drive swap without an edit. If the join stops matching (a
# relabel, an encrypted /var on a dm device) the series is empty, and
# the wear alerts return for EPHEMERAL drives: it fails loud, not
# silent.
- record: node_device:nvme_ephemeral_volume:info
expr: >-
count by (node, device) (
label_replace(
label_replace(
node_filesystem_size_bytes{mountpoint="/var"}
* on (instance) group_left (nodename) node_uname_info,
"node", "$1", "nodename", "(.+)"
),
"device", "$1", "device", "/dev/(nvme[0-9]+)n[0-9]+(p[0-9]+)?"
)
)
# Wear percentages are self-reported: 100 means the rated endurance is
# spent, not that the drive stops.
#
@@ -70,7 +45,6 @@ spec:
)
) < 180
and smartctl_device_percentage_used >= 80
unless on (node, device) node_device:nvme_ephemeral_volume:info
# Long, because a projection off a 7d rate should not react to a day.
for: 6h
labels:
@@ -81,17 +55,13 @@ spec:
The drive is past 80% of its rated endurance and the write rate
of the last week would spend the rest within six months. This is
a lead-time warning, not a fault: decide whether to replace the
drive or move what is writing to it. Drives carrying EPHEMERAL
are excluded; they run to failure. The Disk Health dashboard
drive or move what is writing to it. The Disk Health dashboard
shows which drives carry which role.
# Kept as a flat threshold on purpose: past rated endurance the drive
# runs outside what the manufacturer stands behind regardless of how
# fast it got there, and the alert clears by replacing it. EPHEMERAL
# drives are excluded for the same reason as above.
# fast it got there, and the alert clears by replacing it.
- alert: NvmeDriveWearCritical
expr: >-
smartctl_device_percentage_used >= 95
unless on (node, device) node_device:nvme_ephemeral_volume:info
expr: smartctl_device_percentage_used >= 95
for: 1h
labels:
severity: critical
@@ -30,7 +30,6 @@ retired the depth rule in favour of one persistence rule (`> 5` for `20m`).
| `external-secrets_test.yaml` | Hand-written suite for `prometheusrule-external-secrets.yaml` (REN-103). |
| `home-assistant_test.yaml` | Hand-written suite for `prometheusrule-home-assistant.yaml` (REN-245). |
| `ceph_test.yaml` | Hand-written suite for the `CephPoolLostRedundancyBlind` canary in `prometheusrule-ceph.yaml` (REN-93). |
| `disk-wear_test.yaml` | Hand-written suite for the EPHEMERAL exclusion in `prometheusrule-disk-wear.yaml` (REN-419). |
| `mutations.txt` | Negative controls. Each must make the suite fail. |
## Adding a suite
@@ -1,124 +0,0 @@
# promtool test rules for the EPHEMERAL exclusion in prometheusrule-disk-wear.yaml
# (REN-419).
#
# Run by the `promtool` step of .woodpecker/ci-manifests-test.yml through
# scripts/ci/promtool-rules.sh, which strips `spec:` from the manifest into a
# mirrored tree — hence `../` below, resolved from this file's directory. See
# README.md here. The negative controls for this suite are the `disk-wear-*`
# entries in mutations.txt.
#
# WHAT IT IS FOR. The two wear alerts skip the drive that carries /var, found
# by joining node-exporter's /var mount onto smartctl-exporter's (node, device).
# Each case puts an EPHEMERAL drive and a non-EPHEMERAL drive side by side with
# identical SMART data, so the only thing that can separate them is the join.
#
# Fixture shape: smartctl-exporter labels drives `nvme0`, `nvme1` and carries
# `node` from the ServiceMonitor relabel; node-exporter carries `instance` and
# reports /var on the partition, `/dev/nvme1n1p1`.
rule_files:
- ../prometheusrule-disk-wear.yaml
evaluation_interval: 1m
tests:
# A. Projection. k8s-1 nvme0 and nvme1 wear identically: 89%, 1.5e9 bytes
# per 5m (4.32e11 a day), 9e14 lifetime bytes at 192h. That is 0.04272 % a
# day, 6 % to go, 140 days — under 180. Only nvme0 may fire; nvme1
# carries /var. Neither is at 95%, so the critical alert stays quiet.
#
# Sampled at 5m, not 1h: the instant selectors look back 5m, so hourly
# samples leave the series absent for most of each hour and the 6h `for`
# never completes. Not 1m either: a 7d rate over 1m samples exceeds
# promtool's query sample limit.
- interval: 5m
input_series:
- series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
values: '89+0x2320'
- series: 'smartctl_device_bytes_written{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
values: '896544000000000+1500000000x2320'
- series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
values: '89+0x2320'
- series: 'smartctl_device_bytes_written{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
values: '896544000000000+1500000000x2320'
- series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/nvme1n1p1",fstype="xfs"}'
values: '1000000000000+0x2320'
- series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
values: '1+0x2320'
alert_rule_test:
- eval_time: 192h
alertname: NvmeDriveWearProjectedExhaustion
exp_alerts:
- exp_labels:
severity: warning
node: k8s-1
device: nvme0
instance: 10.0.0.1:9633
exp_annotations:
summary: "NVMe nvme0 on 10.0.0.1:9633 projects to 95% wear in 140 days"
description: "The drive is past 80% of its rated endurance and the write rate of the last week would spend the rest within six months. This is a lead-time warning, not a fault: decide whether to replace the drive or move what is writing to it. Drives carrying EPHEMERAL are excluded; they run to failure. The Disk Health dashboard shows which drives carry which role."
- eval_time: 192h
alertname: NvmeDriveWearCritical
exp_alerts: []
# B. Critical, two nodes. Every drive is at 96%. On k8s-1 /var is nvme1; on
# k8s-2 it is nvme2, which reports no SMART here. So k8s-1 nvme1 is the only
# one excluded, and k8s-2 nvme1 must still fire — the exclusion is per
# node, not per device name.
- interval: 1m
input_series:
- series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme0",instance="10.0.0.1:9633"}'
values: '96+0x200'
- series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
values: '96+0x200'
- series: 'smartctl_device_percentage_used{node="k8s-2",device="nvme1",instance="10.0.0.2:9633"}'
values: '96+0x200'
- series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/nvme1n1p1",fstype="xfs"}'
values: '1000000000000+0x200'
- series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
values: '1+0x200'
- series: 'node_filesystem_size_bytes{instance="10.0.0.2:9100",mountpoint="/var",device="/dev/nvme2n1p1",fstype="xfs"}'
values: '1000000000000+0x200'
- series: 'node_uname_info{instance="10.0.0.2:9100",nodename="k8s-2"}'
values: '1+0x200'
alert_rule_test:
- eval_time: 120m
alertname: NvmeDriveWearCritical
exp_alerts:
- exp_labels:
severity: critical
node: k8s-1
device: nvme0
instance: 10.0.0.1:9633
exp_annotations:
summary: "NVMe nvme0 on 10.0.0.1:9633 is at 96% rated wear"
description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
- exp_labels:
severity: critical
node: k8s-2
device: nvme1
instance: 10.0.0.2:9633
exp_annotations:
summary: "NVMe nvme1 on 10.0.0.2:9633 is at 96% rated wear"
description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
# C. Fails loud. /var is on a device the join cannot map to an NVMe name (an
# encrypted volume on dm-0). The exclusion then matches nothing and the
# EPHEMERAL drive alerts again, which is the intended direction to fail.
- interval: 1m
input_series:
- series: 'smartctl_device_percentage_used{node="k8s-1",device="nvme1",instance="10.0.0.1:9633"}'
values: '96+0x200'
- series: 'node_filesystem_size_bytes{instance="10.0.0.1:9100",mountpoint="/var",device="/dev/dm-0",fstype="xfs"}'
values: '1000000000000+0x200'
- series: 'node_uname_info{instance="10.0.0.1:9100",nodename="k8s-1"}'
values: '1+0x200'
alert_rule_test:
- eval_time: 120m
alertname: NvmeDriveWearCritical
exp_alerts:
- exp_labels:
severity: critical
node: k8s-1
device: nvme1
instance: 10.0.0.1:9633
exp_annotations:
summary: "NVMe nvme1 on 10.0.0.1:9633 is at 96% rated wear"
description: "The drive is at or past the end of its rated endurance. It will likely keep working for a while, but it is now operating outside what the manufacturer stands behind, and replacement should be scheduled rather than considered."
@@ -149,21 +149,3 @@ retention-no-size-clause|kubernetes/apps/monitoring-rules/prometheusrule-prometh
retention-ratio-0.9|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/^\([[:space:]]*\)> 0.8$/\1> 0.9/
retention-lookback-1d|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/)\[7d:1h\]$/)[1d:1h]/
retention-cap-at-peak-time|kubernetes/apps/monitoring-rules/prometheusrule-prometheus-retention.yaml|s/\/ max(prometheus_tsdb_retention_limit_bytes{job="kube-prometheus-stack-prometheus"})$/\/ min_over_time(max(prometheus_tsdb_retention_limit_bytes{job="kube-prometheus-stack-prometheus"})[7d:1h])/
# prometheusrule-disk-wear.yaml (REN-419): the EPHEMERAL exclusion on the two
# wear alerts. Detected by the cases in disk-wear_test.yaml.
#
# Gate inverted on the projection: the EPHEMERAL drive alone fires.
# Detected by: case A.
disk-wear-projection-gate-inverted|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearProjectedExhaustion/,/for: 6h/s/unless on (node, device)/and on (node, device)/
# Gate inverted on the critical rule.
# Detected by: case B.
disk-wear-critical-gate-inverted|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearCritical/,/for: 1h/s/unless on (node, device)/and on (node, device)/
# Matched on device name only: k8s-1's /var on nvme1 then silences nvme1 on
# every node.
# Detected by: case B, k8s-2 nvme1.
disk-wear-critical-device-only|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|/alert: NvmeDriveWearCritical/,/for: 1h/s/unless on (node, device)/unless on (device)/
# Partition suffix not stripped: /dev/nvme1n1p1 no longer maps to nvme1 and the
# exclusion matches nothing.
# Detected by: cases A and B.
disk-wear-ephemeral-keeps-partition|kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml|s/n\[0-9\]+(p\[0-9\]+)?"/n[0-9]+"/
@@ -1,214 +0,0 @@
---
# Per-agent kubectl identities for agent pods (REN-412, parent REN-284).
#
# Each agent gets its own ServiceAccount instead of the shared pod token of
# `paperclip-prometheus-read`. Atlas, Iris, Hestia and Argus get exactly what
# they hold today through that shared account: built-in `view` cluster-wide
# (no Secrets, no write verb) plus the `monitoring/paperclip-prometheus-read`
# Role (see prometheus-read-rbac.yaml). Hermes gets only `get` on the five
# token Secrets below, to issue and rotate the kubeconfigs.
#
# Token Secrets carry no data in git; kube-controller-manager fills
# `data.token`. Flux applies server-side and does not own `data`, so it does
# not wipe the token. Revoke one agent: delete its Secret (the token dies at
# once; Flux recreates it with a fresh token on the next reconcile).
#
# The shared account may read the tokens too (Hermes still runs in the shared
# pod). Each token is worth at most what the shared account already holds, so
# that is no escalation. Drop that subject once Hermes runs in its own pod.
#
# PRODUCTION
#
# ServiceAccounts, bindings and empty token Secrets only. No workload changes,
# nothing restarts. Revert: delete this file and its line in kustomization.yaml.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-atlas
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-iris
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-hestia
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-argus
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-hermes
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
automountServiceAccountToken: false
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: agent-atlas-token
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
kubernetes.io/service-account.name: agent-atlas
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: agent-iris-token
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
kubernetes.io/service-account.name: agent-iris
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: agent-hestia-token
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
kubernetes.io/service-account.name: agent-hestia
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: agent-argus-token
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
kubernetes.io/service-account.name: agent-argus
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: agent-hermes-token
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
annotations:
kubernetes.io/service-account.name: agent-hermes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: paperclip-agents-view
labels:
app.kubernetes.io/name: paperclip
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
subjects:
- kind: ServiceAccount
name: agent-atlas
namespace: paperclip
- kind: ServiceAccount
name: agent-iris
namespace: paperclip
- kind: ServiceAccount
name: agent-hestia
namespace: paperclip
- kind: ServiceAccount
name: agent-argus
namespace: paperclip
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: paperclip-agents-prometheus-read
namespace: monitoring
labels:
app.kubernetes.io/name: paperclip
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
# Existing Role in prometheus-read-rbac.yaml, unchanged.
name: paperclip-prometheus-read
# Subjects live in `paperclip`, the binding in `monitoring`: `namespace:` is
# load-bearing.
subjects:
- kind: ServiceAccount
name: agent-atlas
namespace: paperclip
- kind: ServiceAccount
name: agent-iris
namespace: paperclip
- kind: ServiceAccount
name: agent-hestia
namespace: paperclip
- kind: ServiceAccount
name: agent-argus
namespace: paperclip
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: agent-token-read
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
resourceNames:
- agent-atlas-token
- agent-iris-token
- agent-hestia-token
- agent-argus-token
- agent-hermes-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: agent-token-read
namespace: paperclip
labels:
app.kubernetes.io/name: paperclip
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: agent-token-read
subjects:
# Hermes still runs in the shared pod; drop this subject once Hermes runs in
# its own pod.
- kind: ServiceAccount
name: paperclip-prometheus-read
namespace: paperclip
- kind: ServiceAccount
name: agent-hermes
namespace: paperclip
@@ -10,7 +10,6 @@ resources:
- prometheus-read-rbac.yaml
- cluster-view-rbac.yaml
- sandbox-rbac.yaml
- agent-kubectl-rbac.yaml
- paperclip.yaml
- gate-service.yaml
- ingress.yaml
+1 -1
View File
@@ -34,7 +34,7 @@ spec:
type: RuntimeDefault
containers:
- name: paperclip
image: gitea.renner.uno/konairius/paperclip:build.1790806287@sha256:f0b4b4e09cc02fbc472faedf3edb6d39339ade7e2932c6a435f601209b92d646 # {"$imagepolicy": "flux-system:paperclip"}
image: ghcr.io/paperclipai/paperclip:nightly@sha256:c02c7a8385fe5e3e9999ec03ecb45f29d60b8bf44dac3fedf032d7ef2046045d # {"$imagepolicy": "flux-system:paperclip"}
ports:
- name: http
containerPort: 3100
+2 -2
View File
@@ -134,7 +134,7 @@ spec:
# is a device-bricking class of risk. Bumps are a reviewed, manual
# act — see kubernetes/infrastructure/image-automation/README.md.
- name: terminus
image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
ports:
- name: http
containerPort: 2345
@@ -197,7 +197,7 @@ spec:
capabilities:
drop: ["ALL"]
- name: sidekiq
image: ghcr.io/usetrmnl/terminus:0.76.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
envFrom:
- configMapRef:
@@ -937,7 +937,7 @@ subjects:
# with no marker. Re-apply this block, or the skew silently returns.
apiVersion: v1
stringData:
SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
kind: Secret
metadata:
name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
secretKeyRef:
key: SIDECAR_IMAGE
name: plugin-barman-cloud-kg5c2795f7
image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.1 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
name: barman-cloud
ports:
- containerPort: 9090
@@ -1,18 +1,13 @@
# Our own build (REN-401): upstream master at a pinned commit plus the k8s
# sandbox fix, built by .woodpecker/build-paperclip.yml. Back to
# ghcr.io/paperclipai/paperclip:nightly once upstream PR #13733 is merged;
# see ci/images/paperclip/README.md.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
name: paperclip
namespace: flux-system
spec:
image: gitea.renner.uno/konairius/paperclip
interval: 10m
image: ghcr.io/paperclipai/paperclip
interval: 1h
---
# The pipeline tags every build `build.<unix-ts>`; latest build wins. Pinned by
# digest in the manifest like before.
# Upstream publishes no semver tags; track the nightly tag by digest.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
@@ -22,10 +17,9 @@ spec:
imageRepositoryRef:
name: paperclip
digestReflectionPolicy: Always
interval: 10m
interval: 1h
filterTags:
pattern: '^build\.(?P<ts>\d+)$'
extract: '$ts'
pattern: '^nightly$'
policy:
numerical:
alphabetical:
order: asc
@@ -16,29 +16,13 @@ spec:
# dropped and `externalsecret_status_condition` held 0 series.
#
# Port-restricted, unlike the equivalent rule in velero-ns.yaml, which admits
# `monitoring` to the whole namespace. Namespace-wide is the norm and this is
# the sole exception: counted per ingress list item, this directory holds 20
# `monitoring` grants across 19 files (influxdb-ns.yaml carries two — one
# whole-namespace, one Grafana-pod), and this rule is the only one of the 20
# with a `toPorts`. home-assistant-ns.yaml reads as port-scoped and is not —
# the `toPorts` below its `monitoring` rule belongs to the `evcc` rule that
# follows.
# 8080 is the metrics port and the only thing `monitoring` has any business
# reaching here; the webhook (10250) and the health ports (8082 on the
# controller, 8081 on the other two) stay closed to it. 8080 is the chart
# default
# `monitoring` to the whole namespace. 8080 is the metrics port and the only
# thing `monitoring` has any business reaching here; the webhook (10250) and
# the health port stay closed to it. 8080 is the chart default
# (`metrics.listen.port`) and helmrelease.yaml does not override it — if that
# is ever overridden, this port and the PodMonitor's named `metrics` endpoint
# both have to follow.
#
# Scoped by PORT, not by pod: this CNP's `endpointSelector` is {}, so the
# grant reaches all three of the chart's deployments — controller, webhook,
# cert-controller — which all serve metrics on 8080. Only the controller is
# scraped today (the PodMonitor's selector is exact on the unsuffixed name),
# so the extra two are reachable-but-unvisited. Deliberate: adding a monitor
# for either later needs no policy change, and neither exposes more than
# controller-runtime internals on that port.
#
# The consumer is apps/monitoring-rules/external-secrets-podmonitor.yaml.
# Additive: this rule cannot deny anything previously allowed, so ESO's
# existing reachability to OpenBao and the apiserver is unchanged.
@@ -7,7 +7,6 @@ resources:
- authentik-ns.yaml
- paperless-ns.yaml
- paperclip-ns.yaml
- paperclip-renner-ns.yaml
- audiobookshelf-ns.yaml
- home-assistant-ns.yaml
- spegel-ns.yaml
@@ -1,70 +0,0 @@
# Agent pods in paperclip-renner (REN-284) reach Gitea through Traefik (REN-411).
#
# gitea.renner.uno resolves to the Traefik LB VIP (fd99::86 / 10.2.99.86).
# Cilium translates the VIP to a Traefik pod before egress policy, so the
# destination is a cluster endpoint, not `world`, and the plugin's `toFQDNs`
# rule for gitea.renner.uno never matches it. Allow the Traefik pods by
# label instead, on the post-DNAT port: service 443 -> targetPort websecure
# = container port 8443.
#
# Not labelled `paperclip.io/managed-by`, so the Paperclip k8s plugin does not
# own or delete it. Cilium unions allows, so this only adds to
# paperclip-egress-fqdn and paperclip-deny-all. Traefik's side is open:
# ingress-default-deny (ingress-isolation.yaml) admits `cluster`.
#
# Revert: delete this file and its line in kustomization.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: agent-egress-traefik
namespace: paperclip-renner
spec:
endpointSelector:
matchLabels:
paperclip.io/role: agent
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8443"
protocol: TCP
---
# Agent pods reach the Kubernetes API server with their own kubeconfig
# (REN-412, parent REN-284; RBAC in apps/paperclip/agent-kubectl-rbac.yaml).
#
# A `toCIDRSet` rule cannot open this path: `policy-cidr-match-mode` is unset
# in cilium-config, so CIDR rules never match node IPs, and the API server
# (10.2.46.11-13:6443, Service 10.96.0.1:443 translated by KPR) runs on the
# nodes. The `kube-apiserver` entity matches it. Port is the post-DNAT 6443.
#
# Selects by agent id, so only the five agents with a kubectl identity get
# the path; other agent pods stay closed. Not labelled
# `paperclip.io/managed-by`, so the plugin does not own or delete it.
#
# Revert: delete this document.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: paperclip-agents-kube-apiserver
namespace: paperclip-renner
spec:
endpointSelector:
matchExpressions:
- key: paperclip.io/agent-id
operator: In
values:
- 27d813a4-ae75-4913-b25b-3516a893d614 # Atlas
- 6eb4d648-83b4-40cb-b16e-d05d3ef372cb # Iris
- 32b3c62a-a5dc-4499-b970-25f5c7f60e8c # Hestia
- b023e112-1466-46f1-9fc1-e87e5cd3f005 # Argus
- b1be203c-d260-4922-8b48-ab9e87e73fb0 # Hermes
egress:
- toEntities:
- kube-apiserver
toPorts:
- ports:
- port: "6443"
protocol: TCP
+1 -1
View File
@@ -41,12 +41,12 @@
"ghcr.io/immich-app/immich-server",
"ghcr.io/matter-js/python-matter-server",
"ghcr.io/music-assistant/server",
"ghcr.io/paperclipai/paperclip",
"ghcr.io/paperless-ngx/paperless-ngx",
"ghcr.io/usetrmnl/terminus",
"ghcr.io/usetrmnl/trmnl-ha-amd64",
"gitea.renner.uno/k.renner/shelly-manager",
"gitea.renner.uno/k.renner/vsftpd",
"gitea.renner.uno/konairius/paperclip",
"gitea.renner.uno/konairius/wiki",
"gotenberg/gotenberg",
"lscr.io/linuxserver/lidarr",