Compare commits

...
Author SHA1 Message Date
claude-bot 7666d220c5 perf(woodpecker): put CI workspaces on the unreplicated CephFS
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.

Measured 2026-08-21, polling both CephFS pools every 15s:

  CI active   1.5 min (6 intervals):  data0 143.2 MB   metadata 17.2 MB
  CI idle    16.0 min (63 intervals): data0   0.0 MB   metadata 11.0 MB
  CI share of writes: data0 100%, metadata 61%

data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.

Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.

RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.

Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.

The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.

cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.
2026-08-21 09:31:01 +02:00
k.renner baa5e62bbe Merge pull request 'feat(rook-ceph): add an unreplicated CephFS for CI workspaces' (#542) from ci-throwaway-cephfs into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#542
2026-08-21 07:27:47 +00:00
claude-bot 503d2a4876 feat(rook-ceph): add an unreplicated CephFS for CI workspaces
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.

Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.

  one pipeline, a single 15s interval:  data0 +71.6 MB   metadata +8.2 MB
  every interval with no CI pod:        data0   ~0        metadata   ~0

So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.

size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.

`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.

Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.

Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.

The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
2026-08-21 08:55:33 +02:00
security-bot 48407aa202 chore(image-automation): bump redis:8.10.1-alpine
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-20 17:37:50 +00:00
security-bot ce6e8ba2cb chore(image-automation): bump redis:7.4.11-alpine
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-20 17:19:40 +00:00
k.renner 96387108ff Merge pull request 'perf(cnpg): roll archive_timeout out to the remaining six clusters' (#541) from rollout-archive-timeout into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#541
2026-08-20 13:39:00 +00:00
5 changed files with 121 additions and 7 deletions
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
# Pinned to 7.4-alpine to match the LXC 225 compose exactly.
# Any redis major-version bump is a separate migration.
- name: redis
image: redis:7.4.10-alpine # {"$imagepolicy": "flux-system:redis-immich"}
image: redis:7.4.11-alpine # {"$imagepolicy": "flux-system:redis-immich"}
args:
- redis-server
- --save
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
type: RuntimeDefault
containers:
- name: redis
image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
args:
- redis-server
- --save
+32 -5
View File
@@ -50,12 +50,39 @@ spec:
# clone + subsequent steps share the workspace PVC and the k8s backend
# can schedule them on different nodes -> an RWO volume then fails to
# mount on the second node. CephFS (RWX) lets the workspace follow
# the pipeline across nodes. The -ephemeral variant (reclaim Delete,
# rook-ceph-cluster/storageclass.yaml) is REQUIRED here: workspaces
# are throwaway, and the Retain class leaked one Released PV per
# finished pipeline (issue #262 follow-up).
# the pipeline across nodes. Reclaim MUST be Delete: workspaces are
# throwaway, and a Retain class leaked one Released PV per finished
# pipeline (issue #262 follow-up).
#
# ceph-filesystem-ci, not ceph-filesystem-ephemeral: a dedicated
# CephFS whose pools are BOTH size 1 (infrastructure/
# rook-ceph-cluster/cephfs-ci.yaml). The shared filesystem
# replicates data0 2x and metadata 3x, which is the correct trade
# for real data and pure waste for a git clone that is deleted
# minutes later.
#
# Measured 2026-08-21, polling both pools every 15s (a 5-minute
# interval misses an entire pipeline -- PVC created, used and
# deleted inside ~2 minutes):
# CI active 1.5min: data0 143.2 MB, metadata 17.2 MB
# CI idle 16.0min: data0 0.0 MB, metadata 11.0 MB
# data0 write volume is 100% CI. Per pipeline this moves the drive
# cost from 71.6x2 + 8.2x3 = ~168 MB to ~80 MB.
#
# ceph-filesystem-ephemeral is NOT repointed, and the reason is a
# technical constraint rather than a safety one. Its other consumer
# is velero/restore-test.sh, whose probe PVCs are snapshot CLONES:
# `ceph fs subvolume snapshot clone` operates within one filesystem,
# so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
#
# Those probes are just as throwaway as a CI workspace -- the
# authoritative copy is the backup on Hetzner, and a lost probe just
# means re-running the test -- so they are equally undeserving of
# replication. Landing them unreplicated would mean a second size-1
# data pool inside `cephfs`, and restore-test runs WEEKLY against
# CI's ~96 pipelines/day, so it is not worth the complexity.
- name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
value: ceph-filesystem-ephemeral
value: ceph-filesystem-ci
- name: WOODPECKER_BACKEND_K8S_STORAGE_RWX
value: "true"
# MUST be a JSON object literal — the backend json.Unmarshal's this
@@ -0,0 +1,86 @@
---
# A SEPARATE CephFS whose only purpose is Woodpecker CI workspaces, with both
# pools at size 1.
#
# WHY THIS EXISTS
# Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
# checkout plus build output, and deletes it when the pipeline ends. Measured
# 2026-08-21 by polling both pools at 15s (a 5-minute interval misses a whole
# pipeline — its PVC is created, used and deleted inside ~2 minutes):
#
# one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
# every interval without a CI pod: data0 ~0 metadata ~0
#
# CI drove essentially all of it. On the shared filesystem that costs
# 71.6x2 + 8.2x3 = ~168 MB of drive writes per pipeline. Here it costs ~80 MB.
#
# WHY size 1 IS CORRECT HERE, HAVING BEEN REJECTED FOR THE MAIN FILESYSTEM
# `cephfs` keeps metadataPool at size 3 because losing it destroys the whole
# filesystem, not the objects on one OSD, and 550 GiB in cephfs-data0 depends
# on it. None of that applies to this filesystem. Every byte in it is a fresh
# git clone; the worst case of total loss is that in-flight pipelines fail and
# get re-run. Replicating throwaway data is the thing being paid for, so it is
# the thing removed.
#
# Losing an OSD takes these PGs offline and kills running pipelines. That is
# not a regression: CI already breaks during Ceph maintenance.
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: cephfs-ci
namespace: rook-ceph
spec:
metadataPool:
replicated:
size: 1
# Rook refuses size 1 without this — it is asserting that an unreplicated
# pool is deliberate rather than a typo. It is.
requireSafeReplicaSize: false
dataPools:
- name: data0
replicated:
size: 1
requireSafeReplicaSize: false
metadataServer:
activeCount: 1
# No standby-replay, unlike `cephfs`. A standby exists there so a failed
# MDS does not hang every CephFS mount cluster-wide; here the only casualty
# is a running pipeline, and a second daemon to protect throwaway work is
# not worth its memory.
activeStandby: false
resources:
# Deliberately NOT the 4Gi that `cephfs` needs. That figure is driven by
# immich's millions of cached inodes; a CI workspace is a few thousand
# short-lived files that vanish with the pipeline.
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 1Gi
---
# CI-only class. `ceph-filesystem-ephemeral` is deliberately left pointing at
# the main filesystem: velero/restore-test.sh uses it to restore REAL data for
# verification, which must not land on unreplicated pools.
#
# A new class rather than repointing the old one — StorageClass.parameters are
# immutable, so `fsName` cannot be edited in place.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-filesystem-ci
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
clusterID: rook-ceph
fsName: cephfs-ci
pool: cephfs-ci-data0
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
# Delete, for the same reason ceph-filesystem-ephemeral uses it: with Retain,
# every finished pipeline leaks a Released PV pinning a subvolume (191 of them
# by 2026-07-16, issue #262 follow-up).
reclaimPolicy: Delete
allowVolumeExpansion: true
@@ -4,5 +4,6 @@ resources:
- cephcluster.yaml
- toolbox.yaml
- storageclass.yaml
- cephfs-ci.yaml
- volumesnapshotclass.yaml
- objectstore.yaml