Compare commits

...
Author SHA1 Message Date
claude-bot 503d2a4876 feat(rook-ceph): add an unreplicated CephFS for CI workspaces
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.

Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.

  one pipeline, a single 15s interval:  data0 +71.6 MB   metadata +8.2 MB
  every interval with no CI pod:        data0   ~0        metadata   ~0

So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.

size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.

`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.

Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.

Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.

The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
2026-08-21 08:55:33 +02:00
security-bot 48407aa202 chore(image-automation): bump redis:8.10.1-alpine
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-20 17:37:50 +00:00
security-bot ce6e8ba2cb chore(image-automation): bump redis:7.4.11-alpine
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-08-20 17:19:40 +00:00
k.renner 96387108ff Merge pull request 'perf(cnpg): roll archive_timeout out to the remaining six clusters' (#541) from rollout-archive-timeout into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#541
2026-08-20 13:39:00 +00:00
4 changed files with 89 additions and 2 deletions
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
# Pinned to 7.4-alpine to match the LXC 225 compose exactly.
# Any redis major-version bump is a separate migration.
- name: redis
image: redis:7.4.10-alpine # {"$imagepolicy": "flux-system:redis-immich"}
image: redis:7.4.11-alpine # {"$imagepolicy": "flux-system:redis-immich"}
args:
- redis-server
- --save
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
type: RuntimeDefault
containers:
- name: redis
image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
args:
- redis-server
- --save
@@ -0,0 +1,86 @@
---
# A SEPARATE CephFS whose only purpose is Woodpecker CI workspaces, with both
# pools at size 1.
#
# WHY THIS EXISTS
# Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
# checkout plus build output, and deletes it when the pipeline ends. Measured
# 2026-08-21 by polling both pools at 15s (a 5-minute interval misses a whole
# pipeline — its PVC is created, used and deleted inside ~2 minutes):
#
# one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
# every interval without a CI pod: data0 ~0 metadata ~0
#
# CI drove essentially all of it. On the shared filesystem that costs
# 71.6x2 + 8.2x3 = ~168 MB of drive writes per pipeline. Here it costs ~80 MB.
#
# WHY size 1 IS CORRECT HERE, HAVING BEEN REJECTED FOR THE MAIN FILESYSTEM
# `cephfs` keeps metadataPool at size 3 because losing it destroys the whole
# filesystem, not the objects on one OSD, and 550 GiB in cephfs-data0 depends
# on it. None of that applies to this filesystem. Every byte in it is a fresh
# git clone; the worst case of total loss is that in-flight pipelines fail and
# get re-run. Replicating throwaway data is the thing being paid for, so it is
# the thing removed.
#
# Losing an OSD takes these PGs offline and kills running pipelines. That is
# not a regression: CI already breaks during Ceph maintenance.
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: cephfs-ci
namespace: rook-ceph
spec:
metadataPool:
replicated:
size: 1
# Rook refuses size 1 without this — it is asserting that an unreplicated
# pool is deliberate rather than a typo. It is.
requireSafeReplicaSize: false
dataPools:
- name: data0
replicated:
size: 1
requireSafeReplicaSize: false
metadataServer:
activeCount: 1
# No standby-replay, unlike `cephfs`. A standby exists there so a failed
# MDS does not hang every CephFS mount cluster-wide; here the only casualty
# is a running pipeline, and a second daemon to protect throwaway work is
# not worth its memory.
activeStandby: false
resources:
# Deliberately NOT the 4Gi that `cephfs` needs. That figure is driven by
# immich's millions of cached inodes; a CI workspace is a few thousand
# short-lived files that vanish with the pipeline.
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 1Gi
---
# CI-only class. `ceph-filesystem-ephemeral` is deliberately left pointing at
# the main filesystem: velero/restore-test.sh uses it to restore REAL data for
# verification, which must not land on unreplicated pools.
#
# A new class rather than repointing the old one — StorageClass.parameters are
# immutable, so `fsName` cannot be edited in place.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-filesystem-ci
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
clusterID: rook-ceph
fsName: cephfs-ci
pool: cephfs-ci-data0
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
# Delete, for the same reason ceph-filesystem-ephemeral uses it: with Retain,
# every finished pipeline leaks a Released PV pinning a subvolume (191 of them
# by 2026-07-16, issue #262 follow-up).
reclaimPolicy: Delete
allowVolumeExpansion: true
@@ -4,5 +4,6 @@ resources:
- cephcluster.yaml
- toolbox.yaml
- storageclass.yaml
- cephfs-ci.yaml
- volumesnapshotclass.yaml
- objectstore.yaml