gitea: grow data volume 40Gi -> 60Gi via Flux-owned PVC, add low-space alert (REN-583) #1116

2026-10-03T21:16:25Z

Ghost commented

2026-10-03 21:16:25 +00:00

What

apps/gitea/pvc.yaml: Flux now owns the live claim data-gitea-0 at 60Gi (was 40Gi). prune: disabled, so deleting the file never deletes the claim.
statefulset.yaml: template stays 40Gi (immutable); comment now points to pvc.yaml.
monitoring-rules/prometheusrule-gitea.yaml: GiteaDataVolumeLow, warning when under 25% free for 15m. Has a promtool suite (tests/gitea_test.yaml) and two mutation controls.

Why

The volume filled on REN-580: image pushes returned HTTP 500 and SQLite writes failed. volumeClaimTemplates is immutable, so the size can't change there. A PVC's requests.storage can grow in place, and Flux SSA on the existing claim triggers a ceph-csi online expansion with no pod restart (same mechanism as the July #336 resize, done by hand then). Name, accessModes, storageClassName and volumeMode match the live claim. If anything mismatched, the apply would fail and the claim would stay as it is.

The chart's KubePersistentVolumeFillingUp warning fired at 20:26Z and the volume was full at 20:59Z, so it gave 33 minutes of notice. Growth comes in steps: 30 -> 38 GiB in hours on 2026-10-03.

Ceph capacity (Prometheus via apiserver proxy, REN-153 path, 2026-10-03 ~21:10Z)

replicapool (ceph-block): stored 283 GiB, max_avail 372.5 GiB. +20 GiB is ~5% of that.
Raw: 1430 / 2683 GiB used.
Gitea now: 38.7 GiB used of 39.2 GiB.

Gitea volume before resize (21:16Z)

kubelet volume stats (the statfs of the mount, same as df /var/lib/gitea; I have no pod exec): size 39.2 GiB, used 36.6 GiB, avail 2.6 GiB. I'll add the after numbers once it's live.

Gitea volume after resize (21:35Z, live via #1117 at `77c8566c`)

kubelet volume stats for data-gitea-0 (statfs of the mount): size 58.9 GiB, used 36.6 GiB, avail 22.3 GiB. PVC spec and status are 60Gi. The filesystem grew online; gitea-0 was not restarted (it has run since 10-01 15:08Z). GiteaDataVolumeLow is loaded, health ok, inactive.

Verified

kustomize build apps/gitea renders the PVC; apps/monitoring-rules renders the rule.
scripts/ci/promtool-rules.sh (promtool 3.5.0): 7 suites green, 31 mutations caught, including gitea-for-10m and gitea-floor-le.
Linted only. The live resize happens on merge; I'll check PVC capacity and df in the pod after Flux reconciles.

Revert

Revert the commit. The claim stays at 60Gi (it can't shrink, and prune is disabled), and Flux stops managing its size. The alert is removed.

🤖 Generated with Claude Code

## What - `apps/gitea/pvc.yaml`: Flux now owns the live claim `data-gitea-0` at **60Gi** (was 40Gi). `prune: disabled`, so deleting the file never deletes the claim. - `statefulset.yaml`: template stays 40Gi (immutable); comment now points to `pvc.yaml`. - `monitoring-rules/prometheusrule-gitea.yaml`: `GiteaDataVolumeLow`, warning when under 25% free for 15m. Has a promtool suite (`tests/gitea_test.yaml`) and two mutation controls. ## Why The volume filled on REN-580: image pushes returned HTTP 500 and SQLite writes failed. `volumeClaimTemplates` is immutable, so the size can't change there. A PVC's `requests.storage` can grow in place, and Flux SSA on the existing claim triggers a ceph-csi online expansion with no pod restart (same mechanism as the July #336 resize, done by hand then). Name, accessModes, storageClassName and volumeMode match the live claim. If anything mismatched, the apply would fail and the claim would stay as it is. The chart's `KubePersistentVolumeFillingUp` warning fired at 20:26Z and the volume was full at 20:59Z, so it gave 33 minutes of notice. Growth comes in steps: 30 -> 38 GiB in hours on 2026-10-03. ## Ceph capacity (Prometheus via apiserver proxy, REN-153 path, 2026-10-03 ~21:10Z) - `replicapool` (ceph-block): stored 283 GiB, **max_avail 372.5 GiB**. +20 GiB is ~5% of that. - Raw: 1430 / 2683 GiB used. - Gitea now: 38.7 GiB used of 39.2 GiB. ## Gitea volume before resize (21:16Z) kubelet volume stats (the statfs of the mount, same as `df /var/lib/gitea`; I have no pod exec): size 39.2 GiB, used 36.6 GiB, avail 2.6 GiB. I'll add the after numbers once it's live. ## Gitea volume after resize (21:35Z, live via #1117 at 77c8566c) kubelet volume stats for data-gitea-0 (statfs of the mount): size 58.9 GiB, used 36.6 GiB, avail 22.3 GiB. PVC spec and status are 60Gi. The filesystem grew online; gitea-0 was not restarted (it has run since 10-01 15:08Z). `GiteaDataVolumeLow` is loaded, health ok, inactive. ## Verified - `kustomize build apps/gitea` renders the PVC; `apps/monitoring-rules` renders the rule. - `scripts/ci/promtool-rules.sh` (promtool 3.5.0): 7 suites green, 31 mutations caught, including `gitea-for-10m` and `gitea-floor-le`. - Linted only. The live resize happens on merge; I'll check PVC capacity and `df` in the pod after Flux reconciles. ## Revert Revert the commit. The claim stays at 60Gi (it can't shrink, and prune is disabled), and Flux stops managing its size. The alert is removed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-10-03 21:16:26 +00:00

gitea: grow data volume 40Gi -> 60Gi via a Flux-owned PVC, add low-space alert (REN-583)

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline failed

Details

433719bfc0

The volume filled on REN-580. volumeClaimTemplates is immutable, so own the
live claim data-gitea-0 in apps/gitea/pvc.yaml (prune disabled) and let Flux
server-side apply grow it; ceph-block expands online. Add GiteaDataVolumeLow
(<25% free for 15m) with a promtool suite and two mutation controls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ghost approved these changes

2026-10-03 21:26:20 +00:00

Ghost left a comment

Approved on 433719bfc0. Checked: live data-gitea-0 matches name/accessModes/storageClassName/volumeMode, 40Gi, no ownerRefs; ceph-block allowVolumeExpansion: true; Flux gitea Kustomization force:false (immutable mismatch fails, no recreate), prune annotation keeps claim on revert. kustomize build renders the PVC at 60Gi. promtool-rules.sh (promtool 3.5.0): 21 manifests, 7 suites green, 31 mutations caught incl. gitea-for-10m, gitea-floor-le. Prometheus now: replicapool max_avail 372.5 GiB; data-gitea-0 2.6 GiB free of 39.2 GiB.

Approved on 433719bfc0cfbb1f4c3cde82914482ce7fbc2234. Checked: live data-gitea-0 matches name/accessModes/storageClassName/volumeMode, 40Gi, no ownerRefs; ceph-block allowVolumeExpansion: true; Flux gitea Kustomization force:false (immutable mismatch fails, no recreate), prune annotation keeps claim on revert. kustomize build renders the PVC at 60Gi. promtool-rules.sh (promtool 3.5.0): 21 manifests, 7 suites green, 31 mutations caught incl. gitea-for-10m, gitea-floor-le. Prometheus now: replicapool max_avail 372.5 GiB; data-gitea-0 2.6 GiB free of 39.2 GiB.

Ghost merged commit 4de7b0c49b into master

2026-10-03 21:26:21 +00:00

Ghost referenced this issue from a commit

2026-10-03 21:26:22 +00:00

Merge pull request 'gitea: grow data volume 40Gi -> 60Gi via Flux-owned PVC, add low-space alert (REN-583)' (#1116) from atlas/ren-583-gitea-volume-60gi into master

Ghost referenced this issue from a commit

2026-10-03 21:29:22 +00:00

gitea: keep volumeName on the data-gitea-0 PVC so the 60Gi apply passes (REN-583)

Ghost referenced this pull request

2026-10-03 21:29:31 +00:00

gitea: keep volumeName on data-gitea-0 so the 60Gi apply passes (REN-583, fixes #1116) #1117

Ghost referenced this issue from a commit