The VeleroBackupFailed alert fired because the apps-daily backup was PartiallyFailed for two consecutive days (2026-07-15 and 2026-07-16). The root cause is that kopia reported "structure needs cleaning" (ENOTCLEAN) when attempting to snapshot the data-woodpecker-server-0 PVC (Woodpecker CI server's data volume). This happens because the current node-agent-config configures ceph-block PVCs to be exposed as direct read-only RBD snapshot mounts using the ceph-block-snapshot-readonly StorageClass (which adds mountOptions: [noload]). The noload option bypasses ext4 journal replay on mount — sufficient for a dirty-but-intact journal, but not for a snapshot that captures structurally inconsistent ext4 metadata mid-write (e.g. an in-progress directory update). Kopia's directory traversal then hits ENOTCLEAN on the mounted snapshot and fails. The fix removes the ceph-block entry from backupPVC in node-agent-config, reverting ceph-block backups to the default RW clone path. Ceph RBD clones are copy-on-write (instantaneous metadata copy), so the overhead is negligible; the clone is mounted read-write, ext4 replays the journal and resolves any inconsistencies, and kopia reads a clean filesystem.
Changes
kubernetes/infrastructure/velero/node-agent-config.yaml — removed the ceph-block backupPVC entry (storageClass: ceph-block-snapshot-readonly, readOnly: true); ceph-block PVCs now use the default Velero clone path (RW clone with source StorageClass). The ceph-filesystem readOnly entry is retained since CephFS snapshots do not have the ext4 journal issue and the 2.5h clone savings for large CephFS subvolumes remain valuable.
Verification
Observed via ./.ci-diag/kubectl:
kubectl get datauploads.velero.io -n velero apps-daily-20260716071534-2s5zd -o jsonpath='{.status}' → "phase":"Failed","message":"...Failed to run kopia backup: Failed to upload the kopia snapshot for si default@default:snapshot-data-upload-download/kopia/apps/data-woodpecker-server-0: structure needs cleaning" — same error on apps-daily-20260715071533-tf5rb (yesterday).
kubectl get backups.velero.io -n velero apps-daily-20260716071534 -o jsonpath='{.status.phase}' → PartiallyFailed (one DataUpload out of four failed).
kubectl logs -n velero node-agent-hpd2k (k8s-2) — confirmed the data-mover pod starts, the backup PVC binds and mounts successfully, the kopia repo opens, but kopia fails at Searching for parent snapshot / Unable to estimate with the "structure needs cleaning" error. The CSI layer works: PVC provisioned, snapshot created, pod scheduled.
kubectl get backups.velero.io -n velero home-assistant-daily-20260716064534 -o jsonpath='{.status.phase}' → Completed (all 10 DataUploads Completed). The triage agent's hypothesis that home-assistant's orphaned attachments blocked apps-daily is incorrect.
kubectl get pvc -n apps data-woodpecker-server-0 → Bound, 2Gi, ceph-block, 3d12h old. kubectl get pods -n apps woodpecker-server-0 → 1/1 Running, 0 restarts, 2d2h — the live PVC is healthy; the issue is specific to the RBD snapshot's captured on-disk state.
How to confirm after Flux reconciles:
Flux will reconcile the ConfigMap within its next sync interval (~30m). Verify with: kubectl get configmap -n velero node-agent-config -o jsonpath='{.data.node-agent-config}' — the ceph-block key should be absent.
The next apps-daily schedule runs at 07:15 UTC. The data-woodpecker-server-0 DataUpload should show Completed instead of Failed.
kubectl get backups.velero.io -n velero apps-daily-<next-date>071534 -o jsonpath='{.status.phase}' should be Completed.
The VeleroBackupFailed alert should stop firing once the backup completes without errors.
Notes
The triage agent's root-cause hypothesis was wrong. It attributed the failure to stuck VolumeAttachments and orphaned PVCs from the home-assistant-daily run. Live verification shows: home-assistant completed successfully (all 10 DataUploads Completed); there are no orphaned Velero PVCs; the node-agent logs confirm the CSI snapshot and PVC bind both succeed — the failure is a kopia filesystem read error on the mounted snapshot, not a scheduling or attachment issue.
The SnapshotContentObjectDeleteError and VolumeFailedDelete events cited in the issue were not present in the current cluster state and appear to have been from a transient prior condition that has since resolved.
The ceph-block-snapshot-readonly StorageClass (snapshot-readonly-storageclass.yaml) is no longer referenced by node-agent-config after this change but remains in the cluster without harm. It can be deleted in a separate cleanup PR once confirmed unneeded.
If the data-woodpecker-server-0 backup continues to fail after this change (unlikely but possible if the filesystem corruption is deeper than a dirty journal), the recommended manual action is to run fsck on the source volume offline: scale down woodpecker-server, attach the PVC to a debug pod on a different node without noload, and run fsck.ext4 -p <device>.
## What this PR does
The `VeleroBackupFailed` alert fired because the `apps-daily` backup was `PartiallyFailed` for two consecutive days (2026-07-15 and 2026-07-16). The root cause is that kopia reported "structure needs cleaning" (`ENOTCLEAN`) when attempting to snapshot the `data-woodpecker-server-0` PVC (Woodpecker CI server's data volume). This happens because the current `node-agent-config` configures ceph-block PVCs to be exposed as direct read-only RBD snapshot mounts using the `ceph-block-snapshot-readonly` StorageClass (which adds `mountOptions: [noload]`). The `noload` option bypasses ext4 journal replay on mount — sufficient for a dirty-but-intact journal, but not for a snapshot that captures structurally inconsistent ext4 metadata mid-write (e.g. an in-progress directory update). Kopia's directory traversal then hits `ENOTCLEAN` on the mounted snapshot and fails. The fix removes the `ceph-block` entry from `backupPVC` in `node-agent-config`, reverting ceph-block backups to the default RW clone path. Ceph RBD clones are copy-on-write (instantaneous metadata copy), so the overhead is negligible; the clone is mounted read-write, ext4 replays the journal and resolves any inconsistencies, and kopia reads a clean filesystem.
## Changes
- `kubernetes/infrastructure/velero/node-agent-config.yaml` — removed the `ceph-block` backupPVC entry (`storageClass: ceph-block-snapshot-readonly`, `readOnly: true`); ceph-block PVCs now use the default Velero clone path (RW clone with source StorageClass). The `ceph-filesystem` readOnly entry is retained since CephFS snapshots do not have the ext4 journal issue and the 2.5h clone savings for large CephFS subvolumes remain valuable.
## Verification
**Observed via `./.ci-diag/kubectl`:**
- `kubectl get datauploads.velero.io -n velero apps-daily-20260716071534-2s5zd -o jsonpath='{.status}'` → `"phase":"Failed","message":"...Failed to run kopia backup: Failed to upload the kopia snapshot for si default@default:snapshot-data-upload-download/kopia/apps/data-woodpecker-server-0: structure needs cleaning"` — same error on `apps-daily-20260715071533-tf5rb` (yesterday).
- `kubectl get backups.velero.io -n velero apps-daily-20260716071534 -o jsonpath='{.status.phase}'` → `PartiallyFailed` (one DataUpload out of four failed).
- `kubectl logs -n velero node-agent-hpd2k` (k8s-2) — confirmed the data-mover pod starts, the backup PVC binds and mounts successfully, the kopia repo opens, but kopia fails at `Searching for parent snapshot` / `Unable to estimate` with the "structure needs cleaning" error. The CSI layer works: PVC provisioned, snapshot created, pod scheduled.
- `kubectl get backups.velero.io -n velero home-assistant-daily-20260716064534 -o jsonpath='{.status.phase}'` → `Completed` (all 10 DataUploads Completed). The triage agent's hypothesis that home-assistant's orphaned attachments blocked apps-daily is incorrect.
- `kubectl get pvc -n apps data-woodpecker-server-0` → Bound, 2Gi, ceph-block, 3d12h old. `kubectl get pods -n apps woodpecker-server-0` → 1/1 Running, 0 restarts, 2d2h — the live PVC is healthy; the issue is specific to the RBD snapshot's captured on-disk state.
**How to confirm after Flux reconciles:**
1. Flux will reconcile the ConfigMap within its next sync interval (~30m). Verify with: `kubectl get configmap -n velero node-agent-config -o jsonpath='{.data.node-agent-config}'` — the `ceph-block` key should be absent.
2. The next `apps-daily` schedule runs at 07:15 UTC. The `data-woodpecker-server-0` DataUpload should show `Completed` instead of `Failed`.
3. `kubectl get backups.velero.io -n velero apps-daily-<next-date>071534 -o jsonpath='{.status.phase}'` should be `Completed`.
4. The `VeleroBackupFailed` alert should stop firing once the backup completes without errors.
## Notes
**The triage agent's root-cause hypothesis was wrong.** It attributed the failure to stuck VolumeAttachments and orphaned PVCs from the `home-assistant-daily` run. Live verification shows: home-assistant completed successfully (all 10 DataUploads `Completed`); there are no orphaned Velero PVCs; the node-agent logs confirm the CSI snapshot and PVC bind both succeed — the failure is a kopia filesystem read error on the mounted snapshot, not a scheduling or attachment issue.
The `SnapshotContentObjectDeleteError` and `VolumeFailedDelete` events cited in the issue were not present in the current cluster state and appear to have been from a transient prior condition that has since resolved.
The `ceph-block-snapshot-readonly` StorageClass (`snapshot-readonly-storageclass.yaml`) is no longer referenced by `node-agent-config` after this change but remains in the cluster without harm. It can be deleted in a separate cleanup PR once confirmed unneeded.
If the `data-woodpecker-server-0` backup continues to fail after this change (unlikely but possible if the filesystem corruption is deeper than a dirty journal), the recommended manual action is to run `fsck` on the source volume offline: scale down `woodpecker-server`, attach the PVC to a debug pod on a different node without `noload`, and run `fsck.ext4 -p <device>`.
Closes #251
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What this PR does
The
VeleroBackupFailedalert fired because theapps-dailybackup wasPartiallyFailedfor two consecutive days (2026-07-15 and 2026-07-16). The root cause is that kopia reported "structure needs cleaning" (ENOTCLEAN) when attempting to snapshot thedata-woodpecker-server-0PVC (Woodpecker CI server's data volume). This happens because the currentnode-agent-configconfigures ceph-block PVCs to be exposed as direct read-only RBD snapshot mounts using theceph-block-snapshot-readonlyStorageClass (which addsmountOptions: [noload]). Thenoloadoption bypasses ext4 journal replay on mount — sufficient for a dirty-but-intact journal, but not for a snapshot that captures structurally inconsistent ext4 metadata mid-write (e.g. an in-progress directory update). Kopia's directory traversal then hitsENOTCLEANon the mounted snapshot and fails. The fix removes theceph-blockentry frombackupPVCinnode-agent-config, reverting ceph-block backups to the default RW clone path. Ceph RBD clones are copy-on-write (instantaneous metadata copy), so the overhead is negligible; the clone is mounted read-write, ext4 replays the journal and resolves any inconsistencies, and kopia reads a clean filesystem.Changes
kubernetes/infrastructure/velero/node-agent-config.yaml— removed theceph-blockbackupPVC entry (storageClass: ceph-block-snapshot-readonly,readOnly: true); ceph-block PVCs now use the default Velero clone path (RW clone with source StorageClass). Theceph-filesystemreadOnly entry is retained since CephFS snapshots do not have the ext4 journal issue and the 2.5h clone savings for large CephFS subvolumes remain valuable.Verification
Observed via
./.ci-diag/kubectl:kubectl get datauploads.velero.io -n velero apps-daily-20260716071534-2s5zd -o jsonpath='{.status}'→"phase":"Failed","message":"...Failed to run kopia backup: Failed to upload the kopia snapshot for si default@default:snapshot-data-upload-download/kopia/apps/data-woodpecker-server-0: structure needs cleaning"— same error onapps-daily-20260715071533-tf5rb(yesterday).kubectl get backups.velero.io -n velero apps-daily-20260716071534 -o jsonpath='{.status.phase}'→PartiallyFailed(one DataUpload out of four failed).kubectl logs -n velero node-agent-hpd2k(k8s-2) — confirmed the data-mover pod starts, the backup PVC binds and mounts successfully, the kopia repo opens, but kopia fails atSearching for parent snapshot/Unable to estimatewith the "structure needs cleaning" error. The CSI layer works: PVC provisioned, snapshot created, pod scheduled.kubectl get backups.velero.io -n velero home-assistant-daily-20260716064534 -o jsonpath='{.status.phase}'→Completed(all 10 DataUploads Completed). The triage agent's hypothesis that home-assistant's orphaned attachments blocked apps-daily is incorrect.kubectl get pvc -n apps data-woodpecker-server-0→ Bound, 2Gi, ceph-block, 3d12h old.kubectl get pods -n apps woodpecker-server-0→ 1/1 Running, 0 restarts, 2d2h — the live PVC is healthy; the issue is specific to the RBD snapshot's captured on-disk state.How to confirm after Flux reconciles:
kubectl get configmap -n velero node-agent-config -o jsonpath='{.data.node-agent-config}'— theceph-blockkey should be absent.apps-dailyschedule runs at 07:15 UTC. Thedata-woodpecker-server-0DataUpload should showCompletedinstead ofFailed.kubectl get backups.velero.io -n velero apps-daily-<next-date>071534 -o jsonpath='{.status.phase}'should beCompleted.VeleroBackupFailedalert should stop firing once the backup completes without errors.Notes
The triage agent's root-cause hypothesis was wrong. It attributed the failure to stuck VolumeAttachments and orphaned PVCs from the
home-assistant-dailyrun. Live verification shows: home-assistant completed successfully (all 10 DataUploadsCompleted); there are no orphaned Velero PVCs; the node-agent logs confirm the CSI snapshot and PVC bind both succeed — the failure is a kopia filesystem read error on the mounted snapshot, not a scheduling or attachment issue.The
SnapshotContentObjectDeleteErrorandVolumeFailedDeleteevents cited in the issue were not present in the current cluster state and appear to have been from a transient prior condition that has since resolved.The
ceph-block-snapshot-readonlyStorageClass (snapshot-readonly-storageclass.yaml) is no longer referenced bynode-agent-configafter this change but remains in the cluster without harm. It can be deleted in a separate cleanup PR once confirmed unneeded.If the
data-woodpecker-server-0backup continues to fail after this change (unlikely but possible if the filesystem corruption is deeper than a dirty journal), the recommended manual action is to runfsckon the source volume offline: scale downwoodpecker-server, attach the PVC to a debug pod on a different node withoutnoload, and runfsck.ext4 -p <device>.Closes #251