konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

fix(rook-ceph): ship per-driver CSI ServiceAccounts for v1.20 operator

Browse Source 
The rook-ceph HelmRelease's floating `>=1.16.0` constraint silently
upgraded to v1.20.0, which extracts CSI driver management into the
upstream ceph-csi-operator subchart. That subchart's generated
ctrlplugin Deployments and nodeplugin DaemonSets reference per-driver
ServiceAccounts (`rbd-ctrlplugin-sa`, `cephfs-ctrlplugin-sa`,
`rbd-nodeplugin-sa`, `cephfs-nodeplugin-sa`) that the main rook-ceph
chart no longer ships — they moved to a separate ceph-csi-drivers
chart. With the SAs missing, both ctrlplugin Deployments sat at 0/2
with FailedCreate for 5 days, blocking PVC attach/detach across the
cluster (Grafana, Alertmanager, libation stuck; Home Assistant pinned
to the edge node which never got an RBD nodeplugin pod).

The ceph-csi-drivers chart cannot be installed cleanly here:
helm-controller v1.5.4 predates `disableTakeOwnership`, so Helm would
refuse to adopt the operator-managed `Driver` and `OperatorConfig` CRs
the v1.18 -> v1.20 auto-migration created. Instead, render only the
SAs + RBAC the operator needs from the chart's templates as static
manifests under kubernetes/infrastructure/ceph-csi-drivers/. The
existing operator-managed CRs continue to be the source of truth.

Also pin the rook-ceph chart to `~1.20.0` to prevent the same
floating-constraint surprise on the next minor bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
k.renner
2026-06-08 20:50:16 +02:00
co-authored by Claude Opus 4.7
parent f52b69152a
commit babed07871
5 changed files with 491 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/infrastructure/ceph-csi-drivers/kustomization.yaml
+5
View File
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- serviceaccounts.yaml
- rbac.yaml
kubernetes/infrastructure/ceph-csi-drivers/rbac.yaml
+442
View File
@@ -0,0 +1,442 @@
# RBAC for the per-driver ServiceAccounts in serviceaccounts.yaml.
# Rendered from ceph-csi-operator/ceph-csi-drivers chart v1.0.1 templates,
# with $normalizedDriverName fixed to the operator's hardcoded driver-type
# names (`rbd`, `cephfs`). Sync with upstream if the chart's RBAC changes.
# Source: https://github.com/ceph/ceph-csi-operator/tree/v1.0.1/deploy/charts/ceph-csi-drivers/templates
# ---------- rbd-ctrlplugin ----------
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rbd-ctrlplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list, watch, create, delete, patch, update]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get, list, watch, update]
- apiGroups: [storage.k8s.io]
resources: [storageclasses]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list, watch, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments/status]
verbs: [patch]
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, watch]
- apiGroups: [storage.k8s.io]
resources: [csinodes]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumeclaims/status]
verbs: [patch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshots]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents]
verbs: [get, list, watch, patch, update]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationcontents]
verbs: [get, list, watch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationclasses]
verbs: [get, list, watch]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
- apiGroups: [authorization.k8s.io]
resources: [subjectaccessreviews]
verbs: [create]
- apiGroups: [cbt.storage.k8s.io]
resources: [snapshotmetadataservices]
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rbd-ctrlplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rbd-ctrlplugin-cr
subjects:
- kind: ServiceAccount
name: rbd-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-ctrlplugin-r
namespace: rook-ceph
rules:
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [get, watch, list, delete, update, create]
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-ctrlplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-ctrlplugin-r
subjects:
- kind: ServiceAccount
name: rbd-ctrlplugin-sa
namespace: rook-ceph
# ---------- rbd-nodeplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rbd-nodeplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [""]
resources: [nodes]
verbs: [get]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rbd-nodeplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rbd-nodeplugin-cr
subjects:
- kind: ServiceAccount
name: rbd-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-nodeplugin-r
namespace: rook-ceph
rules:
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-nodeplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-nodeplugin-r
subjects:
- kind: ServiceAccount
name: rbd-nodeplugin-sa
namespace: rook-ceph
# ---------- cephfs-ctrlplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephfs-ctrlplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, watch]
- apiGroups: [storage.k8s.io]
resources: [csinodes]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list, watch, create, delete, patch, update]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get, list, watch, patch, update]
- apiGroups: [storage.k8s.io]
resources: [storageclasses]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list, watch, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments/status]
verbs: [patch]
- apiGroups: [""]
resources: [persistentvolumeclaims/status]
verbs: [patch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshots]
verbs: [get, list]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents]
verbs: [get, list, watch, patch, update]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationcontents]
verbs: [get, list, watch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationclasses]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephfs-ctrlplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephfs-ctrlplugin-cr
subjects:
- kind: ServiceAccount
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-ctrlplugin-r
namespace: rook-ceph
rules:
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [get, watch, list, delete, update, create]
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-ctrlplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-ctrlplugin-r
subjects:
- kind: ServiceAccount
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
# ---------- cephfs-nodeplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephfs-nodeplugin-cr
rules:
- apiGroups: [""]
resources: [nodes]
verbs: [get]
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [""]
resources: [persistentvolumes, persistentvolumeclaims]
verbs: [get]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephfs-nodeplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephfs-nodeplugin-cr
subjects:
- kind: ServiceAccount
name: cephfs-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-nodeplugin-r
namespace: rook-ceph
rules:
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-nodeplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-nodeplugin-r
subjects:
- kind: ServiceAccount
name: cephfs-nodeplugin-sa
namespace: rook-ceph
kubernetes/infrastructure/ceph-csi-drivers/serviceaccounts.yaml
+35
View File
@@ -0,0 +1,35 @@
# Per-driver ServiceAccounts consumed by the workloads that the
# ceph-csi-controller-manager generates for each `Driver` CR. Rook v1.20
# extracted CSI driver management into the upstream ceph-csi-operator
# (subchart of `rook-ceph`); these SAs are normally provisioned by the
# separate `ceph-csi-drivers` chart, but installing that chart would also
# create a parallel set of `Driver` / `OperatorConfig` CRs and Helm cannot
# adopt the operator-managed ones on helm-controller v1.5.4. We render
# only the SAs + RBAC the existing operator-managed `Driver` CRs need.
#
# Names are fixed by the operator (CSI_SERVICE_ACCOUNT_PREFIX="" on
# ceph-csi-controller-manager): `<driver-type>-{ctrl,node}plugin-sa`,
# where driver-type is `rbd` or `cephfs`. Do not rename.
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-nodeplugin-sa
namespace: rook-ceph
kubernetes/infrastructure/kustomization.yaml
+1
View File
@@ -13,6 +13,7 @@ resources:
- crowdsec/
- traefik/
- rook-ceph/
- ceph-csi-drivers/
- cloudnative-pg/
- cloudflared/
- coredns-config/
kubernetes/infrastructure/rook-ceph/helmrelease-operator.yaml
+8 -1
View File
@@ -17,7 +17,14 @@ spec:
chart:
spec:
chart: rook-ceph
version: ">=1.16.0"
# Pinned to a single minor. The previous `>=1.16.0` floating
# constraint silently upgraded to v1.20.0, which extracts CSI driver
# management into the upstream ceph-csi-operator subchart but no
# longer ships the per-driver ServiceAccounts the generated CSI
# workloads reference — those now come from a separate
# `ceph-csi-drivers` chart (we render the SAs + RBAC statically
# in ../ceph-csi-drivers). Bump deliberately on the next minor.
version: "~1.20.0"
sourceRef:
kind: HelmRepository
name: rook-release