feat(ceph): add the toolbox, move image GC off Ceph's warn line, alert on health #500
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#500
Reference in New Issue
Block a user
Addresses #434 — Gap 2, the parked alerting follow-up, and a threshold bug found while checking the issue's status.
Gap 1 was already closed
PR #432 landed and worked:
ceph_health_statusreturns from Prometheus today, against "zeroceph_*series for all 79 days" when the issue was filed. Nothing to do.Gap 2 — the toolbox
Still no
rook-ceph-tools;ceph status,ceph health detail,ceph osd df tree,ceph osd perf,ceph device lsare all unreachable during an incident. Adds the upstream manifest from rook v1.20.4, matching the running operator imagedocker.io/rook/ceph:v1.20.4, taken verbatim with three deliberate changes:quay.io/ceph/ceph:v19.2.5(not upstream's:v20) with the same$imagepolicymarker ascephcluster.yaml, so the client cannot drift ahead of the mons it talks toPrerequisites verified present, not assumed:
rook-ceph-defaultSA,rook-ceph-mon-endpointsConfigMap, andceph-secret+ceph-usernamein therook-ceph-monsecret. Namespace PSA isprivileged, and the namespace CNP restricts ingress only (no egress rules), so it can reach the hostNetwork mons.⚠️ It holds the client.admin keyring — a full-privilege credential, not the read-only path the issue wished for. Rook ships no lesser-privileged toolbox. Ingress is gated by the namespace
default-deny-ingressCNP.The threshold bug
The cluster was in
HEALTH_WARN / MON_DISK_LOWwhen I checked. The cause is not Ceph growth:Mons are on
hostPath, so this is node/var. Ceph'smon_data_avail_warnis ≤30% free.imageGCHighThresholdPercentwas 70 — i.e. GC first fires at exactly 30% free.The thresholds had already been lowered once from the kubelet defaults for precisely this reason; the comment says 70/60 "keeps /var safely under 30%-free", but 70% used is 30% free. It sits on the line rather than below it, so the cache fills to the warn point every time and MON_DISK_LOW trips before kubelet reclaims anything. It cleared to 31.64% on its own once GC ran — which is the oscillation, not a refutation.
60/50 fires at 40% free and reclaims to 50%, leaving 10 points of margin.
I checked that the lower threshold will actually bite rather than just trigger earlier and stall:
/var— the dominant local consumertalosctlattributes to/var/lib/kubeletis mounted Ceph PVCs counted through mount points, not local disk)Alerting
The issue parked this until
ceph_*series existed. They do now.CephHealthWarn—== 1for 6h, deliberately long. Ceph warns for self-clearing things (a slow BlueStore op during the Sunday restore sweep, a mon briefly under its disk threshold); paging on those teaches people to ignore the alert. A warning that does not clear is the signal.CephHealthError—== 2for 5m.CephMonDiskLow— watches/varfree directly rather than via Ceph, because this fires for a full image cache, not Ceph growth. Naming that keeps the next person from debugging the wrong subsystem.Verification
kubectl kustomizebuilds clean forrook-ceph-clusterandmonitoring-rules.kubectl apply --dry-run=serveraccepts the Deployment and the PrometheusRule.CephMonDiskLow's expression runs against live Prometheus — currently 0 nodes below 30%, the correct present-state answer.cluster.yamlparses.Read before merging
cluster.yamlis a bootstrap patch — merging changes nothing on running nodes. The live kubelet value needstalosctl patch machineconfig. Same trap the reviewer caught on #495; flagging it rather than letting it be discovered later.Not done
The issue's other follow-up — auditing whether operator-managed ServiceMonitors besides Rook's are being dropped by the
release: kube-prometheus-stackselector — is untouched and still open. It deserves its own pass rather than being bundled here.🤖 Automated review — model
CLI default(complexity: complex), commit16360aacf94f33d686301aa3f9504a3b80590c62.Summary
Three independent, well-scoped changes: (1) adds a
rook-ceph-toolsDeployment sourced from upstream Rook v1.20.4 with a pinned Ceph image and resource limits, (2) lowers Talos kubelet image-GC thresholds from 70/60 to 60/50 so GC fires before crossing Ceph'smon_data_avail_warn(30% free) line, and (3) adds aPrometheusRulewithCephHealthWarn,CephHealthError, andCephMonDiskLowalerts. The GC-threshold arithmetic checks out (a 70% "high" threshold triggers GC exactly at the 30%-free warn line, so 60% gives real margin), the new PrometheusRule follows the existing repo conventions (release: kube-prometheus-stacklabel, alert/severity/for style matchesprometheusrule-cilium.yaml), and the toolbox manifest'squay.io/ceph/ceph:v19.2.5image plus{"$imagepolicy": "flux-system:ceph"}marker matchescephcluster.yamland thecephImagePolicyinkubernetes/infrastructure/image-automation/ceph.yaml. No blocking issues found.🟡 Suggestions
kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:64-65—CephMonDiskLow's expression compares against Ceph'smon_data_avail_warn(documented/described as "≤30% free") using a strict< 30. At exactly 30.00% free Ceph would already be warning but this alert would not yet have crossed its own threshold. In practice float noise onnode_filesystem_*_bytesmakes this edge case very unlikely to matter, but<= 30would match the semantics described in the alert's own comment more precisely.kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml:126—dnsPolicy: ClusterFirstWithHostNetis normally paired withhostNetwork: true; this pod does not sethostNetwork, so it runs on the regular pod network withClusterFirstWithHostNetDNS policy (which still resolves cluster DNS correctly, just via a slightly different code path than plainClusterFirst). Since the PR states the toolbox is "taken verbatim" from upstream, this is presumably copied as-is fromrook/rook's example manifest rather than a change introduced here — worth a one-line note in the header comment if it was a conscious decision to keep it despite not using host networking, otherwise harmless.https://raw.githubusercontent.com/rook/rook/v1.20.4/deploy/examples/toolbox.yamlto independently confirm the "verbatim + three deliberate changes" claim; the script body and structure read as a faithful copy of the well-known Rook toolbox template, so this is a low-risk gap, not a specific concern.🟢 Nits
None.
Tests
No automated tests exist for this repo's Kubernetes/Talos manifests (it's a GitOps homelab config repo, not an application with a test suite), and none of the sibling PrometheusRule/manifest additions carry tests either — consistent with repo norms. The PR body documents concrete manual verification instead (
kubectl kustomizebuild,kubectl apply --dry-run=server, live Prometheus query forCephMonDiskLow,cluster.yamlparse), which is the right level of rigor for this kind of change and mirrors how other infra PRs in this repo are validated. I was not able to independently re-run these checks (nokubectl/cluster access in this environment), but the manifests are internally consistent with the rest of the checkout: the kustomization files reference the new resources in correct alphabetical order, theflux-system:cephimage-policy marker matches the existingImagePolicy, and the referencedrook-ceph-monsecret keys /rook-ceph-mon-endpointsConfigMap names match howcephcluster.yamland the rest of therook-ceph-clusterkustomization refer to them.One thing worth flagging explicitly (the PR body already calls this out prominently under "Read before merging"):
kubernetes/talos/patches/cluster.yamlis a bootstrap patch, so merging this PR alone does not change the live kubeletimageGCHighThresholdPercent/imageGCLowThresholdPercenton the running nodes — atalosctl patch machineconfig(or equivalent) is required separately. That's a deployment-process note rather than a code defect, but it's the kind of thing that's easy to lose track of after merge.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Correcting a number in the PR body, and adding the safety analysis behind the GC change.
The "roughly two-thirds reclaimable" figure is misleading. That was two-thirds by image count, and the unused images are the small ones. By size, measured on k8s-2:
Also worth separating:
talosctl usageattributes 29 GB to/var/lib/containerd, but only 9.93 GB of that is image content. The remaining ~19 GB is unpacked overlayfs snapshots. Image GC removes an image and its snapshots, so the on-disk reclaim is roughly 2.9x the compressed figure — about 6 GB, not 2 GB.Does 60/50 still work? Yes, because the trigger moves earlier rather than the cache getting bigger. GC fires at 60% used (36 GB of the 60 GB disk). By then a week of image-automation churn has accumulated its ~6 GB of unused layers, so the sweep lands near 30 GB — the 50% low mark. The node then oscillates roughly 40–50% free instead of 30–40%, which keeps it clear of Ceph's 30% line.
The margin is thinner than "two-thirds reclaimable" implied. If unused layers ever accumulate to less than ~6 GB between sweeps, kubelet will get below the 60% high threshold but not down to 50%, and will log
failed to garbage collect required amount of imageseach cycle. That is noisy, not dangerous, and it still achieves the goal of keeping Ceph quiet.Security of the toolbox — checked, not assumed:
system:serviceaccount:monitoring:ci-diagnosticscannotcreate pods/execinrook-cephand cannotget secretsthere. CI agents cannot use the toolbox to reach Ceph admin.rook-ceph-default, holds no meaningful Kubernetes permissions — onlyselfsubject*review APIs. Its capability is entirely the mounted keyring.rook-ceph-monsecret already exists in the namespace, so anyone who can exec inrook-cephcould already read it and build a keyring by hand. The toolbox makes that convenient, not newly possible.Applying the kubelet change should be staged.
talosctl patch machineconfigrestarts kubelet on the target node, and the first sweep afterwards deletes most of the unused cache. Do one node, confirm it settles above 40% free and that pods still start, then the next — rather than all three at once.