feat(ceph): OSD-Auswahl über Labels aus der Machine-Config #718
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#718
Reference in New Issue
Block a user
Die CephCluster nannte drei Hostnamen und drei Laufwerks-Seriennummern. Das band die Storage-Schicht genauso fest an diese drei Kisten, wie es die Talos-Config vorher war — und aus demselben Grund: es stand in einem Kubernetes-Manifest, was nur die Maschine weiß.
Jetzt sagt es die Maschine, zweimal:
nodeLabels: storage.renner.uno/ceph-osdplacement.osd.nodeAffinityliest das — und ist beiuseAllNodes: truedas Einzige, was OSDs von plattenlosen Nodes fernhältRawVolumeConfig ceph-osd/dev/disk/by-partlabel/r-ceph-osd, und das konsumiert RookdevicePathFilterträgt übergangsweise beide Formen. Die drei Produktionsplatten tragen Bluestore roh, ohne Partitionstabelle — es gibt dort schlicht kein Partitionslabel zu treffen. Sie werden separat umgezogen, ein Node nach dem anderen; dann fällt dieby-id-Hälfte weg. Die verbleibende Hälfte trifft jetzt schon nach Modell statt nach drei einzelnen Seriennummern, ein Plattentausch ist damit kein Repo-Edit mehr.Resource-Requests
Gemessen am laufenden Cluster: mon ~420Mi bei 2Gi Request, mgr ~650Mi bei 4Gi, OSDs ~2,2Gi bei 4Gi — rund 17 GiB clusterweit reserviert und nie angefasst. Requests folgen jetzt dem Bedarf, Limits bleiben unverändert, der Burst-Kopfraum also auch. Guaranteed QoS wird bewusst aufgegeben;
priorityClassNames: system-node-criticalsteht schon da und ordnet die Verdrängung tatsächlich.Der OSD-Request liegt absichtlich über der Messung: ein OSD wächst in Cephs
osd_memory_targethinein, 2,2Gi ist ein Boden, keine Decke.Was der Test dabei gefunden hat
Die
vmtest-Phase fährt jetzt eine Control-Plane und drei Worker — Worker, weil der Provisioner die zweite Platte nur an sie vergibt, und drei, weil die Mons mitallowMultiplePerNode: falselaufen und die Poolsreplica 3überfailureDomain: hostsind.Jeder Anlauf hat einen echten Fehler im Produktionspfad aufgedeckt, und alle waren stumm:
talosctl bootstraplief über die Endpunktliste. etcd-Bootstrap ist der eine Aufruf, den apid nicht weiterleitet —PermissionDenied: no request forwarding. Ohne-n/-esucht talosctl sich einen Endpunkt aus, bei drei Control-Planes also ein Münzwurf. Und weiltalosctl kubeconfigdanach trotzdem erfolgreich ist, lief das Skript weiter, als stünde der Cluster.Die Formation-Wartung hing an
discovery.talos.dev. Mitgliedschaft kommt nur von dort, weil die Kubernetes-Registry deaktiviert ist. Ein Cold Rebuild wartete damit auf einen Internetdienst, bevor der Cluster existiert, den er wiederherstellen soll — als Tor vor dem Bootstrap. Sie fragt jetzt jeden Node direkt nach seinem Hostnamen und erwartet den, den sein eigener Patch vergibt; das erwischt nebenbei einen Patch auf der falschen Maschine.Der
Ready-Zähler zählteNotReadymit.grep -c "Ready"trifft den Teilstring. Die Schleife wartete also auf Registrierung, nicht auf Bereitschaft, und hätte mit lauterNotReady-Nodes Erfolg gemeldet. Deshalb war die „3 nodes Ready"-Aussage aus #716 kein Beleg.Alle drei waren nur sichtbar, weil
2>/dev/nullaus dieser Phase verschwunden ist. Fehler werden während der Wiederholungen zurückgehalten und ausgegeben, wenn es endgültig nicht klappt.Verifiziert
Vier Nodes
Readyauf der Repo-eigenen Config, und die Phase prüft ihre eigene Voraussetzung:vm-2,vm-3,vm-4tragen je die Partitionr-ceph-osdund das Label. Ohne das schlägt der Test fehl, statt später ohne Erklärung keine OSDs zu haben.Nicht verifiziert
Ob Ceph auf den VMs tatsächlich
Readywird — das ist der nächste Schritt, die Flux-Phase gegen diesen Branch. Und die drei Produktions-Patches sind generiert und validiert, aber der Plattenumzug dort steht noch aus.The CephCluster named three hostnames and three drive serials. That tied the storage layer to these three boxes as firmly as the Talos config used to be tied to them, and for the same reason: it said in a Kubernetes manifest something only the machine knows. Now the machine says it, twice: nodeLabels storage.renner.uno/ceph-osd which machines carry storage. placement.osd.nodeAffinity reads it, so useAllNodes can be true without putting OSDs everywhere — that nodeAffinity is the only thing keeping them off diskless nodes. RawVolumeConfig ceph-osd which disk, chosen by a CEL expression per machine. The partition surfaces at /dev/disk/by-partlabel/r-ceph-osd and that is what Rook consumes. devicePathFilter carries both that partlabel and the OSD model for now. The three production drives hold bluestore written straight onto the raw disk, with no partition table and so no partition label to match; they are migrated separately, one node at a time, and the by-id half goes when they are. Matching the remaining half by model rather than by three individual serials already means a drive swap is not a repo edit. Resource requests now follow what the daemons use, measured on the running cluster: mon ~420Mi against a 2Gi request, mgr ~650Mi against 4Gi, OSDs ~2.2Gi against 4Gi — roughly 17GiB reserved cluster-wide and never touched. Limits stay where they were, so burst headroom is unchanged; only the reservation moves. Guaranteed QoS is given up deliberately, and priorityClassNames already pins these to system-node-critical, which is what actually orders eviction. The OSD request is set above measured use on purpose: an OSD grows into Ceph's osd_memory_target, so 2.2Gi is a floor rather than a ceiling. The vmtest phase becomes one control plane and three workers. Workers because the QEMU provisioner gives the second disk only to worker machines and an OSD needs one; three because the mons run with allowMultiplePerNode false and the pools are replica 3 over failureDomain host. The apply loop now walks ALL_NODES once instead of once per role, so NODE_ADDRS stays a single ordered list. Not yet verified: whether Ceph actually reaches Ready on the VMs. That is the point of the run this enables, and the memory budget is tight enough that it may not — 2.5GiB plus three times 4GiB against 15.4GiB available.The four-VM run reported: error executing bootstrap: rpc error: code = PermissionDenied desc = no request forwarding and the cluster then sat idle. Nothing downstream noticed: `talosctl kubeconfig` still succeeded, because Talos issues credentials as soon as it can rather than when kube-apiserver serves, so the run carried on to the API wait as if it had bootstrapped. etcd bootstrap is the one call apid will not forward — reached through any node other than the target it fails outright. `talosctl bootstrap` with neither -n nor -e picks an endpoint from the configured list, and that list holds every control plane, plus the workers in the vmtest phase. Whether it landed on the right node was chance, in a real rebuild as much as on VMs. Only visible because the previous commit stopped discarding this error.The formation wait failed with every node listed as unregistered, and the error it now prints was the same one that broke the bootstrap call: PermissionDenied: no request forwarding `talosctl get members` went through the configured endpoint list rather than to a specific machine, and apid will not forward it. It happened to work on some runs and not others, depending on which endpoint was picked. The second problem is worse and is not about VMs. Membership comes from the discovery service, because the Kubernetes registry is disabled in the generated config. So this step made a cold rebuild wait on discovery.talos.dev being reachable, before the cluster being rebuilt exists — an internet dependency in the middle of the disaster path, gating the bootstrap. Neither is needed. What has to be true before etcd is bootstrapped is local: the config took, the machine rebooted into it, and it is the machine we think it is. The wait now asks each address directly for its hostname and expects the name its own patch assigns — which also catches a patch landing on the wrong machine, something the member list never checked. The failure names the address, what it expected, and what it last answered.🤖 Automated review — model
CLI default(complexity: complex), commit3b178068a845afcd407de513b6aa4c3faec2c1ba.Summary
This PR moves Ceph OSD node selection from a hardcoded hostname/serial list in
cephcluster.yamlto astorage.renner.uno/ceph-osdnode label (set per-machine in the Talos patches) plus anodeAffinity, adds a partition-label baseddevicePathFilterfed by a newRawVolumeConfig, trims Ceph resource requests to measured usage, and extends thevmtestbootstrap phase from 3 control planes to 1 CP + 3 OSD-carrying workers so the new selection logic is actually exercised. It also fixes three real, previously-silent bugs inbootstrap.sh(etcd bootstrap going through the endpoint list instead of a direct node, cluster-formation wait depending on the external discovery service, and agrep -c "Ready"that countedNotReadytoo). Thebootstrap.shchanges are well-reasoned and the fixes read as correct against the code around them. No blocking issues found, but the PR leaves two pieces of existing documentation contradicting the new design.🟡 Suggestions
kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:107-115— this comment block was not touched by the diff and still argues againstuseAllNodes, explaining that underuseAllNodesthe operator "re-provisions an OSD on any node whose device still carries a BlueStore signature, so purging an OSD before pulling its drive is impossible." The new code 15 lines below (useAllNodes: true,storage:at line 130) does exactly what this comment warns about, with no note reconciling the two. Since the PR body says the next step is migrating the three production drives "ein Node nach dem anderen" (one node at a time) — which will require purging an OSD before repartitioning its drive — this is exactly the scenario the old comment describes as broken. Either the concern no longer applies (and the stale comment should be removed/updated to say why) or it does apply and should be called out as a caveat for that upcoming migration.kubernetes/RUNBOOK-baremetal-conversion.md:352-361— "Re-add the OSD" step 9 still instructs editingcephcluster.yamlunderstorage.nodes, e.g.- name: k8s-N / devices: - fullpath: /dev/disk/by-id/.... That key no longer exists after this PR (replaced byuseAllNodes: true+ label +devicePathFilter). This runbook was written seven days before this PR (per its own header, 2026-08-25, "saturn and neptune remain") and reads as still actively in use for the two remaining node conversions — following it as written would silently do nothing, since OSD placement is no longer driven by that list.kubernetes/bootstrap.sh:355-358—grep -qi 'already\|AlreadyExists' <<<"$bootstrap_err"— the-iflag already makes the first alternative case-insensitive, soAlreadyExistsis already covered byalready; the second alternative is redundant. Not a bug, just dead pattern.🟢 Nits
kubernetes/bootstrap.sh:196-200—vmtestnow provisions 1×2560MiB control plane + 3×4GiB workers (vs. the old 3×2560MiB, no workers) and 3×(10+8)GiB of disk instead of 3×10GiB — a real increase in what a machine running./bootstrap.sh vmtestneeds (~14.5GiB RAM, ~54GiB disk vs. ~7.7GiB/30GiB before). Worth a line in the usage comment at the top of the file if this is meant to run on CI or a shared box.Tests
The PR's own test is the change to the
vmtestphase, and the PR body states it was run and caught three real bugs, plus a final green run with the new OSD-prerequisite assertions (partition + label) added atkubernetes/bootstrap.sh:457-470. That's appropriate for this kind of infra change — there's no separate unit-test layer in this repo to extend. I did not re-runvmtestmyself (needs root, QEMU, and ~15 min); the logic reads consistently against the rest of the script, so I'm relying on the author's stated verification for the parts I can't execute (in particular thetalosctl get discoveredvolumesoutput actually ending each row with the partition label, matched byawk '$NF == "r-ceph-osd"'atkubernetes/bootstrap.sh:463).Recommendation
Looks reasonable — no blocking issues; comments above are optional but the runbook drift is worth fixing before it's next needed (saturn/neptune conversions).