chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion #597

2026-08-25T22:48:12Z

claude-bot commented

2026-08-25 22:48:12 +00:00

First step of the last conversion. Do not merge until the saturn backfill has finished — all 217 PGs active+clean. Merging while PGs are still remapped would start removing an OSD mid-backfill.

neptune converts to bare metal and its OSD drive is swapped in the same window. Removing k8s-3 from the explicit node list first means Rook stops managing that OSD before the drive it lives on goes away, instead of recreating it within a reconcile and backfilling onto a disk that is about to be pulled. Same shape as #586 for saturn.

The device comment for k8s-3 goes with it — it described the PCI-passed-through drive that node saw as a Proxmox guest, which stops being true.

Expect

Ceph drops to two OSDs and size-3 pools stay degraded until neptune rejoins. cephfs-metadata is already at 2/1 for exactly this window, and .mgr at 3/2. Both get restored at the end, once all three nodes are converted.

Note osd.1 is the worn Crucial rather than a PM983 — neptune is the one node that still has its original drive:

 1   nvme  0.90970              osd.1       up   1.00000  1.00000   # 1TB Crucial
 2   nvme  0.87329              osd.2       up   1.00000  1.00000   # 960GB PM983
 0   nvme  0.87329              osd.0       up   1.00000  1.00000   # 960GB PM983

Rest of the sequence

After this merges and PGs settle: drain k8s-3, power off, install the PM983 and flip the power-mode switch, then boot the Talos stick and install in two stages per the runbook — minimal network config plus EPHEMERAL, then the full config on the running node. Then re-add the OSD and restore the pool sizes.

No Proxmox work is needed. neptune is the last PVE node, so the cluster ceases to exist rather than shrinking and the disks get wiped regardless. Tofu state keeps a stale VM and the PVE resources, to be cleaned up separately.

First step of the last conversion. **Do not merge until the saturn backfill has finished** — all 217 PGs `active+clean`. Merging while PGs are still remapped would start removing an OSD mid-backfill. neptune converts to bare metal and its OSD drive is swapped in the same window. Removing `k8s-3` from the explicit node list first means Rook stops managing that OSD before the drive it lives on goes away, instead of recreating it within a reconcile and backfilling onto a disk that is about to be pulled. Same shape as #586 for saturn. The device comment for `k8s-3` goes with it — it described the PCI-passed-through drive that node saw as a Proxmox guest, which stops being true. ## Expect Ceph drops to two OSDs and size-3 pools stay degraded until neptune rejoins. `cephfs-metadata` is already at 2/1 for exactly this window, and `.mgr` at 3/2. Both get restored at the end, once all three nodes are converted. Note `osd.1` is the worn Crucial rather than a PM983 — neptune is the one node that still has its original drive: ``` 1 nvme 0.90970 osd.1 up 1.00000 1.00000 # 1TB Crucial 2 nvme 0.87329 osd.2 up 1.00000 1.00000 # 960GB PM983 0 nvme 0.87329 osd.0 up 1.00000 1.00000 # 960GB PM983 ``` ## Rest of the sequence After this merges and PGs settle: drain k8s-3, power off, install the PM983 and flip the power-mode switch, then boot the Talos stick and install in **two stages** per the runbook — minimal network config plus `EPHEMERAL`, then the full config on the running node. Then re-add the OSD and restore the pool sizes. No Proxmox work is needed. neptune is the last PVE node, so the cluster ceases to exist rather than shrinking and the disks get wiped regardless. Tofu state keeps a stale VM and the PVE resources, to be cleaned up separately.

claude-bot added 1 commit 2026-08-25 22:48:12 +00:00

chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

8e97253d6c

neptune is the last node to convert to bare metal, and its OSD drive is swapped
in the same window. Removing k8s-3 from the explicit node list first means Rook
stops managing that OSD before the drive it lives on goes away, rather than
recreating it mid-reconcile and backfilling onto a disk that is about to be
pulled.

The device comment for k8s-3 goes with it: it described the PCI-passed-through
drive that node saw as a Proxmox guest, which stops being true here.

Ceph runs on two OSDs and size-3 pools stay degraded until the node rejoins.
cephfs-metadata is already at 2/1 for exactly this window.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-08-25 22:50:50 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 8e97253d6c2b9611d2882ad17c883144e1c0a2c0.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Removes k8s-3 (neptune) from cephcluster.yaml's explicit storage.nodes list ahead of neptune's Proxmox→bare-metal conversion and drive swap, and deletes the now-stale comment describing that node's PCI-passed-through device. This is a documentation-and-config-only change (4 added / 7 removed lines, one file) that mirrors the pattern already used for the prior jupiter/saturn conversions. No blocking issues found.

🟡 Suggestions

kubernetes/RUNBOOK-baremetal-conversion.md:5 says "Written after doing jupiter on 2026-08-25; saturn and neptune remain," but cephcluster.yaml (both before and after this diff) already lists k8s-2 (saturn) with a /dev/disk/by-id path, i.e. saturn's conversion is already done per the PR body ("Same shape as #586 for saturn"). The runbook line is stale and could confuse whoever runs the neptune conversion next. Not introduced by this PR and outside its diff, but since this PR is the trigger for the next runbook execution, it may be worth a one-line fixup either here or in a follow-up.

Tests

Not applicable — this is a declarative infra config change with no automated test coverage in this repo for CephCluster node topology, consistent with how the equivalent jupiter/saturn changes were made. The PR body substitutes manual verification (pre-conditions on PG health, explicit "do not merge until backfill finished" warning), which is the right control for a change like this.

Verification performed

Confirmed the removed comment block (old lines 9-12, describing nvme0n1 on k8s-3 as neptune's PCI-passed-through drive) is fully self-contained and not referenced elsewhere in the file — no dangling references to k8s-3 or that comment remain in cephcluster.yaml.
Confirmed the PR body's claim that cephfs-metadata is already at size 2 / min_size 1: kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:110-114 shows metadataPool.replicated.size: 2 / min_size: "1", explicitly flagged as temporary for exactly this multi-node-conversion window (storageclass.yaml:90-109). replicapool (storageclass.yaml:29-32) is likewise already at size 2/1, independent of this change (drive-wear rationale, not conversion-window rationale).
Confirmed the new replacement comment (k8s-3 is absent while neptune's OSD drive is swapped...) is accurate against the surrounding context: bare-metal nodes (k8s-1, k8s-2) do see three NVMe drives each and are correctly named by /dev/disk/by-id, matching the top-of-file rationale block (cephcluster.yaml:170-186) and the re-add instructions in RUNBOOK-baremetal-conversion.md:351-365.
Checked other repo references to k8s-3 (opentofu VM definition, Talos patches, bootstrap.sh) — these intentionally remain untouched since neptune is being converted, not removed from the cluster; the PR body explicitly scopes Proxmox/Tofu cleanup as a separate step ("No Proxmox work is needed... Tofu state keeps a stale VM... to be cleaned up separately"). Consistent with the diff's scope.
YAML indentation of the new comment block and remaining nodes: entries is consistent (verified by reading the full file post-patch); the list still parses as two entries (k8s-1, k8s-2) with the same structure as before.

Recommendation

Looks reasonable — no blocking issues; comment above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `8e97253d6c2b9611d2882ad17c883144e1c0a2c0`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Removes `k8s-3` (neptune) from `cephcluster.yaml`'s explicit `storage.nodes` list ahead of neptune's Proxmox→bare-metal conversion and drive swap, and deletes the now-stale comment describing that node's PCI-passed-through device. This is a documentation-and-config-only change (4 added / 7 removed lines, one file) that mirrors the pattern already used for the prior jupiter/saturn conversions. No blocking issues found. ## 🟡 Suggestions - `kubernetes/RUNBOOK-baremetal-conversion.md:5` says "Written after doing jupiter on 2026-08-25; saturn and neptune remain," but `cephcluster.yaml` (both before and after this diff) already lists `k8s-2` (saturn) with a `/dev/disk/by-id` path, i.e. saturn's conversion is already done per the PR body ("Same shape as #586 for saturn"). The runbook line is stale and could confuse whoever runs the neptune conversion next. Not introduced by this PR and outside its diff, but since this PR is the trigger for the next runbook execution, it may be worth a one-line fixup either here or in a follow-up. ## Tests Not applicable — this is a declarative infra config change with no automated test coverage in this repo for `CephCluster` node topology, consistent with how the equivalent jupiter/saturn changes were made. The PR body substitutes manual verification (pre-conditions on PG health, explicit "do not merge until backfill finished" warning), which is the right control for a change like this. ## Verification performed - Confirmed the removed comment block (old lines 9-12, describing `nvme0n1` on `k8s-3` as neptune's PCI-passed-through drive) is fully self-contained and not referenced elsewhere in the file — no dangling references to `k8s-3` or that comment remain in `cephcluster.yaml`. - Confirmed the PR body's claim that `cephfs-metadata` is already at size 2 / min_size 1: `kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml:110-114` shows `metadataPool.replicated.size: 2` / `min_size: "1"`, explicitly flagged as temporary for exactly this multi-node-conversion window (storageclass.yaml:90-109). `replicapool` (storageclass.yaml:29-32) is likewise already at size 2/1, independent of this change (drive-wear rationale, not conversion-window rationale). - Confirmed the new replacement comment (`k8s-3 is absent while neptune's OSD drive is swapped...`) is accurate against the surrounding context: bare-metal nodes (`k8s-1`, `k8s-2`) do see three NVMe drives each and are correctly named by `/dev/disk/by-id`, matching the top-of-file rationale block (cephcluster.yaml:170-186) and the re-add instructions in `RUNBOOK-baremetal-conversion.md:351-365`. - Checked other repo references to `k8s-3` (opentofu VM definition, Talos patches, `bootstrap.sh`) — these intentionally remain untouched since neptune is being *converted*, not removed from the cluster; the PR body explicitly scopes Proxmox/Tofu cleanup as a separate step ("No Proxmox work is needed... Tofu state keeps a stale VM... to be cleaned up separately"). Consistent with the diff's scope. - YAML indentation of the new comment block and remaining `nodes:` entries is consistent (verified by reading the full file post-patch); the list still parses as two entries (`k8s-1`, `k8s-2`) with the same structure as before. ## Recommendation Looks reasonable — no blocking issues; comment above is optional.

k.renner merged commit 1f15062b80 into master

2026-08-25 22:54:19 +00:00

k.renner referenced this issue from a commit

2026-08-25 22:54:20 +00:00

Merge pull request 'chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion' (#597) from ceph-drop-neptune into master

alert-monitor-bot referenced this pull request

2026-08-25 23:24:56 +00:00

[alert] TargetDown (kube-system) #601

alert-monitor-bot referenced this pull request

2026-08-25 23:39:46 +00:00

[alert] KubeNodeUnreachable (monitoring) #602

Sign in to join this conversation.