Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 49c33b9b0f fix(ceph): state the cluster CIDR — host-device leaves the canary no NIC
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Since the host-device rollout (#801) Rook cannot finish a CephCluster
reconcile: it runs a canary pod with OSD placement to read the cluster
CIDR off the attachment, the OSD on that node already holds the NIC, and
the canary fails "error adding container to network ceph-san" until the
15-minute discovery timeout, then the next reconcile repeats it. Under
macvlan the canary could sit beside the OSD; a whole-device attachment has
exactly one consumer per node.

Discovery is skipped when addressRanges.cluster is set (Rook: "if user
specifies CIDRs in the network config, that is the definitive source of
truth"). The value is what discovery produced and what ceph config already
holds, so no daemon changes. The old comment claiming addressRanges is
honoured only under provider: host was wrong — the CRD documents it for
host and multus.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 07:21:36 +02:00
k.renner 99827fb776 Merge pull request 'fix(ceph): OSD raw volume label must not contain "ceph" (ceph-volume rejects it)' (#803) from osd-partlabel-no-ceph into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#803
2026-09-07 05:15:17 +00:00
k.renner 19f8d5bc51 Merge pull request 'fix(infrastructure): LAPI ServersTransport behind the Traefik CRDs (phase 1/2, add)' (#802) from lapi-transport-post-crds into master
Reviewed-on: k.renner/homelab#802
2026-09-07 05:00:59 +00:00
k.renner a88d3cd224 Merge pull request 'feat(ceph): hand the SAN NIC to the OSD pod (rollout 2/2, maintenance window)' (#801) from san-host-device-nad into master
ci/woodpecker/cron/e2e Pipeline is running
ci/woodpecker/cron/update-audit Pipeline is running
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#801
2026-09-07 04:51:00 +00:00
claude-botandClaude Fable 5.1 93c0e33dac fix(ceph): OSD raw volume label must not contain "ceph"
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
All three OSD prepare jobs on the Hetzner rehearsal finished with zero
OSDs:

  cephosd: skipping device "sdb1": ["Used by ceph-disk"]

The partition is exactly what the machine config asked for (PARTLABEL
r-ceph-osd, plain Linux partition type). ceph-volume rejects it anyway —
its ceph-disk-member test is `"ceph" in PARTLABEL or PARTTYPE in
ceph_disk_guids` (ceph_volume/util/device.py) — so any label containing
"ceph" reads as a legacy ceph-disk partition and is never provisioned.

Talos derives the label from the RawVolumeConfig name (r-<name>), so the
name changes: ceph-osd -> osd-data, label r-osd-data. This is why the
rebuild branch called it osd-data; the #798 review renamed it back for
symmetry with the node label without this failure being known.

Production is unaffected today: the MS-01 OSDs match on the by-id half of
devicePathFilter and no r-ceph-osd partition exists there. The partlabel
half is the migration target and would have failed on the first
repartitioned drive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 23:43:25 +02:00
claude-botandClaude Fable 5.1 c595451548 fix(infrastructure): LAPI ServersTransport behind the Traefik CRDs (phase 1: add)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
A cold rebuild stops permanently at the infrastructure Kustomization:

  ServersTransport/crowdsec/crowdsec-lapi dry-run failed: no matches for
  kind "ServersTransport" in version "traefik.io/v1alpha1"

Flux dry-runs the whole apply set, so this one unknown kind rejects all of
it — including the Traefik HelmRelease that installs the CRD. Measured on
the Hetzner rehearsal from master: zero HelmReleases, zero CRDs. The
public LAPI endpoint (2026-09-02) landed after the post-crds sweep and put
a traefik.io consumer back into the layer that creates traefik.io.

Phase 1 only adds the object under post-crds; crowdsec/lapi-ingress.yaml
keeps its copy until infrastructure-crds has applied once, because both
Kustomizations prune and a one-commit move deletes before the new owner
applies. Phase 2 removes the original.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 23:24:46 +02:00
k.renner 078b250fbd Merge pull request 'feat(talos): name the SAN NIC by its role on every site (rollout 1/2, inert)' (#800) from san-role-rename into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#800
2026-09-06 21:11:47 +00:00
10 changed files with 44 additions and 19 deletions
+2 -2
View File
@@ -1379,8 +1379,8 @@ EOF
vm_addr="${NODE_ADDRS[$idx]}"
vols=$(talosctl -n "$vm_addr" -e "$vm_addr" get discoveredvolumes 2>&1) \
|| fail "could not read volumes from $vm_node ($vm_addr): ${vols//$'\n'/ | }"
awk '$NF == "r-ceph-osd" {found=1} END {exit !found}' <<<"$vols" \
|| fail "$vm_node has no r-ceph-osd partition — RawVolumeConfig did not take"
awk '$NF == "r-osd-data" {found=1} END {exit !found}' <<<"$vols" \
|| fail "$vm_node has no r-osd-data partition — RawVolumeConfig did not take"
kubectl get node "$vm_node" \
-o jsonpath='{.metadata.labels.storage\.renner\.uno/ceph-osd}' \
| grep -q true \
@@ -0,0 +1,11 @@
# traefik.io CRD consumer for the public LAPI endpoint (crowdsec/lapi-ingress.yaml
# holds the Ingress). The CA it names is the chart's self-signed LAPI CA.
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: crowdsec-lapi
namespace: crowdsec
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAs:
- secret: crowdsec-root-secret
@@ -31,4 +31,7 @@ resources:
- traefik-crowdsec-middleware.yaml
- traefik-rate-limit-middleware.yaml
- traefik-security-headers-middleware.yaml
# Phase 1 of a two-Kustomization move: also still in crowdsec/lapi-ingress.yaml
# until this layer has applied it once, so prune never runs ahead of apply.
- crowdsec-lapi-serverstransport.yaml
- ../network-policies/
@@ -95,12 +95,18 @@ spec:
# Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
# Multus. The attachment is san-network-attachment.yaml.
#
# Only `cluster` is selected. Do not add `public`, and do not reach for
# `addressRanges` — that is honoured only under provider: host, which puts
# every daemon including the mons on host networking. A mon's address then
# becomes the node's DHCP lease, and when a lease moves the mon crash-loops
# on "Cannot assign requested address" with no node free to fail over to,
# because all three already hold the mon host ports.
# Only `cluster` is selected. Do not add `public`: under provider: host
# that puts every daemon including the mons on host networking. A mon's
# address then becomes the node's DHCP lease, and when a lease moves the mon
# crash-loops on "Cannot assign requested address" with no node free to
# fail over to, because all three already hold the mon host ports.
#
# addressRanges.cluster is stated, not discovered. Without it Rook runs a
# canary pod on an OSD node to read the CIDR off the attachment on EVERY
# reconcile; the attachment is host-device, so the OSD on that node already
# holds the NIC and the canary can never attach — the reconcile blocks for
# 15 minutes, fails, and repeats. The value must equal the whereabouts range
# in san-network-attachment.yaml.
#
# With `public` unset, client traffic stays on the pod network and VLAN 49
# carries Ceph-internal traffic only — which is also why Rook's Multus
@@ -117,6 +123,9 @@ spec:
provider: multus
selectors:
cluster: rook-ceph/ceph-san
addressRanges:
cluster:
- 10.99.0.0/29
# Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
# re-provisions an OSD on any node whose device still carries a BlueStore
# signature, so purging an OSD before pulling its drive is impossible — it
@@ -144,10 +153,12 @@ spec:
useAllDevices: false
# Two forms while the production drives are migrated:
#
# by-partlabel/r-ceph-osd the target. Talos names the drive in the
# machine config (RawVolumeConfig ceph-osd), so
# by-partlabel/r-osd-data the target. Talos names the drive in the
# machine config (RawVolumeConfig osd-data), so
# this file needs to know nothing about models,
# serials or hostnames.
# serials or hostnames. The label must not
# contain "ceph": ceph-volume rejects any such
# partition as a legacy ceph-disk member.
# by-id/nvme-SAMSUNG… the three MS-01 drives as they are today:
# bluestore written straight onto the raw disk,
# with no partition table and therefore no
@@ -159,7 +170,7 @@ spec:
#
# The second half goes when those three are repartitioned, one node at a
# time. Until then it is what keeps the running OSDs claimed.
devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
devicePathFilter: "^/dev/disk/by-(partlabel/r-osd-data|id/nvme-SAMSUNG_MZQLB960HAJR)"
config:
osdsPerDevice: "1"
# Highest built-in class: kubelet evicts these last and the kernel gives
+1 -1
View File
@@ -57,7 +57,7 @@ up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
# Two disks: the system disk and the attached volume. Matching on transport
+1 -1
View File
@@ -42,7 +42,7 @@ up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
match: '!system_disk'
+1 -1
View File
@@ -42,7 +42,7 @@ up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
match: '!system_disk'
+2 -2
View File
@@ -21,7 +21,7 @@ auto: off
hostname: vm-2
---
# The OSD device, named by the machine that has it. Ceph then consumes
# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
# /dev/disk/by-partlabel/r-osd-data and needs to know nothing about models,
# serials or hostnames — the same CephCluster works here and on the MS-01s.
#
# The selector is a CEL expression, so each machine says whatever identifies its
@@ -29,7 +29,7 @@ hostname: vm-2
# real nodes it will name the Samsung.
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk
+1 -1
View File
@@ -15,7 +15,7 @@ hostname: vm-3
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk
+1 -1
View File
@@ -19,7 +19,7 @@ hostname: vm-4
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: ceph-osd
name: osd-data
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk