Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 05b20ec74c feat(talos): name the SAN NIC by its role on every site
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Only the Talos machine config knows hardware. Each machine renames its own
Ceph cluster-network link to `ceph-san` at boot, so shared manifests can
address the role and no kernel interface name leaves this directory:

- k8s-1/2/3 (MS-01): a udev rule matches the X710 port 1 by burned-in MAC
  and renames it before the network stack opens links. The LinkConfig then
  names ceph-san (up, mtu 9000). A rule applied to a RUNNING node cannot
  rename (Resource busy) and takes effect at the next reboot, so this
  change is inert on production until each node is rebooted — that is the
  planned rollout, node by node.
- vm-1..4 (qemu site): one NIC only, so the segment is VLAN 30 named
  ceph-san directly. The DHCPv4Config on the parent is load-bearing:
  declaring any network document stops the implicit DHCP that addresses
  the node.
- hcloud-1/2/3 already carry the same shape (udev by PCI bus path) since
  the rebuild-tooling merge.

Validated on Hetzner with the host-device attachment: OSD pods held the
real fabric MACs, Ceph reached HEALTH_OK.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 22:54:19 +02:00
7 changed files with 88 additions and 11 deletions
+14 -6
View File
@@ -28,6 +28,13 @@ machine:
# this label instead of listing hostnames, so which boxes carry storage is
# stated once, here, by the machine that knows.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
# role name at boot, before the network stack opens links — a rule applied
# to a RUNNING node cannot rename (Resource busy) and takes effect at the
# next reboot.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -124,14 +131,15 @@ parent: enp2s0f0np0
up: true
---
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
# wire. OSD-to-OSD replication and nothing else.
# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
# section above renames the port to ceph-san at boot, so shared manifests name
# the role and only this file knows the hardware.
#
# The node holds NO address here on purpose. Rook attaches the link to the OSD
# pods through a Multus macvlan, and a macvlan parent only needs to be up.
# Giving the node an address would put it on a segment reserved for
# Ceph-internal traffic.
# The node holds NO address here on purpose: the link is handed whole to the
# OSD pod (host-device attachment). Giving the node an address would put it on
# a segment reserved for Ceph-internal traffic.
apiVersion: v1alpha1
kind: LinkConfig
name: enp2s0f1np1
name: ceph-san
up: true
mtu: 9000
+8 -3
View File
@@ -23,6 +23,10 @@ machine:
topology.kubernetes.io/zone: saturn
# Holds a Ceph OSD drive — see k8s-1.yaml.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port to the role name at boot — see k8s-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -95,11 +99,12 @@ up: true
---
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
# wire. OSD-to-OSD replication only, not routed off its own segment.
# Renamed to the role name by the udev rule above — see k8s-1.yaml.
#
# The node holds NO address here on purpose. Rook attaches the link to the OSD
# pods through a Multus macvlan, and a macvlan parent only needs to be up.
# The node holds NO address here on purpose: the link is handed whole to the
# OSD pod (host-device attachment).
apiVersion: v1alpha1
kind: LinkConfig
name: enp2s0f1np1
name: ceph-san
up: true
mtu: 9000
+6 -2
View File
@@ -19,6 +19,10 @@ machine:
topology.kubernetes.io/zone: neptune
# Holds a Ceph OSD drive — see k8s-1.yaml.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port to the role name at boot — see k8s-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -78,9 +82,9 @@ parent: enp2s0f0np0
up: true
---
# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
# here on purpose; the link is only a macvlan parent for the OSD pods.
# here on purpose; the link is handed whole to the OSD pod (host-device).
apiVersion: v1alpha1
kind: LinkConfig
name: enp2s0f1np1
name: ceph-san
up: true
mtu: 9000
+12
View File
@@ -43,3 +43,15 @@ apiVersion: v1alpha1
kind: HostnameConfig
auto: off
hostname: vm-1
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+24
View File
@@ -35,3 +35,27 @@ provisioning:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# The Ceph cluster network link. Production gives it a second physical port;
# this machine has one NIC, so the segment is a VLAN on it.
#
# A dummy link would be simpler and worthless — no traffic between nodes, so
# Ceph would replicate over an interface going nowhere and the test would pass
# having proven nothing. A VLAN rides the provisioner's bridge that every VM is
# on, so the SAN carries real frames on its own tag.
#
# No MTU, unlike production's 9000: the qemu NIC is created with host_mtu=1500.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
# Load-bearing. Declaring any network document makes Talos configure this link
# explicitly, and the implicit DHCP that gives the node its address stops. A
# first attempt with only the VLANConfig took vm-4 off the network — NotReady,
# unreachable over the Talos API. Verified on a second throwaway node that this
# document keeps the parent addressed.
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+12
View File
@@ -21,3 +21,15 @@ provisioning:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+12
View File
@@ -25,3 +25,15 @@ provisioning:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2