Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 315c9aeb46 fix(crowdsec): request 50m CPU instead of the chart's 500m
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.

crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.

Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 17:05:05 +02:00
k.renner 3e57ed5c2d Merge pull request 'feat(k8s-1): run Talos on bare-metal jupiter and swap the OSD to U.2' (#566) from jupiter-baremetal-u2 into master
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#566
2026-08-25 14:51:53 +00:00
claude-botandClaude Opus 5 6380f02ca4 feat(k8s-1): run Talos on bare-metal jupiter and swap the OSD to U.2
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the
MS-01's only U.2-capable slot means moving the boot drive anyway. Doing the
bare-metal conversion in the same window avoids opening the machine twice and
throws away no work: the osd-ssd PCI mapping that a Proxmox-side swap would
need to be re-pointed at ceases to exist.

Networking uses the v1.12 multi-document config rather than the deprecated
machine.network.interfaces. The docs do not define what happens when one link
is configured by both schemas, so k8s-1 keeps every link in one of them. This
also buys VLANConfig.name, which is free-form: naming the VLAN 30 link ens19
keeps the four v30iot macvlan NADs working unchanged across VM and bare-metal
nodes, so the cluster can stay mixed without pinning those workloads.

Physical NICs are matched by permanent MAC, not driver — both X710 ports bind
i40e here, unlike the single passed-through port k8s-2/k8s-3 see.

cephfs-metadata drops to size 2 / min_size 1 for the duration. At size 3 on
two remaining hosts every PG sits undersized with min_size 2, so one further
OSD loss would block IO on every CephFS mount cluster-wide. This must go back
to size 3 once k8s-1 rejoins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 16:44:21 +02:00
k.renner 47c1170d51 Merge pull request 'perf(velero): run the restore-test monthly instead of weekly' (#565) from velero-restore-test-monthly into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#565
2026-08-25 13:42:55 +00:00
claude-botandClaude Opus 5 44aa0d6919 perf(velero): run the restore-test monthly instead of weekly
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The sweep restores every discovered Velero schedule, immich's ~180 GiB
library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and
measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per
week — ~47% of the endurance budget, against a ~65 GB/day weekday
baseline.

That baseline is itself the result of the 2026-08-18..21 write-reduction
work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs,
paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive
writes ~70% from ~195 GB/day. With the application tier down, the restore
test is now the single largest consumer of remaining QLC endurance; the
three OSD drives are at 81/82/87% used.

Monthly still exercises every restore path and cuts the sweep's share of
drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.

Cadence-only change:
- 0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the
  03:15-07:15 daily backup window (see the comment in the manifest).
- ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still
  inspectable when the next one fires.
- Comments and docs that state the cadence updated to match, including
  the BLUESTORE_SLOW_OP_ALERT runbook text, which tells the reader to
  check whether a stall fell inside the restore-test window.

No day-of-week form is available: Kubernetes parses cron with
robfig/cron's standard parser, whose day-of-week field allows only
`* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields
ORs them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 07:44:21 +02:00
k.renner a8a7aa9765 Merge pull request 'docs(zigbee2mqtt): record the WS90 ZCL reporting configuration' (#564) from ws90-reporting-config into master
ci/woodpecker/cron/e2e Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#564
2026-08-24 17:58:48 +00:00
14 changed files with 201 additions and 133 deletions
+9 -2
View File
@@ -129,13 +129,20 @@ steps:
- echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
- |
# Ensure Talos image exists on all Proxmox nodes via API
# Ensure Talos image exists on all Proxmox nodes via API.
#
# jupiter is NOT in this list: it runs Talos on bare metal and is no
# longer a Proxmox node, so the storage API returns nothing for it and
# the download-wait loop below would burn its full 60x10s before
# giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
# still need the nocloud image; bare-metal nodes boot the metal-amd64
# image from their own installed disk instead.
TALOS_VERSION="v1.12.6"
TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
for node in jupiter saturn neptune; do
for node in saturn neptune; do
EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
-H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
if [ "$EXISTS" -gt 0 ]; then
+1 -1
View File
@@ -167,7 +167,7 @@ permanent recovery.
## CNPG restore-test (verify backups are restorable — non-destructive)
The weekly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
code path). Run this drill periodically to prove the CNPG backups are actually
restorable, **without touching the live clusters**. It restores into a throwaway
+24
View File
@@ -79,6 +79,30 @@ which also pre-seeds the Talos image onto every Proxmox node. Wait until the
node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
Step 2 — `bootstrap.sh` polls DNS but will not create the records.
**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
Before Step 2, boot jupiter from the bare-metal image below so it comes up in
maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
all three nodes identically from there, since `apply-config --insecure` does
not care whether the target is a VM or bare metal.
```
https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
```
That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
— without it the predictable interface names that `talos/patches/k8s-1.yaml`
matches its VLAN parents against do not exist — and differs in three ways:
- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
`machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
present only so enabling the MS-01 Thunderbolt mesh later does not require a
new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
device attached to the machine once those modules are loaded.
## Step 2 — talos: form the cluster + get kubeconfig
```sh
@@ -34,9 +34,11 @@ spec:
# both episodes in the 30d to 2026-08-23 ran ~26h and cleared
# untouched, the first ~24h after the last stall.
#
# Both began inside the weekly Velero restore-test window (Sun
# 08:00-13:00Z) — a 5h sustained restore that outruns the SLC cache
# on these QLC NVMe. Expected for the hardware, but not excludable:
# Both began inside the Velero restore-test sweep — a ~5h sustained
# restore that outruns the SLC cache on these QLC NVMe. The sweep
# runs monthly, 08:00-14:00Z on the 1st.
#
# Expected for the hardware, but not excludable:
# this is also the earliest signal of a genuinely failing drive, and
# all three are at 80-87% endurance used (see cephcluster.yaml).
expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
@@ -45,7 +47,7 @@ spec:
severity: warning
annotations:
summary: "Ceph health check {{ $labels.name }} active for 6h"
description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the Sunday 08:00-13:00Z Velero restore-test window, which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
- alert: CephPoolLostRedundancy
# Names the pool, which POOL_NO_REDUNDANCY does not.
+1 -1
View File
@@ -79,7 +79,7 @@ spec:
# authoritative copy is the backup on Hetzner, and a lost probe just
# means re-running the test -- so they are equally undeserving of
# replication. Landing them unreplicated would mean a second size-1
# data pool inside `cephfs`, and restore-test runs WEEKLY against
# data pool inside `cephfs`, and restore-test runs MONTHLY against
# CI's ~96 pipelines/day, so it is not worth the complexity.
- name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
value: ceph-filesystem-ci
+1 -1
View File
@@ -268,7 +268,7 @@ EOF
# 120x15s = 30 min, not the old 60x5s = 5 min. node-agent has no
# loadConcurrency override, so Velero's default of one data path per node
# applies: while the Sunday restore-test sweep holds the slot, even this 1 GiB
# applies: while the restore-test sweep holds the slot, even this 1 GiB
# DataUpload sits in phase Prepared for well over 5 minutes. The old 300 s wait
# then let the teardown below delete the source namespace and the Backup
# mid-upload ("Cancel du under phase Prepared because it is being deleted"),
@@ -87,6 +87,16 @@ spec:
lapi:
replicas: 1
# The chart defaults request 500m. Measured steady-state draw is 2m, and
# the request is what the scheduler reserves, so the default parks half a
# core of unusable capacity per LAPI. That is affordable on three nodes
# and is not on two: with one node out for maintenance the remaining pair
# sit at 100% of requested CPU while actually using ~22%, and nothing new
# can be placed. The limit stays at the chart default so a log flood can
# still burst; only the reservation shrinks.
resources:
requests:
cpu: 50m
podAnnotations:
# Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
# env-var secret, so a restart is required to pick up a rotated key). See
@@ -122,6 +132,12 @@ spec:
key: key
agent:
# Same reasoning as lapi.resources above, multiplied by three: this is a
# DaemonSet, so the chart's 500m default reserves half a core on every
# node to run a process that measures 1-2m.
resources:
requests:
cpu: 50m
# The agent parses the ingress controller's access logs (read from
# the pod's stdout) to detect attacks. Repointed from the retired
# ingress-nginx (decommissioned 2026-05-17) to Traefik. The
@@ -164,7 +164,15 @@ spec:
useAllNodes: true
useAllDevices: false
devices:
- name: nvme0n1 # NVMe drive PCI-passed-through from each PVE node
# On k8s-2/k8s-3 this is the NVMe drive PCI-passed-through from saturn
# and neptune, and it is the only block device the guest can see. k8s-1
# is bare metal on jupiter and sees two: the OSD drive in MS-01 slot 1
# (PCI 01:00.0) and the Talos system disk in slot 2 (58:00.0). Slot 1
# enumerates first, so nvme0n1 is still the OSD drive there — but that
# is enumeration order, not a guarantee. The system disk is protected
# from the other direction instead: talos/patches/k8s-1.yaml pins the
# installer to it by serial, and Rook skips a disk carrying partitions.
- name: nvme0n1
config:
osdsPerDevice: "1"
# Storage layer must never be evicted by kubelet under node pressure
@@ -87,15 +87,27 @@ metadata:
name: cephfs
namespace: rook-ceph
spec:
# metadataPool STAYS at size 3 while the data pool drops to 2. It holds
# 467 MiB against the data pool's 275 GiB, so it contributes nothing
# measurable to drive wear — the entire reason for the size reduction — while
# losing it would destroy the whole filesystem, not just the objects on the
# lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
# trade; the wear saving comes from data0.
# TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
#
# The standing trade for this pool is size 3: it holds ~253 MiB against the
# data pool's 280 GiB, so it contributes nothing measurable to drive wear —
# the entire reason data0 dropped to 2 — while losing it would destroy the
# whole filesystem, not just the objects on the lost OSD. Paying 3x on a
# quarter of a gigabyte to keep that intact is the right trade in steady
# state.
#
# It does not hold while one of three hosts is out for the jupiter bare-metal
# conversion. size 3 on two remaining hosts leaves every PG undersized at two
# copies with min_size 2, so a single further OSD loss drops the pool below
# min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
# two copies are active+clean instead, and a further loss degrades rather
# than blocks — availability bought at the cost of accepting writes onto a
# single copy while degraded.
metadataPool:
replicated:
size: 3
size: 2
parameters:
min_size: "1"
dataPools:
- name: data0
# size 2 / min_size 1 for the same reasons as replicapool above — see
@@ -1,19 +1,30 @@
---
# Weekly Velero restore-test CronJob.
# Monthly Velero restore-test CronJob — 08:00 UTC on the 1st.
#
# Sunday 08:00 UTC — *after* Sunday's own daily backups have finished
# (paperless 03:15 through apps 07:15, each normally ~1 min). The previous
# 02:00 slot was chosen as "well before Monday's backups", which missed
# that the daily schedules also fire on Sunday: a sweep starting at 02:00
# and running ~5 h sat on top of the entire 03:15-07:15 backup window and
# The sweep restores every discovered schedule, immich's ~180 GiB library
# included. At `size: 2` that is ~360 GiB raw, ~120 GiB per OSD, against a
# ~65 GB/day weekday baseline on three QLC NVMe at 81-87% endurance used —
# ~47% of each drive's weekly write budget. Monthly keeps every restore path
# exercised and cuts that share ~4x.
#
# 08:00 — *after* that day's own daily backups have finished (paperless
# 03:15 through apps 07:15, each normally ~1 min). The previous 02:00 slot
# was chosen as "well before the next day's backups", which missed that the
# daily schedules fire on the same day: a sweep starting at 02:00 and
# running ~5 h sat on top of the entire 03:15-07:15 backup window and
# starved it of node-agent DataUpload concurrency. Observed effect on
# 2026-07-19 and 2026-07-26 — audiobookshelf-daily took 2 h 18 m and
# 2 h 33 m respectively against a ~1 min baseline, with the books
# DataUpload not starting until 06:18. Both are Sundays; no other day
# shows it.
# DataUpload not starting until 06:18. Both are Sundays, the only day the
# sweep ran then; no other day shows it.
#
# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of
# Monday's 03:15.
# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of the
# next day's 03:15.
#
# Day-of-month rather than a fixed weekday: Kubernetes parses cron with
# robfig/cron's standard parser, which has no `#` nth-weekday operator, and
# restricting both day fields (`0 8 1-7 * 0`) ORs them — every Sunday plus
# the 1st through 7th.
#
# Concurrency policy Forbid: the immich restore (~180 GiB, millions of
# small files) alone can run multiple hours, so the full sweep is sized
@@ -29,7 +40,7 @@ metadata:
name: restore-test
namespace: velero
spec:
schedule: "0 8 * * 0"
schedule: "0 8 1 * *"
timeZone: Etc/UTC
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
@@ -39,7 +50,7 @@ spec:
spec:
backoffLimit: 0
activeDeadlineSeconds: 21600 # 6 h hard cap (immich alone can take ~3 h)
ttlSecondsAfterFinished: 1209600 # 14 d — enough to inspect after the next weekly run
ttlSecondsAfterFinished: 3456000 # 40 d — enough to inspect after the next monthly run
template:
spec:
serviceAccountName: restore-test
@@ -1,5 +1,5 @@
---
# RBAC for the weekly restore-test CronJob.
# RBAC for the monthly restore-test CronJob.
#
# Permissions required by restore-test.sh:
# - velero.io: read schedules/backups/BSL, create+delete restores
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Weekly verification that Velero PVC backups are actually restorable.
# Monthly verification that Velero PVC backups are actually restorable.
#
# What this exercises (per service):
# 1. Velero Restore reaches phase=Completed against yesterday's
@@ -53,7 +53,7 @@ check() {
# and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
# production PVC deletions can be undone manually. The restore-test has the
# opposite need: every PVC it creates is throwaway and MUST not survive the
# iteration. Without this, each weekly run leaks O(100 GiB) of CephFS
# iteration. Without this, each run leaks O(100 GiB) of CephFS
# subvolumes and the cluster eventually goes OSD_FULL (incident 2026-05-17).
#
# Why this still exists after ceph-*-ephemeral landed. Three categories of
+89 -50
View File
@@ -1,62 +1,101 @@
# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
# still Proxmox VMs. Every machine-specific selector below differs from theirs
# for that reason: there are no virtio NICs, no PCI-passthrough layer, and both
# NVMe drives are visible to Talos at once.
#
# Networking uses the v1.12 multi-document config rather than the deprecated
# machine.network.interfaces. That is deliberate: the docs do not define what
# happens when the same link is configured by both schemas, so this file keeps
# every link in one schema instead of straddling them.
machine:
network:
# Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
# below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
# as k8s.renner.uno.
hostname: k8s-1
# Shared control-plane VIP — etcd-elected leader hosts 10.2.40.60 as a
# secondary IP. Exposed via UniFi DNS as k8s.renner.uno.
#
# Two virtio NICs (mgmt + v30iot). `driver: virtio_net` would match
# both, so each is pinned by its PCI bus path:
# - 0000:06:12.0 — original mgmt NIC (the only virtio before)
# - 0000:06:13.0 — extra_network_bridges = ["v30iot"] in tofu
# Bus paths are consistent across k8s-{1,2,3}; verified live 2026-05-13.
# The X710 selector stays driver-based (i40e is unambiguous) and
# tolerates kernel renaming.
interfaces:
- deviceSelector:
busPath: "0000:06:12.0"
dhcp: true
vip:
ip: 10.2.40.60
# v30iot: LINK ONLY — no host addressing. The NIC is the macvlan
# parent for the v30iot pod attachments (matter/otbr/shelly-manager),
# whose children do their own DHCP on the L2; host addresses were
# never needed for that. Host DHCP (formerly dhcp + routeMetric 4096)
# installed a connected 10.2.30.0/24 route, so node replies to
# IoT-subnet clients bypassed the UDM and broke every UDM-DNATed
# flow (hairpin NAT: un-translated SYN-ACK -> client kernel RST,
# verified live 2026-07-23). The purpose the old comment claimed —
# Cilium L2 announce for an IoT LB pool — does not exist (no
# L2AnnouncementPolicy, no such pool). v6 RA addressing on this NIC
# is disabled via machine.sysctls for the same reason.
- deviceSelector:
busPath: "0000:06:13.0"
dhcp: false
# Dedicated Ceph cluster network on the X710 SFP+ port (PCI passthrough,
# mapping x710-port1 in opentofu/vm_150_k8s_1.tf). Separate /29 on VLAN 49,
# MTU 9000, no default route — used by Rook for OSD↔OSD replication via
# CephCluster.spec.network.addressRanges.cluster.
- deviceSelector:
driver: i40e
addresses:
- 10.99.0.1/29
mtu: 9000
sysctls:
# v30iot NIC carries no host addressing (see interfaces above); this
# disables v6 entirely (incl. link-local) so the host is unreachable on that L2.
#
# Keyed on the STABLE predictable name `ens19` (busPath 0000:06:13.0 =
# v30iot). Predictable naming is re-enabled via the Image Factory
# schematic (net.ifnames=1; see .woodpecker/deploy-infrastructure.yml),
# so `ens19` is topology-derived and does NOT shuffle across reboots —
# unlike the former `eth0/1/2` (driver-probe order) which renamed on a
# k8s-3 reboot and silently mis-targeted this sysctl onto mgmt. The four
# v30iot macvlan NADs key on the same `ens19`.
# ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
# the macvlan parent for the v30iot pod attachments
# (matter/otbr/shelly-manager/music-assistant), whose children do their own
# DHCP on that L2. Host addresses were never needed and are actively
# harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
# clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
# un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
# This disables v6 entirely on that link, including link-local.
net.ipv6.conf.ens19.disable_ipv6: "1"
install:
disk: /dev/vda
# Both NVMe drives are now ordinary block devices to Talos: the Kingston
# OM8PGP41024Q-A0 system disk and the Samsung PM983 that Rook claims as the
# Ceph OSD. Selecting the installer target by serial means it cannot land
# on the OSD drive regardless of which one the kernel enumerates first.
diskSelector:
serial: 50026B7382B48B86
certSANs:
- k8s-1.srv.renner.uno
nodeLabels:
topology.kubernetes.io/zone: jupiter
cluster:
allowSchedulingOnControlPlanes: true
---
# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
# driver selector k8s-2/k8s-3 use for their single passed-through port would
# match two links here. MAC matching is unambiguous and survives udev renaming.
apiVersion: v1alpha1
kind: LinkAliasConfig
name: lan0
selector:
match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
---
apiVersion: v1alpha1
kind: LinkAliasConfig
name: ceph0
selector:
match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
---
# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
# before, which is why the VM saw two already-untagged virtio NICs instead.
apiVersion: v1alpha1
kind: LinkConfig
name: lan0
up: true
---
apiVersion: v1alpha1
kind: VLANConfig
name: mgmt0
vlanID: 40
parent: lan0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: mgmt0
---
apiVersion: v1alpha1
kind: Layer2VIPConfig
name: 10.2.40.60
link: mgmt0
---
# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
# serve both VM and bare-metal nodes for as long as the cluster is mixed.
# Nothing else on this machine is called ens19: the physical NICs are
# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp89s0.
apiVersion: v1alpha1
kind: VLANConfig
name: ens19
vlanID: 30
parent: lan0
up: true
---
# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
# on X710 port 1. Used by Rook for OSD<->OSD replication via
# CephCluster.spec.network.addressRanges.cluster. No default route.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph0
up: true
mtu: 9000
addresses:
- address: 10.99.0.1/29
-51
View File
@@ -1,51 +0,0 @@
module "k8s_1" {
source = "./modules/proxmox_vm"
name = "k8s-1.srv.renner.uno"
target_node = "jupiter"
vmid = 150
memory = 49152
cores = 8
template_name = null
cloud_init = false
full_clone = false
bios = "ovmf"
machine = "q35"
boot = "order=virtio0"
agent_enabled = true
# Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
# VLAN from this node (previously only k8s-edge had v30iot, so any
# IoT-pool LB lease landing here was unreachable from devices).
extra_network_bridges = ["v30iot"]
disks = [
{
storage = "local-zfs"
size = "64G"
format = "raw"
slot = "virtio0"
file_id = local.talos_image
}
]
hostpci = [
{
mapping = "osd-ssd"
},
{
mapping = "x710-port1"
}
]
tags = "kubernetes;talos;controlplane"
ssh_public_keys = local.common_ssh_keys
vm_state = "running"
# Rebuild pipeline issues `-replace`; skip graceful shutdown so a
# non-responsive Talos guest can't hang the destroy for 30 minutes.
stop_on_destroy = true
}