Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 c868b8cd9d feat(intel-gpu-plugin): expose the Intel iGPU as gpu.intel.com/i915
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Adds the Intel GPU device plugin as a DaemonSet in kube-system, making the
on-board Iris Xe of each MS-01 node schedulable as `gpu.intel.com/i915`.

Pods can then request the iGPU without a /dev/dri hostPath or privilege
escalation — the plugin passes the render node through the kubelet
device-plugin API. The resource covers QuickSync transcoding as well as GPU
compute.

Vendored from upstream v0.36.0 rather than installed from the
`intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
custom resource requiring the Intel Device Plugin Operator, its CRDs and a
cert-manager webhook. One device type on three identical nodes gives the
operator nothing to manage that the DaemonSet does not.

Runs in kube-system because the plugin mounts hostPaths; the cluster enforces
PodSecurity `baseline`, which forbids those, and kube-system is the only
namespace in the API server's exemption list. NFD is not deployed, so
upstream's nfd_labeled_nodes overlay and its nodeSelector are omitted.

`-shared-dev-num=4` lets several containers share the single iGPU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 12:38:42 +02:00
security-bot 99445753d1 chore(image-automation): bump valkey/valkey:9.1.2-alpine
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Updates managed by image-automation-controller.
2026-09-01 02:21:46 +00:00
67 changed files with 2536 additions and 3011 deletions
+6 -97
View File
@@ -87,30 +87,6 @@ steps:
--destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
--destination gitea.renner.uno/gitea_admin/ci-k8s:latest
- name: build-vmtest
image: gcr.io/kaniko-project/executor:v1.23.2-debug
directory: ci/images/vmtest
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/vmtest/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/ci-vmtest:$TAG \
--destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
- name: build-tofu
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/tofu
@@ -171,28 +147,6 @@ steps:
# not for who uses it — its first consumer is the zigbee2mqtt seed-data init
# container, but nothing about it is zigbee-specific. Exists so tools come from
# the image rather than an `apk add` on every container start.
#
# Published THREE times, and the third one is not a typo. Its consumers — the
# zigbee2mqtt seed-data init container and the trmnl renderer — start while
# the cluster is still being rebuilt, and at that moment gitea.renner.uno does
# not exist, because Gitea runs inside the cluster being built. A cold-rebuild
# rehearsal caught both in ImagePullBackOff.
#
# So there is a copy on ghcr.io. But the manifests must NOT point at it:
# normal operation pulls from the local registry, and an external reference
# would make every steady-state pull leave the network. The recovery source
# and the everyday source are different things, exactly as they are for Flux's
# git source.
#
# Talos resolves that with machine.registries.mirrors, which takes an ordered
# endpoint list — gitea first, ghcr second (bootstrap.sh). Containerd keeps
# the repository path when it falls through, so the two registries have to
# agree on it: `gitea_admin/alpine-jq` cannot be served by ghcr, where the
# namespace is fixed to the account name. Hence the second Gitea destination
# under konairius/, which the manifests reference and which matches the ghcr
# path exactly.
#
# The gitea_admin/ copy stays until nothing references it any more.
- name: build-alpine-jq
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/alpine-jq
@@ -205,47 +159,23 @@ steps:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
GHCR_USER: konairius
GHCR_TOKEN:
from_secret: ghcr_mirror_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
--destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
--destination gitea.renner.uno/konairius/alpine-jq:$TAG \
--destination gitea.renner.uno/konairius/alpine-jq:latest \
--destination ghcr.io/konairius/alpine-jq:$TAG \
--destination ghcr.io/konairius/alpine-jq:latest
--destination gitea.renner.uno/gitea_admin/alpine-jq:latest
# Runtime image, not CI tooling: upstream containernetworking/plugins binaries
# baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
# node-side fetch from GitHub that had NO checksum verification and sat on the
# critical path of every node rebuild. The download now happens here, once,
# where the sha256 is checked — see the Containerfile for the full rationale.
#
# This one ALSO publishes to ghcr.io. A cold-rebuild rehearsal proved why: the
# DaemonSet could not pull from gitea.renner.uno, because Gitea runs inside the
# very cluster being rebuilt. Without the plugins there is no macvlan, so
# Multus cannot attach ceph-san, so Rook's network canary never starts, so the
# CephCluster sits at "Configuring Ceph Mons" — and OpenBao waits forever for a
# PVC that will never bind. Moving the download here fixed the integrity gap
# but quietly swapped github.com on the critical path for something strictly
# worse: a registry that by definition does not exist during a recovery. The
# GHCR copy is public, so a bare node can pull it with no credential and no
# Gitea.
#
# The test that proved this also proved the rule is not specific to this
# image. alpine-jq and alpine-openssl mirror for the same reason — see the
# note on build-alpine-jq. Anything an app needs while the cluster is still
# being built cannot live only in that cluster's own registry.
- name: build-cni-plugins
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/cni-plugins
@@ -258,36 +188,23 @@ steps:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
GHCR_USER: konairius
GHCR_TOKEN:
from_secret: ghcr_mirror_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
--destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
--destination gitea.renner.uno/konairius/cni-plugins:$TAG \
--destination gitea.renner.uno/konairius/cni-plugins:latest \
--destination ghcr.io/konairius/cni-plugins:$TAG \
--destination ghcr.io/konairius/cni-plugins:latest
--destination gitea.renner.uno/gitea_admin/cni-plugins:latest
# Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
# needing s_client/x509. Named for what it is; first consumer is the paperless
# fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
# every paperless pod start. The openssl CLI is genuinely absent from the alpine
# base, so this one needs an image rather than a busybox substitute.
#
# Mirrored to ghcr.io for the reason given on build-alpine-jq: it is an init
# container, so paperless does not start without it, and during a rebuild
# gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
# paperless in Init:ErrImagePull on exactly this image.
- name: build-alpine-openssl
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/alpine-openssl
@@ -300,25 +217,17 @@ steps:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
GHCR_USER: konairius
GHCR_TOKEN:
from_secret: ghcr_mirror_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
--destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
--destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
--destination gitea.renner.uno/konairius/alpine-openssl:latest \
--destination ghcr.io/konairius/alpine-openssl:$TAG \
--destination ghcr.io/konairius/alpine-openssl:latest
--destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
# Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
# run as two services on edge-1. Replaces pip-into-venv on the host, where an
-84
View File
@@ -1,84 +0,0 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Everything `bootstrap.sh vmtest` needs, so the rebuild test can run as a Job in
# the cluster instead of on a workstation. The workstation has 31G and the test
# wants ~14.5G for four VMs — a single run left under 1G headroom, and two
# overlapping runs OOM-killed the session. The nodes have ~94G with 70G free.
#
# Separate from ci-k8s rather than built on it: that image pins talosctl v1.12.6,
# and this test creates a cluster at the version talos/patches/cluster.yaml
# installs. A talosctl older than the Talos it provisions is exactly the kind of
# skew the test exists to catch, so it must not be baked in here.
#
# KEEP TALOS_VERSION IN STEP with the installer image tag in
# kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
# file at run time and passes it to the provisioner; this pin only decides which
# client talks to it.
ARG TALOS_VERSION=v1.13.9
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KOPIA_VERSION=0.23.1
# qemu-system-x86_64 and qemu-img are the provisioner's actual backend, and ovmf
# supplies the UEFI firmware those VMs boot from — without it the provisioner
# stops with "the required flash image was not found in any of the expected
# paths", having already downloaded the ISO and generated the PKI. Note the
# firmware comes from ovmf specifically: qemu-system-x86_64 ships an edk2 CODE
# image but no VARS, which is the half the provisioner needs.
#
# iptables is shelled out to by the CNI plugins that build the VM network; the
# provisioner fails with "error accessing iptables" without it. The rest is
# what the phase shells out to: helm for the Cilium install, yq to read the
# HelmRelease and the installer image, jq for OpenBao's seal status.
#
# Versions come from the digest-pinned base rather than being pinned one by one,
# the same way ci-ansible does it.
RUN apk add --no-cache \
bash \
ca-certificates \
curl \
git \
helm \
iptables \
jq \
openssl \
ovmf \
qemu-img \
qemu-system-x86_64 \
yq
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
"https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
&& curl -sfL -o /tmp/kubectl.sha256 \
"https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl.sha256" \
&& echo "$(cat /tmp/kubectl.sha256) /usr/local/bin/kubectl" | sha256sum -c - \
&& rm -f /tmp/kubectl.sha256 \
&& chmod +x /usr/local/bin/kubectl
RUN curl -sfL -o /tmp/flux.tar.gz \
"https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
&& echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3 /tmp/flux.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
&& chmod +x /usr/local/bin/flux \
&& rm -f /tmp/flux.tar.gz
# kopia reads the Velero backup repository directly, which is what the OpenBao
# restore does when there is no Velero yet to do it: the credentials Velero
# needs live in OpenBao, so OpenBao cannot be restored by Velero. Its absence
# was found by a rebuild rehearsal reaching the restore for the first time and
# stopping at "kopia not found".
#
# Same checksum-verified download as the tools above; the tarball unpacks into
# a versioned directory, so the binary is pulled out by name.
RUN curl -sfL -o /tmp/kopia.tar.gz \
"https://github.com/kopia/kopia/releases/download/v${KOPIA_VERSION}/kopia-${KOPIA_VERSION}-linux-x64.tar.gz" \
&& echo "416d0f84a3dbb321a8b2d8f0997b1a0a6e915babe79ee76fa6e4d2bd1e1c5178 /tmp/kopia.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/kopia.tar.gz -C /tmp \
&& mv "/tmp/kopia-${KOPIA_VERSION}-linux-x64/kopia" /usr/local/bin/kopia \
&& chmod +x /usr/local/bin/kopia \
&& rm -rf /tmp/kopia.tar.gz "/tmp/kopia-${KOPIA_VERSION}-linux-x64"
@@ -15,6 +15,7 @@ resources:
- alertmanager-gate-egress.yaml
- prometheus-gate-egress.yaml
- wordpress-gate-egress.yaml
- outpost-servicemonitors.yaml
- ldap-outpost-cert.yaml
- ldap-outpost-lb.yaml
- ldap-outpost-cnp.yaml
@@ -57,14 +57,7 @@ spec:
# and leaves this one untouched (tags are not deleted), so this keeps
# resolving — but the image then came from the branch build rather than
# master. Re-point it at a master-built tag when convenient.
#
# konairius/, not gitea_admin/: this init container runs during a cold
# rebuild, when Gitea does not exist because it runs inside the
# cluster being rebuilt. The node then falls back to a ghcr.io copy
# (machine.registries.mirrors, applied by bootstrap.sh), and containerd
# keeps the repository path when it falls through — so the path has to
# exist on both registries. Steady state still pulls from Gitea.
image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
command:
- sh
- -c
@@ -1,46 +0,0 @@
# Cilium's agent ServiceMonitor, declared here rather than by the chart.
#
# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
# taken field for field off the live object so scraping is unchanged: same name,
# same namespace, same selector, same 10s interval and node relabeling.
#
# It lives here because the chart's template is gated on monitoring.coreos.com/v1
# existing at install time, and on an empty cluster it does not — those CRDs
# arrive with kube-prometheus-stack, which reconciles after infrastructure.
# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
#
# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
# why; this is the same move for the component where it mattered most.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: cilium-agent
namespace: kube-system
labels:
app.kubernetes.io/name: cilium-agent
app.kubernetes.io/part-of: cilium
spec:
selector:
matchLabels:
app.kubernetes.io/name: cilium-agent
namespaceSelector:
matchNames:
- kube-system
# The chart's Service keeps rendering (prometheus.enabled), so the target and
# its `metrics` port are there regardless of who declares the ServiceMonitor.
targetLabels:
- k8s-app
endpoints:
- port: metrics
path: /metrics
interval: 10s
honorLabels: true
relabelings:
# Agent metrics are per-node; without this every series lands without a
# node label and BPF map pressure cannot be attributed to a machine.
- action: replace
sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: node
replacement: ${1}
@@ -23,20 +23,9 @@ resources:
# endpoint, which proves reachability, not that anything was written.
- prometheusrule-cnpg-backup.yaml
- alloy-servicemonitor.yaml
- cilium-servicemonitor.yaml
- cnpg-podmonitors.yaml
- home-assistant-servicemonitor.yaml
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- probes-blackbox.yaml
# Velero's own monitors: monitoring.coreos.com CRDs arrive with
# kube-prometheus-stack, which reconciles after infrastructure, so these
# cannot live beside the Velero HelmRelease.
- velero-podmonitor.yaml
- velero-servicemonitor.yaml
- authentik-outpost-servicemonitors.yaml
# Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
# when it cannot create these, which held the CephCluster in Progressing and
# deadlocked the whole apps layer. See the file for the cycle.
- rook-servicemonitors.yaml
@@ -1,93 +0,0 @@
# Rook's two ServiceMonitors, declared here rather than by the operator.
#
# Reproduced field for field off the live objects in the production cluster, so
# scraping is unchanged: same names, same namespace, same selectors, same 10s
# interval and `cluster` relabeling. The only thing dropped is the CephCluster
# ownerReference, which Rook set and Flux cannot.
#
# They live here because CephCluster.spec.monitoring.enabled is now false, and
# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
# reconcile.go, v1.20.7):
#
# if !cephCluster.Spec.Monitoring.MetricsDisabled {
# // exporter reconciled; its errors are logged and reconcile continues
# if cephCluster.Spec.Monitoring.Enabled {
# if err := EnableCephExporterServiceMonitor(...); err != nil {
# return errors.Wrap(err, "failed to enable service monitor")
# }
# }
# }
#
# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
# fails with "the server could not find the requested resource (post
# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
# aborts, and the CephCluster never leaves status.phase Progressing. That is
# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
# closes on itself:
#
# infrastructure-ceph-cluster → CephCluster phase Ready
# → Rook reconcile settles
# → monitoring.coreos.com CRDs exist
# → kube-prometheus-stack (monitoring)
# → grafana-db
# → infrastructure-ceph-cluster
#
# Nothing outside that loop can break it, so a cold rebuild stopped here with
# every one of the twenty app Kustomizations behind it.
#
# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
# get, and for the same reason: the metrics stay where they are, only the
# scrape wiring moves to the layer where its CRDs exist.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: rook-ceph-exporter
namespace: rook-ceph
labels:
team: rook
spec:
selector:
matchLabels:
app: rook-ceph-exporter
rook_cluster: rook-ceph
namespaceSelector:
matchNames:
- rook-ceph
endpoints:
- port: ceph-exporter-http-metrics
path: /metrics
interval: 10s
honorLabels: true
# Rook stamps the cluster name so series from several Ceph clusters stay
# separable. One cluster here, but dropping it would rename every series.
relabelings:
- action: replace
replacement: rook-ceph
targetLabel: cluster
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: rook-ceph-mgr
namespace: rook-ceph
labels:
team: rook
spec:
selector:
matchLabels:
app: rook-ceph-mgr
rook_cluster: rook-ceph
namespaceSelector:
matchNames:
- rook-ceph
endpoints:
- port: http-metrics
path: /metrics
interval: 10s
honorLabels: true
relabelings:
- action: replace
replacement: rook-ceph
targetLabel: cluster
+1 -14
View File
@@ -93,20 +93,7 @@ spec:
# on every paperless pod start: tools belong in the image, not
# installed at runtime. Dated tag so a change is a reviewable diff
# here, matching the ci-ansible / alpine-jq convention.
#
# konairius/, not gitea_admin/, and that is not cosmetic. This is an
# INIT container, so paperless does not start without it — and during
# a cold rebuild gitea.renner.uno does not exist, because Gitea runs
# inside the cluster being rebuilt. A rehearsal left this pod in
# Init:ErrImagePull on exactly this image.
#
# The node falls back to a ghcr.io copy in that window
# (machine.registries.mirrors, applied by bootstrap.sh). Containerd
# keeps the repository path when it falls through, so the path has to
# exist on both registries — hence konairius/, which matches the ghcr
# namespace. Steady state still pulls from Gitea and never leaves the
# network.
image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04
image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
command:
- sh
- -c
+1 -8
View File
@@ -177,14 +177,7 @@ spec:
# webhook start failing with "expired JWT access token" about
# thirty minutes after seeding.
- name: render-schedule
# konairius/, not gitea_admin/: this container starts during a cold
# rebuild, when Gitea does not exist because it runs inside the
# cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff.
# The node falls back to a ghcr.io copy then
# (machine.registries.mirrors, applied by bootstrap.sh), and containerd
# keeps the repository path, so it has to exist on both registries.
# Steady state still pulls from Gitea.
image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
command:
- sh
- -c
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
# Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
# is the only consumer; a major bump is a separate migration.
- name: valkey
image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
image: valkey/valkey:9.1.2-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
args:
- valkey-server
- --save
+15
View File
@@ -1,3 +1,18 @@
# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
# isolated from the apps namespace.
apiVersion: v1
kind: Namespace
metadata:
name: woodpecker-ci
labels:
app.kubernetes.io/name: woodpecker
# Image-build steps run a privileged docker/buildx plugin pod (see the
# server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
# without this the privileged step pods are rejected at admission. The
# namespace is dedicated to ephemeral CI step pods, so scoping the
# exemption here keeps it off apps/.
pod-security.kubernetes.io/enforce: privileged
---
apiVersion: v1
kind: ServiceAccount
metadata:
+60 -1377
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -8,10 +8,10 @@ metadata:
spec:
interval: 1m0s
ref:
branch: vmtest-flux
branch: master
secretRef:
name: flux-system
url: https://gitea.renner.uno/gitea_admin/homelab.git
url: https://gitea.renner.uno/k.renner/homelab.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -28,19 +28,6 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
# CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
# kind is unknown when this path is first applied, and a server-side
# dry-run rejects the ENTIRE set over the one object it cannot recognise:
#
# ObjectStore/home-assistant/ha-pg-store dry-run failed:
# no matches for kind "ObjectStore" in version ...
#
# Every other ObjectStore user — authentik, immich, paperless, trmnl,
# woodpecker — already declares it. This was the only one that did not, and
# it took speech-to-phrase, zone-configurator and matter-server down with
# it, since all three wait on home-assistant.
- name: infrastructure-cnpg-barman
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -122,20 +122,8 @@ spec:
path: ./kubernetes/infrastructure/rook-ceph-cluster
prune: true
wait: true
# Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
# existing cluster could ever hit and no new one could ever escape:
# infrastructure is Ready only once its openbao healthCheck passes, openbao-0
# only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
# defined in this very Kustomization. A cold rebuild stopped here for good,
# with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
#
# The edge was never needed. infrastructure runs wait: false, so it applies
# the rook-ceph namespace and the operator immediately and only its readiness
# waits on the healthChecks — everything this path needs exists long before
# infrastructure reports Ready. On an empty API server this Kustomization
# retries on retryInterval until the operator installs its CRDs, the same
# self-healing shape post-crds already relies on; on a live cluster the CRDs
# are always present and nothing changes.
dependsOn:
- name: infrastructure
# kstatus doesn't know how to read Rook CR readiness, so without these
# this Kustomization reports Ready as soon as the manifests apply — long
# before OSDs/mons/MDS are actually serving I/O. Apps depending on this
@@ -1,115 +0,0 @@
# The upstream ceph-csi-drivers chart, which is how Rook expects CSI drivers to
# be provided since v1.20: "CSI drivers are admin-managed via the
# ceph-csi-operator. Rook no longer deploys CSI."
#
# This replaces a static rendering of the chart's ServiceAccounts and RBAC. That
# workaround (#babed078) was written during a five-day outage in which both
# ctrlplugin Deployments sat at 0/2 and PVC attach/detach was blocked
# cluster-wide, and it justified itself with:
#
# "helm-controller v1.5.4 predates disableTakeOwnership, so Helm would refuse
# to adopt the operator-managed Driver and OperatorConfig CRs"
#
# That is not so. disableTakeOwnership landed in helm-controller v1.2.0, three
# minors BEFORE the version running here, and it is an opt-OUT: taking ownership
# of existing resources is the default. Nothing stood in the way of the chart.
#
# What the workaround could not do is create the Driver CRs, and a cold-rebuild
# rehearsal is where that surfaced. Rook's v1.18 -> v1.20 migration had created
# them once on this cluster, so they looked permanent; on an empty cluster Rook
# creates the CephConnection and ClientProfile and stops there, no Driver is
# ever made, the ceph-csi-operator deploys no provisioner, and every PVC waits
# forever. OpenBao's among them, which is where the rebuild stopped.
#
# cephConnections and clientProfiles stay EMPTY here on purpose. Rook owns both:
# the CephConnection carries the mon Service IPs Rook keeps current, and the
# ClientProfile points at Secrets Rook creates. Handing either to Helm would
# freeze a value that is supposed to move. Verified on a fresh cluster — Rook
# creates exactly those two and nothing else.
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: ceph-csi-operator
namespace: rook-ceph
spec:
interval: 24h
url: https://ceph.github.io/ceph-csi-operator/
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: ceph-csi-drivers
namespace: rook-ceph
spec:
interval: 30m
# Wait for the rook-ceph HelmRelease to finish, not just for the CRDs to
# appear. The two are not the same thing, and an earlier version of this file
# argued the opposite: "nothing here installs those CRDs, so a dependsOn would
# only move the wait". True as far as existence goes — but the csi.ceph.io
# CRDs come from the ceph-csi-operator subchart inside the rook-ceph release,
# which runs with `crds: CreateReplace` and `retries: 3`, so they are not
# created once. They are created again on every attempt, and everything living
# under them goes with them.
#
# A cold rebuild landed in exactly that window:
#
# 19:24:21 ceph-csi-drivers install succeeded — both Driver CRs created
# 19:25:50 csi.ceph.io CRDs recreated by a later operator attempt
# 19:29:33 rook-ceph-operator install finally succeeded
#
# The Driver objects were gone, but Helm still had them in its stored release
# and would not re-render before the 30m interval. No controller reported
# anything: the ceph-csi-operator sat idle with nothing to reconcile, no
# provisioner Deployment was ever created, and every PVC on ceph-block stayed
# Pending on "waiting for a volume to be created by the external provisioner
# rook-ceph.rbd.csi.ceph.com". openbao-0 then never got a node, and the run
# failed 15 minutes later reporting that OpenBao would not answer.
dependsOn:
- name: rook-ceph-operator
chart:
spec:
chart: ceph-csi-drivers
version: "~1.0.4"
sourceRef:
kind: HelmRepository
name: ceph-csi-operator
namespace: rook-ceph
install:
# Kept as a backstop. dependsOn removes the race this was covering for, but
# a CRD replace outside an install of this release would still empty it, and
# a retry is cheaper than a 30-minute wait for the next interval.
remediation:
retries: 5
upgrade:
remediation:
retries: 3
values:
# Rook's, not ours — see the header.
cephConnections: []
clientProfiles: []
# Names must match what the StorageClasses already name as their
# provisioner (rook-ceph.rbd.csi.ceph.com), which is also what Rook's
# migration produced on the live cluster. The chart's default drops the
# namespace prefix and would leave every StorageClass pointing at a driver
# that does not exist.
# clusterName is load-bearing and easy to miss: it becomes --clustername on
# the plugin, and ceph-csi hands it to rados_create2 as the Ceph cluster
# name. The chart defaults it to "", which librados rejects outright — the
# provisioner then fails every request with
# failed to get connection: connecting failed: rados: ret=-22
# which reads like a credential or network fault and is neither. The
# toolbox keeps working throughout, because it falls back to the default
# name "ceph", so the cluster looks healthy while nothing can provision.
drivers:
rbd:
name: rook-ceph.rbd.csi.ceph.com
clusterName: rook-ceph
enabled: true
cephfs:
name: rook-ceph.cephfs.csi.ceph.com
clusterName: rook-ceph
enabled: true
nfs:
enabled: false
nvmeof:
enabled: false
@@ -1,6 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# The ServiceAccounts and RBAC that used to be rendered here by hand now come
# from the chart that owns them, together with the Driver CRs they exist for.
resources:
- helmrelease.yaml
- serviceaccounts.yaml
- rbac.yaml
@@ -0,0 +1,442 @@
# RBAC for the per-driver ServiceAccounts in serviceaccounts.yaml.
# Rendered from ceph-csi-operator/ceph-csi-drivers chart v1.0.1 templates,
# with $normalizedDriverName fixed to the operator's hardcoded driver-type
# names (`rbd`, `cephfs`). Sync with upstream if the chart's RBAC changes.
# Source: https://github.com/ceph/ceph-csi-operator/tree/v1.0.1/deploy/charts/ceph-csi-drivers/templates
# ---------- rbd-ctrlplugin ----------
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rbd-ctrlplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list, watch, create, delete, patch, update]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get, list, watch, update]
- apiGroups: [storage.k8s.io]
resources: [storageclasses]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list, watch, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments/status]
verbs: [patch]
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, watch]
- apiGroups: [storage.k8s.io]
resources: [csinodes]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumeclaims/status]
verbs: [patch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshots]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents]
verbs: [get, list, watch, patch, update]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationcontents]
verbs: [get, list, watch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationclasses]
verbs: [get, list, watch]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
- apiGroups: [authorization.k8s.io]
resources: [subjectaccessreviews]
verbs: [create]
- apiGroups: [cbt.storage.k8s.io]
resources: [snapshotmetadataservices]
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rbd-ctrlplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rbd-ctrlplugin-cr
subjects:
- kind: ServiceAccount
name: rbd-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-ctrlplugin-r
namespace: rook-ceph
rules:
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [get, watch, list, delete, update, create]
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-ctrlplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-ctrlplugin-r
subjects:
- kind: ServiceAccount
name: rbd-ctrlplugin-sa
namespace: rook-ceph
# ---------- rbd-nodeplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rbd-nodeplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [""]
resources: [nodes]
verbs: [get]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rbd-nodeplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rbd-nodeplugin-cr
subjects:
- kind: ServiceAccount
name: rbd-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-nodeplugin-r
namespace: rook-ceph
rules:
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-nodeplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-nodeplugin-r
subjects:
- kind: ServiceAccount
name: rbd-nodeplugin-sa
namespace: rook-ceph
# ---------- cephfs-ctrlplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephfs-ctrlplugin-cr
rules:
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, watch]
- apiGroups: [storage.k8s.io]
resources: [csinodes]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [persistentvolumes]
verbs: [get, list, watch, create, delete, patch, update]
- apiGroups: [""]
resources: [persistentvolumeclaims]
verbs: [get, list, watch, patch, update]
- apiGroups: [storage.k8s.io]
resources: [storageclasses]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments]
verbs: [get, list, watch, patch]
- apiGroups: [storage.k8s.io]
resources: [volumeattachments/status]
verbs: [patch]
- apiGroups: [""]
resources: [persistentvolumeclaims/status]
verbs: [patch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshots]
verbs: [get, list]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents]
verbs: [get, list, watch, patch, update]
- apiGroups: [snapshot.storage.k8s.io]
resources: [volumesnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.k8s.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotclasses]
verbs: [get, list, watch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents]
verbs: [get, list, watch, update, patch]
- apiGroups: [groupsnapshot.storage.openshift.io]
resources: [volumegroupsnapshotcontents/status]
verbs: [update, patch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationcontents]
verbs: [get, list, watch]
- apiGroups: [replication.storage.openshift.io]
resources: [volumegroupreplicationclasses]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephfs-ctrlplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephfs-ctrlplugin-cr
subjects:
- kind: ServiceAccount
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-ctrlplugin-r
namespace: rook-ceph
rules:
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [get, watch, list, delete, update, create]
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-ctrlplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-ctrlplugin-r
subjects:
- kind: ServiceAccount
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
# ---------- cephfs-nodeplugin ----------
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephfs-nodeplugin-cr
rules:
- apiGroups: [""]
resources: [nodes]
verbs: [get]
- apiGroups: [""]
resources: [secrets]
verbs: [get, list, watch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts]
verbs: [get]
- apiGroups: [""]
resources: [serviceaccounts/token]
verbs: [create]
- apiGroups: [""]
resources: [events]
verbs: [list, watch, create, update, patch]
- apiGroups: [""]
resources: [persistentvolumes, persistentvolumeclaims]
verbs: [get]
- apiGroups: [authentication.k8s.io]
resources: [tokenreviews]
verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephfs-nodeplugin-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephfs-nodeplugin-cr
subjects:
- kind: ServiceAccount
name: cephfs-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-nodeplugin-r
namespace: rook-ceph
rules:
- apiGroups: [csiaddons.openshift.io]
resources: [csiaddonsnodes]
verbs: [get, watch, list, create, update, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get]
- apiGroups: [apps]
resources: [deployments/finalizers, daemonsets/finalizers]
verbs: [update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-nodeplugin-rb
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-nodeplugin-r
subjects:
- kind: ServiceAccount
name: cephfs-nodeplugin-sa
namespace: rook-ceph
@@ -0,0 +1,35 @@
# Per-driver ServiceAccounts consumed by the workloads that the
# ceph-csi-controller-manager generates for each `Driver` CR. Rook v1.20
# extracted CSI driver management into the upstream ceph-csi-operator
# (subchart of `rook-ceph`); these SAs are normally provisioned by the
# separate `ceph-csi-drivers` chart, but installing that chart would also
# create a parallel set of `Driver` / `OperatorConfig` CRs and Helm cannot
# adopt the operator-managed ones on helm-controller v1.5.4. We render
# only the SAs + RBAC the existing operator-managed `Driver` CRs need.
#
# Names are fixed by the operator (CSI_SERVICE_ACCOUNT_PREFIX="" on
# ceph-csi-controller-manager): `<driver-type>-{ctrl,node}plugin-sa`,
# where driver-type is `rbd` or `cephfs`. Do not rename.
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-nodeplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-ctrlplugin-sa
namespace: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-nodeplugin-sa
namespace: rook-ceph
@@ -70,23 +70,9 @@ spec:
# map only shows up as "no space left on device" in the log while pod
# sandbox creation fails (#452). cilium_bpf_map_pressure makes it
# alertable beforehand.
#
# That reasoning is about the metrics, which this line provides. The
# ServiceMonitor is only the scrape wiring and is declared standalone in
# apps/monitoring-rules instead — the same treatment velero/helmrelease
# already gives its own, and for the same reason: the chart's template is
# gated on monitoring.coreos.com/v1 existing at install time.
#
# Here it is worse than for velero. Cilium's chart does not skip the
# template, it aborts rendering — "Service Monitor requires
# monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
# healthChecks the infrastructure Kustomization waits on. So on an empty
# cluster the CNI blocked infrastructure, which blocked grafana-db, which
# blocked monitoring, which is where those CRDs come from. The lowest
# layer in the cluster was waiting on the highest.
enabled: true
serviceMonitor:
enabled: false
enabled: true
bpf:
# Per-endpoint policy map size: one entry per (identity, direction,
# protocol, port) allowed. When it fills, endpoint regeneration fails
@@ -54,7 +54,7 @@ spec:
# required.
#
# NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
# bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
# bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
# so there is no live impact, but re-enabling it will require pointing the
# plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
tls:
@@ -1,4 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- bouncer-key-externalsecret.yaml
- helmrelease.yaml
@@ -1,4 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- helmrelease.yaml
@@ -9,26 +9,9 @@ spec:
---
# v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
# (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
# filter pins us to the bare three-component release.
#
# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
# upgrade tied to the Rook operator version. That coupling had already come
# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
# the mons with a client a whole major ahead of them.
#
# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
# Nothing broke only because the four CSI users predate the cipher change and
# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
# because a fresh cluster has no legacy keys at all.
#
# So the cutover happens now, with the version the client already is. Rook
# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
# upstream reports data corruption there when read affinity is enabled.
# filter pins us to the bare three-component release. Range stays at
# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
# upgrade tied to the Rook operator version.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
@@ -42,4 +25,4 @@ spec:
extract: '$ver'
policy:
semver:
range: ">=20.2.1 <21.0.0"
range: "19.x"
@@ -0,0 +1,118 @@
---
# Advertises the on-board Intel iGPU as the schedulable resource
# `gpu.intel.com/i915`, so pods can request it without a /dev/dri hostPath or
# any privilege escalation — the plugin hands the render node to the container
# through the kubelet device-plugin API. The same resource covers both GPU
# compute (LLM inference) and QuickSync video transcoding; it is not
# compute-specific.
#
# All three nodes are MS-01 with an i9-12900H (Alder Lake-P, PCI 8086:46a6,
# Iris Xe 96 EU) at 0000:00:02.0. Talos carries the driver in the `i915` system
# extension, already present in the schematic, so no machineconfig change is
# needed.
#
# Vendored from upstream rather than installed via the
# `intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
# custom resource, which only works alongside the Intel Device Plugin Operator
# and its CRDs, cert-manager webhook and controller. With one device type on
# three identical nodes, the operator manages nothing this DaemonSet does not.
#
# Source: intel/intel-device-plugins-for-kubernetes, v0.36.0,
# deployments/gpu_plugin/base/intel-gpu-plugin.yaml
# Upgrading means re-syncing this file against that path at the new tag and
# bumping the image tag below.
#
# Namespace is kube-system because the plugin mounts hostPaths (/dev/dri,
# /sys/class/drm, the kubelet device-plugin socket and the CDI path). The
# cluster enforces PodSecurity `baseline`, which forbids hostPath volumes, and
# kube-system is the only namespace in the API server's exemption list.
#
# Node Feature Discovery is not deployed, so upstream's `nfd_labeled_nodes`
# overlay is omitted along with its `intel.feature.node.kubernetes.io/gpu`
# nodeSelector. Every node carries the same iGPU, leaving nothing to discover;
# the plugin exits cleanly on a node without /dev/dri.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-plugin
namespace: kube-system
labels:
app: intel-gpu-plugin
spec:
selector:
matchLabels:
app: intel-gpu-plugin
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
template:
metadata:
labels:
app: intel-gpu-plugin
spec:
containers:
- name: intel-gpu-plugin
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
image: intel/intel-gpu-plugin:0.36.0
imagePullPolicy: IfNotPresent
args:
# Slots the single iGPU is handed out as, letting that many
# containers share it concurrently. A slot count, not fractional
# accounting — nothing partitions the device between sharers, so
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
resources:
requests:
memory: "45Mi"
cpu: 40m
limits:
memory: "90Mi"
cpu: 100m
volumeMounts:
- name: devfs
mountPath: /dev/dri
readOnly: true
- name: sysfsdrm
mountPath: /sys/class/drm
readOnly: true
- name: kubeletsockets
mountPath: /var/lib/kubelet/device-plugins
- name: cdipath
mountPath: /var/run/cdi
volumes:
- name: devfs
hostPath:
path: /dev/dri
- name: sysfsdrm
hostPath:
path: /sys/class/drm
- name: kubeletsockets
hostPath:
path: /var/lib/kubelet/device-plugins
- name: cdipath
hostPath:
path: /var/run/cdi
type: DirectoryOrCreate
nodeSelector:
kubernetes.io/arch: amd64
@@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- daemonset.yaml
@@ -10,6 +10,7 @@ resources:
- cert-manager/
- snapshot-controller/
- metrics-server/
- intel-gpu-plugin/
- crowdsec/
- traefik/
- rook-ceph/
@@ -30,11 +30,6 @@ spec:
labels:
app: cni-plugins-installer
spec:
# Same reasoning as multus-daemonset.yaml. This one installs the macvlan
# binary the ceph-san attachment is made of, so a node that cannot
# schedule it has no plugin to delegate to — the failure the init
# container comment below already describes, reached by a different road.
priorityClassName: system-node-critical
hostNetwork: true
tolerations:
- operator: Exists
@@ -49,32 +44,7 @@ spec:
# at all and put github.com on the critical path of every node rebuild.
# Dated tag so a version change is a reviewable diff here, matching the
# ci-ansible / alpine-jq convention.
#
# konairius/, not gitea_admin/, and the difference is what makes a
# rebuild possible. Gitea runs inside this cluster, so during a rebuild
# it does not exist: a rehearsal sat here in ImagePullBackOff with "no
# route to host", and because the plugins never installed there was no
# macvlan, so Multus could not attach ceph-san, so Rook's network
# canary never started, so the CephCluster hung at "Configuring Ceph
# Mons" and OpenBao waited forever on a PVC that could not bind. The
# failure surfaced four layers from its cause.
#
# This used to point straight at ghcr.io, which fixed the rebuild and
# broke the normal case: every steady-state pull then left the network
# for an image the local registry already holds. The fallback belongs
# to the node, not to the manifest — machine.registries.mirrors names
# Gitea first and ghcr.io second (bootstrap.sh), so nothing here has to
# know about the recovery path.
#
# Containerd keeps the repository path when it falls through, so the
# path must resolve on both registries; the build pushes one digest to
# konairius/ on each, verified equal. Bumping the tag means building
# first — an unpublished tag is unpullable on both.
#
# No imagePullSecret: the ghcr namespace is private and its credential
# is in machine.registries too, because a Secret would have to come
# from OpenBao, which needs Ceph, which needs these binaries.
image: gitea.renner.uno/konairius/cni-plugins:2026-09-04
image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
command:
- sh
- -c
@@ -154,40 +154,6 @@ spec:
app: multus
name: multus
spec:
# Upstream ships this DaemonSet with no priority class, so it runs at 0
# and can preempt nothing. Any event that makes the pod need a fresh
# scheduling decision — a rollout, a node restart, an eviction — then has
# to fit it onto a node that may since have filled up, and if it does not
# fit it simply stays Pending. The node has no multus after that, and
# nothing reports it: pods still start, they just never get their
# secondary attachment, and the annotation everything waits on never
# appears.
#
# Seen on a cold rebuild: two of four nodes sat at 99% memory requests
# when the DaemonSet rolled, both multus pods went Pending, and Rook's
# network canary hung in Init:0/1 waiting for a network-status annotation
# that could never come. The CephCluster stayed in Progressing, which is
# what infrastructure-ceph-cluster gates on, so the whole apps layer
# stopped behind a scheduling decision three layers away.
#
# node-critical, the same class cilium and cni-dhcp-daemon carry, because
# that is what this is: without multus a node cannot attach a secondary
# network, and only that node is affected. Nothing cluster-wide fails.
#
# This does outrank kube-apiserver, which Talos runs as a static pod at
# system-cluster-critical (2000000000, against 2000001000 here), and on a
# control plane that has run out of memory the kubelet will pick the API
# server as the pod to evict. That was observed twice, at 6GiB and at
# 8GiB.
#
# It is not a reason to demote this DaemonSet. Kubernetes puts the CNI
# above the control plane deliberately — a node without its CNI cannot run
# anything at all — and a class name is a statement about what the pod is,
# not a dial for picking a number. The eviction says the control plane is
# starved, and that is where it has to be fixed: enough allocatable
# memory, or kubeReserved/systemReserved so the static pods keep what they
# need. Demoting this to cluster-critical would only move the victim.
priorityClassName: system-node-critical
hostNetwork: true
hostPID: true
tolerations:
@@ -146,11 +146,6 @@ spec:
app: whereabouts
name: whereabouts
spec:
# Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
# secondary-network path, so a node without it cannot hand out an address
# on ceph-san and the attachment fails just as completely as if multus
# itself were missing. Upstream leaves the priority class unset.
priorityClassName: system-node-critical
hostNetwork: true
serviceAccountName: whereabouts
# Every node here is a control plane node, so the NoSchedule toleration
@@ -183,31 +183,3 @@ apiVersion: v1
kind: Namespace
metadata:
name: woodpecker
---
# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
# from `apps`. Distinct from `woodpecker`, which holds the service itself.
#
# Here rather than in apps/woodpecker for the reason spelled out above that
# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
# ../network-policies/, and infrastructure-crds runs before every app KS. With
# the namespace declared in the workload KS the order inverted, and a rebuild
# stopped with
#
# CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
# namespaces "woodpecker-ci" not found
#
# which held infrastructure-crds, and with it all twenty app Kustomizations
# that depend on it. The same reasoning was already written down for
# `woodpecker` and simply not carried across to its sibling.
#
# Image-build steps run a privileged docker/buildx plugin pod (see the server's
# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
# label the privileged step pods are rejected at admission. The namespace holds
# nothing but ephemeral CI step pods, so the exemption stays off apps/.
apiVersion: v1
kind: Namespace
metadata:
name: woodpecker-ci
labels:
app.kubernetes.io/name: woodpecker
pod-security.kubernetes.io/enforce: privileged
@@ -6,5 +6,6 @@ kind: Kustomization
# offsite lives in post-crds/velero-schedules.yaml; the staleness alert
# in apps/monitoring-rules/prometheusrule-netbird-backup.yaml.
resources:
- externalsecret.yaml
- pvc.yaml
- cronjob.yaml
@@ -1,4 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- daemonset.yaml
@@ -10,8 +10,33 @@
# name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
# (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
#
# The ExternalSecret that publishes the rotated token to Flux lives in
# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
# own it) so a sync failure leaves the last-good value in place; Flux's
# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
# provisioned by hand (see policies/rotator-gitea-flux.hcl).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: flux-system-git-auth
namespace: flux-system
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: flux-system
creationPolicy: Merge
data:
- secretKey: username
remoteRef:
key: infrastructure/flux-git
property: username
- secretKey: password
remoteRef:
key: infrastructure/flux-git
property: token
---
apiVersion: v1
kind: ServiceAccount
@@ -1,60 +0,0 @@
# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
# WITHOUT owning it, so a sync failure leaves the last-good value in place;
# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
#
# See creationPolicy below for why that is Orphan and not Merge — the
# distinction is what makes a recovery from the break-glass source possible.
#
# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
# rotation CronJob: external-secrets.io CRDs are installed by the ESO
# HelmRelease in the infrastructure Kustomization, so this manifest cannot
# pass server-side dry-run there on a fresh cluster.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: flux-system-git-auth
namespace: flux-system
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: flux-system
# Orphan, not Merge: create the Secret if it is absent, but never own it.
#
# Merge cannot create — it only writes into a Secret that already exists,
# on the assumption that `flux bootstrap` made one. That assumption breaks
# in exactly the situation this cluster is rebuilt from. The break-glass
# bootstrap pulls from the GitHub mirror and keeps its token in
# flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
# so this ExternalSecret has nothing to merge into — and reports Ready while
# producing no Secret at all. The operator then performs the documented
# handover back to Gitea and finds the GitRepository pointing at a Secret
# that does not exist.
#
# Owner would create it but attach an ownerReference, so deleting this
# object would take Flux's credential with it. Orphan creates and updates
# without ownership, which is what the paragraph at the top of this file
# actually wanted: a sync failure, or the loss of this object, leaves the
# last-good value in place.
#
# Safe on a live cluster: the Secret holds exactly the two keys written
# below (verified against production), so writing it whole is the same
# result as merging into it.
creationPolicy: Orphan
# Explicit rather than relying on the default, because it is the half that
# matters here: if the OpenBao entry ever disappears, keep serving the
# credential Flux already has rather than leaving it with none.
deletionPolicy: Retain
data:
- secretKey: username
remoteRef:
key: infrastructure/flux-git
property: username
- secretKey: password
remoteRef:
key: infrastructure/flux-git
property: token
@@ -10,25 +10,4 @@ resources:
# configured (2026-07-13): a store that can't turn Ready would block this
# KS's wait gate and freeze the dependsOn chain.
- openbao-clustersecretstore.yaml
# ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
# the infrastructure Kustomization, so these cannot pass its server-side
# dry-run on a fresh cluster — Flux rejects the whole apply set, including
# the ESO HelmRelease that would have installed the CRDs. They sat next to
# their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
# nothing had ever applied this layer against an empty API server before.
- crowdsec-bouncer-key-externalsecret.yaml
- external-dns-unifi-externalsecret.yaml
- flux-git-externalsecret.yaml
- netbird-backup-externalsecret.yaml
- netbird-ingress-externalsecret.yaml
- velero-externalsecret.yaml
- velero-hetzner-creds-externalsecret.yaml
# traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
# infrastructure. Traefik is deliberately absent from that Kustomization's
# healthChecks, so this KS may dry-run before the chart lands; it retries on
# retryInterval rather than deadlocking, because nothing here installs the
# CRDs it is waiting for.
- traefik-crowdsec-middleware.yaml
- traefik-rate-limit-middleware.yaml
- traefik-security-headers-middleware.yaml
- ../network-policies/
@@ -5,7 +5,7 @@ metadata:
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
allowUnsupported: false
dataDirHostPath: /var/lib/rook
# Defaults to false. With three OSDs and size 3 there is no spare copy, so
@@ -78,20 +78,7 @@ spec:
enabled: true
ssl: false
monitoring:
# false, and this is not a decision about metrics. `metricsDisabled` is the
# switch that turns metrics off; this one only decides whether Rook creates
# the ServiceMonitors — and creating them is fatal to the reconcile when it
# fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
# Rook's node-daemon reconcile aborted, the CephCluster never left
# status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
# waits on exactly that — with grafana-db → monitoring → those same CRDs
# behind it, closing the loop.
#
# The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
# only the scrape wiring moved, to apps/monitoring-rules/
# rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
# Both objects are reproduced there field for field off the live cluster.
enabled: false
enabled: true
# Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
# Multus. The attachment is san-network-attachment.yaml.
#
@@ -144,7 +131,7 @@ spec:
useAllDevices: false
# Two forms while the production drives are migrated:
#
# by-partlabel/r-osd-data the target. Talos names the drive in the
# by-partlabel/r-ceph-osd the target. Talos names the drive in the
# machine config (RawVolumeConfig ceph-osd), so
# this file needs to know nothing about models,
# serials or hostnames.
@@ -159,7 +146,7 @@ spec:
#
# The second half goes when those three are repartitioned, one node at a
# time. Until then it is what keeps the running OSDs claimed.
devicePathFilter: "^/dev/disk/by-(partlabel/r-osd-data|id/nvme-SAMSUNG_MZQLB960HAJR)"
devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
config:
osdsPerDevice: "1"
# Highest built-in class: kubelet evicts these last and the kernel gives
@@ -3,26 +3,33 @@
# without every Ceph daemon landing on host networking to get there. The
# alternative and why it was rejected is in cephcluster.yaml.
#
# host-device: the SAN NIC is dedicated to Ceph and has exactly one consumer
# per node — the OSD pod — so it is handed to that pod whole, not virtualised
# under it. The device keeps its identity (MAC, MTU) and moves back to the
# host when the pod goes away.
#
# `device` names the role, not hardware: every machine renames its own SAN
# interface to ceph-san at boot (udev rule in its Talos patch). No kernel
# interface name appears in this file.
#
# ISOLATION
#
# VLAN 49 carries Ceph-internal traffic and nothing else. No gateway and no
# routes: there is no router on this segment and there must not be one.
# whereabouts adds neither unless asked, so nothing here can leak onto or off
# the SAN.
# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
# are deliberate:
#
# The pool is the whole usable /29, .1 through .6 — headroom for a leaked
# allocation to sit until the reconciler collects it. One address per node is
# in use; a second OSD on a node would need a second SAN interface, which is a
# hardware change, not a config one.
# No gateway and no routes. There is no router on this segment and there
# must not be one. whereabouts adds neither unless asked, so nothing here
# can leak onto or off the SAN. This also means no `sbr` chaining is
# needed — unlike the v30iot attachments, there is no DHCP-supplied default
# route to confine, because there is no DHCP and no default route.
#
# The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
# migration ran, because the nodes still held .1-.3 for as long as Ceph
# needed host networking to reach this link — three addresses for three
# OSDs, no spare. The nodes have since given those up (the link is only a
# macvlan parent now and needs no address of its own), so the reservation
# has nothing left to avoid.
#
# Keeping the pool inside the /29 rather than moving it elsewhere on the
# segment is not cosmetic. There is no router here, so an address outside
# the /29 would be unreachable from anything still using one inside it —
# which is exactly what would break a future rolling change, where migrated
# and un-migrated OSDs have to keep talking to each other.
#
# Six addresses for three OSDs is the headroom the migration could not
# afford: room for a leaked allocation to sit until the reconciler collects
# it, and room to add a second OSD per node without touching this file.
#
# IPAM
#
@@ -30,18 +37,29 @@
# a server that must not exist here, and host-local keeps its state per node
# with no cluster-wide coordination, so two nodes would hand out the same
# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
#
# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
# it is stated so that a change to the parent does not silently halve
# replication throughput without anything in the tree recording the
# expectation.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: ceph-san
namespace: rook-ceph
spec:
# `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
# a busPath-derived predictable name (net.ifnames=1 via the Image Factory
# schematic) that is identical on k8s-1/2/3 and cannot shuffle across
# reboots.
config: |
{
"cniVersion": "0.3.1",
"name": "ceph-san",
"type": "host-device",
"device": "ceph-san",
"type": "macvlan",
"master": "enp2s0f1np1",
"mode": "bridge",
"mtu": 9000,
"ipam": {
"type": "whereabouts",
"range": "10.99.0.0/29",
@@ -38,7 +38,7 @@ spec:
serviceAccountName: rook-ceph-default
containers:
- name: rook-ceph-tools
image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
command:
- /bin/bash
- -c
@@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- helmrelease.yaml
- crowdsec-middleware.yaml
- security-headers-middleware.yaml
- rate-limit-middleware.yaml
@@ -1,9 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- snapshot-readonly-storageclass.yaml
- node-agent-config.yaml
- helmrelease.yaml
- servicemonitor.yaml
- podmonitor.yaml
- restore-test-rbac.yaml
- restore-test-cronjob.yaml
# Velero Schedule CRs live in kubernetes/infrastructure/post-crds/
-69
View File
@@ -1,69 +0,0 @@
# First machine of the throwaway cluster on Hetzner.
#
# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
# order bootstrap.sh attaches them.
machine:
install:
disk: /dev/sda
certSANs:
- hcloud-1
nodeLabels:
# Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN NIC to the role name at boot. Matched by PCI bus path
# rather than MAC: the address follows from the order bootstrap.sh attaches
# the networks, while the MAC is assigned by Hetzner at attach time and is
# not known when this file is written.
rules:
- SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
# In the generated schema rather than machine.network.hostname: naming the host
# in both is rejected at apply time. `auto: off` switches off the generated
# `auto: stable`, which is mutually exclusive with a static name.
apiVersion: v1alpha1
kind: HostnameConfig
auto: off
hostname: hcloud-1
---
# Every interface is stated, the public one included. Talos stops running its
# default DHCP operators for the whole machine as soon as one new-style network
# document exists, so an omitted interface gets no address.
apiVersion: v1alpha1
kind: LinkConfig
name: enp1s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp1s0
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp7s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp7s0
---
# Ceph cluster network, renamed to the role name by the udev rule in the
# machine section. The node holds no address here; the link exists for the OSD
# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
provisioning:
diskSelector:
# Two disks: the system disk and the attached volume. Matching on transport
# as the qemu patches do selects nothing here — a Hetzner volume arrives
# over virtio-scsi, and a selector matching zero disks is skipped silently.
match: '!system_disk'
# bootstrap.sh reads this size and creates the volume to match.
minSize: 50GiB
grow: true
-50
View File
@@ -1,50 +0,0 @@
# Second machine. See hcloud-1.yaml.
machine:
install:
disk: /dev/sda
certSANs:
- hcloud-2
nodeLabels:
storage.renner.uno/ceph-osd: "true"
udev:
# See hcloud-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
apiVersion: v1alpha1
kind: HostnameConfig
auto: off
hostname: hcloud-2
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp1s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp1s0
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp7s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp7s0
---
# See hcloud-1.yaml.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
provisioning:
diskSelector:
match: '!system_disk'
minSize: 50GiB
grow: true
-50
View File
@@ -1,50 +0,0 @@
# Third machine. See hcloud-1.yaml.
machine:
install:
disk: /dev/sda
certSANs:
- hcloud-3
nodeLabels:
storage.renner.uno/ceph-osd: "true"
udev:
# See hcloud-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
apiVersion: v1alpha1
kind: HostnameConfig
auto: off
hostname: hcloud-3
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp1s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp1s0
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp7s0
up: true
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp7s0
---
# See hcloud-1.yaml.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
up: true
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
provisioning:
diskSelector:
match: '!system_disk'
minSize: 50GiB
grow: true
+6 -14
View File
@@ -28,13 +28,6 @@ machine:
# this label instead of listing hostnames, so which boxes carry storage is
# stated once, here, by the machine that knows.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
# role name at boot, before the network stack opens links — a rule applied
# to a RUNNING node cannot rename (Resource busy) and takes effect at the
# next reboot.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -131,15 +124,14 @@ parent: enp2s0f0np0
up: true
---
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
# section above renames the port to ceph-san at boot, so shared manifests name
# the role and only this file knows the hardware.
# wire. OSD-to-OSD replication and nothing else.
#
# The node holds NO address here on purpose: the link is handed whole to the
# OSD pod (host-device attachment). Giving the node an address would put it on
# a segment reserved for Ceph-internal traffic.
# The node holds NO address here on purpose. Rook attaches the link to the OSD
# pods through a Multus macvlan, and a macvlan parent only needs to be up.
# Giving the node an address would put it on a segment reserved for
# Ceph-internal traffic.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
name: enp2s0f1np1
up: true
mtu: 9000
+3 -8
View File
@@ -23,10 +23,6 @@ machine:
topology.kubernetes.io/zone: saturn
# Holds a Ceph OSD drive — see k8s-1.yaml.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port to the role name at boot — see k8s-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -99,12 +95,11 @@ up: true
---
# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
# wire. OSD-to-OSD replication only, not routed off its own segment.
# Renamed to the role name by the udev rule above — see k8s-1.yaml.
#
# The node holds NO address here on purpose: the link is handed whole to the
# OSD pod (host-device attachment).
# The node holds NO address here on purpose. Rook attaches the link to the OSD
# pods through a Multus macvlan, and a macvlan parent only needs to be up.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
name: enp2s0f1np1
up: true
mtu: 9000
+2 -6
View File
@@ -19,10 +19,6 @@ machine:
topology.kubernetes.io/zone: neptune
# Holds a Ceph OSD drive — see k8s-1.yaml.
storage.renner.uno/ceph-osd: "true"
udev:
# Renames the SAN port to the role name at boot — see k8s-1.yaml.
rules:
- SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
cluster:
allowSchedulingOnControlPlanes: true
---
@@ -82,9 +78,9 @@ parent: enp2s0f0np0
up: true
---
# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
# here on purpose; the link is handed whole to the OSD pod (host-device).
# here on purpose; the link is only a macvlan parent for the OSD pods.
apiVersion: v1alpha1
kind: LinkConfig
name: ceph-san
name: enp2s0f1np1
up: true
mtu: 9000
-25
View File
@@ -1,25 +0,0 @@
# Site patch for the throwaway cluster on Hetzner Cloud.
#
# This site exists for one thing the other two cannot test: whether a node that
# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
# answers, so containerd never reaches the second endpoint in
# machine.registries.mirrors and the fallback stays unproven.
#
# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
# frames from it do not cross between servers — measured: same-node ARP
# resolves, cross-node does not. OSD replication therefore does not run here.
#
# What belongs in this file is what the cluster is. Interfaces, disks and names
# are machine properties and live in the node patches.
machine:
kubelet:
nodeIP:
# The segment bootstrap.sh provisions for node addresses. Unpinned, the
# kubelet can pick the public NIC.
validSubnets:
- 10.90.0.0/16
cluster:
# Three control planes that also carry workloads, which is the production
# shape. The qemu site cannot mirror it: that provisioner gives a second disk
# to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
allowSchedulingOnControlPlanes: true
-12
View File
@@ -43,15 +43,3 @@ apiVersion: v1alpha1
kind: HostnameConfig
auto: off
hostname: vm-1
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+2 -26
View File
@@ -21,7 +21,7 @@ auto: off
hostname: vm-2
---
# The OSD device, named by the machine that has it. Ceph then consumes
# /dev/disk/by-partlabel/r-osd-data and needs to know nothing about models,
# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
# serials or hostnames — the same CephCluster works here and on the MS-01s.
#
# The selector is a CEL expression, so each machine says whatever identifies its
@@ -29,33 +29,9 @@ hostname: vm-2
# real nodes it will name the Samsung.
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
name: ceph-osd
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# The Ceph cluster network link. Production gives it a second physical port;
# this machine has one NIC, so the segment is a VLAN on it.
#
# A dummy link would be simpler and worthless — no traffic between nodes, so
# Ceph would replicate over an interface going nowhere and the test would pass
# having proven nothing. A VLAN rides the provisioner's bridge that every VM is
# on, so the SAN carries real frames on its own tag.
#
# No MTU, unlike production's 9000: the qemu NIC is created with host_mtu=1500.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
# Load-bearing. Declaring any network document makes Talos configure this link
# explicitly, and the implicit DHCP that gives the node its address stops. A
# first attempt with only the VLANConfig took vm-4 off the network — NotReady,
# unreachable over the Talos API. Verified on a second throwaway node that this
# document keeps the parent addressed.
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+1 -13
View File
@@ -15,21 +15,9 @@ hostname: vm-3
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
name: ceph-osd
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
+1 -13
View File
@@ -19,21 +19,9 @@ hostname: vm-4
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
name: ceph-osd
provisioning:
diskSelector:
match: disk.transport == "virtio" && !system_disk
minSize: 5GiB
grow: true
---
# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
# DHCPv4Config below is load-bearing.
apiVersion: v1alpha1
kind: VLANConfig
name: ceph-san
parent: enp0s2
vlanID: 30
---
apiVersion: v1alpha1
kind: DHCPv4Config
name: enp0s2
-128
View File
@@ -1,128 +0,0 @@
# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
#
# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
# one-shot test, and a Kustomization would recreate it after every completion.
# It is applied by the vmtest pipeline, or by hand for a single run.
#
# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
# a single run left under 1G free and two overlapping runs OOM-killed the
# session. The nodes have ~94G each with about 70G unused.
#
# The whole script runs here, flux and OpenBao included, so that what a rebuild
# rehearses is what a real recovery would execute. The unseal key is not baked
# in and not stored as a Secret: the operator streams all five secrets into an
# in-memory emptyDir at the moment the run asks for them, and they die with the
# Job without ever reaching etcd or a backup of it.
apiVersion: v1
kind: Namespace
metadata:
name: vmtest
labels:
# The pod is privileged; say so rather than have the admission policy
# reject it for reasons that look unrelated.
pod-security.kubernetes.io/enforce: privileged
---
apiVersion: batch/v1
kind: Job
metadata:
name: vmtest
namespace: vmtest
spec:
backoffLimit: 0 # a failed rebuild test is a result, not something to retry
ttlSecondsAfterFinished: 86400
template:
metadata:
labels:
app.kubernetes.io/name: vmtest
spec:
restartPolicy: Never
# Nothing here should be able to reach the cluster it runs in. The phase
# points KUBECONFIG at its own throwaway cluster, and without a mounted
# token a stray kubectl cannot fall back to this one.
automountServiceAccountToken: false
containers:
- name: vmtest
# Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
# serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
# three builds old, which predated OVMF, and died with "the required
# flash image was not found" before the first VM booted. The tag is
# carried alongside for readability — containerd resolves the digest.
image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
# Privileged for /dev/kvm and for the bridges and taps the provisioner
# builds. It needs no hostNetwork: the provisioner's network lives in
# this pod's own namespace, and the script that talks to 10.5.0.x runs
# in the same pod.
securityContext:
privileged: true
# A hard ceiling, because the failure mode is known: an unbounded run
# is what OOM-killed the workstation. 40G is the VMs (16G control
# plane + 3x8G workers), the rest is qemu overhead and the tooling.
# The workers carry an OSD, a mon and a mgr each; at 4G they ran out
# of room for the CNI DaemonSets themselves. The control plane went
# 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
# memory and the kubelet evicted kube-apiserver to admit a
# node-critical pod — bootstrap.sh carries the full account next to
# the sizing flags. The hosts have ~95GiB each, so this fits.
resources:
requests:
cpu: "6"
memory: 42Gi
limits:
memory: 48Gi
env:
# talosctl caches the image and the CNI bundle under $HOME; point it
# at the scratch volume rather than the container layer.
- name: HOME
value: /work
# The VM disks. Short on purpose — the provisioner puts a qemu
# monitor socket under it and sun_path caps the path at 108 bytes.
- name: VM_STATE
value: /work/vm
# Which branch to test. master unless a run is checking a change.
- name: BRANCH
value: master
# Secrets are streamed in here at the moment the run needs them,
# never baked into the image and never stored as a Secret. They live
# in this emptyDir for the length of the Job and go with it — so
# they do not reach etcd, or any backup of it.
- name: OPENBAO_SECRETS_DIR
value: /secrets
command: ["/bin/bash", "-ec"]
args:
- |
# From the GitHub mirror, not in-cluster Gitea. A rebuild test
# that clones from the cluster it is rehearsing the loss of
# cannot start when it matters — and a run already died here on
# resolving gitea.renner.uno.
git clone --depth 1 --branch "$BRANCH" \
https://github.com/konairius/Homelab.git /work/homelab
exec /work/homelab/kubernetes/bootstrap.sh vmtest
volumeMounts:
- name: work
mountPath: /work
- name: secrets
mountPath: /secrets
- name: dev-kvm
mountPath: /dev/kvm
volumes:
# The VM disks and the provisioner state. emptyDir on the node's disk
# rather than a PVC: it is scratch that dies with the Job, and putting it
# on Ceph would have the test's own writes compete with the storage it is
# meant to leave alone.
- name: work
emptyDir:
# The VM images are sparse, so this bounds what is actually written,
# not the 232GiB the geometry declares. 60Gi was already half spent
# by four 10GiB systems disks; the node carries this comfortably with
# ~810GB free on the filesystem behind it.
sizeLimit: 150Gi
# In memory, so the material never touches a node's disk.
- name: secrets
emptyDir:
medium: Memory
sizeLimit: 1Mi
- name: dev-kvm
hostPath:
path: /dev/kvm
type: CharDevice
+1 -12
View File
@@ -13,18 +13,7 @@ set -euo pipefail
ENDPOINT="https://fsn1.your-objectstorage.com"
BUCKET="renner-uno-k8s-secrets"
PREFIX="cluster-access"
# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
# The default is unchanged and deliberately so: "secrets" resolves to working
# credentials and this script has always used it. Both "secrets" and "hetzner"
# exist in ~/.aws/credentials on the workstation.
#
# The defect is only the assignment. Forcing the value defeated the env-based
# path this file's own header offers, and it also overrode a caller who had
# already selected a profile — which is how a 403 from HeadObject can appear
# while the credentials the caller intended were perfectly good. That error
# reads as a bucket permissions problem and sends you looking in the wrong
# place.
export AWS_PROFILE="${AWS_PROFILE:-secrets}"
export AWS_PROFILE="secrets"
command -v aws >/dev/null || { echo "aws-cli not found"; exit 1; }