Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 ac16e53098 fix(flux): four cold-rebuild ordering deadlocks
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.

infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.

home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.

woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-06 20:23:05 +02:00
7 changed files with 61 additions and 40 deletions
-15
View File
@@ -1,18 +1,3 @@
# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
# isolated from the apps namespace.
apiVersion: v1
kind: Namespace
metadata:
name: woodpecker-ci
labels:
app.kubernetes.io/name: woodpecker
# Image-build steps run a privileged docker/buildx plugin pod (see the
# server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
# without this the privileged step pods are rejected at admission. The
# namespace is dedicated to ephemeral CI step pods, so scoping the
# exemption here keeps it off apps/.
pod-security.kubernetes.io/enforce: privileged
---
apiVersion: v1
kind: ServiceAccount
metadata:
@@ -28,6 +28,19 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
# CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
# kind is unknown when this path is first applied, and a server-side
# dry-run rejects the ENTIRE set over the one object it cannot recognise:
#
# ObjectStore/home-assistant/ha-pg-store dry-run failed:
# no matches for kind "ObjectStore" in version ...
#
# Every other ObjectStore user — authentik, immich, paperless, trmnl,
# woodpecker — already declares it. This was the only one that did not, and
# it took speech-to-phrase, zone-configurator and matter-server down with
# it, since all three wait on home-assistant.
- name: infrastructure-cnpg-barman
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -122,8 +122,20 @@ spec:
path: ./kubernetes/infrastructure/rook-ceph-cluster
prune: true
wait: true
dependsOn:
- name: infrastructure
# Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
# existing cluster could ever hit and no new one could ever escape:
# infrastructure is Ready only once its openbao healthCheck passes, openbao-0
# only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
# defined in this very Kustomization. A cold rebuild stopped here for good,
# with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
#
# The edge was never needed. infrastructure runs wait: false, so it applies
# the rook-ceph namespace and the operator immediately and only its readiness
# waits on the healthChecks — everything this path needs exists long before
# infrastructure reports Ready. On an empty API server this Kustomization
# retries on retryInterval until the operator installs its CRDs, the same
# self-healing shape post-crds already relies on; on a live cluster the CRDs
# are always present and nothing changes.
# kstatus doesn't know how to read Rook CR readiness, so without these
# this Kustomization reports Ready as soon as the manifests apply — long
# before OSDs/mons/MDS are actually serving I/O. Apps depending on this
@@ -9,26 +9,9 @@ spec:
---
# v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
# (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
# filter pins us to the bare three-component release.
#
# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
# upgrade tied to the Rook operator version. That coupling had already come
# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
# the mons with a client a whole major ahead of them.
#
# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
# Nothing broke only because the four CSI users predate the cipher change and
# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
# because a fresh cluster has no legacy keys at all.
#
# So the cutover happens now, with the version the client already is. Rook
# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
# upstream reports data corruption there when read affinity is enabled.
# filter pins us to the bare three-component release. Range stays at
# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
# upgrade tied to the Rook operator version.
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
@@ -42,4 +25,4 @@ spec:
extract: '$ver'
policy:
semver:
range: ">=20.2.1 <21.0.0"
range: "19.x"
@@ -190,3 +190,31 @@ apiVersion: v1
kind: Namespace
metadata:
name: ai
---
# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
# from `apps`. Distinct from `woodpecker`, which holds the service itself.
#
# Here rather than in apps/woodpecker for the reason spelled out above that
# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
# ../network-policies/, and infrastructure-crds runs before every app KS. With
# the namespace declared in the workload KS the order inverted, and a rebuild
# stopped with
#
# CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
# namespaces "woodpecker-ci" not found
#
# which held infrastructure-crds, and with it all twenty app Kustomizations
# that depend on it. The same reasoning was already written down for
# `woodpecker` and simply not carried across to its sibling.
#
# Image-build steps run a privileged docker/buildx plugin pod (see the server's
# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
# label the privileged step pods are rejected at admission. The namespace holds
# nothing but ephemeral CI step pods, so the exemption stays off apps/.
apiVersion: v1
kind: Namespace
metadata:
name: woodpecker-ci
labels:
app.kubernetes.io/name: woodpecker
pod-security.kubernetes.io/enforce: privileged
@@ -5,7 +5,7 @@ metadata:
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
allowUnsupported: false
dataDirHostPath: /var/lib/rook
# Defaults to false. With three OSDs and size 3 there is no spare copy, so
@@ -38,7 +38,7 @@ spec:
serviceAccountName: rook-ceph-default
containers:
- name: rook-ceph-tools
image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
command:
- /bin/bash
- -c