The SAN is a dedicated interface with exactly one consumer per node, so
the attachment becomes host-device: the OSD pod gets the NIC whole, with
its real MAC and MTU, instead of a macvlan child on top of it. The
device field names the role (ceph-san, the boot-time rename), so the NAD
still contains no hardware. The earlier "failed to move link" was
measured against the bridge, which cannot move — not against the NIC;
the move against the renamed NIC is untested until the next throwaway
run.
The Hetzner provisioner derives its network ranges instead of inventing
them: the cluster segment from the subnet the site patch pins node
addresses to, the storage segment from the SAN pool the attachment
allocates from. Invented ranges are how this week a fabric ended up
knowing 10.91.x while the workloads spoke 10.99.x — the same
two-statements-drift as the 10GiB/50GB disk, which already has this
rule.
Known, deliberately unresolved: the fabric reserves the first address of
a network range as its gateway, which for the storage net is the SAN
pool's first address. It matters only if cross-node SAN traffic on that
site is ever attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A machine-config udev rule renames the port to ceph-san before the
network stack opens links, so the attachment resolves a real kernel name
and the bridge indirection goes away entirely.
Measured on a plain two-NIC qemu VM booting the metal ISO built from
this cluster's own schematic, Talos v1.13.9: the rule renamed the
secondary NIC at boot, and a LinkConfig naming ceph-san found and
brought up the renamed link. #8018's "Resource busy" is the RUNTIME
case — rules applied to a running node cannot rename an open link, which
is also why rolling this out takes a reboot per node.
Match keys are machine-local, as the design demands: burned-in MAC on
the MS-01s, PCI bus path on Hetzner where the MAC is assigned at attach
time. The qemu VMs need no rule at all — their SAN is a VLAN child and
VLANConfig names it directly.
NOT rolled out to production. The rule lands inert on a running node and
takes effect at the next reboot; until then the renamed LinkConfig
matches nothing and the running link keeps its state. Per-node reboot
with drain, Hetzner first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lost with the revert of the udev experiment; unrelated to it. The warning
tested whether the sweep function is defined, which it always is, so the
qemu phase printed it too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Job cloned from in-cluster Gitea, so the rehearsal depended on the
thing it rehearses losing. A run died on resolving gitea.renner.uno
before the first VM booted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The warning tested whether the sweep function is defined, which it always
is, so the qemu phase printed it too.
Also adds a udev rule to vm-2 only, to measure whether renaming the SAN
link gives the CNI a resolvable name and removes the need for the bridge.
vm-3 and vm-4 are identical without it, so the comparison is local to one
node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comments state what holds, not how I got there. The bridge and
provisioning comments had grown into accounts of what failed first and
why I chose what I chose — commit-message material, not config.
Roughly a third of the comment lines added today are gone; the facts a
reader needs to change the file safely are kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A link alias is the shorter answer and does not reach the CNI. Talos
writes it to IFLA_IFALIAS and deliberately not as an altname — the
maintainer position is that udevd owns altnames — so a plugin resolving
its master by kernel name fails with "link not found".
siderolabs/talos#12604 is the same setup, Multus macvlan against an
aliased link, and is still open. Measured here too: a node booted with
LinkAliasConfig showed alias=ceph-san with altNames unchanged, and the
attachment failed with that error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The billing warning fires from the exit handler whenever the sweep
function is defined, so `hcloudclean` printed it too — directly after
deleting every server it could find.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phase ended by writing node addresses to a cache file and exiting, so
the cluster build had to be a second invocation that read it. Nothing
needed that: the talos phase already takes NODES, WORKERS, K8S_API_HOST
and SITE, and the comment on those four says they are the whole of what a
test site may change.
What was wrong with the old handover was not the handover — it was
FLUX_SOURCE, HCLOUD_FULL and a ConfigMap riding along with it. Those are
gone. The four remain, as vmtest uses them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The attachment named enp2s0f1np1, a kernel interface, in configuration
every cluster shares. That forced every other site to produce an
interface with that name: the qemu VMs built a VLAN and called it that,
and its own comment admitted "the name looks physical and is not". A site
whose hardware could not be made to match had no way in at all.
Each machine now declares a ceph-san bridge over whatever its own SAN
hardware is — the second X710 port on k8s-1/2/3, enp8s0 on the Hetzner
servers, a VLAN on the qemu VMs — and the attachment names the bridge.
The interface name appears only in the file that describes the machine
that has it, which is the shape the OSD disks already use: the machine
labels itself, the CephCluster selects by label.
Aliases were the obvious route and do not work: LinkAliasConfigController
sets IFLA_IFALIAS, which is descriptive, so nothing resolves an interface
by it. A bridge is a real link with a name we choose.
The bridge repeats mtu 9000 on the production nodes. A bridge takes the
smallest MTU of its ports, but it is created before the port is enslaved,
so leaving it unset pins it at 1500 and silently caps the segment.
NOT rolled out to production by this commit. Moving a live macvlan master
onto a bridge is a per-node, drain-first change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowSchedulingOnControlPlanes and kubelet.nodeIP.validSubnets were in
every node patch. They are properties of the cluster: the first is what
shape the cluster has, the second is which segment the cluster puts node
addresses on. Both move to the site patch, where site-qemu and
site-homelab already pin their node subnets.
Interfaces, install disk, hostname, certSANs, labels and the OSD volume
stay per node. Three machines having the same NICs does not make the NICs
a cluster property — it makes them alike.
The site patch also said a cloud server has one address and therefore
nothing to pin, and that this site has no second network and no second
disk. All three stopped being true when provisioning started creating
them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enp1s0 is the public NIC, enp7s0 and enp8s0 are the private networks in
the order bootstrap.sh attaches them. Read off all three servers with
talosctl in maintenance mode and matched against the MAC addresses
Hetzner reported, rather than assumed from attachment order.
Every interface is stated, the public one included. Talos stops running
its default DHCP operators for the whole machine as soon as one new-style
network document exists, so an omitted interface gets no address at all
and the node is reachable only from the console.
kubelet.nodeIP.validSubnets pins the node address to the cluster network.
Production keeps its node IP on the management LAN, and it takes etcd,
the API and pod east-west with it; without this the kubelet can pick the
public address and every traffic class rides the internet-facing NIC.
enp8s0 carries no address, as the second X710 port does in production.
No jumbo MTU on it: 9000 is what a switched segment carries, and a
Hetzner network is not one.
Also fixed in provisioning, both found by running it: the exit trap tore
down the machines the moment the phase succeeded, and the pre-run sweep
deleted the servers the adopt path exists to reuse. Networks get one /16
each, since a server cannot be attached to two overlapping ranges, and
an existing network is reused only if its range is the one this run
wants. Server and network lookups read JSON instead of Go templates,
which compared a struct against an ID and silently matched nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phase booked servers and then fell through into the Talos and Flux
phases as SITE=hcloud, which put site-specific branches into everything
downstream. It now provisions and exits; the cluster is built by the
ordinary phases, which carry no knowledge of this provider. The isolation
from the production cluster is printed as the caller commands rather than
set here, because both halves are needed and only the caller outlives
this phase.
Networks: two segments, matching where the traffic classes are today
rather than where they are going — the node address carries management,
cluster-internal and external alike, and Ceph replication has its own.
Attachment order is load-bearing and stated once; the run records each
segment address and MAC so the machine config can be written from what
the platform actually did instead of an assumed interface order.
Shape comes from the machine config. hcloud_volumes_for_node reads the
RawVolumeConfig out of the node patch, so the disk a node gets is the
disk Talos was told about. The size was previously stated in both places
and they disagreed: 10GiB against a 50GB volume.
Removed with it: the cluster-settings-local ConfigMap and the
CEPH_NETWORK_PROVIDER substitution, the break-glass source in
gotk-sync.yaml, the substitute-disabled annotation on the toolbox, and
the commentary I had written into the configuration files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GITHUB_REPO is "Homelab", so `flux bootstrap` writes
github.com/konairius/Homelab.git. The previous commit wrote it lowercase,
which GitHub resolves to the same repository — so nothing would have
looked broken. Flux compares strings, not repositories: the file would
have differed from what bootstrap generates on every run, been rewritten
and committed to the mirror, and the next push to Gitea would have
force-pushed that commit away again. Exactly the loop the previous commit
set out to close.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things, both about being able to iterate without paying for it.
Reuse: capacity is the binding constraint. Every node of the last run
waited through resource_unavailable before it was placed, so a machine
already in hand is worth more than a clean slate. Servers carrying this
test label are now adopted instead of created, and rebuilt from the Talos
snapshot to get back to maintenance mode — talosctl reset cannot be used
for that, because the previous run PKI is gone with its temp directory.
The OSD volume is still discarded and recreated: a reused one carries a
bluestore signature that Rook adopts, which would quietly turn a cold
rebuild into a restart test. Volumes are cheap, servers are not.
Sync source: gotk-sync.yaml now names the break-glass source it is
actually bootstrapped with on this branch. It pointed at in-cluster
Gitea, so `flux bootstrap github` had to rewrite and commit it on every
run — and that commit lives only on the GitHub mirror. The next push to
Gitea force-pushed the mirror, destroyed it, and Flux reconciled itself
back onto a Gitea URL the throwaway cluster has no secret for. A push
during a run then breaks the rebuild, which is exactly what this test
must not do. With git already matching what bootstrap generates, the
bootstrap commit is a no-op and there is nothing for a push to lose.
No comments added to gotk-sync.yaml on purpose: bootstrap regenerates
that file wholesale, so anything extra would diverge, commit, and start
the same loop again.
This file must not reach master — it already carries branch:
vmtest-flux, and now a GitHub URL as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The empty override rendered as YAML null, and Rook rejected the whole
CephCluster: "spec.network.provider: Unsupported value: null".
Quoting the placeholder in the manifest does not survive the pipeline.
Kustomize builds before Flux substitutes, and it re-emits `${VAR}` as a
bare scalar because that string needs no quotes. Substituting an empty
value into a bare scalar yields `provider:`, which is null rather than
the empty string the CRD requires.
So the ConfigMap values carry their own quote characters and the
manifest field is deliberately unquoted — quoting both would double them
into `""multus""`. Verified end to end with `flux build kustomization`
piped through `flux envsubst`, in both the multus and the empty case.
That full path is the point: the previous commit checked envsubst on its
own, which is why this was missed — kustomize runs first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An empty override can never beat a substitution default. Flux follows
POSIX parameter expansion and cannot distinguish "set to empty" from
"unset", so ${CEPH_NETWORK_PROVIDER:=multus} handed an empty value
yields multus — and ${VAR=multus} without the colon behaves identically.
Both checked with `flux envsubst`.
That mattered because the empty string is the only provider value
meaning "pod network", so the one site that needs it had no way to ask
for it. The site override was therefore inert: Hetzner would have been
handed the multus SAN it has no layer 2 to build.
Production values move into cluster-settings and the manifest carries a
bare ${VAR}, so the override wins by there being nothing to fall back to.
The cost is that deleting those keys silently moves production Ceph off
the SAN; the warning lives with them. A missing cluster-settings is not
a risk — it is non-optional in substituteFrom, so Flux fails and retries
rather than applying half-substituted values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Indented one level too deep they are not a mis-sized volume but a fatal
config: Talos rejects the whole document with "unknown keys found during
decoding", so hcloud-1 refused every apply-config and the node took no
configuration at all. hcloud-2 and hcloud-3 were unaffected — the drift
came in with the comments explaining the selector.
Also corrects the volume size in that comment, which still said 20GB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things stood between the Hetzner rehearsal and a cluster that comes up, and
they had to be solved together rather than one run at a time.
DISKS. The nodes had none beyond their system disk, so ceph-block existed as a
StorageClass with nothing behind it and OpenBao's PVC sat Pending. Each node now
gets a 50GB volume, attached raw — not formatted, not automounted, because Rook
needs the bare device and a filesystem makes Ceph skip the disk as in use — and
carries the storage.renner.uno/ceph-osd label the CephCluster selects on. That
label is why this works at all: OSD nodes are chosen by label rather than
hostname, so a cluster built elsewhere gets OSDs.
50GB, not the 10GB minimum I first wrote. The reference is the qemu site, which
gives each OSD worker 24GiB and gets twelve app Kustomizations up; production's
2.2TB is years of real data that neither site restores. 10GB would leave ~10GB
usable across three replicas — less than OpenBao's two PVCs ask for.
The disk selector drops the `disk.transport == "virtio"` clause the qemu
patches carry. A Hetzner volume arrives over virtio-scsi, and a selector
matching zero disks is skipped silently — the OSD would simply never appear.
NETWORK. The CephCluster demands multus with the ceph-san macvlan attachment,
whose master is the kernel link enp2s0f1np1. That cannot exist on Hetzner:
their networks are layer 3 only — "the servers are not directly connected with
each other, instead they are connected via a gateway" — so there is no layer 2
for macvlan and no VLAN to manufacture the name.
Rather than fork the manifests, provider and selector become substituted values
defaulting to what production runs. The override arrives in a
cluster-settings-local ConfigMap that exists only on the throwaway cluster,
listed second and `optional: true` so production is untouched. An empty provider
is a valid CRD value and puts Ceph on the pod network.
That required enabling postBuild substitution on a path that never had it, and
toolbox.yaml is full of shell ${MON_CONFIG}-style variables that Flux would
replace with empty strings. It carries
kustomize.toolkit.fluxcd.io/substitute: disabled for exactly that reason —
without it the toolbox would write an empty ceph.conf and never find a monitor.
The break-glass bootstrap died on
GET https://api.github.com/repos/konairius/Homelab: 401 Bad credentials
after the cluster was already built. The token entry read
${GITHUB_TOKEN:-${GITEA_TOKEN:-}} — whichever variable happened to be set. On
this workstation GITEA_TOKEN is exported and GITHUB_TOKEN is not, so a run with
FLUX_SOURCE=github handed a Gitea token to the GitHub API.
The comment directly above it already said "which token depends on FLUX_SOURCE,
and that is not a detail". The code did not read it. git_token_for_source now
chooses by the forge that will actually be authenticated against, with
`gh auth token` as the GitHub fallback — the credential the operator already
has here, and the one the vmtest secret feeder has always used.
Also removes the two `2>/dev/null` I had written today.
- gh is called plainly. Swallowing its stderr would turn a failed login into
an empty string, and require_secrets would report "came back empty" for a
credential that exists but could not be read. Its non-zero exit propagates
instead, which is what makes the reason visible.
- the firewall address lookup keeps both curl attempts' stderr and prints it
if BOTH families fail. One failing is ordinary; both failing is a fault
whose cause is the diagnosis.
Nine `2>/dev/null` remain in this file, all older than today. Left alone here
rather than swept up mid-test, but they are the same hazard: each one makes a
zero indistinguishable from a failure.
Two faults, one found by the provider and one by me.
RETRY ON PLACEMENT. The first cx43 run died on
hcloud: error during placement (resource_unavailable, ...)
which is Hetzner being momentarily out of that type in that location, not a
wrong request — probing the same type in the same location minutes later
created a server fine. For a test meant to run on a schedule that distinction
is the whole difference between a real failure and a red run nobody needs to
look at. Placement errors are now retried for ten minutes; anything else still
fails immediately, with the provider's message rather than a bare exit code.
KEEP MUST NOT SUPPRESS THE SWEEP. cleanup_hcloudtest has two callers: the sweep
BEFORE a run, which exists so a previous crash cannot leak servers, and the
teardown after it. I put the HCLOUD_KEEP check inside the function, which
disabled both — so a kept cluster would survive into the next run, and that run
would then fail creating servers whose names were already taken. The check
moves to on_exit, the only caller that means "after".
Also drops the talosconfig hint from the keep message: it printed "?" because
the variable is not in scope there, and a placeholder is worse than saying
nothing.
cx23 has 4 GB, 3.2 GB of it allocatable. The three nodes sat at 59-78% of that
with the infrastructure skeleton alone — no Ceph OSDs, no OpenBao running, not
one app — with limits already at 134-207% of allocatable and both
kube-controller-manager and kube-scheduler restarted twice. There was no room
for the thing the cluster is built to run.
I chose it as the successor to the cx22 the Talos Hetzner guide names, which is
a minimal example rather than a sizing statement for this workload. That is the
same mistake I had already made and corrected today on the local control plane,
which went 6GiB → 8GiB → 16GiB before it stopped evicting its own API server.
cx43 instead: 16 GB, 8 vCPU. The local rehearsal is the reference — a 16GiB
control plane and three 8GiB workers, 40GiB in total — and three nodes here
carry that same workload plus a control plane each, so 48GiB is the comparable
figure.
Constraints that narrowed it: cpx11/cpx21 are US-only, and cax* are ARM, which
would not match the amd64 schematic this cluster installs.
require_secrets picks its branch on whether OPENBAO_SECRETS_DIR is set — fetch
from pass when it is empty, wait for streamed-in files when it is not. The
function sets that variable itself, so the test was reading state it had
already changed.
The subset support added for the talos phase made that visible. hcloudtest asks
for ghcr-pull and hcloud-token before anything else; from then on the variable
was populated, and the full call for the flux phase took the wait branch. On a
workstation with pass right there, the run printed streaming instructions
naming a `kubectl -n vmtest exec` Job that does not exist, and stopped — while
three Hetzner nodes sat in maintenance because the wait comes before the talos
phase applies any config.
SECRETS_SELF_FETCH records which world the run is in, decided once on first
entry: an empty directory means pass is available, an inherited one means the
in-cluster Job. Later calls follow that decision rather than re-deriving it.
The fetch loop also skips what a previous call already collected. A subset call
fetches only what it asked for, so the full call after it fills the gaps — and
re-running `pass show` for a key already on disk is how a locked store prompts
twice and an unattended run hangs.
The three-node run provisioned a cluster, watched it form, and destroyed it
without anything having run on it. Provisioning is the first step of the task,
not the task.
HCLOUD_FULL=1 continues into flux and wait-apps instead of stopping after
Talos, so the run reaches whatever actually blocks it rather than the blocker
being guessed at in advance. It needs all six secrets, including the OpenBao
unseal key — which works from a workstation, where require_secrets reads pass,
and is exactly what does not work unattended. That is why it is a switch and
not the default.
HCLOUD_KEEP=1 leaves the servers up. The teardown fires the moment a run ends,
including when it ends badly, which is precisely when the machines are worth
looking at — the same reasoning as VMTEST_KEEP, with the difference that these
bill by the hour and nothing else reclaims them. The message says so and names
`bootstrap.sh hcloudclean`.
FLUX_SOURCE is forced to github in this phase, not defaulted: gitea.renner.uno
has no external ingress, so from a Hetzner node it does not answer at all. The
break-glass source is the only one this site has — which is the position a real
recovery is in, and the reason the test is worth running from outside.
The three-node Hetzner run printed
==> Waiting for K8s nodes...
Talos phase complete.
with nothing between, and a kubectl I ran seconds earlier had listed no nodes at
all. Whether the wait saw three Ready nodes in that gap or matched zero against
zero cannot be decided from the log, because the step is silent on success.
That is the defect: a check nobody can audit is not much of a check.
It now prints the count it accepted, which is the same number its pass
condition uses.
And it refuses to run with an empty expectation. `expected` comes from
ALL_NODES, handed over by whichever phase provisioned the machines; if that
array were ever empty, `0 -eq 0` satisfies the loop's exit condition on the
first pass and the script reports a complete Talos phase for a cluster with no
nodes in it. An empty ALL_NODES is a bug in the provisioning phase, not a
cluster that is ready, so it fails loudly instead.
Neither change alters what a correct run does. Both change what a broken one
looks like.
One machine is a node. etcd has no quorum to lose, nothing schedules anywhere
else, and a rebuild that only ever ran on a single box proves nothing about the
one that has three. The phase now provisions three control planes, all carrying
workloads — which is the production shape: k8s-1/2/3 are all control planes and
all run apps.
The qemu site cannot mirror that. Its provisioner hands a second disk to
workers only, so a Ceph OSD forces 1 control plane and 3 workers there. Hetzner
attaches volumes to any server, so this site can match production instead of
the emulator. allowSchedulingOnControlPlanes moves to the site patch, where a
cluster-wide property belongs, and the node patches carry only what is true of
each machine.
Also opens the firewall between the nodes, which one node did not need and
three do: etcd peers on 2380, the apiserver, kubelet on 10250 and Cilium's
VXLAN all run between them, and with only the this-host rules etcd never forms
a quorum. The rule is added after creation because the addresses do not exist
before it, and it admits all ports between the members rather than an
enumerated list — guessing the full port set of Talos plus Cilium is how a
rebuild fails on the one port nobody wrote down. The outside still sees 50000
and 6443, and only from the host running this.
Not yet run at three nodes. The single-node run before this reached "Talos
phase complete" with the API answering and Cilium installed, and left nothing
behind.
26092ba5 changed the default to "hetzner" and justified it with "no such
profile exists on the workstation". That is false. Another session ran this
script with the original hardcoded "secrets" and it succeeded, and both
"secrets" and "hetzner" are present in ~/.aws/credentials here.
So the default goes back to "secrets" — unchanged behaviour for everyone it
already worked for, and the owner's long-standing choice. Changing it on the
strength of one failed run in one environment risked breaking the sessions
where it works, which is exactly what the previous commit did.
What was genuinely broken stays fixed: the assignment was unconditional, so it
overrode a caller who had already selected a profile, and defeated the
env-credential path this file's own header documents. `${AWS_PROFILE:-secrets}`
keeps the default and restores the override.
My own 403 therefore had some other cause in this session's environment, which
I have not identified and am not going to guess at in a commit message.
import-cluster-access.sh hardcoded AWS_PROFILE="secrets". No such profile
exists on the workstation; the one holding the Hetzner Object Storage keypair
is "hetzner". The failure this produced is worse than a missing profile
usually is:
fatal error: An error occurred (403) when calling the HeadObject
operation: Forbidden
which reads as a permissions problem on the bucket, and sends you looking at
bucket policy rather than at a profile that was never there.
`${AWS_PROFILE:-hetzner}` rather than a second hardcoded name, so a caller can
override it — the header has always documented env credentials as an equally
valid path, and the unconditional export silently defeated that.
Found while restoring ~/.kube/config and ~/.talos/config after the hcloudtest
phase overwrote them (fixed in 174e0862). Verified end to end: both files
restored, and the real cluster answers with all three control planes.
Two faults the first real runs exposed, one of them damaging.
ISOLATION. The phase exported TALOSCONFIG and KUBECONFIG to a temp directory
and stopped there. But TALOSCONFIG_PATH and KUBECONFIG_PATH are derived at the
top of this script, from those variables, before any phase runs — so exporting
them afterwards moves nothing, and the talos phase writes to the _PATH ones.
The throwaway cluster's credentials were written over ~/.kube/config and
~/.talos/config, and another session working on the real cluster lost its view
of it mid-task. The vmtest phase has always set both halves; this one now does
too.
BOOTSTRAP RETRY. `talosctl bootstrap` was called once, and anything that was
not "already exists" was reported and then ignored. "bootstrap is not available
yet" is a moment, not a fault: the preceding wait only establishes that the node
answers with its hostname, while etcd accepting the call comes later and how
much later depends on the machine. On qemu the gap is short enough to miss. On
a Hetzner cloud server it is not — the single attempt failed with
FailedPrecondition, the run carried on, and then sat out its full timeout
waiting for an apiserver that could never start because etcd had never been
bootstrapped. It now retries for ten minutes, treats "already exists" as
success, and fails loudly if the call never lands.
Also adds a check that the generated registry patch is a readable file, after
a bug where it was not: require_secrets prints "All secrets present." on
stdout, and calling it inside registry_auth_patch made that line part of the
captured path. talosctl reported `open All secrets present. | /tmp/…: no such
file` sixty times while the node sat in maintenance. The call moved out of the
substitution; a function whose value is its stdout must be the only thing
writing there.
Building the snapshot exercised the API for real and found the server type
wrong. The Talos Hetzner guide names cx22 and `hcloud server create` rejects it
outright — the type is gone. In the German locations the current shared x86
types are cpx12 and cx23; cpx11/cpx21 are US-only, and cax* are ARM, which
would not match the amd64 schematic this cluster installs. Default is now cx23.
The snapshot the phase requires exists: image 427962556, labelled os=talos,
built from factory.talos.dev with this cluster's own schematic and version read
out of cluster.yaml rather than pinned separately —
factory.talos.dev/image/0f6fb8bd.../v1.13.9/hcloud-amd64.raw.xz
written to /dev/sda in the rescue system, which produced the expected four-
partition Talos layout. Verified findable by exactly the query the phase runs.
Everything used to build it was removed afterwards: the builder server, the
ephemeral SSH key on both ends. The only server left in the project is
netbird.renner.uno, which was never touched and is delete-protected.
The token is now in pass, so the phase could be checked against the real API
for the first time — and the first thing it showed is that this project is not
empty. netbird.renner.uno has been running in it for 196 days and is the VPN
way in.
The label was the only thing separating a throwaway server from it, and
`hcloud server list -l ""` lists EVERY server, so an unset HCLOUD_LABEL would
have turned the sweep into a project wipe. Verified read-only that the selector
matches nothing today, which is correct but is a property of one variable
holding its value.
Two independent conditions now: the label is checked for emptiness before it is
used, and every candidate must additionally carry the name prefix this phase
assigns. A server that has the label but not the prefix is reported and left
alone rather than deleted.
Also corrects the token path. It is homelab/hetzner/cloud-api-secret — spelled
correctly, next to the pre-existing homelab/hetztner/ that holds the restore
keys. Two folders, one typo, both real; the comment says so, because the
obvious "fix" of unifying them would break whichever path was not moved.
The structural version of this is a separate Hetzner project with a token
scoped to it, so a bug here cannot reach production at all. Noted in the code;
worth doing before this runs unattended.
Still not executed: no snapshot labelled os=talos exists yet, which is the last
prerequisite and the one the phase fails on with the recipe.
The vmtest phase cannot test the one thing the registry mirror exists for. The
nodes carry machine.registries.mirrors with gitea.renner.uno first and ghcr.io
second, and the second endpoint only matters in the window where the first does
not exist. At home the first always exists — the production cluster is running
and answers on the LAN — so containerd never falls through and the fallback
stays unproven. From Hetzner, Gitea does not answer at all: it has no external
ingress.
The phase mirrors vmtest exactly: it provisions, sets the five variables the
talos phase reads, and falls through into it. A rehearsal that runs different
code proves less than one that does not.
Teardown is the part that had to be right, because this one bills by the hour.
It runs from the single EXIT trap, and it deletes by LABEL rather than by
remembered ID — so a run killed between `server create` and its own bookkeeping
is still cleanable, by the next run (which sweeps before it creates) or by
`bootstrap.sh hcloudclean`, which does nothing else.
The Talos API is unauthenticated in maintenance mode, which is how apply-config
reaches a node with no config. On a public address that is closed to everything
but the runner: the phase learns its own public IP and creates a firewall
admitting 50000 and 6443 from that address alone.
Scope, and why it is not the full topology: Hetzner is not the limit — a cloud
server takes up to 16 volumes and 3 private networks, so extra disks and NICs
are available. ceph-san is, because it binds to the kernel link name
enp2s0f1np1 and macvlan resolves its master by kernel name; LinkAliasConfig was
already tried on a throwaway node and does not help (vm-2.yaml). site-qemu
manufactures the name with a VLAN interface, and whether a Hetzner private
network passes VLAN tags is unverified, so it is not assumed. Everything
downstream of Multus therefore stays out, as does Flux — that phase needs the
OpenBao unseal key, which deliberately never reaches CI, and this test is meant
to run unattended.
The Talos snapshot is a prerequisite rather than something this builds; the
error names the exact one-off recipe, with this cluster's schematic and version
substituted from cluster.yaml rather than left as placeholders.
NOT YET RUN. hcloud is not installed here, there is no Hetzner Cloud token in
pass, and no snapshot exists, so none of this has been executed — only parsed.
The three prerequisites are what the first run will find.
require_secrets is gated on `phase_runs flux`. The node registry credential I
added in 2946dfcd is read in the TALOS phase, so `bootstrap.sh talos` on its own
— documented as "the part that can run unattended", and the default the vmtest
job uses without VMTEST_FULL — would reach registry_auth_patch with no secrets
collected and fail on a file nobody had asked for.
It went unnoticed because every run since has set VMTEST_FULL=1, which pulls in
the flux phase and therefore the full require_secrets first.
require_secrets now takes an optional list of names, and the talos phase asks
for `ghcr-pull` alone. Demanding the whole list there would be worse than the
bug: it would make an unattended talos run depend on the OpenBao unseal key,
which is deliberately kept out of CI — the reason the secret streaming is
manual in the first place.
The subset is validated against the list, so a typo in a name fails loudly
instead of silently demanding nothing.
creationPolicy: Merge cannot create a Secret. It only writes into one that
already exists, on the assumption that `flux bootstrap` made it — and that
assumption fails in precisely the situation this cluster gets rebuilt from.
The break-glass bootstrap pulls from the GitHub mirror and keeps its token in
flux-system-breakglass, so `flux-system` is never created. This ExternalSecret
then has nothing to merge into, produces no Secret, and still reports
Ready=True. Observed on a rebuild rehearsal: nine of nine ExternalSecrets Ready,
`flux-system` absent, and the moment anything pointed the GitRepository at it:
GitOperationFailed: failed to get secret 'flux-system/flux-system':
secrets "flux-system" not found
So the documented handover back to Gitea — switch the source once Gitea is
serving — could not be performed at all. The credential it needs was never
going to be there.
Orphan creates the Secret when absent and updates it when present, without an
ownerReference. That is what the header comment already claimed to want:
losing this object, or a failed sync, leaves Flux's last-good credential in
place. Owner would have created it too, but with an ownerReference that takes
the credential down with the ExternalSecret.
Safe on the live cluster: `flux-system` holds exactly the two keys written
here — verified, username and password — so writing it whole and merging into
it produce the same object. deletionPolicy is stated explicitly because it is
the half that matters: if the OpenBao entry disappears, keep serving what Flux
already has.
Upstream will not solve this for us. fluxcd/flux2#2761 asked for a fallback URL
on GitRepository and was rejected, with the reason that lands exactly here —
"the deploy keys are unique per repo so the auth will fail, same with HTTPS
access" (stefanprodan). A single source with two credentials is the shape Flux
has, so the credentials have to be kept apart by us.
Four images are needed while the cluster is still being built: the CNI plugins
every pod depends on, paperless's fetch-bridge-cert init container,
zigbee2mqtt's seed-data init container and the trmnl renderer. None of them can
come from gitea.renner.uno during a rebuild, because Gitea runs inside the
cluster being rebuilt.
cni-plugins solved that by referencing ghcr.io directly, and the earlier commits
in this branch were about to do the same to the other three. That fixes the
rebuild and breaks the normal case: every steady-state pull then leaves the
network for an image the local registry already holds. The recovery source and
the everyday source are different things — the same distinction Flux's git
source makes between the in-cluster Gitea and the GitHub break-glass mirror.
So the fallback moves into machine.registries.mirrors, where containerd takes an
ordered endpoint list: gitea.renner.uno first, ghcr.io second. Normal operation
is unchanged and local; ghcr answers only in the window where the first cannot.
No manifest names an external registry any more, cni-plugins included.
Containerd keeps the repository path when it falls through, so both registries
have to agree on it. `gitea_admin/alpine-jq` cannot be served by ghcr, where the
namespace is fixed to the account name — hence the konairius/ path on both, one
digest pushed to each by the same build. Verified equal on this run:
Pushed gitea.renner.uno/konairius/alpine-jq@sha256:87d0bf88...
Pushed ghcr.io/konairius/alpine-jq@sha256:87d0bf88...
The ghcr namespace stays private — three of the images this will eventually
cover are own applications, not alpine+jq — so the node also carries a
credential, in machine.registries.config, read from pass as the sixth required
secret. An imagePullSecret cannot work here: it would need to exist in five
namespaces before any of them does, and it would come from OpenBao, which needs
Ceph, which needs these very images.
Production still needs the machine config applied before a rebuild relies on
it; in steady state the references resolve from Gitea with or without it.
The previous commit put alpine-jq, alpine-openssl and cni-plugins on ghcr.io so
a rebuilding cluster can reach them. Pointing the manifests there was the wrong
conclusion: normal operation must pull from the local registry, and an external
reference sends every steady-state pull out of the network. The recovery source
and the everyday source are different things — the same distinction Flux's git
source already makes between the in-cluster Gitea and the GitHub break-glass
mirror.
Talos expresses that with machine.registries.mirrors and an ordered endpoint
list: gitea first, ghcr second. Containerd keeps the repository path when it
falls through to the next endpoint, so both registries have to agree on it.
`gitea_admin/alpine-jq` cannot be served by ghcr, where the namespace is fixed
to the account name.
Hence a second Gitea destination under konairius/, matching the ghcr path
exactly. The manifests will reference that one; the mirror makes it resolve
during a rebuild without any of them naming an external registry.
Pipeline only, again. The images do not exist under the new path until this
runs, so the references move in a separate commit afterwards. The gitea_admin/
copies stay until nothing points at them.
Needs the konairius org in Gitea (created) and package write for the pipeline's
token in it — this run will show whether ci_images_token already has it.
Three images are needed while the cluster is still being built — paperless's
fetch-bridge-cert init container, zigbee2mqtt's seed-data init container and
the trmnl renderer. None of them can come from gitea.renner.uno, which runs
inside the cluster being rebuilt; a rehearsal left all three in ImagePullBackOff
and took their Kustomizations down with them.
They are mirrored to ghcr.io (previous commit) but the namespace is private,
and it stays private: three of the five images this will eventually cover are
own application images, not alpine+jq. Making them world-readable is not
something the rebuild requires.
The credential goes into machine.registries.config, not an imagePullSecret.
An imagePullSecret fails here twice over: it would have to exist in five
namespaces before any of them exists, and it would come from OpenBao, which
needs Ceph, which needs these very images. containerd's own auth is in place
before the first pod is scheduled and applies to every pull on the node
regardless of namespace (pkg/machinery/config/types/v1alpha1,
RegistryAuthConfig — field names follow .docker/config.json).
The token is the sixth entry in OPENBAO_REQUIRED_SECRETS, so a run demands it
up front rather than discovering it missing after the disks are wiped. The
generated patch lives in the in-memory secrets dir and dies with the Job.
Only alpine-jq and alpine-openssl move in this commit, because only they are
published to the mirror so far. vsftpd, shelly-manager and
authentik-samba-bridge-ldap-outpost are built by other repositories and need
the same push there first; their references stay on gitea until then.
Production rollout is NOT this commit. The nodes need machine.registries
applied before any manifest points at ghcr, or the pods cannot pull. Order:
apply the machine config, verify a pull, then move the references.
cni-plugins has been publishing to ghcr.io since a cold-rebuild rehearsal
showed that gitea.renner.uno cannot serve a cluster that does not exist yet.
The comment there claimed it was "the only image here that must". A later
rehearsal disproved that: paperless sat in Init:ErrImagePull on
alpine-openssl, and trmnl's renderer on alpine-jq, both for the identical
reason.
The rule is not about any one image. Anything an app needs while the cluster
is still being built cannot live only in that cluster's own registry —
zigbee2mqtt's seed-data init container, the trmnl renderer and paperless's
fetch-bridge-cert init container all start during the rebuild.
Gitea stays the primary destination; ghcr.io is the copy a bare cluster can
reach. It has to stay credential-free to pull, because an imagePullSecret
would come from OpenBao, which needs Ceph, which needs the cluster to be up.
Pipeline only. The manifests still reference gitea.renner.uno and will keep
working; they move to ghcr in a separate commit once the images are actually
published there, because switching the references first would break the apps
immediately.
Needs the ghcr_mirror_token secret, now present as a global Woodpecker secret
scoped to write:packages and available on push and deployment — the two events
this pipeline runs on.
kubernetes/apps/home-assistant/ha-pg-objectstore.yaml declares an ObjectStore,
whose CRD comes from the CNPG Barman plugin in infrastructure-cnpg-barman.
The Kustomization never said so, and on a cold cluster that means the kind is
unknown when the path is first applied. Flux's server-side dry-run then
rejects the whole set over the single object it cannot recognise:
ObjectStore/home-assistant/ha-pg-store dry-run failed:
no matches for kind "ObjectStore" in version ...
Six app paths declare an ObjectStore — authentik, home-assistant, immich,
paperless, trmnl, woodpecker. Five of them already list the dependency. This
was the one that did not.
The blast radius is four Kustomizations, not one: speech-to-phrase and
zone-configurator depend on home-assistant directly, and matter-server health-
checks a Deployment inside it, so all three stalled behind an ObjectStore none
of them uses.
The csi.ceph.io CRDs come from the ceph-csi-operator subchart inside the
rook-ceph HelmRelease, which runs with `crds: CreateReplace` and `retries: 3`.
They are therefore not created once — they are created again on every attempt,
and every custom resource living under them goes with them.
A cold rebuild landed squarely in that window:
19:24:21 ceph-csi-drivers install succeeded — both Driver CRs created
19:25:50 csi.ceph.io CRDs recreated by a later operator attempt
19:29:33 rook-ceph-operator install finally succeeded
Helm still had the two Driver objects in its stored release, so nothing
re-rendered them and nothing reported them missing. The ceph-csi-operator sat
idle with no Drivers to reconcile, no provisioner Deployment was created, and
every PVC on ceph-block stayed Pending on "waiting for a volume to be created
by the external provisioner rook-ceph.rbd.csi.ceph.com". openbao-0 never got a
node, and the run failed 15 minutes later reporting that OpenBao would not
answer — five layers from the cause, with Ceph itself reporting Ready and
HEALTH_OK throughout.
This file previously argued against ordering it: "nothing here installs those
CRDs, so a dependsOn would only move the wait". That holds for the CRDs
existing, which was the failure being fixed at the time. It does not hold for
them being replaced, which is a different event and the one that bites here.
dependsOn on rook-ceph-operator waits for the last attempt instead of the
first.
The install retries stay as a backstop: a CRD replace outside an install of
this release would still empty it, and a retry is cheaper than waiting out the
30-minute interval.
6GiB was not enough and neither was 8: at both sizes vm-1 reached 99% of
allocatable memory once the infrastructure layer was up. A control plane with
nothing left over stops being one — the kubelet admits the node-critical CNI
DaemonSets by evicting whatever ranks below them, and kube-apiserver, a static
pod at system-cluster-critical, is what ranks below them:
message: Pod was preempted by Kubelet to accommodate a critical pod.
reason: Preempting
That was misread twice as a priority problem. It is not. The ordering is
deliberate and correct — a node without its CNI cannot run anything — and
nothing about it blocks: the memory shortfall blocks, and priority only picks
who pays for it. Run 26 showed this cleanly inside a single run: with the CNI
demoted to cluster-critical the API server survived bootstrap, and it was
evicted within a minute of the DaemonSets going back to node-critical, on an
unchanged node. Same pressure, different victim.
So the control plane gets 16GiB and the Job's ceiling moves with it (40G of
guests). The hosts carry ~95GiB each with 43-66GiB free, so this costs
nothing that matters.
vm-1 is the only control plane and still takes workload, so it carries the
static pods, the CNI and its share of the infrastructure layer at once. That
is the asymmetry with production, where every node is a control plane and none
of them is small.
This reverts the priority classes set in ce31dace and the wording in
d0de3798, putting multus, whereabouts and cni-plugins-installer back on
system-node-critical.
Demoting them was the wrong move. The class name says what a pod is, and
these are node-critical by what they do: without multus a node cannot attach
a secondary network, and only that node is affected — nothing cluster-wide
fails. Cilium and cni-dhcp-daemon are in the same class for the same reason.
Choosing cluster-critical instead picked a number for its ordering and made
the manifest state something untrue about the component.
It also aimed at the wrong thing. The kubelet evicted kube-apiserver because
the control plane had run out of memory; priority only decided which pod was
chosen once that had happened. Equal priority took the API server out of the
candidate set without removing a single byte of the pressure — the next
victim is simply someone else, or the CNI pod stalls at Pending, which is the
defect 0b76c130 set out to fix.
So the preemption stays open, and it is open where it belongs: the control
plane is starved during a rebuild, when the whole infrastructure layer lands
on it at once. That is fixed with memory or with kubeReserved/systemReserved
in the machine config, not by reordering the components on top of it.
Recorded next to the sizing flags in bootstrap.sh.
The comment added in ce31dace claimed "preemption is not a capacity
question". That inverts the cause. The kubelet preempts because the node is
out of memory; without a shortfall nothing is evicted, whatever the priorities
say. Priority only selects the victim once the shortfall exists.
What the two runs isolate is exactly that and no more: at 8GiB with
node-critical the API server was chosen, at 8GiB with cluster-critical it was
not. The pressure did not change — the candidate set did. Under pressure the
kubelet now displaces an ordinary workload instead, or this pod stays Pending,
which is the original defect and the acceptable one of the two outcomes.
Also records why equal priority is enough: Preemptable() compares with a
strict >, and the API server is a static pod, so IsCriticalPod holds for it
and the critical-beats-noncritical branch never applies.
0b76c130 gave the three secondary-network DaemonSets system-node-critical,
copying cilium and cni-dhcp-daemon. That is 2000001000, and Talos runs
kube-apiserver as a static pod at system-cluster-critical, 2000000000 — so
the change made the CNI outrank the control plane. On a full node the kubelet
did exactly what that ordering tells it to:
message: Pod was preempted by Kubelet to accommodate a critical pod.
reason: Preempting
phase: Failed
Both runs after the change died that way, and 097a0fac's larger control plane
did not help: it reproduced unchanged at 8GiB. During a rebuild the whole
infrastructure layer lands on a single-node control plane at once, and in that
window the node is full whatever its size. Preemption is an ordering, not a
capacity question, and no amount of memory makes a lower-priority pod safe
from a higher-priority one.
system-cluster-critical instead. Equal priority cannot preempt, so the API
server is out of reach of these pods. What 0b76c130 was for still holds:
2000000000 is far above any ordinary workload, so a rollout onto a busy node
displaces application pods rather than stalling at Pending forever — which was
the original defect, where multus sat unscheduled and every secondary network
on that node silently stopped working.
The residual is symmetric: the API server cannot displace multus either. That
is the correct trade when the alternative is losing the control plane.
The 8GiB control plane in bootstrap.sh stays, on its own merits — at 6GiB it
sat at 99% of allocatable memory with nothing left for late arrivals — but its
comment no longer claims to fix the preemption, because it never did.
Note: cni-dhcp-daemon.yaml still carries system-node-critical, from long
before this. It is the same hazard and is left alone here rather than widened
into unrequested scope.
Making the CNI DaemonSets system-node-critical (0b76c130) had a consequence I
did not account for. That class is 2000001000; Talos runs kube-apiserver as a
static pod at system-cluster-critical, 2000000000. The CNI pods therefore
outrank the API server, and on a node with no headroom the kubelet resolves
the next admission by evicting it.
That is what happened on the first run after the change. The 6GiB control
plane was at 99% of allocatable memory once the infrastructure layer was up;
the kubelet logged
preemption/preemption.go:127 "Pod evicted successfully"
pod: kube-apiserver-vm-1
and the API server never came back, because the room it needed never freed.
The run reported "openbao-0 never answered bao status" three steps later,
which is true and useless — the pod could not answer because there was no API
server to reach it through.
The priority ordering stays. A node without its CNI cannot run anything,
which is why cilium and cni-dhcp-daemon already carry the same class. What it
demands in return is capacity: a control plane must keep enough room that
admitting a node-critical pod never has to displace the control plane itself.
6GiB did not, so the throwaway control plane gets 8GiB, and the Job's own
ceiling moves with it (32G of guests now, not 30).
The obligation is written down in both places it can be violated from —
next to the sizing flags in bootstrap.sh, and next to the priority class in
multus-daemonset.yaml.
The GitHub break-glass bootstrap let `flux bootstrap` use its default secret
name, `flux-system`. That is also the target of the flux-system-git-auth
ExternalSecret in post-crds, which carries the GITEA credentials — correct for
the steady state, where the source is the in-cluster Gitea.
Sharing the name means that the moment infrastructure-crds reconciles and ESO
resolves, the GitHub token is replaced with Gitea credentials while the
GitRepository still points at github.com. Flux immediately loses the ability
to read its own source:
GitOperationFailed: unable to list remote for
'https://github.com/konairius/Homelab.git': authentication required:
Invalid username or token
So the cold rebuild destroys its git source at the exact moment it starts
working. The better the run goes, the sooner it kills itself — which is why
this stayed hidden through twenty-two rebuild attempts: not one of them had
ever got the ExternalSecrets to resolve.
--secret-name=flux-system-breakglass keeps the two apart. `flux-system` still
receives the Gitea credentials from OpenBao and simply goes unused until the
source is pointed back at Gitea, which is already a deliberate manual step
(RUNBOOK-cold-rebuild.md), announced by the warning printed right above this
call.
Nothing in the steady-state path changes: the Gitea bootstrap keeps the
default name, and the committed gotk-sync.yaml keeps referencing
`flux-system`.
Three of the four DaemonSets that make up the secondary-network path ran at
priority 0. cni-dhcp-daemon.yaml, sitting in the same directory, has been
system-node-critical all along, and so is cilium's own DaemonSet — multus,
whereabouts and cni-plugins-installer were the odd ones out.
At priority 0 they cannot preempt anything. Any event that forces a fresh
scheduling decision — a rollout, a node restart, an eviction — has to fit the
pod onto a node that may since have filled up, and if it does not fit the pod
just stays Pending. The node then has no multus, or no IPAM, or no macvlan
binary, and nothing reports it: pods still start, they simply never receive
their secondary attachment.
Found on a cold rebuild. Two of four nodes sat at 99% memory requests when the
DaemonSet rolled; both new multus pods went Pending with "1 Insufficient
memory, 3 node(s) didn't satisfy plugin(s) [NodeAffinity]" and stayed there.
Rook's network canary on one of those nodes hung in Init:0/1 in
wait-for-network-status-annotation — waiting for an annotation multus was not
running to write. Its CephCluster reconcile stopped at "discovering ceph
cluster network CIDR(s)" and never returned to status.phase Ready, which is
exactly what infrastructure-ceph-cluster's healthCheckExpr gates on, so the
entire apps layer stalled behind a scheduling decision three layers away.
cni-plugins-installer.yaml already carries a comment describing this same
canary-hang, reached from a different cause (the plugin image was pulled from
the in-cluster registry, which does not exist during a rebuild). Same symptom,
same four-layer distance between cause and evidence.
A cold rebuild deadlocked with every app Kustomization behind it:
infrastructure-ceph-cluster → CephCluster status.phase == Ready
→ Rook's node-daemon reconcile settles
→ monitoring.coreos.com CRDs exist
→ kube-prometheus-stack (monitoring)
→ grafana-db
→ infrastructure-ceph-cluster
Nothing outside that loop can break it. The operator logged
"node reconcile failed: failed to enable service monitor: the server could
not find the requested resource (post servicemonitors.monitoring.coreos.com)"
every two minutes while the CephCluster sat in Progressing, and
infrastructure-ceph-cluster's healthCheckExpr waits on exactly that phase.
Rook treats the two ServiceMonitors differently, and only one of them is
fatal (v1.20.7):
- mgr.go:344 logs and continues — "we don't want to return an error to
block the cluster reconcile since monitoring is an optional service"
- nodedaemon/reconcile.go:254 returns the error, which aborts the whole
per-node reconcile and requeues it forever
So the exporter's ServiceMonitor, the one Rook considers mandatory, is what
held the cluster.
`metricsDisabled` is not the switch to reach for — that turns the metrics
themselves off. `monitoring.enabled: false` only stops Rook from declaring
the scrape targets: configurePrometheusModule is gated on !MetricsDisabled
alone, so the mgr prometheus module and the ceph-exporter DaemonSet keep
running and serving exactly as before.
Both objects are reproduced in apps/monitoring-rules field for field off the
live production cluster — same names, namespace, selectors, ports, 10s
interval and `cluster` relabeling. The only thing lost is the CephCluster
ownerReference, which Flux cannot set.
This is the third component to need this treatment, after velero and cilium,
and the same sentence covers all three: a chart or operator that renders a
monitoring.coreos.com object at install time cannot sit below the layer that
provides those CRDs.
Note for the production rollout: Rook does not delete a ServiceMonitor when
this flag flips, so the existing objects stay and Flux adopts them rather
than creating new ones.
The operator chart needs no change. Its own `monitoring.enabled` gates only
the RBAC that permits ServiceMonitor writes, and `csi.serviceMonitor.enabled`
is already false, so neither creates a monitoring.coreos.com object.
On a cold start only openbao-0 exists when unseal_openbao runs, so the loop
over `pods` unsealed exactly one node and returned. That deadlocks the
StatefulSet: podManagementPolicy is OrderedReady and a sealed pod never
passes its readiness probe, so openbao-1 is created only once openbao-0 is
unsealed — after this step has finished — and openbao-2 is never created at
all. The cluster settles at 1 of 3 replicas.
What makes it fatal rather than merely degraded is that the openbao Service
sets publishNotReadyAddresses, which the chart does so replicas can reach
each other to join raft before any of them is unsealed. The sealed openbao-1
is therefore a live backend on the service IP, and clients get balanced onto
a node that cannot answer. The symptom surfaces nowhere near OpenBao:
external-secrets' ClusterSecretStore flaps between "store validated" and
`dial tcp 10.105.146.11:8200: connect: no route to host`, all nine
ExternalSecrets report "could not get secret data from provider", and
infrastructure-crds — whose `wait: true` covers them — never goes Ready,
blocking all twenty app Kustomizations behind it.
The unseal step now converges instead of iterating once: re-list the pods
each round, unseal whatever is sealed, and finish when spec.replicas of them
are unsealed. A pod still in Init cannot be unsealed yet and is retried
rather than treated as a failure, which is how the OrderedReady chain is
meant to be walked.
Also drops the `2>/dev/null` on the seal-status read; stderr goes to the
same .baoerr file the wait above already uses.
infrastructure-crds stopped with
CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
namespaces "woodpecker-ci" not found
and took all twenty app Kustomizations with it, since every one of them
dependsOn it. post-crds ships that namespace's policies through
../network-policies/ and runs before any app KS; the namespace itself was
declared in apps/woodpecker, which runs after.
namespaces.yaml already explains this exact trap, two objects above, for the
sibling namespace: 'Declared here rather than alongside the workloads ...
Creating it in the woodpecker workload KS would invert that order.' It was
written for `woodpecker` and never carried across to `woodpecker-ci`.
Moving a Namespace between Kustomizations prunes and recreates it, taking its
contents with it — the hazard that ruled out splitting the rook operator
earlier. Checked before doing it: production's woodpecker-ci holds no
workloads and no PVCs, only per-step Secrets left behind by pipelines that
finished weeks ago. Losing those costs nothing.
Third instance of the same shape today, after velero/cilium and the
CRD-dependent manifests: the lesson written down beside the code, and not
applied to the case next to it.
infrastructure never went Ready on an empty cluster. Six of its seven
healthChecks were met; cilium's HelmRelease was not:
execution error at (cilium/templates/validate.yaml:79:13):
Service Monitor requires monitoring.coreos.com/v1 CRDs
Those CRDs come from kube-prometheus-stack, which lives in monitoring, which
dependsOn grafana-db, which dependsOn infrastructure. The CNI — the first thing
that has to exist — was waiting on the last.
Cilium needs none of this. prometheus.enabled is what exposes the agent metrics
the comment there argues for, and it needs no CRDs. serviceMonitor.enabled only
asks the chart to also write the scrape wiring, which is a monitoring-stack
concern expressed in the CNI's release. It moves to apps/monitoring-rules, the
layer that dependsOn monitoring and therefore has the CRDs, alongside the
ServiceMonitors for alloy, loki, immich, smartctl, velero and authentik.
The manifest reproduces cilium 1.20.1's own output field for field, read off
the live object: same name, namespace, selector, 10s interval, node relabeling.
Scraping is unchanged.
velero/helmrelease.yaml already did exactly this and explains why in a comment
that names the rebuild case precisely. It was never applied to cilium, where
the same gate is worse: velero's chart skips its template silently, cilium's
aborts the render, and cilium is a healthCheck.
Third shape, and this one is what OpenBao asks for in its own error text:
you are executing outside of a terminal (tty) ... If this is not an option,
the unseal key can be provided as the first argument
`unseal -` was Vault syntax and sent the dash as the key. Dropping the
argument makes OpenBao prompt, and the prompt wants a real terminal — from a
Job it answers 'file descriptor 0 is not a terminal'. There is no piped form.
So the key goes in as an argument and is visible in the pod's process list for
the length of the call, which the code here deliberately avoided before. That
is a real cost, taken knowingly: reading that list needs exec into the very
pod holding the key in memory, and the alternative is a port-forward and curl
against /v1/sys/unseal from outside — the openbao image carries no curl, wget
or nc — which puts a background process in the most critical step of a
recovery. Say so if you would rather pay that instead.
Run 18 reached this again with everything before it working: Ceph HEALTH_OK,
three OSDs, PVCs bound, the kopia repository read and
openbao-raft-20260902-010002.snap extracted.
Both unseal call sites ran `bao operator unseal -`, which is Vault syntax.
OpenBao's operator_unseal takes the key as argv[1] and has no case for "-":
case 1:
unsealKey = strings.TrimSpace(args[0])
so the dash itself was sent as the key and the server answered
* 'key' must be a valid hex or base64 string
Dropping the argument is what makes it read stdin, which is what the comment
beside it always claimed and what keeps the key out of the pod's process list.
Both sites fixed; nothing else in the script uses a dash that way.
Neither path had ever been reached before, so neither had ever run. The
restore got this far for the first time on run 17 and did the hard part
correctly: it read the kopia repository at Hetzner read-only, found
openbao-raft-20260902-010002.snap, pulled 839768 bytes out of it and
initialised the empty server — then could not unseal what it had just
initialised.
A rehearsal reached the restore for the first time and stopped at
FAILED: kopia not found — needed to read the Velero backup repository
without Velero
which is the one step that cannot be delegated: Velero's credentials live in
OpenBao, so Velero cannot be what restores OpenBao. bootstrap.sh reads the
kopia repository directly instead, and on a workstation kopia is simply
installed. In the Job it has to be in the image.
Added the same way as talosctl, kubectl and flux: pinned version, checksum
verified against upstream's checksums.txt, binary lifted out of the versioned
directory the tarball unpacks into.
Everything before this point now works unaided — Ceph HEALTH_OK with three
OSDs, all PVCs bound, openbao-0 running and unsealed. This is the next thing
in the way, not a regression.
One line after the previous fix, the same assumption again:
initialized=$(jq -r '.initialized // empty' <<<"$status")
jq's // is the alternative operator and treats false exactly like null, so the
one answer a cold rebuild always produces — {"initialized": false} — came out
as an empty string. The guard below then reported that OpenBao's state could
not be read at all, on a pod that had answered correctly and completely.
Presence and value are now asked separately. Verified against all three
inputs: false yields "false", true yields "true", and a genuinely missing
field still yields empty so the guard keeps working.
Run 15 got past the wait fixed in the previous commit and died here instead,
with Ceph HEALTH_OK, three OSDs, every PVC bound and openbao-0 Running. The
cluster has now come up unaided twice; both times the script threw it away one
step later. The remaining swept-for pattern is clean — the only other jq -e
tests a comparison, which is a real boolean.
The wait for openbao-0 asked jq -e '.initialized', which tests whether the
value is true. A freshly created OpenBao answers
{"initialized": false, "sealed": true}
which is complete, correct, and the only answer a rebuild can produce. jq -e
returned 1, the loop treated it as silence, and after 90 rounds against a
perfectly healthy pod the run died with 'never answered bao status'.
So the check could never pass in the one situation it exists for. It only ever
succeeded against an OpenBao that was already initialised — which on a cold
rebuild is precisely what is not there.
The comment two lines above already said an exit code of 2 is an answer and
not a failure. The code then asked the wrong question anyway. It now asks
whether the field is present, which is what 'did it answer' means; a truncated
or non-JSON reply still fails, as it should.
Found on run 14, one step after the cluster came up unaided for the first
time: Ceph HEALTH_OK, three OSDs, all PVCs bound, openbao-0 Running. The
storage chain was finally whole and this wait threw it away.
Every CSI pod in this cluster has been talking to the mons with client
libraries a whole major version ahead of them. The ceph-csi image ships Ceph
20.2.1 (Tentacle); the cluster runs 19.2.6 (Squid). Verified in both the live
cluster and a rebuild rehearsal — same images, same skew.
The image policy said the 19 -> 20 cutover should be deliberate and tied to the
Rook operator version. The coupling it protected was already gone: Rook's
floating constraint pulled v1.20 in June, which is where the v20 client came
from, and the cluster stayed on 19. The rule held while its premise quietly
did not.
It is not cosmetic. The 19.2.6 mons mint auth keys as aes256k and the 20.2.1
client cannot decode them:
auth: failed to decode key 'AgB...'
Storage works today only because the four CSI users predate the cipher change
and still carry legacy aes keys — the ones Ceph's own health check flags as
"5 auth client entities with insecure key types". Recreate any one of them, by
rotation or a deleted Secret or Rook resyncing, and the provisioner stops: no
PVC binds and nothing mounts. A cold rebuild has no legacy keys at all, so it
hits this immediately, which is how it was found. Proven both ways on the test
cluster: with aes256k every request fails, and after rotating the same user to
aes the same provisioner reports success and the volumes bind.
So the cutover happens now, to the version the client already is. Rook v1.20
supports Tentacle from v20.2.1. v20.2.0 is excluded on purpose — upstream
reports data corruption there when read affinity is enabled.
Unverified and deliberately so: this is a Ceph major upgrade against live
storage. It is on this branch to be rehearsed first, not merged.
Every PVC failed with
failed to get connection: connecting failed: rados: ret=-22, Invalid argument
and it was mine. The Driver CRs on the live cluster carry clusterName:
rook-ceph; I copied the driver names out of them and not this field, and the
chart defaults it to the empty string. It becomes --clustername on the plugin
and goes to rados_create2 as the Ceph cluster name, which librados refuses.
The failure is a good imitation of a credential problem. Ruled out, each
against the live cluster: the CSI user exists and is named identically in both
clusters, the key in the Secret hashes equal to the one Ceph holds, that same
user and key connect fine from the toolbox, monitors in ceph-csi-config match
the mon Services exactly, ClientProfile is byte-identical to production, and
the images are the same cephcsi v3.17.0. A detour through Ceph key ciphers
looked convincing — production's CSI users still carry the legacy aes type
while a fresh cluster mints aes256k — and was wrong: the aes256k key
authenticates when passed the way ceph-csi passes it.
What gave it away was diffing the plugin's own arguments against production's,
which is where the missing flag is plainly visible.
A cold rebuild never gets storage. Rook stopped deploying CSI in v1.20 and
hands it to the ceph-csi-operator, which does nothing until someone supplies
Driver CRs. On this cluster Rook's v1.18 -> v1.20 migration created them once,
116 days ago, so they look like part of the furniture; on an empty cluster Rook
creates the CephConnection and the ClientProfile and stops. No Driver, no
provisioner, and every PVC waits forever — OpenBao's included, which is exactly
where the rehearsal stopped, three times, after Ceph itself was healthy with
three OSDs.
The workaround this replaces (babed078) rendered the chart's ServiceAccounts and
RBAC as static manifests and left the CRs to the migration. It was written
during a five-day outage with PVC attach blocked cluster-wide, and it gave a
reason that does not hold: helm-controller v1.5.4 does not "predate
disableTakeOwnership". That landed in v1.2.0, three minors earlier, and it is an
opt-out — taking ownership of existing resources is the default. The chart was
installable all along.
cephConnections and clientProfiles are left empty deliberately. Rook owns both:
the CephConnection carries mon Service IPs Rook keeps current, and the
ClientProfile references Secrets Rook creates. Helm managing either would freeze
a value meant to move. Confirmed against a fresh cluster, where Rook creates
precisely those two and no Driver.
The driver names are pinned to rook-ceph.{rbd,cephfs}.csi.ceph.com, matching
what the StorageClasses already name as their provisioner. The chart's default
omits the prefix, which would leave every StorageClass pointing at nothing.
On a cluster that already has these objects, Helm adopts them. That has not been
exercised yet — this branch is not for master until it has.
With the memory fix in place the workers moved from 99% memory requests to
49%, and straight into the next wall at 69% of CPU. What could not be
scheduled this time was rook-ceph-operator itself, plus the multus
DaemonSet's new pods during a rollout.
An unscheduled operator is a cluster with nothing driving it: two OSDs
crash-looped with no one to reconcile them, the MDS never placed, and the
PVCs sat on 'Waiting for a volume to be created by the external provisioner'
for half an hour because the provisioner's operator was Pending. Ceph
reported Ready throughout, which is true of the CR and says nothing about
whether storage works.
Workers get 4 CPUs and the pod requests 6. k8s-3 is at 24% of its CPU
requests, so this is not close to anything.
Ceph came up with three OSDs and then went HEALTH_ERR. The OSDs crash-looped
on
unable to find any IPv4 address in networks '10.99.0.0/29' interfaces ''
Failed to pick cluster address.
which reads like an addressing fault and is not one. The whereabouts pool
had zero allocations and the OSD pods carried the ceph-san request
annotation with no network-status to match it, because kube-multus-ds and
whereabouts were themselves Pending: a worker sat at 99% of its memory
requests once it carried an OSD, a mon and a mgr, and there was no room left
for the CNI DaemonSets. The MDS pods could not schedule either, which is
where the HEALTH_ERR came from.
Workers go to 8GiB and the pod ceiling to 36Gi. k8s-3 carries it at about
half its allocatable memory.
The alternative was shrinking Ceph's requests until they fit, which would
have made the test pass by measuring something production does not run.
Ceph reached Ready with zero OSDs. All three osd-prepare jobs completed
cleanly and refused the disk they were pointed at:
skipping device "vdb1": ["Used by ceph-disk"]
skipping OSD configuration as no devices matched the storage settings
Rook classifies a partition as belonging to an old ceph-disk deployment by
its label, and Talos names a RawVolumeConfig's partition r-<name> — so
'ceph-osd' became 'r-ceph-osd' and Rook read its own name in it and backed
off. The partition type GUID is the generic Linux one, so the label is the
only thing it could have matched on.
Renamed to osd-data, giving r-osd-data. Nothing about the volume changes but
the string. Production is unaffected either way: there the OSD is a whole
NVMe matched by device id, not a partition, which is why this never showed
up until a VM ran the same CephCluster.
Ceph itself now comes up — mons, both mgrs, mds, and the network canary all
pass, which is what the last three commits were about. This is the next link
after that: still no storage, so OpenBao's PVC still cannot bind.
The NetworkAttachmentDefinition set mtu 9000, and so does the LinkConfig on
enp2s0f1np1 that it attaches to. Two places for one number, which have to
agree and are not checked against each other.
They disagreed on the rebuild rehearsal and the attachment was refused
outright: 'invalid MTU 9000, must be [0, master MTU(1500)]'. A VM's NIC is
capped at 1500 by the host_mtu qemu advertises, and the virtio driver will
not let the guest exceed it, so jumbo frames are not reachable there at all.
The plugin's own error names the fix — 0, meaning unset, is a valid value
and inherits from the master. Production behaviour is unchanged, because the
master already carries 9000; the number is now stated once, on the link that
actually has to carry the frames.
Third error in the same chain, each one a layer further in: the plugin was
missing (mirrored), then the link was missing (VLAN), now the MTU. Ceph is
still not up.
Ceph never came up in the rehearsal. Rook's network canary could not attach
ceph-san and the CephCluster sat at 'Configuring Ceph Mons', so ceph-block
never existed, so OpenBao's PVC never bound. The canary's error moved from
'failed to find plugin macvlan' to 'Link not found' once the CNI plugins
were mirrored: macvlan ran and found no enp2s0f1np1, which is a
bus-path-derived name from an X710 card that no VM will ever produce.
Production has a second physical port for that link. A VM has one NIC, so
here it is a VLAN on it, deliberately carrying the same name — the test then
runs san-network-attachment.yaml verbatim rather than a patched copy, which
is the whole point of the exercise.
Two dead ends first, both found on throwaway nodes rather than in
production. LinkAliasConfig looks like the right tool and is not: Talos
records the alias in its own field, not in altNames, so macvlan — which
resolves its master by kernel link name — cannot see it. And a VLANConfig on
its own took vm-4 off the network entirely: declaring any network document
makes Talos configure the link explicitly and the implicit DHCP that
addresses the node stops. The DHCPv4Config restores it, verified on a second
node before being written down.
A dummy link would have been simpler and worthless: no traffic between
nodes, so Ceph would replicate over an interface going nowhere and the test
would go green having proven nothing.
This is machine-level config, not site-level. The link belongs to the
machine that has it.
A cold-rebuild rehearsal stopped dead here. The installer DaemonSet pulls
gitea.renner.uno/gitea_admin/cni-plugins, and Gitea runs inside the cluster
being rebuilt, so all four nodes sat in ImagePullBackOff with 'no route to
host'. Without the plugins there is no macvlan; without macvlan Multus
cannot attach ceph-san; without that network Rook's canary never starts,
the CephCluster hangs at 'Configuring Ceph Mons', and OpenBao waits forever
for a PVC that can never bind. Four layers between the cause and the symptom.
Baking the binaries into an image fixed a real integrity gap — the old path
fetched a release tarball on the node with no checksum at all. What it also
did, unnoticed, was trade github.com on the critical path for something
strictly worse: a registry that by definition does not exist during a
recovery. The mirror keeps the checksum and removes the dependency.
ghcr.io/konairius/cni-plugins:2026-07-30 is the same image, verified by
digest (sha256:85e83e32...) against the Gitea copy, and public: an
unauthenticated pull was confirmed to work. It has to stay credential-free,
because an imagePullSecret would come from OpenBao, which needs Ceph, which
needs these binaries.
Every other image in this directory already came from a public registry.
The build pipeline does not yet publish to ghcr.io — that needs a token this
commit cannot provide, so today's mirror was pushed by hand. Bumping the tag
means mirroring first until that lands.
At 10GiB the system disk yielded an 8.36GB /var, which the infrastructure
image set filled to 82%. kubelet then tainted all four nodes with
node.kubernetes.io/disk-pressure and nothing could schedule at all —
Ceph's own monitor sat Pending, and the symptom surfaced three layers
away as a Cilium HelmRelease stuck InProgress, which is not where anyone
would look. Now 40GiB.
The second disk is the Ceph OSD. Three 8GiB OSDs replicated 3x leave 8GiB
usable, less than OpenBao's two PVCs ask for on their own, so the storage
could not have satisfied the cluster it was being tested with. Now 24GiB.
Both are raw sparse files: the larger geometry costs only what is written.
The emptyDir ceiling moves to 150Gi, which still bounds actual usage well
below the ~810GB free on the filesystem behind it.
infrastructure-ceph-cluster no longer dependsOn infrastructure. That single edge
closed a loop, found by rebuilding from an empty API server:
infrastructure is Ready only once its openbao healthCheck passes
-> the openbao HelmRelease is Ready only once openbao-0 runs
-> openbao-0 stays Pending on "unbound immediate PersistentVolumeClaims"
-> data-openbao-0 wants StorageClass ceph-block
-> ceph-block is defined in infrastructure/rook-ceph-cluster/
-> applied by infrastructure-ceph-cluster, which waited on infrastructure
Nothing in that chain can move. Every StorageClass in the cluster is Ceph-backed,
so there is no storage OpenBao could use that exists before Ceph, and the loop
cannot be broken from the storage side at all.
The edge was never load-bearing. infrastructure runs wait: false, so it applies
the rook-ceph namespace and the operator the moment it reconciles; only its
readiness waits on the healthChecks. Verified on the deadlocked test cluster:
removing the dependsOn created the CephCluster and CephBlockPool within seconds
and ceph-block appeared, with infrastructure still not Ready. On an empty API
server this Kustomization now retries until the operator installs its CRDs — the
shape post-crds already relies on — and on a live cluster nothing changes,
because the CRDs are always there.
The alternative was dropping the openbao healthCheck, which breaks the cycle
just as well. It was rejected: the guarantee it encodes belongs somewhere, and
splitting the rook operator into its own Kustomization was rejected too, because
it would move a Namespace between Kustomizations and a prune on a Namespace
takes everything inside it.
Also moves authentik's outpost ServiceMonitors to apps/monitoring-rules/. Their
CRDs come from kube-prometheus-stack, and the authentik Kustomization depends on
infrastructure* but not on monitoring — so on a cold cluster it raced the chart
and applied nothing until it happened to lose. Unlike the Ceph loop this one
resolves itself, but authentik is the login provider a lot waits behind. No
ServiceMonitor, PodMonitor, PrometheusRule or Probe now sits outside the layer
whose dependency installs its CRD.
The whole script has run in the pod since the flux and OpenBao half moved
there, but the comment still told a reader that half stays on the
workstation, and still quoted the old 2.5G control plane.
Two harness defects, both found by a run rather than by reading.
The control plane ran on 2560MiB. Measured on the live VM mid-run: 250MiB
available, page cache squeezed to 386MiB, 94MiB free. etcd's fsyncs therefore
landed on a system in permanent reclaim, and linearized reads took 5 to 20
seconds against an expected 100ms. The apiserver became unreachable, recovered,
went again; the run sat in a bounded wait printing nothing, because its bound
counts iterations and each iteration was blocking on API timeouts. Nothing in
that failure says whether the repo can rebuild a cluster, which is the only
question this test exists to answer, so the control plane gets 6GiB and the pod
ceiling moves to 24Gi. k8s-3 carries it comfortably: requests were at 30% of
93GiB allocatable.
The image is pinned by digest. Spegel mirrors mutable tags and keeps serving
what it cached, so a run on k8s-3 pulled a ci-vmtest:latest from three builds
back — one that predated OVMF — and died with "the required flash image was not
found" before the first VM booted. The registry's :latest was a different digest
at that moment, which is what made the diagnosis unambiguous. The dated tag rides
along for readability; containerd resolves the digest.
The wait for OpenBao is supposed to end on one of two cluster facts: the
StatefulSet appears, or Flux reports the infrastructure Kustomization has failed.
Only the first ever fired. The second read READY as the fourth field of
`flux get kustomization --no-header`, whose columns are NAME REVISION SUSPENDED
READY MESSAGE — but REVISION is empty until the first successful reconcile, which
is exactly the state this check exists to catch. The columns shift left, the
fourth field lands on the message, and matching it against "False" never fires.
So the persistent-failure counter never moved. Yesterday's run sat for 45 minutes
on an infrastructure Kustomization that had been failing its dry-run since its
first minute, and would have run the full 720 iterations — two hours — before
saying anything. The bound meant as a backstop was doing all the work, which is
the failure this wait was written to avoid.
The Ready condition has no positional ambiguity, so all three waits now read it
directly via kubectl. ks_ready_condition echoes "<status>|<message>"; an empty
status means the object does not exist yet, and kubectl's own error is carried
through as the message rather than discarded, keeping "not created yet"
distinguishable from "the lookup itself broke". Verified against the live cluster
for both the found and not-found paths.
The HelmRelease tally in the summary gains a denominator. "12 ready" reads as
success whether the cluster has 12 HelmReleases or 40, and a rebuild is precisely
where that distinction matters. Both numbers now come from the objects.
A cold-rebuild test against an empty API server stopped here permanently:
infrastructure False ExternalSecret/apps/external-dns-unifi-secret dry-run
failed: no matches for kind "ExternalSecret" in version "external-secrets.io/v1"
Flux server-side dry-runs the whole apply set, so a single unknown kind rejects
all of it — including the ESO HelmRelease that installs the very CRD the dry-run
is missing. The VM cluster confirmed it: zero HelmReleases, zero ESO CRDs. The
deadlock cannot resolve itself, and no amount of waiting changes that.
Eight ExternalSecrets and three Traefik Middlewares sat beside their consumers
under infrastructure/, whose own HelmReleases install their CRDs. They move to
post-crds/, which exists for exactly this and already held the OpenBao
ClusterSecretStore and the Velero Schedules on the same grounds.
Velero's ServiceMonitor and PodMonitor could never have applied there at all:
monitoring.coreos.com CRDs come from kube-prometheus-stack under apps/, which
reconciles after infrastructure. They move to apps/monitoring-rules/, whose Flux
Kustomization already carries a comment describing this precise failure for
PrometheusRule — the same diagnosis, made once and never carried across.
None of this was reachable in production, where the CRDs have existed since the
day each operator was introduced. Ordering only becomes load-bearing when
something applies it against nothing, and until now nothing ever had.
The ExternalSecret in openbao/rotation/flux-git.yaml is split out; the
ServiceAccount and rotation CronJob stay where they are.
Merging this transfers ownership between Kustomizations, so Flux prunes each
object from infrastructure and recreates it under infrastructure-crds or
monitoring-rules. The Secrets ESO owns vanish for that window — running pods
keep their mounted copies, but a pod starting in the gap will not find them.
Found by reviewing rather than by running, after several twenty-minute runs each
surfaced one more. Every one of these was introduced today.
The documented cold-rebuild command could not work. require_secrets resolved the
git token from FLUX_TOKEN — a name that appears nowhere else in the repo, while
the usage header and RUNBOOK-cold-rebuild tell the operator to export
GITHUB_TOKEN or GITEA_TOKEN. Worse than a typo: `${VAR:?}` inside eval terminates
the shell outright, so `FLUX_SOURCE=github GITHUB_TOKEN=… ./bootstrap.sh flux`
printed "Collecting secrets... " and exited, with no message, no diagnosis, and
no cleanup — the EXIT trap does not run either. It now reads the variables the
documentation names.
A second `trap ... EXIT` replaces the first rather than chaining, and the
secrets-cleanup handler was registered after the vmtest teardown. A workstation
rehearsal therefore left four VMs, their bridges and the sudo keepalive running,
and ignored VMTEST_KEEP — exactly the "next run collides with the last" failure
the teardown exists to prevent, and only on the laptop path. There is now one
handler, and the others register with it.
wait_kustomization and wait_certificates returned 1 on timeout, called bare under
`set -e`. They printed "continuing anyway" and then ended the run — every wait
after the first slow app never ran, and the result read as "bootstrap broke at
apps" when apps was merely slower than a guessed bound.
The `grep -c Ready` bug survived in the verification block, where it was reported
as fixed elsewhere: NotReady contains Ready, so a cluster with no ready node
printed 3/3. `grep -c True` on Kustomizations and HelmReleases matched the
SUSPENDED column, so a Kustomization Flux is not reconciling read as ready. Both
now match the field whole, verified against live output — READY is column 4 for
Kustomizations and 5 for HelmReleases with -A. awk also avoids grep's exit 1 on
zero matches, which under pipefail had ended the run with no output.
The in-cluster run failed with the VM cluster unable to reach Gitea at
10.2.99.86 — and that is not a fault in the test environment. Gitea runs inside
the cluster being rebuilt, so in a real cold rebuild it is exactly as
unreachable. The test reproduced the disaster condition faithfully; the answer is
the one already in this script, FLUX_SOURCE=github.
Worth recording that the first diagnosis was wrong: nested NAT and a VXLAN MTU
looked like the cause, and the provisioner has no MTU flag, so the next step was
going to be measuring inside the VMs. The reachability was never the problem.
The secrets list demanded a Gitea token regardless of FLUX_SOURCE, so a run
rehearsing the break-glass path could not have obtained its token — the rehearsal
was wired to need the one thing a disaster removes. One entry now covers both and
is handed to whichever variable the chosen source reads.
The recovery path must not be the one that only runs during a recovery. Secrets
were read where they were needed — from pass off-cluster, from files in the Job —
and the upfront check existed only in the Job. So a missing key surfaced an hour
into a run, and the laptop branch, the one that matters most, had no check at
all.
One function now collects all of them into one directory before anything else
happens: on a laptop by running the commands the list names, in the Job by
waiting for the operator to stream them in. Everything downstream reads files and
cannot tell the two apart, which is the point — the difference between a
rehearsal and a real recovery is confined to those few lines.
The paths live in that list and nowhere else. They were also sitting in
OPENBAO_PASS_* constants, which is how one value ends up defined twice and a copy
goes stale — as had already happened here with a gitea pass path that does not
exist.
Verified on the workstation: all five collected, the directory is 700, and the
trap removes it on exit.
The test wants about 14.5G — one control plane at 2.5G and three OSD workers at
4G. The workstation has 31G with a desktop on it: a single run left under 1G
free, and when a killed run's teardown could not get sudo and the next run
started beside it, the OOM killer took the session. The nodes have ~94G each
with roughly 70G unused, and /dev/kvm is present on them.
Three pieces:
ci/images/vmtest carries what the phase shells out to — qemu, talosctl, kubectl,
flux, helm, yq, jq. Separate from ci-k8s rather than built on it, because that
image pins talosctl v1.12.6 while this creates a cluster at the version
cluster.yaml installs; a client older than the Talos it provisions is the kind of
skew this test exists to catch.
kubernetes/vmtest-job.yaml runs it. Deliberately outside kubernetes/clusters and
so not reconciled by Flux — a Kustomization would recreate a one-shot Job after
every completion.
The phase learned that root is not always sudo: in the Job the process is already
root and sudo is not installed, so the elevation became a variable. The
keep-alive that stops sudo's timestamp expiring mid-run applies only off-cluster.
Two things are deliberate rather than incidental:
The pod is privileged. It needs /dev/kvm and it builds bridges and taps. It does
NOT need hostNetwork — the provisioner's network lives in the pod's own
namespace and the script talking to 10.5.0.x runs in the same pod — and it does
not mount a service account token, so a stray kubectl cannot reach the cluster it
is running in.
The memory limit is not decoration. The failure it guards against has already
happened once, and on a node it would take real workloads rather than a session.
What stays on the workstation is the flux and OpenBao half. That needs the unseal
key from the operator's pass store, and putting the key that protects OpenBao
into a pod inside the cluster OpenBao secures gives up the reason it lives
outside. It costs almost no memory, which is the half the workstation can afford.
The documented cold rebuild could not work. Velero's bucket credentials come from
an ExternalSecret, which needs the ClusterSecretStore, which needs an unsealed
OpenBao — so Velero cannot be the thing that restores OpenBao. That circle is why
the runbook falls back to a human with `kubectl cp`, and why the migration doc
still says "Still owed: one tested restore".
Nothing new is produced to break it. The snapshots are already offsite: the
nightly CronJob writes them to a PVC and the openbao-daily Schedule ships that
PVC through Velero's data mover into a kopia repository. Verified from outside
the cluster — seven daily snapshots, newest 839KB from this morning.
The phase reads them directly with kopia. All three things it needs live outside
the cluster, which is the whole point: bucket credentials from pass (their own
key, not Velero's), the kopia binary, and the repository password — which is not
a secret, Velero hardcodes `static-passw0rd` in pkg/repository/keys/keys.go.
Worth stating plainly: the offsite backup's confidentiality therefore rests on
bucket access control alone.
The restore sequence follows from how snapshots and seal keys interact. A
snapshot can only be loaded into a running server, so the empty one is
initialised and unsealed with a throwaway key that exists for a few seconds and
is never stored. `-force` is then required, because OpenBao otherwise refuses a
snapshot whose keyring does not match the cluster's own — which is exactly the
case here. Restoring replaces the keyring, leaving the server sealed against the
original key, and the existing unseal step carries on from there with the key
from pass.
Guarded on the one case that destroys data: this runs only when OpenBao reports
initialized: false. Against an initialised server nothing happens. And with no
unseal key in pass it stops rather than initialising a fresh OpenBao, because
that is a decision and not a step.
Adds kopia and jq as requirements, both checked with a message rather than
assumed.
Never executed yet — the run this enables is the test.
A cold rebuild could not reach a reconciled state without someone at a keyboard.
infrastructure-crds waits on the ClusterSecretStore, that needs an unsealed
OpenBao, and every app Kustomization waits on infrastructure-crds — so eight
typed commands sat in the middle of the disaster path. The VM test made this
concrete: everything downstream simply stops there.
The flux phase now unseals every replica before it starts waiting on
infrastructure. The unseal key stays exactly where it was, in the operator's
`pass` store, and is read at run time; it is never written to the repo and never
reaches CI, which is why this was left manual to begin with. It goes to the pod
on stdin rather than as an argument, so it does not appear in a process list.
Idempotent by design: a replica that is already unsealed is skipped, so this is
safe inside `bootstrap.sh all` rather than being a one-shot.
What it deliberately does NOT do is initialise. `bao status` distinguishes sealed
from uninitialised, and the two are different situations: a rebuild restores a
raft snapshot and reuses the existing key, a genuinely new cluster runs `bao
operator init`. Initialising over an OpenBao that already holds data is how the
data is lost, so an uninitialised server stops the run with a message naming
which of the two applies. Restoring the snapshot is still a human decision and
still manual.
Also clears a cluster left behind by a killed run before creating a new one. A
run that is killed never reaches its EXIT trap, and the leftover VMs keep the
provisioner's CIDR — the next start then failed with what looked like a
networking fault.
The phase stopped after the Talos phase, which is the part that needs nothing but
the machines and can therefore run unattended. VMTEST_FULL=1 continues into flux
and wait-apps instead — the same code a real rebuild runs, against VMs.
It is opt-in because it needs two things the Talos half does not: GITEA_TOKEN,
and a branch for `flux bootstrap` to commit its sync manifests into. That branch
has to exist already and must not be master, so the phase refuses to start
otherwise rather than discovering it after the VMs are up.
Nothing else changes: the phase still sets the site variables and falls through
into the real phases rather than reimplementing them, and the teardown still runs
whatever happens.
The formation wait failed with every node listed as unregistered, and the error
it now prints was the same one that broke the bootstrap call:
PermissionDenied: no request forwarding
`talosctl get members` went through the configured endpoint list rather than to a
specific machine, and apid will not forward it. It happened to work on some runs
and not others, depending on which endpoint was picked.
The second problem is worse and is not about VMs. Membership comes from the
discovery service, because the Kubernetes registry is disabled in the generated
config. So this step made a cold rebuild wait on discovery.talos.dev being
reachable, before the cluster being rebuilt exists — an internet dependency in
the middle of the disaster path, gating the bootstrap.
Neither is needed. What has to be true before etcd is bootstrapped is local: the
config took, the machine rebooted into it, and it is the machine we think it is.
The wait now asks each address directly for its hostname and expects the name its
own patch assigns — which also catches a patch landing on the wrong machine,
something the member list never checked.
The failure names the address, what it expected, and what it last answered.
Four nodes Ready says nothing about whether Ceph could use them. The CephCluster
picks OSD nodes by a label out of the machine config and their device by the
partition label RawVolumeConfig creates, so the phase now asserts both on every
worker: the r-ceph-osd partition exists, and the Kubernetes node carries
storage.renner.uno/ceph-osd.
Checking this by hand afterwards does not work — the run's EXIT trap destroys the
VMs as soon as it finishes, and that race was lost on the last attempt, leaving
the partition confirmed on exactly one worker from an earlier failed run.
Also drops the second statement of how many VMs there are. The node patch lists
are the count now, rather than the flags and the progress line each carrying
their own copy, and the addresses follow from the list instead of being spelled
out. That list had already gone stale once when the topology moved from three
control planes to one plus three workers.
The four-VM run reported:
error executing bootstrap: rpc error: code = PermissionDenied
desc = no request forwarding
and the cluster then sat idle. Nothing downstream noticed: `talosctl kubeconfig`
still succeeded, because Talos issues credentials as soon as it can rather than
when kube-apiserver serves, so the run carried on to the API wait as if it had
bootstrapped.
etcd bootstrap is the one call apid will not forward — reached through any node
other than the target it fails outright. `talosctl bootstrap` with neither -n nor
-e picks an endpoint from the configured list, and that list holds every control
plane, plus the workers in the vmtest phase. Whether it landed on the right node
was chance, in a real rebuild as much as on VMs.
Only visible because the previous commit stopped discarding this error.
The four-VM run died on "Talos cluster didn't form" after 150s. The message was
wrong twice over: the wait was too short, and it could not have told the
difference between an incomplete member list and no node answering at all,
because talosctl's error went to /dev/null.
Membership here comes only from the discovery service — the Kubernetes registry
is disabled — so a node registers after it has installed to disk and rebooted.
Four machines do not finish that in 150s. The timeout is now 10 minutes, and the
failure names the nodes that never registered, the last talosctl error, and the
member list it actually saw.
Three more of the same shape in this phase:
`talosctl bootstrap` discarded every error. The `|| true` exists because
re-running against an already-bootstrapped cluster is fine, but it hid genuine
failures too, and the run then continued to break somewhere further on with
nothing pointing back here. Anything that is not "already exists" is now
reported.
`talosctl kubeconfig` held its error while retrying and then failed with
"Couldn't get kubeconfig" and no reason. The reason is now printed when the
retries run out.
The node-ready wait counted with `grep -c "Ready"`, which also matches NotReady —
the substring is in it. So it waited for nodes to register, not to become ready,
and would report success with every node still NotReady. It now matches the
STATUS column whole. This is why the previous run's "3 nodes Ready" claim was
not evidence of readiness; whether those nodes were in fact ready is unknown.
The CephCluster named three hostnames and three drive serials. That tied the
storage layer to these three boxes as firmly as the Talos config used to be tied
to them, and for the same reason: it said in a Kubernetes manifest something only
the machine knows.
Now the machine says it, twice:
nodeLabels storage.renner.uno/ceph-osd which machines carry storage.
placement.osd.nodeAffinity reads it,
so useAllNodes can be true without
putting OSDs everywhere — that
nodeAffinity is the only thing
keeping them off diskless nodes.
RawVolumeConfig ceph-osd which disk, chosen by a CEL
expression per machine. The
partition surfaces at
/dev/disk/by-partlabel/r-ceph-osd
and that is what Rook consumes.
devicePathFilter carries both that partlabel and the OSD model for now. The
three production drives hold bluestore written straight onto the raw disk, with
no partition table and so no partition label to match; they are migrated
separately, one node at a time, and the by-id half goes when they are. Matching
the remaining half by model rather than by three individual serials already means
a drive swap is not a repo edit.
Resource requests now follow what the daemons use, measured on the running
cluster: mon ~420Mi against a 2Gi request, mgr ~650Mi against 4Gi, OSDs ~2.2Gi
against 4Gi — roughly 17GiB reserved cluster-wide and never touched. Limits stay
where they were, so burst headroom is unchanged; only the reservation moves.
Guaranteed QoS is given up deliberately, and priorityClassNames already pins
these to system-node-critical, which is what actually orders eviction. The OSD
request is set above measured use on purpose: an OSD grows into Ceph's
osd_memory_target, so 2.2Gi is a floor rather than a ceiling.
The vmtest phase becomes one control plane and three workers. Workers because
the QEMU provisioner gives the second disk only to worker machines and an OSD
needs one; three because the mons run with allowMultiplePerNode false and the
pools are replica 3 over failureDomain host. The apply loop now walks ALL_NODES
once instead of once per role, so NODE_ADDRS stays a single ordered list.
Not yet verified: whether Ceph actually reaches Ready on the VMs. That is the
point of the run this enables, and the memory budget is tight enough that it may
not — 2.5GiB plus three times 4GiB against 15.4GiB available.
Both bootstrap calls hardcoded --branch=master, which is why the flux phase could
never be tried before it was needed: `flux bootstrap` commits its own sync
manifests into the branch it reconciles, so running it wrote to the source a
rebuild depends on.
FLUX_BRANCH defaults to master and can be pointed at a throwaway branch. That
branch has to exist first — `flux bootstrap` clones it and fails with
`couldn't find remote ref` rather than creating it, which is not obvious and is
why the comment spells the two commands out.
Rehearsed against a scratch cluster to establish that: `flux bootstrap` clones
the named branch, commits its manifests there, and installs the controllers. The
waits after it need real disks and were not exercised.
Nothing else about the phase changes, which is the point — a rehearsal that takes
a different code path proves less than one that takes the same one.
bootstrap.sh generated a config and then edited it with awk before applying it,
cutting out the HostnameConfig document `talosctl gen config` emits. That was
added in April when Talos 1.12 started emitting it, and it has been load-bearing
since: without it apid rejects apply-config outright with
static hostname is already set in v1alpha1 config
The document was never the problem. Naming the host in two schemas at once is:
machine.network.hostname and machine.features.stableHostname are the v1alpha1
spellings of what the generated document already says. Setting the name in the
document instead — with `auto: off`, which is otherwise mutually exclusive with
a static name — removes the conflict, and with it the awk.
That matters beyond tidiness. Post-processing the generator's output meant the
config could only be produced by this one script: anything else that ran
`talosctl gen config` against these patches, including `talosctl cluster create`,
got a config the nodes refuse. Which is why none of this had ever been tried
outside a real rebuild.
Two more values moved out of cluster.yaml for the same reason — they described
this network rather than this cluster, so a config generated for anywhere else
was wrong before it was applied:
kubelet.nodeIP.validSubnets now talos/patches/site-homelab.yaml, a third
layer between the cluster and the machine. It
pinned every node to 10.2.40.0/24 whether or not
it has an interface there.
cluster.controlPlane.endpoint removed outright. `gen config` takes the
endpoint as its second argument and bootstrap.sh
already passes it; restating it here overrode
whatever a caller had generated for.
With that, `./bootstrap.sh vmtest` provisions three throwaway QEMU VMs and runs
the talos phase against them unchanged. It is deliberately not a copy of that
phase: it sets NODES, NODE_ADDRS, SITE and K8S_API_HOST and falls through, so a
passing test says something about a rebuild rather than about itself. Talos
version and Image Factory schematic are read out of install.image in
cluster.yaml rather than named again.
It found two real defects on its first run, both fixed here:
The phase used the kubeconfig the moment `talosctl kubeconfig` returned, which is
when Talos can issue credentials — not when kube-apiserver is serving on the
endpoint the file points at. Helm hit `EOF` against a port nothing was listening
on. It now waits for `kubectl get --raw /version`, which needs the server;
`kubectl version` would have answered from the client and passed against nothing.
`echo` without -e printed raw escape codes instead of colouring the per-node
result.
TALOSCONFIG/KUBECONFIG are honoured for the access files instead of hardcoded
$HOME paths, unset resolving to the same two paths as before. That is what keeps
a test run from replacing the operator's production access with a VM's; access-
staging is skipped under vmtest for the same reason.
Verified: three VMs Ready on Cilium 1.20.1 from the HelmRelease, and all four
node patches generate and validate with no post-processing at all.
Not covered, and still only provable on hardware: Ceph (the provisioner gives
extra disks to workers only), the mgmt and v30iot VLANs, the Layer2 VIP, Flux,
and applying the three changed production patches to the real nodes.
The name says the bootstrap depends on a registry mirror. Reading the patch list
in the runbook, that is the obvious conclusion — and it would be a serious
problem, because a cold rebuild has no mirror: an in-cluster one cannot exist
before the cluster does.
The file configures no mirror. It writes one containerd setting,
discard_unpacked_layers = false, so that pulled layers stay in the content store
and Spegel can advertise them to peers later. Spegel writes the per-registry
hosts.toml itself at runtime; the file's own comment says a
machine.registries.mirrors block is not needed, and there is none anywhere in
talos/patches. Image pulls at bootstrap go straight to the real registries.
So the alarm the name raises is false, but it is the name doing it, and the cost
of that is someone deciding a rebuild is blocked when it is not.
containerd-layer-retention.yaml describes the single thing inside.
Renamed with git mv and the three references updated. Verified `talosctl gen
config` still produces a config with the renamed patch in the list.
The Cilium install had become a block of shell in a runbook for a human to
paste. That is the same rot as a pinned version — nobody runs it between
rebuilds, so nothing tells you when it stopped working. It belongs in the script
that already drives the rest of the bootstrap.
The talos phase was unusable since #581: it passed cilium-cni.yaml whole, and
that file's `$patch: delete` document fails against a config with no
inlineManifests. The file is gone, so the failure is gone with it — reviving the
phase was mostly deleting two lines that referenced it.
It now installs Cilium between `talosctl bootstrap` and the node-Ready wait,
which is the only place it can go: with `cni: none` the kubelet stays NotReady
until a CNI exists, so the wait below would otherwise time out against a cluster
that was never going to converge.
Chart repo, version constraint and values are read out of helmrelease.yaml at
run time, so the script cannot install something different from what Flux
reconciles minutes later. helm takes the semver range verbatim. There is a guard
on the extracted values — the HelmRelease is a two-document file and selecting
the wrong one yields an empty values set that helm silently renders as chart
defaults, which is a failure mode worth catching loudly rather than discovering
in a datapath.
`--wait` because the next step needs a working network, and `upgrade --install`
so a re-run after a partial bootstrap converges instead of erroring.
Verified as far as it can be without wiping a cluster: `talosctl gen config`
with exactly the patch list the script now passes produces a config carrying
cni: none and proxy: disabled, no Cilium manifests, and it validates for metal
mode. The apply-and-bootstrap half is untested since the rewrite, and the
runbook says so — it keeps the manual procedure as the thing to read first and
to fall back on.
The install command in Step 2b hardcoded --version 1.20.1 and the chart repo
URL. That is a copy of the HelmRelease in a text file nobody runs between
rebuilds, so it would have gone stale unnoticed and the next rebuild would have
installed something Flux immediately fought — the same failure the deleted
bootstrap seed produced, in a new place.
Everything now comes out of helmrelease.yaml at the moment the command runs:
the repo URL from the HelmRepository, the values from .spec.values, and
--version from .spec.chart.spec.version verbatim. Helm accepts the semver range
and resolves it exactly as Flux does, so the two cannot pick different releases.
Verified by running the block as written:
--version ">=1.20.0 <1.21.0" -> cilium:v1.20.1, operator-generic:v1.20.1
enable-bgp-control-plane: "true" bpf-lb-sock: "true"
bpf-policy-map-max: "65536" ipv4-native-routing-cidr: 10.0.0.0/8
ServiceMonitor objects: 0
which is the config the cluster is running.
The only literal left is --set prometheus.serviceMonitor.enabled=false, and it
is a bootstrap constraint rather than a value: the CRD arrives with the
monitoring stack long afterwards. Because nothing else is overridden,
helm-controller adopts this release instead of replacing it.
This does not make the step unnecessary. Flux's controllers are pods and cannot
reconcile a CNI they need in order to run; something has to seed it. What it
does mean is that the step has nothing left in it to maintain.
Both blocking findings from the review on #709, and both were real.
The deleted cilium-cni.yaml was the only place in the repo that set
cluster.network.cni.name: none and cluster.proxy.disabled: true. They sat in its
second document next to the inline manifest, so removing the file removed them
too — a fresh `talosctl gen config` from this repo's patches would have produced
a config with Talos's default CNI and kube-proxy running, against a Cilium
release configured with kubeProxyReplacement: true. Two runbooks meanwhile
asserted that cluster.yaml supplied both, which it did not.
RUNBOOK-baremetal-conversion.md also still passed `@<cilium additive half>` in
its gen-config example, three lines above prose saying there is no CNI patch any
more. The equivalent block in the cold-rebuild runbook had been fixed; this one
was missed.
Verified the way it should have been in the first place — by running the
runbook's own command rather than inspecting the live cluster:
talosctl gen config homelab https://k8s.renner.uno:6443 \
--output-types controlplane \
--config-patch @talos/patches/cluster.yaml \
--config-patch @talos/patches/registry-mirror.yaml \
--config-patch @talos/patches/k8s-1.yaml
-> cni=none proxy.disabled=true
-> 0 cilium manifests
The live cluster could never have caught this. It has carried both settings
since it was built, and the node configs were patched by hand rather than
regenerated, so nothing in the verification touched the path a rebuild actually
uses.
Final step of PROPOSAL-cilium-single-owner.md. Talos no longer carries Cilium in
any form, and the HelmRelease is the only place it is configured.
The cluster side is already done and verified:
- the cilium inline manifest is gone from all three machine configs, applied
node by node with no reboot and no effect on the running CNI
- Kubernetes went 1.35.2 -> 1.36.3 in the same operation, with
--manifests-no-prune so the 22 objects Talos claimed were not deleted
- Talos's bootstrap-manifest inventory went from 41 entries to 13, and the 28
cilium/hubble entries are gone. A prune-enabled `upgrade-k8s --dry-run`
reported 28 deletions before and reports 0 now
That last number is the point of the whole exercise. A bare `upgrade-k8s` used
to destroy the CNI and now does nothing to it.
The inventory entries had to be removed by hand. --manifests-no-prune skips the
prune AND the inventory rewrite, so the run that saved the objects also left
Talos still claiming them — the mine was deferred, not cleared. Talos offers no
way to drop a stale claim without the prune that deletes, so the ConfigMap was
edited directly. It is derived bookkeeping, not configuration, and the result is
verifiable in both directions, which is why this was acceptable where hand-
editing configuration would not be.
Repo side, this commit:
- deletes talos/patches/cilium-cni.yaml and talos/cilium.yaml
- adds Step 2b to RUNBOOK-cold-rebuild.md: install Cilium from the HelmRelease
values with helm, after bootstrap and before Flux. The nodes stay NotReady
until then, which the runbook now says out loud rather than leaving someone
to wonder
- drops the "keep in lockstep" and "THREE COPIES" comments, which described a
duplication that no longer exists, and replaces them with one note saying
where Cilium is configured
- corrects RUNBOOK-baremetal-conversion.md, which told the reader to extract
half of a file that is gone
bootstrap.sh is left alone. Its talos phase was already dead and is now dead for
a second reason; the runbook says so.
The cost, stated plainly: the cluster no longer forms itself from the machine
config. A cold rebuild needs one documented helm install that it did not need
before. That is the price of Cilium existing in exactly one place.
Step 1 of the ownership transition in PROPOSAL-cilium-single-owner.md. Nothing
about ownership changes yet — this only moves three live settings out of nowhere
and into git.
enable-bgp-control-plane and both native-routing CIDRs existed in no file at
all. They were applied by hand and are held by the `kubectl-patch` field manager
on cilium-config; server-side apply left them alone, so nothing broke and
nothing reported them. The peering itself was always in git as CRDs in
post-crds/cilium-bgp.yaml, which is what made the gap invisible: BGP looked
configured because its policy was version-controlled, while the switch that
starts the control plane was not. A cold rebuild installs only what git
describes, so it would have come up with two established-looking peers
configured and no BGP at all.
The CIDRs go with it because they are the same decision. They are a masquerading
exception, not a routing statement: pod traffic to 10.0.0.0/8 and fd00::/8
leaves unmasqueraded so LAN services see real pod IPs, and that only works
because BGP advertises the pod CIDRs and gives the UDM a route back. Tracking
one without the other would leave a half-decision in git.
This is not purely a change of ownership, which is why it is its own step and
not folded into a later one. Enabling bgpControlPlane through the chart emits
five companion keys the hand-patch never set, so Cilium has been running on its
own internal defaults for router-id allocation among others. They should agree
with the chart's; if they do not, the sessions re-establish. Verify the peers
after Flux reconciles before moving to step 2.
The proposal called the tunnel redundant because the underlay already has
routes to the pod CIDRs. That reads the two as alternatives, and they are not:
VXLAN carries pod-to-pod directly between nodes, while the BGP-advertised pod
CIDRs exist so traffic from the LAN can reach pods at all, which necessarily
goes through the UDM.
Removing the encapsulation would therefore not remove a layer. It would move
inter-node pod traffic off a direct node-to-node path and onto the underlay,
through the router.
The observation worth keeping is narrower: routingMode is set nowhere, so the
current behaviour is an unexamined chart default rather than a recorded
decision. It happens to be the right default.
Cilium is currently owned twice. The 22 objects that make up the whole
installation — both DaemonSets, the operator, both ConfigMaps, the RBAC, the
cilium-secrets namespace — are listed in Talos's bootstrap-manifest inventory
and simultaneously carry Helm's release annotations. cilium-config has three
field managers.
That is not a tidiness problem. It produced a bootstrap seed twelve days behind
the chart, where a Kubernetes upgrade would have downgraded the CNI and switched
off socket load balancing, and it left the BGP control-plane switch and both
native-routing CIDRs existing in no file at all, so a cold rebuild would have
come up without BGP while the peering CRDs in git said otherwise.
This records the decision — Flux owns Cilium, Talos provides only cni: none and
proxy: disabled — and the ordered transition, including why the order matters:
Talos prunes what its inventory used to list and no longer does, so removing the
manifest naively deletes the CNI.
Two claims in it are verified rather than assumed. An OS upgrade does not
reconcile bootstrap manifests: Talos went to 1.13.9 four days ago and Cilium
stayed at 1.20.1 despite the seed saying 1.19.2. And the prune can be tested
before it is run, because NoPrune is honoured by the dry run as well as the
apply.
No code changes here. PR #706 stays WIP until this is agreed.
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.
What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.
So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".
Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.
Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.
BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.
The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.
BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.
Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.
Both were unmonitored. Neither is a leaf service: Woodpecker runs every cron
and deploy in this repo, and every ExternalSecret in the cluster resolves
through OpenBao.
OpenBao is probed at /v1/sys/health rather than at the root, because a sealed
instance still serves its UI. Root would answer 307 -> /ui/ -> 200 while every
secret lookup failed — the exact green-while-broken shape probes-blackbox.yaml
already warns about for the gated monitoring UIs.
standbyok=true is required, not cosmetic: the endpoint returns 200 only on the
active node and 429 on a standby, and the external route currently lands on a
standby. Measured — the bare path returns 429 right now, so an http_2xx probe
without the parameter would have been red from the first scrape. With it, 200
means initialized, unsealed and serving; 503 sealed and 501 uninitialized stay
red.
The alert matches OpenBao by instance prefix instead of the full target string,
because that target carries a query parameter and an exact match would fail
silently if the label ever encodes it differently.
Both alerts are new rules rather than additions to an existing file. Worth
noting while here: there is no generic blackbox-down rule. Only fsn1, netbird
and the two relays have alerts, so gitea, immich, auth, home, prometheus,
alerts and dash are probed and graphed but nothing fires if they go down. Not
addressed here — it needs a decision about whether to write one generic rule
and retire the per-instance ones.
The scan loop swallowed every Trivy failure with `|| true`, so a step that
scanned 21 of 22 images reported exactly the same as one that scanned all 22.
The gap was found by the fix agent noticing a missing file in its own input,
not by the pipeline.
The loop still continues past a bad image — one unpullable reference should not
cost the scan of everything after it — but now counts what failed, names it,
and exits non-zero:
--- scanned 21 of 22 images ---
NOT SCANNED: proxmox/proxmox-backup-client:latest
`failure: ignore` stays, which per Woodpecker's semantics reports the step as
failed while letting the workflow continue. That is deliberate: coverage is
reported, not enforced. The step goes red and the fix agent still runs on what
was scanned. Making the whole pipeline fail would mean one unscannable image
blocks all remediation.
The image that prompted this is gone with the nas-1 removal in the previous
commit, so the next run should report 20 of 20. The accounting is what matters
— the next dead reference will announce itself instead of being absorbed.
Refs #694
nas-1 has been off for months. Everything scoped to it went with it:
inventory/host_vars/nas-1.renner.uno/ vars and vault
[synology_nas] in pve_hosts.yml the group's only member
roles/pbs-client/ PBS backup client
plays/install-pbs-client.yml role-based, hosts: synology_nas
plays/install-pbs-client-synology.yml raw-module variant, same host
plays/install-garage-synology.yml Garage S3, replaced by Hetzner
.woodpecker/install-garage-synology.yml the pipeline that ran it
The two PBS plays were already inconsistent with each other before the host
went away: the role pulled aterfax/pbs-client, while the standalone play
pulled proxmox/proxmox-backup-client, which has never existed on Docker Hub
(404). Nothing referenced the standalone play at all. Both are moot now.
pve_hosts.yml keeps its "gone from this file, and why" list, so synology_nas is
recorded there alongside pve_cluster and the in-cluster migrations rather than
vanishing without trace.
Checked: `ansible-inventory --graph` resolves, and grep finds no remaining
reference to synology_nas, nas-1, pbs-client or the garage play outside that
one note. plays/archived/ is left alone — it is history, not live config.
file.renner.uno served HTTP 502. The Service is an ExternalName to
nas.renner.uno:7001, a CNAME to nas-1, and nas-1 has been off for months. The
Ingress and Service had been in the cluster for 110 days pointing at nothing.
Nothing caught it because file.renner.uno is not in the blackbox probe list, so
a publicly resolvable hostname returned 502 for months without a single alert.
The insecure-backend ServersTransport goes with it. It existed to let Traefik
re-encrypt to the NAS web UI's self-signed certificate and apps/file was its
only consumer, which its own header comment documents.
Rolls the security-scan pilot out to the other thirteen. Every workflow-root
gate of the form
- event: manual
evaluate: 'PIPELINE == "x"'
becomes
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "x"'
The target rides in pipeline metadata rather than in the manual-run variable
map, so it reaches the workflow-root filter on Woodpecker 3.18 where PIPELINE
does not. Verified on the pilot: pipeline 7004 selected the security-scan
workflow and nothing else.
Two files also carry step-level manual gates, and those had to move with the
root or the pipelines would have converted into something that runs but cannot
do its job:
deploy-infrastructure the apply step gates on TOFU_APPLY. Left on
`event: manual` it would never fire again, so the
pipeline would plan and silently never apply.
TOFU_APPLY now arrives via `--param`, which lands in
the same non-plugin environment step-level `when`
evaluates against. No step here is a plugin — nothing
in .woodpecker uses `settings:` — so that path holds.
build-ci-images eight per-image steps gate on manual-or-push to decide
whether to rebuild. All eight follow the root.
Cron gates are untouched. They already matched by name everywhere except
security-scan, which the pilot fixed.
Also corrects the trigger documentation, including a claim in update-audit.yml
that the other pipelines gate on "an injected PIPELINE var set at cron-create
time in the UI". Crons cannot carry variables at all — `repo cron add` takes
name, branch, schedule and enabled. That sentence described a mechanism that
never existed and was the same misunderstanding that left the daily security
scan silently not running.
The `timeout: 30` lint warning on the four deploy-* files predates this and is
left alone.
Two separate bugs in one `when` block, only one of which is #582.
The cron gate has never matched. It asked for `PIPELINE == "security-scan"` on
`event: cron`, and crons cannot carry variables — `repo cron add` takes name,
branch, schedule and enabled, and nothing else. So the variable is undefined on
every cron run and the workflow is filtered out. The `daily-security-scan` cron
does fire, and pipeline 6877 shows what actually ran under it: the `e2e`
workflow alone, which opts into that cron by name. The Trivy scan and the
security-fix agent have not been running on schedule. This predates 3.18.0 and
would have survived any fix to it.
The manual gate is the #582 case. Woodpecker 3.18 stopped merging manual-run
variables into the environment the workflow-root filter evaluates against
(builder.go:102-111 builds it from metadata plus global envs; AdditionalEnvs is
passed to the compiler at line 223 but never into this map), so
`PIPELINE == "security-scan"` cannot be true there either.
Both now use selectors the root filter can actually see:
cron -> matched by the cron's NAME, the way alert-monitor and e2e
already do it. First-class filter, no variable involved.
deployment -> CI_PIPELINE_DEPLOY_TARGET, which comes from pipeline metadata
(metadata/environment.go:74) and therefore reaches the root
filter on 3.18.
Dispatch becomes `woodpecker-cli pipeline deploy k.renner/homelab <n>
security-scan` — a named deployment target rather than a magic variable that
exists only to pick which of fourteen files is allowed to run.
Pilot for the remaining thirteen. Nothing else is converted until this one is
confirmed working end to end.
The comment blamed #522 — Backup CRs deleted within a minute of completing, so
CNPG has no record to derive the metric from. That is not what is happening.
The CRs are retained: 31 completed backups for ak-pg, im-pg and pprl-pg, 19 for
tr-pg, 11 for wp-pg, 10 for ha-pg, the newest of each from this morning.
The metric still reads 0 everywhere. The actual cause is one level up: the
collector derives it from Cluster.status.lastSuccessfulBackup, and that field
is absent on every Cluster because they all back up with method: plugin and the
barman-cloud plugin path never writes it back. lastFailedBackup is populated on
three of them, which is what makes the absence look like "no backups" rather
than "no reporting".
The conclusion does not change — the alert still cannot be built on that metric
— but the stated blocker does, and #522 is not it. Whoever picks this up next
would have gone looking at CR retention and found nothing wrong.
Also drops the incident retellings and the dated "verified on" stamps from this
file's comments, which is the cleanup already applied elsewhere in the repo.
Chasing a stale comment turned up a live duplicate. Since serviceMonitorSelector
was loosened to {}, Prometheus discovers the goauthentik.io operator's own
ServiceMonitors — which is why it was loosened. The hand-written copies for ldap
and paperless-gate, added back when the strict selector was dropping the
operator's, now scrape the same endpoints a second time.
Measured, not assumed: 10.244.4.113:9300 and 10.244.4.76:9300 were each
producing 40 samples per 10 minutes against 20 for every other outpost, and
ldap picked up a third target because the hand-written selector matched the
plain Service as well as the -metrics one. No duplicate-timestamp errors,
because the two pools scrape at different offsets — the cost is doubled
ingestion and a target list that lies about how many outposts there are.
Radius stays, and the comment now says why it is not the same case: the
operator's ServiceMonitor for it selects on managed-by=goauthentik.io together
with service-type=metrics, and no Service carries both — the operator's has the
first, the one defined here has the second. It is the only thing scraping
radius, so deleting it would have been a silent loss.
Also corrects the comments that claimed the release label is required for
discovery. It is not, for ServiceMonitors and PodMonitors: those two selectors
are empty. It very much is for Probes, PrometheusRules and ScrapeConfigs, whose
selectors still match on it, so probes-blackbox.yaml now says so explicitly —
otherwise the corrected comments elsewhere read as license to drop the label
there too.
Live selector state, for the record:
serviceMonitorSelector {} podMonitorSelector {}
probeSelector / ruleSelector / scrapeConfigSelector release=kube-prometheus-stack
Deleting the wear warning outright went too far. Wear that is high AND still
climbing is exactly the thing worth knowing about; only the level on its own
is noise. This replaces the flat 80% threshold with the combination.
The rule now projects: at the write rate of the last seven days, when does the
drive reach 95%? Under six months, and above 80% already, it alerts. The
projection clears when writes slow, which the level never could.
The rate is self-calibrating. Each drive reports both lifetime bytes written
and its own wear percentage, so bytes-per-percent is derived from the drive
rather than from a rated-endurance table that would have to be kept in sync
with the hardware. That matters here because the fleet is three different
models with three different endurance ratings sharing one rule.
It also has to work that way. percentage_used is a whole-number percent and
advances roughly once every six weeks on these drives, so any rate taken from
it directly is either zero or a single step — far too coarse to project from,
and the reason a delta- or predict_linear-based rule would have been useless.
bytes_written moves on every scrape.
The >= 80 gate is numerical rather than editorial: bytes-per-percent divides
by an integer percentage, so on a lightly worn drive half a percent of
rounding swings the projection by tens of percent. Idle drives divide by a
zero rate, yield +Inf, and stay silent without needing a special case.
Measured against live Prometheus before committing: the expression returns
nothing today, and with the horizon widened it selects exactly the three
Crucial drives at 81/83/87%, projecting 7822, 6513 and 3544 days to 95%. High,
flat, and roughly a decade of headroom — which is why the old rule was firing
every fifteen minutes about nothing.
The 95% critical alert stays a flat threshold. Past rated endurance the drive
is outside what the manufacturer stands behind however fast it got there, and
that alert clears by replacing the drive.
Closes#686
NvmeDriveWearHigh has produced four identical issues since yesterday — #673,
#675, #682, #686 — and would have kept producing them every fifteen minutes.
The alert-monitor deduplicates against OPEN issues, so closing one just makes
room for the next.
That is not a bug in the pipeline. It is a bug in the rule I wrote yesterday:
wear only ever increases, so a `>= 80` threshold starts firing the day a drive
crosses it and never stops. It is a status dressed as an alert, and no
suppression mechanism fixes a rule that is permanently true by construction.
The second half is worse. The drives it fired on are the EPHEMERAL volumes,
worn deliberately — they hold /var precisely because their remaining endurance
does not matter there. The alert was reporting the design as a fault, three
times over, for drives at 81/83/87% that are doing exactly what they were put
there to do.
So the warning goes and the critical threshold stays. 95% is a decision — the
drive is outside what the manufacturer stands behind — and a decision is what
an alert is for. The wear figure itself lives on the Disk Health dashboard,
per drive and with the role attached, which is where a slowly-moving budget
belongs.
The comment left behind says all of this, because the obvious reaction to
"there is no wear warning" is to add one back.
Not done here, deliberately: the noise log (#326) would also have silenced
this, and it is the designed path for an alert that fires continuously without
being repo-fixable. It is the wrong tool here — the alert was fixable, and
suppressing a bad rule leaves the bad rule.
Closes#686
559 -> 509 comment lines. A far smaller cut than the other files, and that is
the honest result: most of this file is per-entry rationale for a declarative
blueprint, where the comment is the only place the constraint is written down.
Everything load-bearing stays, and stays emphasised. Every ORDER MATTERS note
survives — Application before Provider, Application before Provider before
Outpost — because Application.provider is on_delete=PROTECT and getting the
order wrong leaves the blueprint failing to apply with no obvious cause. So do
both "git owns membership, add and remove people HERE not in the UI" notes,
since blueprint apply silently reverts UI edits to those groups. So does the
reason each gate exists at all: code-server runs --auth=none as root over
plaintext HA secrets, and shelly-manager has no authentication whatsoever.
Cut: the derivations. Why the dual-OAuth2 setup was collapsed into one
provider, the full W3C-spec account of credentials:'omit', a paragraph
reconstructing how the music-assistant gate failed. The instruction that came
out of that last one is kept and sharpened into "do not reintroduce a gate in
front of a PWA".
Two stale things went with it. The paperless proxy comment said the ingress
controller "only routes paperless.renner.uno to the outpost service" and named
ingress-nginx as that controller — there is no ingress-nginx left, Traefik is
the only IngressClass. And a leftover instruction to "delete via the Authentik
UI once everything's confirmed working" from a session long finished.
Verified beyond the usual: these comments live INSIDE the embedded blueprint
string, so a broken indent would render fine and only fail when Authentik
parses it. Extracted the rendered ConfigMap's k8s-outposts.yaml and parsed it
separately — valid, and still 74 entries.
Comments only across otbr, trmnl/renderer, zigbee2mqtt-deployment and the
smartctl ServiceMonitor. All four still parse; monitoring-rules still renders.
Each keeps the instruction and loses the investigation. otbr stays disabled with
the condition for re-enabling it — verified partition merge, one OMR prefix, the
Nest routers visible — instead of the paragraph reconstructing how the split
partition was diagnosed. Its socat bridge keeps both dead ends, because they are
what stops someone simplifying it: forkpty needs a socat the image does not ship,
and a sidecar cannot supply the PTY because devpts is per-container. zigbee2mqtt
keeps "never run alongside ZHA, and never start without the restored coordinator
backup or you form a new network and drop every device", and loses the cutover
diary around it.
The ServiceMonitor correction is the useful part. Four monitors carry the claim
that kube-prometheus-stack's Prometheus "selects ServiceMonitors by this label;
without it the resource exists but is never discovered". The live resource has
serviceMonitorSelector {} and podMonitorSelector {} — empty, so every monitor is
picked up regardless of labels. Only ruleSelector matches on
release=kube-prometheus-stack, which is why the same sentence on the disk-wear
PrometheusRule is correct and this one is not.
Corrected here for the monitor I added yesterday, which is where I introduced it
by copying a neighbour. The other three carry the same wrong sentence and are
left for a pass that can check each rather than sed across them.
mdns-relay 122 -> 64 comment lines, mosquitto 131 -> 118. Comments only; the
diff filtered for non-comment lines is empty and apps/home-assistant still
renders 83 objects.
The mdns-relay header was mine, written yesterday, and was the clearest case
of the problem: a seventy-line essay with sections. What survives is what stops
someone undoing it — that the relay must stay unicast because Cilium runs in
tunnel mode without enable-multicast and drops re-emitted multicast, so an
avahi-style reflector cannot work; that python-zeroconf accepts unicast at all,
which is the non-obvious fact the design rests on; that it restores
announcements but not HA-initiated browsing; and that so-bindtodevice=net1 is
what keeps any sibling pod from injecting records into HA's zeroconf cache.
Cut: the problem statement, the Errno 113 symptom, the exposure analysis
weighing what the IoT VLAN could already forge, and a paragraph explaining that
the relay is not what makes HA's listener reachable in-cluster.
mosquitto keeps every warning and loses the reasoning around them. The one
worth naming is the retained-state note: a roll preserves connectivity but not
the retained store, and Zigbee2MQTT does not recover on its own. That is an
operational consequence someone needs before touching the rollout strategy, so
it stays — the paragraph explaining why maxSurge:1 is safe does not.
crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp
and the openbao rotation kustomization. Comments only; the diff filtered for
non-comment lines is empty and infrastructure/ still builds.
Every instruction survives, because that is the half worth having: do not
switch crowdsec back to password auth, or a wiped LAPI crashloops the agents
and wedges the Kustomization; do not move netbird-ingress off the rootless
image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not
pin the rotators by digest, or a quarterly job fails to pull a manifest that
was garbage-collected months earlier and looks like it simply did not run.
What went is the incident each was learned from, and one duplicate: the
san-network-attachment header re-derived the provider: host argument that
cephcluster.yaml already makes, so it now points there instead.
Two more stale claims found on the way through:
- woodpecker-ns.yaml described the namespace as holding wp-pg alone with
'server and agent arriving in stage 2'. Both have been running there for
some time.
- cilium-bgp.yaml said each node sources its BGP session from eth0. The
management interface is mgmt0; eth0 was the VM-era name.
infrastructure/ is now 2553 comment lines.
cilium 86 -> 63 comment lines, traefik 71 -> 59. Comments only; the diff
filtered for non-comment lines is empty and both files still parse.
Cilium keeps everything that is a trap rather than a story: that bpf-policy-map-max
and vlan-bpf-bypass each exist in three files with nothing enforcing agreement,
that both take an agent restart, that a full policy map surfaces as a pod stuck
in sandbox creation, and why only VLAN 30 belongs in the bypass list. What went
is how each was diagnosed — the Hubble traces, the manual agent deletions, the
device-by-device walk through `cilium-dbg status`.
Traefik keeps the version ceiling and the reason it exists, and keeps the
CrowdSec bouncer note as an instruction: do not re-attach until client-IP
visibility is fixed and a LAN whitelist exists. It loses the retelling of the
outage that produced that instruction.
The traefik header was outright wrong and is the reason this file was worth
opening. It described Traefik as running "alongside ingress-nginx during the
migration", called ingress-nginx "the default IngressClass" and Traefik
"opt-in via ingressClassName: traefik". There is one IngressClass in the
cluster and it is traefik; ingress-nginx has no deployment and no class. A
reader following that comment would have added an ingressClassName they do not
need and looked for a controller that is not there.
152 comment lines to 125 — a much lighter cut than the previous batches, on
purpose. Most of this file is per-rule security rationale, and that is exactly
the category worth keeping: someone deleting the :8124 rule needs to know it is
the trust anchor for X-Forwarded-For, and someone deleting a gate rule needs to
know it is what stands between an --auth=none editor and the LAN.
What was cut is repetition. Four near-identical paragraphs explained the
Authentik gate pattern once per app, three of them re-deriving that
endpointSelector {} makes a :8080 grant namespace-wide. That warning is now
stated once, at the first :8080 rule, and it is stronger for being in one
place: three apps share that port, so changing any of those rules affects all
three.
The world-ingress and ICMP blocks keep their rules and lose their incident
reports — do not add a blanket fromEntities: world to an endpointSelector {}
policy; ICMP needs an explicit allow once any L4 rule exists and Cilium takes
one type per entry.
Two stale references went with it. Egress-to-world was documented as reaching
"garage S3" and "legacy InfluxDB on VM 108": Garage was retired to nas-1 and
there is no Proxmox left to host VM 108. Both replaced by what the rule
actually grants rather than a list of destinations that has already changed.
Comments only; the diff filtered for non-comment lines is empty, and the file
still passes a server-side dry run.
Comments only — filtering the diff for non-comment lines returns empty, and
both files still parse and build.
cephcluster.yaml goes from 224 comment lines to 76. Kept: that `ceph config`
retains values which merely disappear from this file, so deleting a line does
not restore a default and a live provider change needs `ceph config rm` first;
that osdMaxUpdatesInParallel and upgradeOSDRequiresHealthyPGs are the only
things keeping a rolling OSD change off all three replicas at once; that
`addressRanges` drags the mons onto host networking and ties them to a DHCP
lease; that devices are named by-id because kernel names move between boots;
that the bluestore deferral settings must not be raised again.
Cut from it: the paxos write-amplification measurement down to the byte, the
history of why mon_data_avail_warn was once 20, and a table of every drive
serial in the fleet sitting three lines above the by-id paths that already
name them.
velero-schedules.yaml loses a per-PVC inventory of the home-assistant namespace
— eleven names, which had already been wrong once by two — in favour of saying
the schedule sweeps the namespace.
Two stale claims went with it. Both files said PVC contents are uploaded to
Garage; the BackupStorageLocation reads bucket renner-uno-k8s-velero at
fsn1.your-objectstorage.com, and Garage was retired to nas-1. And the ha-pg
exclusion still explained itself by quoting the comment it had replaced.
242 comment lines out, 91 in, across the four hand-written Talos patches. The
diff touches nothing but comments — filtering it for non-comment lines returns
empty.
Kept, because each changes what someone editing that line would do: select
disks by serial or the installer takes the OSD drive; EPHEMERAL volume config
applies only on first provision and is silently ignored afterwards; the
LinkAliasConfig attempt that left a node with no address on any interface;
kube-apiserver not following a node re-IP; the fallback-network links must stay
v6-disabled or the cluster loses IPv6 egress; the installer image is Image
Factory and not the stock one, and its tag must match the version being upgraded
to.
Cut: how each of those was discovered. The four-equal-priority-routes
investigation, the 30 hours of NetBird probes, the image-GC measurement on
k8s-2 down to the percentage point, the per-extension rationale for the
schematic. All of it is in the commit history and the wiki.
Cut as stale rather than merely long, which is the point of doing this at all:
- k8s-1.yaml opened by saying k8s-2 and k8s-3 are still Proxmox VMs. All
three have been bare metal since neptune's conversion; DMI reports
Minisforum on every one.
- two files justified a disk selector with "Talos and Proxmox do not agree
on that order". There is no Proxmox.
- three files carried per-drive wear figures (87%, 81%, 83%) and one an
exact media-error count. Those are measurements, they move, and a config
file never gets re-read when they do.
- the mgmt0 note explained VLAN 40 by way of which address the k8s-1 VM
used to reach over that port.
One thing removed here has no other home: whether k8s-1's 18 media errors are
historical or ongoing was written up in this file and nowhere else, and #549 is
closed. Tracking it separately rather than losing it.
Config files were carrying investigation write-ups: two ruled-out hypotheses in
the smartctl-exporter securityContext, a twenty-line canonical note on the
Hetzner credential, the history of an alert threshold. That is derivation, not
configuration, and it belongs where it does not rot — the commit message for how
we got here, the wiki for what holds.
The distinction this applies: a comment stays if it changes what someone editing
that line would do — a constraint of the tool, a trap, a pointer to an issue. It
goes if it narrates how a conclusion was reached, restates a measurement, or
justifies a value after the fact. The test is whether the surrounding config
could change and leave the comment quietly wrong.
160 lines out, 30 in. Nothing behavioural changed; both manifests still pass a
server-side dry run.
Alert descriptions were tightened rather than removed. Those are not comments —
they are what someone reads when paged, and an essay is worse there than
anywhere.
This covers only what I added today. The repo has roughly 8600 comment lines
with 88 files over 25, and going further needs the criterion above agreed
first — much of that volume is load-bearing warning rather than narration.
Three ExternalSecrets synced a Secret that nothing reads. Established rather
than assumed, from three directions: every live ObjectStore names
hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore
(all seven run the barman-cloud plugin against an ObjectStore CR), and no pod
in the cluster mounts barman-cloud-creds.
They are almost certainly left from before the plugin migration, when the
inline configuration read a Secret of that name.
Worth stating plainly because it is not merely untidy: seed/{authentik,immich,
paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in
use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with
the three copies identical to each other. A valid key with no consumer is
exposure without benefit, so if it is a distinct S3 key it wants revoking at
Hetzner, which only a human can do — deleting our copies does not.
Removing the manifests does NOT remove the objects. All three carry
kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so
Flux skips them during garbage collection by design. The ExternalSecrets, their
Secrets and the three OpenBao paths have to be deleted explicitly after this
merges; git alone leaves them live and orphaned, which is worse than leaving
them declared.
The OpenBao paths deliberately stay until then. Deleting them first would leave
three ExternalSecrets failing to sync against a key that no longer exists, for
no gain.
#397 asked for the opposite: give woodpecker its own path so a rotation of
paperless's credentials would stop silently changing woodpecker's. The premise
does not hold. There is one Hetzner S3 key, and rotating it at Hetzner changes
it for every consumer no matter how many paths hold a copy. A fifth copy would
have bought nothing and made the separation look real.
Seven ExternalSecrets were reading it from four paths — seed/paperless (four
namespaces), seed/authentik, seed/immich, seed/trmnl. They now all read
infrastructure/hetzner-s3, which sits beside the existing
infrastructure/velero-hetzner.
That the four paths are byte-identical was verified rather than assumed, and
without reading them: SHA-256 over the three fields of each synced Secret
returns the same fingerprint in all seven namespaces. The same check says
infrastructure/velero-hetzner carries the same access key id, in a different
shape — a single `cloud` field holding an AWS credentials file.
Velero is deliberately not folded in. Moving it means templating that INI from
these fields, which is worth doing and not worth bundling with a path swap that
touches every database backup at once.
The reasoning lives once, in the paperless manifest, with the others pointing at
it. Repeating it six times is how six copies drift.
Kept from the note this replaces, because it was right and cost someone an
afternoon: the 403 recorded in #397 was a wrong-path error, not missing
permissions. The mount is `homelab`, so the path is homelab/seed/... —
`bao kv list homelab/seed` works with the identity available here.
homelab/infrastructure/hetzner-s3 now exists, copied from the paperless path
inside OpenBao so the value never left it, and fingerprints identical to its
source. Writes turned out to be possible all along: they fail through
openbao.renner.uno with a response the client cannot parse, and succeed against
the Service. That is worth knowing before the next person concludes, as I first
did, that the identity is read-only.
Closes#397
I described the worn drives carrying EPHEMERAL as the dangerous placement,
repeatedly, in the alert text and in the dashboard's caveats. That reads the
volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be
disposable — container images, kubelet state, pod scratch, spegel and Rook's
local state all rebuild. Putting the drives with the least endurance left there
is a sensible destination, not a risk someone should be warned about.
The alert said "a node simply stops when it goes", which is simply wrong. A node
losing /var rebuilds it.
What survives the correction is narrower and checked rather than recalled:
/var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node
costs that member's data. Quorum holds at two of three and the member is removed
and re-added. That is a rebuild, not data loss, and the alert now says so at
that weight.
The fill-level panel keeps its warning but for the right reason: a full /var
stops a node being useful long before anything is lost, and nothing rebalances
it the way Ceph rebalances a full OSD. That is a capacity argument, not an
endurance one.
None of this changes the monitoring — measuring every drive is still right, and
the wear figures are unchanged. It changes what a reader is told the numbers
mean, which is the part that was misleading.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter reports all nine drives now, so the dashboard shows what it
measures: a worst-wear stat in the status row, wear per drive over time, and a
table of the failure-now signals — available spare, media errors, the critical
warning bitfield, power-on time and written bytes.
Every expression was run against the live TSDB and the label names read off real
series rather than a README. That mattered: the exporter labels devices `nvme0`
and carries model and serial on a separate `smartctl_device` info metric, so
each query joins on (node, device) to pull the model in. A legend on the device
name alone would compare different hardware across machines, which is the same
trap the block-layer panels already avoid.
Wear and damage are presented as different things because they are. The wear
percentage is a budget the drive spends and says nothing about tomorrow; spare
exhaustion, media errors and the critical warning mean something is wrong now.
The caveats panel says so, and no longer claims endurance is missing — it was
missing when that text was written and is not any more.
Also fixes the OSD stat, which still rendered two identical numbers side by
side. The commit that fixed it did not make it into #668 before that PR merged,
so the panel went out in the state that prompted the complaint. It now shows
OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about
it separately.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third attempt, and the first two were wrong for the same reason: I kept looking
for the gate in the capability and syscall layers, and it was never there.
Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything
else dropped. It found all three drives per node and could open none of them.
Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the
default action for a syscall an unlisted profile blocks and that the kubelet
runs seccompDefault: true. /proc/1/status confirmed both capabilities effective
and a filter loaded, so the theory was at least testable — and removing the
filter changed nothing.
What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1`
inside the container also fails with EPERM. Reading a block device involves no
ioctl, no capability and nothing a seccomp profile filters. The denial is the
device cgroup, which containerd applies to every non-privileged container
regardless of whether /dev is mounted. Mounting /dev makes the nodes visible;
it does not make them usable.
Kubernetes has no per-device allowlist in a Pod spec — no equivalent of
`docker --device`. So the honest position is that `privileged: true` here is
not a lazy shortcut standing in for a narrower setting; it is the only
mechanism the API offers, short of writing a device plugin. The manifest now
says that, and records what was ruled out and how, so the next reader does not
spend the same two rounds in the capability layer.
The /dev mount loses its readOnly flag with it. smartmontools opens NVMe
devices read-write on some paths, and once the container is privileged that
flag suggests a boundary that is not there.
A device plugin advertising the drives as a schedulable resource is the
genuinely narrower answer and is noted as such. It is worth building when more
than one workload needs raw device access; for a single read-only metrics
exporter it is a component to write, deploy and maintain in exchange for the
same access.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter deployed cleanly, ran 3/3 with no restarts, answered its liveness
probe, and produced no drive metrics at all. Only smartctl_devices and
smartctl_exporter_build_info reached Prometheus — a green DaemonSet measuring
nothing, which is the failure mode the dashboard's own caveats panel warns
about.
The logs said why: it found all three drives per node and could open none of
them, with 'Smartctl open device: /dev/nvme0 failed: Operation not permitted'.
EPERM rather than EACCES is the tell — it is the default seccomp action for a
syscall a profile does not list. Rather than guess between that and the
capabilities, I read /proc/1/status in the running container: CapEff was
0x220000, so SYS_ADMIN and SYS_RAWIO were both effective, while Seccomp was 2
with one filter loaded. The capabilities were never the problem. The kubelet
runs with seccompDefault: true, so every pod that does not say otherwise gets
RuntimeDefault.
SELinux was ruled out the same way: /sys/fs/selinux/enforce reads 0.
This stays narrower than privileged: true, which would additionally grant every
capability, all host devices and write access to /sys. The tighter option is a
custom seccomp profile allowing exactly the ioctl smartctl issues; that is worth
building if this set of exceptions ever grows, and is not worth it for one pod.
Refs #549
Drive wear has been invisible in every signal this cluster has. `ceph -s` reads
active+clean at any wear level, no Ceph health check fires until a device
actually fails, and the block-layer metrics show throughput and latency but
never endurance. That blind spot is how a set of OSD drives reached 81-87%
before anyone looked (#549, #434).
This closes it for every drive, and it costs a deliberate exception to the
no-privileged-containers rule. The owner made that call explicitly after being
shown the alternative.
The alternative was real and is worth recording. Ceph's devicehealth module
already archives complete smartctl output daily — percentage_used,
power_on_hours, media_errors, critical_warning, available_spare — collected
without any elevated privileges, because the OSD pods hold the device access
already. Exporting that needs no exception at all. It also covers OSDs and
nothing else: three of nine drives here, and the six it misses include the ones
retired from Ceph for wear that now carry EPHEMERAL. Monitoring the healthy
drives while leaving the worn ones dark was not worth a new service.
The requirement comes from the kernel, not from the tool. NVMe health is read
through an admin ioctl on the character device, which is gated on CAP_SYS_ADMIN;
sysfs publishes the controller temperature and nothing more. Swapping
smartctl_exporter for another exporter would not change that, so the header says
so — the thing to re-check one day is whether the kernel grew a sysfs interface,
not whether some other tool asks for less.
The exception is kept to the smallest form that works: no `privileged: true`, no
host namespaces, no hostPID. Root plus CAP_SYS_ADMIN and CAP_SYS_RAWIO, all
other capabilities dropped, a read-only root filesystem, and /dev as the only
host mount — required because Kubernetes cannot hand individual character
devices to a pod without a device plugin. The admission policy that forbids
privileged pods is bound to the woodpecker-ci namespace only, so nothing here
circumvents it; monitoring already enforces pod-security `privileged`.
Two details that would otherwise have made the metrics useless:
The ServiceMonitor relabels the pod's node onto every series. Without it each
sample carries only the DaemonSet pod's IP, which says nothing about which
machine the drive is in and changes on every reschedule — an unattributable wear
figure is most of the value gone.
No --smartctl.device list. The exporter scans and rescans on its own, and a
written-down device list would quietly stop covering a drive the next time one
is replaced. Device names are not stable across these machines in the first
place.
The alert rules deliberately separate wear from damage: a wear percentage is a
budget being spent and warns at 80, while spare exhaustion, a critical-warning
flag or new media errors mean the drive is failing now. Media errors alert on
the increase rather than the total, so a drive that recorded errors long ago
does not alert forever while one that starts failing today does.
The completeness check compares the number of reporting drives against
node-exporter's own NVMe inventory rather than a written-down fleet size, so
adding or removing a drive moves both sides together. A rule that alerts on
nothing looks exactly like a healthy cluster, which is the failure mode worth
guarding.
Dashboard panels follow separately, once the metrics are live and the label
names can be read off the real series rather than guessed from a README.
Closes#549
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The descriptions I wrote listed the drives fitted today: which model holds which
role, that nvme0 is the system disk on two named nodes and something else on the
third, that a count below three means a drive is gone. All true this afternoon,
none of it true after the next swap — and prose does not get re-read when
hardware changes. That is the same failure this session already found in the
velero schedule comment, in the ci-claude consumer list, and in an instruction
pointing at a deleted file. I built a fresh instance of it while writing about
the others.
The text now describes mechanisms that outlive any particular set of drives, and
points at the inventory table for anything specific. Roles are assigned by serial
in Talos and by by-id path in Rook; a broad `disk.transport == 'nvme'` selector
matches every drive in the box; a drive retired from Ceph is not necessarily out
of service. None of that changes when the hardware does.
The OSD-count panel had the same problem in a worse form, since it was a
threshold rather than a sentence: green at exactly 3. It now reads OSDs down and
out as a difference against ceph_osd_metadata, so growing or shrinking the
cluster does not turn the panel red.
Fixes a claim that was simply false: the inventory description said the OSD
column came from ceph_disk_occupation, and the query never joined it. There was
no such column. It exists now, as a PromQL outer join — the matched half via
group_left, the unmatched half via `unless`, unioned — so drives with no OSD
still appear. Verified against the live TSDB: nine rows, three carrying a
ceph_daemon.
A Grafana `merge` transformation would have been the shorter route and would not
have worked. Merge only combines rows whose shared fields all agree, and the two
queries disagree on `instance` and `job` — node-exporter reports an IP and port
where ceph-mgr reports a node name — so it would have produced twelve rows
instead of nine, silently. Doing the join in PromQL also made it testable before
commit rather than after deploy.
Temperature thresholds are now labelled as generic NVMe limits rather than any
drive's rating, since they are not derived from a datasheet.
Refs #549
The first version graphed throughput, latency and temperature but never answered
"is anything wrong". Establishing that still meant querying Prometheus and Ceph
by hand — which is exactly the work a dashboard exists to remove, and which the
next person would have had to repeat.
A status row now leads: Ceph health, OSDs up and in, Ceph capacity used, and the
fullest EPHEMERAL volume across the three nodes. Each expression was run against
the live TSDB before being written in.
EPHEMERAL fill level gets its own graph too. It was the one number checked by
hand that had no home here, and it is the failure that has no safety net: Ceph
rebalances away from a full OSD, nothing rebalances away from a full /var — the
node simply stops.
The caveats panel gains three things found while answering the question:
Ceph does hold wear for the OSDs, in its devicehealth module and not in
Prometheus. The panel names the command rather than the current value, because a
number written into a dashboard is stale the moment it is written. It also
records that the retired Crucial drives were purged from Ceph's device registry,
so their history is not recoverable either — verified by asking for it and
getting ENOENT.
Talos boots with nvme_core.io_timeout=4294967295, so NVMe commands never time
out. A dying drive stalls instead of returning errors, and the latency panel
goes flat rather than spiking. That inverts what a reader would otherwise
conclude from a quiet graph.
Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and
a systemd-journal from elsewhere. I found this by searching Loki for NVMe errors,
getting nothing, and checking whether the collector existed before believing the
zero; it did not. talosctl dmesg is the only source, it is a ring buffer, and
this morning's reboots emptied it.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"description":"NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.",
"description":"NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
"description":"Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.",
"description":"Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
"description":"OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
"description":"Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
"description":"Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
"description":"Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
"description":"Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
"description":"Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
"description":"The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
"description":"Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.",
"description":"Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
"content":"### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.",
"content":"### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
# — so Cilium has been running on its own defaults for those. Router-id
# allocation is the one to watch: if the chart's default disagrees with
# Cilium's, the sessions re-establish.
bgpControlPlane:
enabled:true
# Masquerading exception, not a routing statement: pod traffic to these
# ranges leaves unmasqueraded, so LAN services see real pod IPs. That only
# works because BGP advertises the pod CIDRs, which is what gives the UDM a
# route back. The two settings are one decision and have to move together.
#
# Pod CIDRs are 10.244.0.0/16 and fd10:244::/48 (talos/patches/cluster.yaml);
# the supernets here cover the LAN as well, which is the point.
ipv4NativeRoutingCIDR:10.0.0.0/8
ipv6NativeRoutingCIDR:fd00::/8
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot
issue-fix-bot