Backing up an external server is that server's own business. The pull path
made the cluster responsible for it and, worse, put a root key on the VPS
that the cluster held — a standing inbound grant that existed only to move
a SQLite file.
Removed cluster-side: the netbird-backup namespace and everything in it
(the nightly CronJob, the netbird-backup-ssh ExternalSecret, the 5Gi
ceph-block PVC), the namespace's entry in the OpenBao ClusterSecretStore
allow-list, the netbird-backup-daily Velero Schedule, and the
NetbirdVpsBackupStale / NetbirdVpsBackupNeverRan alerts.
Flux prunes the PVC, which destroys the 14 retained dumps (2026-08-15
through 08-28). That is intended. Velero's existing offsite copies age out
on their own under the schedule's 720h TTL; no cleanup is needed there.
Removed VPS-side: the dump script, and the forced-command key authorising
it. Deleting the two tasks that placed them would only stop ansible
managing them and leave both live on the host, so they are replaced with
state: absent tasks that actually revoke them. Those two can go once the
role has run against every host that ever carried them.
The trmnl-daily schedule's stagger comment pointed at netbird-backup-daily
for its slot; it now points at apps-daily, which is the schedule that
actually precedes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump ran sqlite3's .backup with no busy handler. sqlite3 defaults its
busy timeout to 0 ms, so the online-backup API gave up the first time the
management container held the write lock instead of waiting for it, and
set -eu turned that into a failed job. Four nightly runs failed that way
before this, three attempts each, every one refused within seconds of the
ssh handshake.
NetBird sets no journal_mode on the store — its DSN carries only
_busy_timeout=30000 and cache=shared — so store.db uses the default
rollback journal, where a writer's lock blocks a reader outright. Under
WAL the reader would have gone through and this would never have come up.
.timeout arms the busy handler on the source connection, which is the
contended side. The shell's .backup never arms one on the destination and
does not need to: that file is created fresh in $tmp and nothing else
touches it.
VACUUM INTO replaces .backup because a busy timeout on its own only turns
an instant failure into a slow one. .backup copies 100 pages per
sqlite3_backup_step() and SQLite restarts the copy whenever the source is
written between steps, so against a steady writer it returns only once the
writes stop. Measured on a rollback-journal database of the same shape,
with a writer committing continuously for 20 s: .backup returned after
19 s, VACUUM INTO returned in under a second, and integrity_check on its
output passed. Both failed identically with no timeout at all, which is
what reproduced the production error. VACUUM INTO also compacts, so the
dump lands smaller than the live file.
30 s sits well inside the puller's 600 s activeDeadlineSeconds, and the
read transaction VACUUM INTO holds is covered by NetBird's own 30 s busy
timeout on the writing side.
The script lives on the VPS at /usr/local/bin/netbird-backup-dump and is
refreshed only when this role runs, so merging this to master is what
deploys it — deploy-netbird-server triggers on ansible/roles/netbird-server/**.
Closes#714. Refs #712.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runbook said nothing requests /dev/dri and the extension was carried only
so hardware transcoding would later be a config change. Both halves are now
out of date: infrastructure/intel-gpu-plugin advertises gpu.intel.com/i915 from
it, and apps/llama-server requests that resource.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The initContainer's inline shell script named each model a second time, so the
model existed in two places: models.ini and the download script. Changing a
model in the ConfigMap alone left the pod fetching the old file and failing to
start, and the script itself was untested, unlinted YAML-embedded shell.
llama-server resolves Hugging Face repos natively, and `hf-repo` is honoured
inside a preset section: server-models.cpp calls unset_reserved_args(preset,
false) for model instances, so LLAMA_ARG_HF_REPO survives into the child; only
the router's own base preset has it stripped. Pointing LLAMA_CACHE at the models
PVC makes the download a one-time cost as before.
Verified against the pinned image rather than inferred: `-hf
Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0` reaches "model loaded", and the rendered
models.ini yields "Loaded 2 custom model presets" with both `chat` and `embed`
registered and --hf-repo present in the spawned child's args.
models.ini is now the only place a model is named.
Also sets --cors-origins localhost. The server defaults to allowing every
origin and logs a warning about it at startup; without this, any page a LAN
browser visits could call this unauthenticated API from the user's machine.
Every consumer talks to it server-side, and the built-in WebUI is same-origin,
so nothing needs cross-origin access.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds llama.cpp's server in router mode as the cluster's OpenAI-compatible
inference endpoint, in a new `ai` namespace, running entirely on the on-board
Iris Xe via the `gpu.intel.com/i915` resource.
One Deployment serves both a chat model and an embedding model: started with
--models-preset and no -m, llama-server supervises each INI section as its own
subprocess behind a single port, routed by the request body's "model" field.
That gives /v1/chat/completions, /v1/embeddings and /v1/rerank from one pod,
with a crash in one model not taking out the other.
Backend and build are chosen from measurement rather than documentation. On
this hardware (i9-12900H, Iris Xe 96 EU) Vulkan beats the SYCL/oneAPI build by
+50% prefill and +36% decode, so the Vulkan image is pinned to b10731 — the
exact build benchmarked. Full offload (-ngl 99) matches CPU throughput at
~10.8 tok/s decode and ~137 t/s prefill while leaving the node's 14 cores free
for the rest of the cluster; the iGPU is here for that isolation, not speed.
The weights are fetched by an initContainer on first start, so no manual
seeding step stands between a merge and a working endpoint. They land on a
prune-disabled CephFS RWX PVC and are skipped on later starts.
Exposed LAN-only at llm.renner.uno: the API is unauthenticated, so the
`dns.renner.uno/public` label is deliberately omitted and only the UDM
external-dns instance publishes the record.
The pod needs no hostPath and no privilege — the device plugin passes the
render node in through the device-plugin API — so `ai` takes the cluster's
default `baseline` enforcement with no pod-security labels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the Intel GPU device plugin as a DaemonSet in kube-system, making the
on-board Iris Xe of each MS-01 node schedulable as `gpu.intel.com/i915`.
Pods can then request the iGPU without a /dev/dri hostPath or privilege
escalation — the plugin passes the render node through the kubelet
device-plugin API. The resource covers QuickSync transcoding as well as GPU
compute.
Vendored from upstream v0.36.0 rather than installed from the
`intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
custom resource requiring the Intel Device Plugin Operator, its CRDs and a
cert-manager webhook. One device type on three identical nodes gives the
operator nothing to manage that the DaemonSet does not.
Runs in kube-system because the plugin mounts hostPaths; the cluster enforces
PodSecurity `baseline`, which forbids those, and kube-system is the only
namespace in the API server's exemption list. NFD is not deployed, so
upstream's nfd_labeled_nodes overlay and its nodeSelector are omitted.
`-shared-dev-num=4` lets several containers share the single iGPU.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The formation wait failed with every node listed as unregistered, and the error
it now prints was the same one that broke the bootstrap call:
PermissionDenied: no request forwarding
`talosctl get members` went through the configured endpoint list rather than to a
specific machine, and apid will not forward it. It happened to work on some runs
and not others, depending on which endpoint was picked.
The second problem is worse and is not about VMs. Membership comes from the
discovery service, because the Kubernetes registry is disabled in the generated
config. So this step made a cold rebuild wait on discovery.talos.dev being
reachable, before the cluster being rebuilt exists — an internet dependency in
the middle of the disaster path, gating the bootstrap.
Neither is needed. What has to be true before etcd is bootstrapped is local: the
config took, the machine rebooted into it, and it is the machine we think it is.
The wait now asks each address directly for its hostname and expects the name its
own patch assigns — which also catches a patch landing on the wrong machine,
something the member list never checked.
The failure names the address, what it expected, and what it last answered.
Four nodes Ready says nothing about whether Ceph could use them. The CephCluster
picks OSD nodes by a label out of the machine config and their device by the
partition label RawVolumeConfig creates, so the phase now asserts both on every
worker: the r-ceph-osd partition exists, and the Kubernetes node carries
storage.renner.uno/ceph-osd.
Checking this by hand afterwards does not work — the run's EXIT trap destroys the
VMs as soon as it finishes, and that race was lost on the last attempt, leaving
the partition confirmed on exactly one worker from an earlier failed run.
Also drops the second statement of how many VMs there are. The node patch lists
are the count now, rather than the flags and the progress line each carrying
their own copy, and the addresses follow from the list instead of being spelled
out. That list had already gone stale once when the topology moved from three
control planes to one plus three workers.
The four-VM run reported:
error executing bootstrap: rpc error: code = PermissionDenied
desc = no request forwarding
and the cluster then sat idle. Nothing downstream noticed: `talosctl kubeconfig`
still succeeded, because Talos issues credentials as soon as it can rather than
when kube-apiserver serves, so the run carried on to the API wait as if it had
bootstrapped.
etcd bootstrap is the one call apid will not forward — reached through any node
other than the target it fails outright. `talosctl bootstrap` with neither -n nor
-e picks an endpoint from the configured list, and that list holds every control
plane, plus the workers in the vmtest phase. Whether it landed on the right node
was chance, in a real rebuild as much as on VMs.
Only visible because the previous commit stopped discarding this error.
The four-VM run died on "Talos cluster didn't form" after 150s. The message was
wrong twice over: the wait was too short, and it could not have told the
difference between an incomplete member list and no node answering at all,
because talosctl's error went to /dev/null.
Membership here comes only from the discovery service — the Kubernetes registry
is disabled — so a node registers after it has installed to disk and rebooted.
Four machines do not finish that in 150s. The timeout is now 10 minutes, and the
failure names the nodes that never registered, the last talosctl error, and the
member list it actually saw.
Three more of the same shape in this phase:
`talosctl bootstrap` discarded every error. The `|| true` exists because
re-running against an already-bootstrapped cluster is fine, but it hid genuine
failures too, and the run then continued to break somewhere further on with
nothing pointing back here. Anything that is not "already exists" is now
reported.
`talosctl kubeconfig` held its error while retrying and then failed with
"Couldn't get kubeconfig" and no reason. The reason is now printed when the
retries run out.
The node-ready wait counted with `grep -c "Ready"`, which also matches NotReady —
the substring is in it. So it waited for nodes to register, not to become ready,
and would report success with every node still NotReady. It now matches the
STATUS column whole. This is why the previous run's "3 nodes Ready" claim was
not evidence of readiness; whether those nodes were in fact ready is unknown.
The CephCluster named three hostnames and three drive serials. That tied the
storage layer to these three boxes as firmly as the Talos config used to be tied
to them, and for the same reason: it said in a Kubernetes manifest something only
the machine knows.
Now the machine says it, twice:
nodeLabels storage.renner.uno/ceph-osd which machines carry storage.
placement.osd.nodeAffinity reads it,
so useAllNodes can be true without
putting OSDs everywhere — that
nodeAffinity is the only thing
keeping them off diskless nodes.
RawVolumeConfig ceph-osd which disk, chosen by a CEL
expression per machine. The
partition surfaces at
/dev/disk/by-partlabel/r-ceph-osd
and that is what Rook consumes.
devicePathFilter carries both that partlabel and the OSD model for now. The
three production drives hold bluestore written straight onto the raw disk, with
no partition table and so no partition label to match; they are migrated
separately, one node at a time, and the by-id half goes when they are. Matching
the remaining half by model rather than by three individual serials already means
a drive swap is not a repo edit.
Resource requests now follow what the daemons use, measured on the running
cluster: mon ~420Mi against a 2Gi request, mgr ~650Mi against 4Gi, OSDs ~2.2Gi
against 4Gi — roughly 17GiB reserved cluster-wide and never touched. Limits stay
where they were, so burst headroom is unchanged; only the reservation moves.
Guaranteed QoS is given up deliberately, and priorityClassNames already pins
these to system-node-critical, which is what actually orders eviction. The OSD
request is set above measured use on purpose: an OSD grows into Ceph's
osd_memory_target, so 2.2Gi is a floor rather than a ceiling.
The vmtest phase becomes one control plane and three workers. Workers because
the QEMU provisioner gives the second disk only to worker machines and an OSD
needs one; three because the mons run with allowMultiplePerNode false and the
pools are replica 3 over failureDomain host. The apply loop now walks ALL_NODES
once instead of once per role, so NODE_ADDRS stays a single ordered list.
Not yet verified: whether Ceph actually reaches Ready on the VMs. That is the
point of the run this enables, and the memory budget is tight enough that it may
not — 2.5GiB plus three times 4GiB against 15.4GiB available.
Both bootstrap calls hardcoded --branch=master, which is why the flux phase could
never be tried before it was needed: `flux bootstrap` commits its own sync
manifests into the branch it reconciles, so running it wrote to the source a
rebuild depends on.
FLUX_BRANCH defaults to master and can be pointed at a throwaway branch. That
branch has to exist first — `flux bootstrap` clones it and fails with
`couldn't find remote ref` rather than creating it, which is not obvious and is
why the comment spells the two commands out.
Rehearsed against a scratch cluster to establish that: `flux bootstrap` clones
the named branch, commits its manifests there, and installs the controllers. The
waits after it need real disks and were not exercised.
Nothing else about the phase changes, which is the point — a rehearsal that takes
a different code path proves less than one that takes the same one.
bootstrap.sh generated a config and then edited it with awk before applying it,
cutting out the HostnameConfig document `talosctl gen config` emits. That was
added in April when Talos 1.12 started emitting it, and it has been load-bearing
since: without it apid rejects apply-config outright with
static hostname is already set in v1alpha1 config
The document was never the problem. Naming the host in two schemas at once is:
machine.network.hostname and machine.features.stableHostname are the v1alpha1
spellings of what the generated document already says. Setting the name in the
document instead — with `auto: off`, which is otherwise mutually exclusive with
a static name — removes the conflict, and with it the awk.
That matters beyond tidiness. Post-processing the generator's output meant the
config could only be produced by this one script: anything else that ran
`talosctl gen config` against these patches, including `talosctl cluster create`,
got a config the nodes refuse. Which is why none of this had ever been tried
outside a real rebuild.
Two more values moved out of cluster.yaml for the same reason — they described
this network rather than this cluster, so a config generated for anywhere else
was wrong before it was applied:
kubelet.nodeIP.validSubnets now talos/patches/site-homelab.yaml, a third
layer between the cluster and the machine. It
pinned every node to 10.2.40.0/24 whether or not
it has an interface there.
cluster.controlPlane.endpoint removed outright. `gen config` takes the
endpoint as its second argument and bootstrap.sh
already passes it; restating it here overrode
whatever a caller had generated for.
With that, `./bootstrap.sh vmtest` provisions three throwaway QEMU VMs and runs
the talos phase against them unchanged. It is deliberately not a copy of that
phase: it sets NODES, NODE_ADDRS, SITE and K8S_API_HOST and falls through, so a
passing test says something about a rebuild rather than about itself. Talos
version and Image Factory schematic are read out of install.image in
cluster.yaml rather than named again.
It found two real defects on its first run, both fixed here:
The phase used the kubeconfig the moment `talosctl kubeconfig` returned, which is
when Talos can issue credentials — not when kube-apiserver is serving on the
endpoint the file points at. Helm hit `EOF` against a port nothing was listening
on. It now waits for `kubectl get --raw /version`, which needs the server;
`kubectl version` would have answered from the client and passed against nothing.
`echo` without -e printed raw escape codes instead of colouring the per-node
result.
TALOSCONFIG/KUBECONFIG are honoured for the access files instead of hardcoded
$HOME paths, unset resolving to the same two paths as before. That is what keeps
a test run from replacing the operator's production access with a VM's; access-
staging is skipped under vmtest for the same reason.
Verified: three VMs Ready on Cilium 1.20.1 from the HelmRelease, and all four
node patches generate and validate with no post-processing at all.
Not covered, and still only provable on hardware: Ceph (the provisioner gives
extra disks to workers only), the mgmt and v30iot VLANs, the Layer2 VIP, Flux,
and applying the three changed production patches to the real nodes.
The name says the bootstrap depends on a registry mirror. Reading the patch list
in the runbook, that is the obvious conclusion — and it would be a serious
problem, because a cold rebuild has no mirror: an in-cluster one cannot exist
before the cluster does.
The file configures no mirror. It writes one containerd setting,
discard_unpacked_layers = false, so that pulled layers stay in the content store
and Spegel can advertise them to peers later. Spegel writes the per-registry
hosts.toml itself at runtime; the file's own comment says a
machine.registries.mirrors block is not needed, and there is none anywhere in
talos/patches. Image pulls at bootstrap go straight to the real registries.
So the alarm the name raises is false, but it is the name doing it, and the cost
of that is someone deciding a rebuild is blocked when it is not.
containerd-layer-retention.yaml describes the single thing inside.
Renamed with git mv and the three references updated. Verified `talosctl gen
config` still produces a config with the renamed patch in the list.
The Cilium install had become a block of shell in a runbook for a human to
paste. That is the same rot as a pinned version — nobody runs it between
rebuilds, so nothing tells you when it stopped working. It belongs in the script
that already drives the rest of the bootstrap.
The talos phase was unusable since #581: it passed cilium-cni.yaml whole, and
that file's `$patch: delete` document fails against a config with no
inlineManifests. The file is gone, so the failure is gone with it — reviving the
phase was mostly deleting two lines that referenced it.
It now installs Cilium between `talosctl bootstrap` and the node-Ready wait,
which is the only place it can go: with `cni: none` the kubelet stays NotReady
until a CNI exists, so the wait below would otherwise time out against a cluster
that was never going to converge.
Chart repo, version constraint and values are read out of helmrelease.yaml at
run time, so the script cannot install something different from what Flux
reconciles minutes later. helm takes the semver range verbatim. There is a guard
on the extracted values — the HelmRelease is a two-document file and selecting
the wrong one yields an empty values set that helm silently renders as chart
defaults, which is a failure mode worth catching loudly rather than discovering
in a datapath.
`--wait` because the next step needs a working network, and `upgrade --install`
so a re-run after a partial bootstrap converges instead of erroring.
Verified as far as it can be without wiping a cluster: `talosctl gen config`
with exactly the patch list the script now passes produces a config carrying
cni: none and proxy: disabled, no Cilium manifests, and it validates for metal
mode. The apply-and-bootstrap half is untested since the rewrite, and the
runbook says so — it keeps the manual procedure as the thing to read first and
to fall back on.
The install command in Step 2b hardcoded --version 1.20.1 and the chart repo
URL. That is a copy of the HelmRelease in a text file nobody runs between
rebuilds, so it would have gone stale unnoticed and the next rebuild would have
installed something Flux immediately fought — the same failure the deleted
bootstrap seed produced, in a new place.
Everything now comes out of helmrelease.yaml at the moment the command runs:
the repo URL from the HelmRepository, the values from .spec.values, and
--version from .spec.chart.spec.version verbatim. Helm accepts the semver range
and resolves it exactly as Flux does, so the two cannot pick different releases.
Verified by running the block as written:
--version ">=1.20.0 <1.21.0" -> cilium:v1.20.1, operator-generic:v1.20.1
enable-bgp-control-plane: "true" bpf-lb-sock: "true"
bpf-policy-map-max: "65536" ipv4-native-routing-cidr: 10.0.0.0/8
ServiceMonitor objects: 0
which is the config the cluster is running.
The only literal left is --set prometheus.serviceMonitor.enabled=false, and it
is a bootstrap constraint rather than a value: the CRD arrives with the
monitoring stack long afterwards. Because nothing else is overridden,
helm-controller adopts this release instead of replacing it.
This does not make the step unnecessary. Flux's controllers are pods and cannot
reconcile a CNI they need in order to run; something has to seed it. What it
does mean is that the step has nothing left in it to maintain.
Both blocking findings from the review on #709, and both were real.
The deleted cilium-cni.yaml was the only place in the repo that set
cluster.network.cni.name: none and cluster.proxy.disabled: true. They sat in its
second document next to the inline manifest, so removing the file removed them
too — a fresh `talosctl gen config` from this repo's patches would have produced
a config with Talos's default CNI and kube-proxy running, against a Cilium
release configured with kubeProxyReplacement: true. Two runbooks meanwhile
asserted that cluster.yaml supplied both, which it did not.
RUNBOOK-baremetal-conversion.md also still passed `@<cilium additive half>` in
its gen-config example, three lines above prose saying there is no CNI patch any
more. The equivalent block in the cold-rebuild runbook had been fixed; this one
was missed.
Verified the way it should have been in the first place — by running the
runbook's own command rather than inspecting the live cluster:
talosctl gen config homelab https://k8s.renner.uno:6443 \
--output-types controlplane \
--config-patch @talos/patches/cluster.yaml \
--config-patch @talos/patches/registry-mirror.yaml \
--config-patch @talos/patches/k8s-1.yaml
-> cni=none proxy.disabled=true
-> 0 cilium manifests
The live cluster could never have caught this. It has carried both settings
since it was built, and the node configs were patched by hand rather than
regenerated, so nothing in the verification touched the path a rebuild actually
uses.
Final step of PROPOSAL-cilium-single-owner.md. Talos no longer carries Cilium in
any form, and the HelmRelease is the only place it is configured.
The cluster side is already done and verified:
- the cilium inline manifest is gone from all three machine configs, applied
node by node with no reboot and no effect on the running CNI
- Kubernetes went 1.35.2 -> 1.36.3 in the same operation, with
--manifests-no-prune so the 22 objects Talos claimed were not deleted
- Talos's bootstrap-manifest inventory went from 41 entries to 13, and the 28
cilium/hubble entries are gone. A prune-enabled `upgrade-k8s --dry-run`
reported 28 deletions before and reports 0 now
That last number is the point of the whole exercise. A bare `upgrade-k8s` used
to destroy the CNI and now does nothing to it.
The inventory entries had to be removed by hand. --manifests-no-prune skips the
prune AND the inventory rewrite, so the run that saved the objects also left
Talos still claiming them — the mine was deferred, not cleared. Talos offers no
way to drop a stale claim without the prune that deletes, so the ConfigMap was
edited directly. It is derived bookkeeping, not configuration, and the result is
verifiable in both directions, which is why this was acceptable where hand-
editing configuration would not be.
Repo side, this commit:
- deletes talos/patches/cilium-cni.yaml and talos/cilium.yaml
- adds Step 2b to RUNBOOK-cold-rebuild.md: install Cilium from the HelmRelease
values with helm, after bootstrap and before Flux. The nodes stay NotReady
until then, which the runbook now says out loud rather than leaving someone
to wonder
- drops the "keep in lockstep" and "THREE COPIES" comments, which described a
duplication that no longer exists, and replaces them with one note saying
where Cilium is configured
- corrects RUNBOOK-baremetal-conversion.md, which told the reader to extract
half of a file that is gone
bootstrap.sh is left alone. Its talos phase was already dead and is now dead for
a second reason; the runbook says so.
The cost, stated plainly: the cluster no longer forms itself from the machine
config. A cold rebuild needs one documented helm install that it did not need
before. That is the price of Cilium existing in exactly one place.
Step 1 of the ownership transition in PROPOSAL-cilium-single-owner.md. Nothing
about ownership changes yet — this only moves three live settings out of nowhere
and into git.
enable-bgp-control-plane and both native-routing CIDRs existed in no file at
all. They were applied by hand and are held by the `kubectl-patch` field manager
on cilium-config; server-side apply left them alone, so nothing broke and
nothing reported them. The peering itself was always in git as CRDs in
post-crds/cilium-bgp.yaml, which is what made the gap invisible: BGP looked
configured because its policy was version-controlled, while the switch that
starts the control plane was not. A cold rebuild installs only what git
describes, so it would have come up with two established-looking peers
configured and no BGP at all.
The CIDRs go with it because they are the same decision. They are a masquerading
exception, not a routing statement: pod traffic to 10.0.0.0/8 and fd00::/8
leaves unmasqueraded so LAN services see real pod IPs, and that only works
because BGP advertises the pod CIDRs and gives the UDM a route back. Tracking
one without the other would leave a half-decision in git.
This is not purely a change of ownership, which is why it is its own step and
not folded into a later one. Enabling bgpControlPlane through the chart emits
five companion keys the hand-patch never set, so Cilium has been running on its
own internal defaults for router-id allocation among others. They should agree
with the chart's; if they do not, the sessions re-establish. Verify the peers
after Flux reconciles before moving to step 2.
The proposal called the tunnel redundant because the underlay already has
routes to the pod CIDRs. That reads the two as alternatives, and they are not:
VXLAN carries pod-to-pod directly between nodes, while the BGP-advertised pod
CIDRs exist so traffic from the LAN can reach pods at all, which necessarily
goes through the UDM.
Removing the encapsulation would therefore not remove a layer. It would move
inter-node pod traffic off a direct node-to-node path and onto the underlay,
through the router.
The observation worth keeping is narrower: routingMode is set nowhere, so the
current behaviour is an unexamined chart default rather than a recorded
decision. It happens to be the right default.
Cilium is currently owned twice. The 22 objects that make up the whole
installation — both DaemonSets, the operator, both ConfigMaps, the RBAC, the
cilium-secrets namespace — are listed in Talos's bootstrap-manifest inventory
and simultaneously carry Helm's release annotations. cilium-config has three
field managers.
That is not a tidiness problem. It produced a bootstrap seed twelve days behind
the chart, where a Kubernetes upgrade would have downgraded the CNI and switched
off socket load balancing, and it left the BGP control-plane switch and both
native-routing CIDRs existing in no file at all, so a cold rebuild would have
come up without BGP while the peering CRDs in git said otherwise.
This records the decision — Flux owns Cilium, Talos provides only cni: none and
proxy: disabled — and the ordered transition, including why the order matters:
Talos prunes what its inventory used to list and no longer does, so removing the
manifest naively deletes the CNI.
Two claims in it are verified rather than assumed. An OS upgrade does not
reconcile bootstrap manifests: Talos went to 1.13.9 four days ago and Cilium
stayed at 1.20.1 despite the seed saying 1.19.2. And the prune can be tested
before it is run, because NoPrune is honoured by the dry run as well as the
apply.
No code changes here. PR #706 stays WIP until this is agreed.
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.
What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.
So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".
Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.
Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.
BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.
The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.
BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.
Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.
Both were unmonitored. Neither is a leaf service: Woodpecker runs every cron
and deploy in this repo, and every ExternalSecret in the cluster resolves
through OpenBao.
OpenBao is probed at /v1/sys/health rather than at the root, because a sealed
instance still serves its UI. Root would answer 307 -> /ui/ -> 200 while every
secret lookup failed — the exact green-while-broken shape probes-blackbox.yaml
already warns about for the gated monitoring UIs.
standbyok=true is required, not cosmetic: the endpoint returns 200 only on the
active node and 429 on a standby, and the external route currently lands on a
standby. Measured — the bare path returns 429 right now, so an http_2xx probe
without the parameter would have been red from the first scrape. With it, 200
means initialized, unsealed and serving; 503 sealed and 501 uninitialized stay
red.
The alert matches OpenBao by instance prefix instead of the full target string,
because that target carries a query parameter and an exact match would fail
silently if the label ever encodes it differently.
Both alerts are new rules rather than additions to an existing file. Worth
noting while here: there is no generic blackbox-down rule. Only fsn1, netbird
and the two relays have alerts, so gitea, immich, auth, home, prometheus,
alerts and dash are probed and graphed but nothing fires if they go down. Not
addressed here — it needs a decision about whether to write one generic rule
and retire the per-instance ones.
The scan loop swallowed every Trivy failure with `|| true`, so a step that
scanned 21 of 22 images reported exactly the same as one that scanned all 22.
The gap was found by the fix agent noticing a missing file in its own input,
not by the pipeline.
The loop still continues past a bad image — one unpullable reference should not
cost the scan of everything after it — but now counts what failed, names it,
and exits non-zero:
--- scanned 21 of 22 images ---
NOT SCANNED: proxmox/proxmox-backup-client:latest
`failure: ignore` stays, which per Woodpecker's semantics reports the step as
failed while letting the workflow continue. That is deliberate: coverage is
reported, not enforced. The step goes red and the fix agent still runs on what
was scanned. Making the whole pipeline fail would mean one unscannable image
blocks all remediation.
The image that prompted this is gone with the nas-1 removal in the previous
commit, so the next run should report 20 of 20. The accounting is what matters
— the next dead reference will announce itself instead of being absorbed.
Refs #694
nas-1 has been off for months. Everything scoped to it went with it:
inventory/host_vars/nas-1.renner.uno/ vars and vault
[synology_nas] in pve_hosts.yml the group's only member
roles/pbs-client/ PBS backup client
plays/install-pbs-client.yml role-based, hosts: synology_nas
plays/install-pbs-client-synology.yml raw-module variant, same host
plays/install-garage-synology.yml Garage S3, replaced by Hetzner
.woodpecker/install-garage-synology.yml the pipeline that ran it
The two PBS plays were already inconsistent with each other before the host
went away: the role pulled aterfax/pbs-client, while the standalone play
pulled proxmox/proxmox-backup-client, which has never existed on Docker Hub
(404). Nothing referenced the standalone play at all. Both are moot now.
pve_hosts.yml keeps its "gone from this file, and why" list, so synology_nas is
recorded there alongside pve_cluster and the in-cluster migrations rather than
vanishing without trace.
Checked: `ansible-inventory --graph` resolves, and grep finds no remaining
reference to synology_nas, nas-1, pbs-client or the garage play outside that
one note. plays/archived/ is left alone — it is history, not live config.
file.renner.uno served HTTP 502. The Service is an ExternalName to
nas.renner.uno:7001, a CNAME to nas-1, and nas-1 has been off for months. The
Ingress and Service had been in the cluster for 110 days pointing at nothing.
Nothing caught it because file.renner.uno is not in the blackbox probe list, so
a publicly resolvable hostname returned 502 for months without a single alert.
The insecure-backend ServersTransport goes with it. It existed to let Traefik
re-encrypt to the NAS web UI's self-signed certificate and apps/file was its
only consumer, which its own header comment documents.
Rolls the security-scan pilot out to the other thirteen. Every workflow-root
gate of the form
- event: manual
evaluate: 'PIPELINE == "x"'
becomes
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "x"'
The target rides in pipeline metadata rather than in the manual-run variable
map, so it reaches the workflow-root filter on Woodpecker 3.18 where PIPELINE
does not. Verified on the pilot: pipeline 7004 selected the security-scan
workflow and nothing else.
Two files also carry step-level manual gates, and those had to move with the
root or the pipelines would have converted into something that runs but cannot
do its job:
deploy-infrastructure the apply step gates on TOFU_APPLY. Left on
`event: manual` it would never fire again, so the
pipeline would plan and silently never apply.
TOFU_APPLY now arrives via `--param`, which lands in
the same non-plugin environment step-level `when`
evaluates against. No step here is a plugin — nothing
in .woodpecker uses `settings:` — so that path holds.
build-ci-images eight per-image steps gate on manual-or-push to decide
whether to rebuild. All eight follow the root.
Cron gates are untouched. They already matched by name everywhere except
security-scan, which the pilot fixed.
Also corrects the trigger documentation, including a claim in update-audit.yml
that the other pipelines gate on "an injected PIPELINE var set at cron-create
time in the UI". Crons cannot carry variables at all — `repo cron add` takes
name, branch, schedule and enabled. That sentence described a mechanism that
never existed and was the same misunderstanding that left the daily security
scan silently not running.
The `timeout: 30` lint warning on the four deploy-* files predates this and is
left alone.
Two separate bugs in one `when` block, only one of which is #582.
The cron gate has never matched. It asked for `PIPELINE == "security-scan"` on
`event: cron`, and crons cannot carry variables — `repo cron add` takes name,
branch, schedule and enabled, and nothing else. So the variable is undefined on
every cron run and the workflow is filtered out. The `daily-security-scan` cron
does fire, and pipeline 6877 shows what actually ran under it: the `e2e`
workflow alone, which opts into that cron by name. The Trivy scan and the
security-fix agent have not been running on schedule. This predates 3.18.0 and
would have survived any fix to it.
The manual gate is the #582 case. Woodpecker 3.18 stopped merging manual-run
variables into the environment the workflow-root filter evaluates against
(builder.go:102-111 builds it from metadata plus global envs; AdditionalEnvs is
passed to the compiler at line 223 but never into this map), so
`PIPELINE == "security-scan"` cannot be true there either.
Both now use selectors the root filter can actually see:
cron -> matched by the cron's NAME, the way alert-monitor and e2e
already do it. First-class filter, no variable involved.
deployment -> CI_PIPELINE_DEPLOY_TARGET, which comes from pipeline metadata
(metadata/environment.go:74) and therefore reaches the root
filter on 3.18.
Dispatch becomes `woodpecker-cli pipeline deploy k.renner/homelab <n>
security-scan` — a named deployment target rather than a magic variable that
exists only to pick which of fourteen files is allowed to run.
Pilot for the remaining thirteen. Nothing else is converted until this one is
confirmed working end to end.
The comment blamed #522 — Backup CRs deleted within a minute of completing, so
CNPG has no record to derive the metric from. That is not what is happening.
The CRs are retained: 31 completed backups for ak-pg, im-pg and pprl-pg, 19 for
tr-pg, 11 for wp-pg, 10 for ha-pg, the newest of each from this morning.
The metric still reads 0 everywhere. The actual cause is one level up: the
collector derives it from Cluster.status.lastSuccessfulBackup, and that field
is absent on every Cluster because they all back up with method: plugin and the
barman-cloud plugin path never writes it back. lastFailedBackup is populated on
three of them, which is what makes the absence look like "no backups" rather
than "no reporting".
The conclusion does not change — the alert still cannot be built on that metric
— but the stated blocker does, and #522 is not it. Whoever picks this up next
would have gone looking at CR retention and found nothing wrong.
Also drops the incident retellings and the dated "verified on" stamps from this
file's comments, which is the cleanup already applied elsewhere in the repo.
Chasing a stale comment turned up a live duplicate. Since serviceMonitorSelector
was loosened to {}, Prometheus discovers the goauthentik.io operator's own
ServiceMonitors — which is why it was loosened. The hand-written copies for ldap
and paperless-gate, added back when the strict selector was dropping the
operator's, now scrape the same endpoints a second time.
Measured, not assumed: 10.244.4.113:9300 and 10.244.4.76:9300 were each
producing 40 samples per 10 minutes against 20 for every other outpost, and
ldap picked up a third target because the hand-written selector matched the
plain Service as well as the -metrics one. No duplicate-timestamp errors,
because the two pools scrape at different offsets — the cost is doubled
ingestion and a target list that lies about how many outposts there are.
Radius stays, and the comment now says why it is not the same case: the
operator's ServiceMonitor for it selects on managed-by=goauthentik.io together
with service-type=metrics, and no Service carries both — the operator's has the
first, the one defined here has the second. It is the only thing scraping
radius, so deleting it would have been a silent loss.
Also corrects the comments that claimed the release label is required for
discovery. It is not, for ServiceMonitors and PodMonitors: those two selectors
are empty. It very much is for Probes, PrometheusRules and ScrapeConfigs, whose
selectors still match on it, so probes-blackbox.yaml now says so explicitly —
otherwise the corrected comments elsewhere read as license to drop the label
there too.
Live selector state, for the record:
serviceMonitorSelector {} podMonitorSelector {}
probeSelector / ruleSelector / scrapeConfigSelector release=kube-prometheus-stack
Deleting the wear warning outright went too far. Wear that is high AND still
climbing is exactly the thing worth knowing about; only the level on its own
is noise. This replaces the flat 80% threshold with the combination.
The rule now projects: at the write rate of the last seven days, when does the
drive reach 95%? Under six months, and above 80% already, it alerts. The
projection clears when writes slow, which the level never could.
The rate is self-calibrating. Each drive reports both lifetime bytes written
and its own wear percentage, so bytes-per-percent is derived from the drive
rather than from a rated-endurance table that would have to be kept in sync
with the hardware. That matters here because the fleet is three different
models with three different endurance ratings sharing one rule.
It also has to work that way. percentage_used is a whole-number percent and
advances roughly once every six weeks on these drives, so any rate taken from
it directly is either zero or a single step — far too coarse to project from,
and the reason a delta- or predict_linear-based rule would have been useless.
bytes_written moves on every scrape.
The >= 80 gate is numerical rather than editorial: bytes-per-percent divides
by an integer percentage, so on a lightly worn drive half a percent of
rounding swings the projection by tens of percent. Idle drives divide by a
zero rate, yield +Inf, and stay silent without needing a special case.
Measured against live Prometheus before committing: the expression returns
nothing today, and with the horizon widened it selects exactly the three
Crucial drives at 81/83/87%, projecting 7822, 6513 and 3544 days to 95%. High,
flat, and roughly a decade of headroom — which is why the old rule was firing
every fifteen minutes about nothing.
The 95% critical alert stays a flat threshold. Past rated endurance the drive
is outside what the manufacturer stands behind however fast it got there, and
that alert clears by replacing the drive.
Closes#686
NvmeDriveWearHigh has produced four identical issues since yesterday — #673,
#675, #682, #686 — and would have kept producing them every fifteen minutes.
The alert-monitor deduplicates against OPEN issues, so closing one just makes
room for the next.
That is not a bug in the pipeline. It is a bug in the rule I wrote yesterday:
wear only ever increases, so a `>= 80` threshold starts firing the day a drive
crosses it and never stops. It is a status dressed as an alert, and no
suppression mechanism fixes a rule that is permanently true by construction.
The second half is worse. The drives it fired on are the EPHEMERAL volumes,
worn deliberately — they hold /var precisely because their remaining endurance
does not matter there. The alert was reporting the design as a fault, three
times over, for drives at 81/83/87% that are doing exactly what they were put
there to do.
So the warning goes and the critical threshold stays. 95% is a decision — the
drive is outside what the manufacturer stands behind — and a decision is what
an alert is for. The wear figure itself lives on the Disk Health dashboard,
per drive and with the role attached, which is where a slowly-moving budget
belongs.
The comment left behind says all of this, because the obvious reaction to
"there is no wear warning" is to add one back.
Not done here, deliberately: the noise log (#326) would also have silenced
this, and it is the designed path for an alert that fires continuously without
being repo-fixable. It is the wrong tool here — the alert was fixable, and
suppressing a bad rule leaves the bad rule.
Closes#686
559 -> 509 comment lines. A far smaller cut than the other files, and that is
the honest result: most of this file is per-entry rationale for a declarative
blueprint, where the comment is the only place the constraint is written down.
Everything load-bearing stays, and stays emphasised. Every ORDER MATTERS note
survives — Application before Provider, Application before Provider before
Outpost — because Application.provider is on_delete=PROTECT and getting the
order wrong leaves the blueprint failing to apply with no obvious cause. So do
both "git owns membership, add and remove people HERE not in the UI" notes,
since blueprint apply silently reverts UI edits to those groups. So does the
reason each gate exists at all: code-server runs --auth=none as root over
plaintext HA secrets, and shelly-manager has no authentication whatsoever.
Cut: the derivations. Why the dual-OAuth2 setup was collapsed into one
provider, the full W3C-spec account of credentials:'omit', a paragraph
reconstructing how the music-assistant gate failed. The instruction that came
out of that last one is kept and sharpened into "do not reintroduce a gate in
front of a PWA".
Two stale things went with it. The paperless proxy comment said the ingress
controller "only routes paperless.renner.uno to the outpost service" and named
ingress-nginx as that controller — there is no ingress-nginx left, Traefik is
the only IngressClass. And a leftover instruction to "delete via the Authentik
UI once everything's confirmed working" from a session long finished.
Verified beyond the usual: these comments live INSIDE the embedded blueprint
string, so a broken indent would render fine and only fail when Authentik
parses it. Extracted the rendered ConfigMap's k8s-outposts.yaml and parsed it
separately — valid, and still 74 entries.
Comments only across otbr, trmnl/renderer, zigbee2mqtt-deployment and the
smartctl ServiceMonitor. All four still parse; monitoring-rules still renders.
Each keeps the instruction and loses the investigation. otbr stays disabled with
the condition for re-enabling it — verified partition merge, one OMR prefix, the
Nest routers visible — instead of the paragraph reconstructing how the split
partition was diagnosed. Its socat bridge keeps both dead ends, because they are
what stops someone simplifying it: forkpty needs a socat the image does not ship,
and a sidecar cannot supply the PTY because devpts is per-container. zigbee2mqtt
keeps "never run alongside ZHA, and never start without the restored coordinator
backup or you form a new network and drop every device", and loses the cutover
diary around it.
The ServiceMonitor correction is the useful part. Four monitors carry the claim
that kube-prometheus-stack's Prometheus "selects ServiceMonitors by this label;
without it the resource exists but is never discovered". The live resource has
serviceMonitorSelector {} and podMonitorSelector {} — empty, so every monitor is
picked up regardless of labels. Only ruleSelector matches on
release=kube-prometheus-stack, which is why the same sentence on the disk-wear
PrometheusRule is correct and this one is not.
Corrected here for the monitor I added yesterday, which is where I introduced it
by copying a neighbour. The other three carry the same wrong sentence and are
left for a pass that can check each rather than sed across them.
mdns-relay 122 -> 64 comment lines, mosquitto 131 -> 118. Comments only; the
diff filtered for non-comment lines is empty and apps/home-assistant still
renders 83 objects.
The mdns-relay header was mine, written yesterday, and was the clearest case
of the problem: a seventy-line essay with sections. What survives is what stops
someone undoing it — that the relay must stay unicast because Cilium runs in
tunnel mode without enable-multicast and drops re-emitted multicast, so an
avahi-style reflector cannot work; that python-zeroconf accepts unicast at all,
which is the non-obvious fact the design rests on; that it restores
announcements but not HA-initiated browsing; and that so-bindtodevice=net1 is
what keeps any sibling pod from injecting records into HA's zeroconf cache.
Cut: the problem statement, the Errno 113 symptom, the exposure analysis
weighing what the IoT VLAN could already forge, and a paragraph explaining that
the relay is not what makes HA's listener reachable in-cluster.
mosquitto keeps every warning and loses the reasoning around them. The one
worth naming is the retained-state note: a roll preserves connectivity but not
the retained store, and Zigbee2MQTT does not recover on its own. That is an
operational consequence someone needs before touching the rollout strategy, so
it stays — the paragraph explaining why maxSurge:1 is safe does not.
crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp
and the openbao rotation kustomization. Comments only; the diff filtered for
non-comment lines is empty and infrastructure/ still builds.
Every instruction survives, because that is the half worth having: do not
switch crowdsec back to password auth, or a wiped LAPI crashloops the agents
and wedges the Kustomization; do not move netbird-ingress off the rootless
image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not
pin the rotators by digest, or a quarterly job fails to pull a manifest that
was garbage-collected months earlier and looks like it simply did not run.
What went is the incident each was learned from, and one duplicate: the
san-network-attachment header re-derived the provider: host argument that
cephcluster.yaml already makes, so it now points there instead.
Two more stale claims found on the way through:
- woodpecker-ns.yaml described the namespace as holding wp-pg alone with
'server and agent arriving in stage 2'. Both have been running there for
some time.
- cilium-bgp.yaml said each node sources its BGP session from eth0. The
management interface is mgmt0; eth0 was the VM-era name.
infrastructure/ is now 2553 comment lines.
cilium 86 -> 63 comment lines, traefik 71 -> 59. Comments only; the diff
filtered for non-comment lines is empty and both files still parse.
Cilium keeps everything that is a trap rather than a story: that bpf-policy-map-max
and vlan-bpf-bypass each exist in three files with nothing enforcing agreement,
that both take an agent restart, that a full policy map surfaces as a pod stuck
in sandbox creation, and why only VLAN 30 belongs in the bypass list. What went
is how each was diagnosed — the Hubble traces, the manual agent deletions, the
device-by-device walk through `cilium-dbg status`.
Traefik keeps the version ceiling and the reason it exists, and keeps the
CrowdSec bouncer note as an instruction: do not re-attach until client-IP
visibility is fixed and a LAN whitelist exists. It loses the retelling of the
outage that produced that instruction.
The traefik header was outright wrong and is the reason this file was worth
opening. It described Traefik as running "alongside ingress-nginx during the
migration", called ingress-nginx "the default IngressClass" and Traefik
"opt-in via ingressClassName: traefik". There is one IngressClass in the
cluster and it is traefik; ingress-nginx has no deployment and no class. A
reader following that comment would have added an ingressClassName they do not
need and looked for a controller that is not there.
152 comment lines to 125 — a much lighter cut than the previous batches, on
purpose. Most of this file is per-rule security rationale, and that is exactly
the category worth keeping: someone deleting the :8124 rule needs to know it is
the trust anchor for X-Forwarded-For, and someone deleting a gate rule needs to
know it is what stands between an --auth=none editor and the LAN.
What was cut is repetition. Four near-identical paragraphs explained the
Authentik gate pattern once per app, three of them re-deriving that
endpointSelector {} makes a :8080 grant namespace-wide. That warning is now
stated once, at the first :8080 rule, and it is stronger for being in one
place: three apps share that port, so changing any of those rules affects all
three.
The world-ingress and ICMP blocks keep their rules and lose their incident
reports — do not add a blanket fromEntities: world to an endpointSelector {}
policy; ICMP needs an explicit allow once any L4 rule exists and Cilium takes
one type per entry.
Two stale references went with it. Egress-to-world was documented as reaching
"garage S3" and "legacy InfluxDB on VM 108": Garage was retired to nas-1 and
there is no Proxmox left to host VM 108. Both replaced by what the rule
actually grants rather than a list of destinations that has already changed.
Comments only; the diff filtered for non-comment lines is empty, and the file
still passes a server-side dry run.
Comments only — filtering the diff for non-comment lines returns empty, and
both files still parse and build.
cephcluster.yaml goes from 224 comment lines to 76. Kept: that `ceph config`
retains values which merely disappear from this file, so deleting a line does
not restore a default and a live provider change needs `ceph config rm` first;
that osdMaxUpdatesInParallel and upgradeOSDRequiresHealthyPGs are the only
things keeping a rolling OSD change off all three replicas at once; that
`addressRanges` drags the mons onto host networking and ties them to a DHCP
lease; that devices are named by-id because kernel names move between boots;
that the bluestore deferral settings must not be raised again.
Cut from it: the paxos write-amplification measurement down to the byte, the
history of why mon_data_avail_warn was once 20, and a table of every drive
serial in the fleet sitting three lines above the by-id paths that already
name them.
velero-schedules.yaml loses a per-PVC inventory of the home-assistant namespace
— eleven names, which had already been wrong once by two — in favour of saying
the schedule sweeps the namespace.
Two stale claims went with it. Both files said PVC contents are uploaded to
Garage; the BackupStorageLocation reads bucket renner-uno-k8s-velero at
fsn1.your-objectstorage.com, and Garage was retired to nas-1. And the ha-pg
exclusion still explained itself by quoting the comment it had replaced.
242 comment lines out, 91 in, across the four hand-written Talos patches. The
diff touches nothing but comments — filtering it for non-comment lines returns
empty.
Kept, because each changes what someone editing that line would do: select
disks by serial or the installer takes the OSD drive; EPHEMERAL volume config
applies only on first provision and is silently ignored afterwards; the
LinkAliasConfig attempt that left a node with no address on any interface;
kube-apiserver not following a node re-IP; the fallback-network links must stay
v6-disabled or the cluster loses IPv6 egress; the installer image is Image
Factory and not the stock one, and its tag must match the version being upgraded
to.
Cut: how each of those was discovered. The four-equal-priority-routes
investigation, the 30 hours of NetBird probes, the image-GC measurement on
k8s-2 down to the percentage point, the per-extension rationale for the
schematic. All of it is in the commit history and the wiki.
Cut as stale rather than merely long, which is the point of doing this at all:
- k8s-1.yaml opened by saying k8s-2 and k8s-3 are still Proxmox VMs. All
three have been bare metal since neptune's conversion; DMI reports
Minisforum on every one.
- two files justified a disk selector with "Talos and Proxmox do not agree
on that order". There is no Proxmox.
- three files carried per-drive wear figures (87%, 81%, 83%) and one an
exact media-error count. Those are measurements, they move, and a config
file never gets re-read when they do.
- the mgmt0 note explained VLAN 40 by way of which address the k8s-1 VM
used to reach over that port.
One thing removed here has no other home: whether k8s-1's 18 media errors are
historical or ongoing was written up in this file and nowhere else, and #549 is
closed. Tracking it separately rather than losing it.
Config files were carrying investigation write-ups: two ruled-out hypotheses in
the smartctl-exporter securityContext, a twenty-line canonical note on the
Hetzner credential, the history of an alert threshold. That is derivation, not
configuration, and it belongs where it does not rot — the commit message for how
we got here, the wiki for what holds.
The distinction this applies: a comment stays if it changes what someone editing
that line would do — a constraint of the tool, a trap, a pointer to an issue. It
goes if it narrates how a conclusion was reached, restates a measurement, or
justifies a value after the fact. The test is whether the surrounding config
could change and leave the comment quietly wrong.
160 lines out, 30 in. Nothing behavioural changed; both manifests still pass a
server-side dry run.
Alert descriptions were tightened rather than removed. Those are not comments —
they are what someone reads when paged, and an essay is worse there than
anywhere.
This covers only what I added today. The repo has roughly 8600 comment lines
with 88 files over 25, and going further needs the criterion above agreed
first — much of that volume is load-bearing warning rather than narration.
Three ExternalSecrets synced a Secret that nothing reads. Established rather
than assumed, from three directions: every live ObjectStore names
hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore
(all seven run the barman-cloud plugin against an ObjectStore CR), and no pod
in the cluster mounts barman-cloud-creds.
They are almost certainly left from before the plugin migration, when the
inline configuration read a Secret of that name.
Worth stating plainly because it is not merely untidy: seed/{authentik,immich,
paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in
use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with
the three copies identical to each other. A valid key with no consumer is
exposure without benefit, so if it is a distinct S3 key it wants revoking at
Hetzner, which only a human can do — deleting our copies does not.
Removing the manifests does NOT remove the objects. All three carry
kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so
Flux skips them during garbage collection by design. The ExternalSecrets, their
Secrets and the three OpenBao paths have to be deleted explicitly after this
merges; git alone leaves them live and orphaned, which is worse than leaving
them declared.
The OpenBao paths deliberately stay until then. Deleting them first would leave
three ExternalSecrets failing to sync against a key that no longer exists, for
no gain.
#397 asked for the opposite: give woodpecker its own path so a rotation of
paperless's credentials would stop silently changing woodpecker's. The premise
does not hold. There is one Hetzner S3 key, and rotating it at Hetzner changes
it for every consumer no matter how many paths hold a copy. A fifth copy would
have bought nothing and made the separation look real.
Seven ExternalSecrets were reading it from four paths — seed/paperless (four
namespaces), seed/authentik, seed/immich, seed/trmnl. They now all read
infrastructure/hetzner-s3, which sits beside the existing
infrastructure/velero-hetzner.
That the four paths are byte-identical was verified rather than assumed, and
without reading them: SHA-256 over the three fields of each synced Secret
returns the same fingerprint in all seven namespaces. The same check says
infrastructure/velero-hetzner carries the same access key id, in a different
shape — a single `cloud` field holding an AWS credentials file.
Velero is deliberately not folded in. Moving it means templating that INI from
these fields, which is worth doing and not worth bundling with a path swap that
touches every database backup at once.
The reasoning lives once, in the paperless manifest, with the others pointing at
it. Repeating it six times is how six copies drift.
Kept from the note this replaces, because it was right and cost someone an
afternoon: the 403 recorded in #397 was a wrong-path error, not missing
permissions. The mount is `homelab`, so the path is homelab/seed/... —
`bao kv list homelab/seed` works with the identity available here.
homelab/infrastructure/hetzner-s3 now exists, copied from the paperless path
inside OpenBao so the value never left it, and fingerprints identical to its
source. Writes turned out to be possible all along: they fail through
openbao.renner.uno with a response the client cannot parse, and succeed against
the Service. That is worth knowing before the next person concludes, as I first
did, that the identity is read-only.
Closes#397
I described the worn drives carrying EPHEMERAL as the dangerous placement,
repeatedly, in the alert text and in the dashboard's caveats. That reads the
volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be
disposable — container images, kubelet state, pod scratch, spegel and Rook's
local state all rebuild. Putting the drives with the least endurance left there
is a sensible destination, not a risk someone should be warned about.
The alert said "a node simply stops when it goes", which is simply wrong. A node
losing /var rebuilds it.
What survives the correction is narrower and checked rather than recalled:
/var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node
costs that member's data. Quorum holds at two of three and the member is removed
and re-added. That is a rebuild, not data loss, and the alert now says so at
that weight.
The fill-level panel keeps its warning but for the right reason: a full /var
stops a node being useful long before anything is lost, and nothing rebalances
it the way Ceph rebalances a full OSD. That is a capacity argument, not an
endurance one.
None of this changes the monitoring — measuring every drive is still right, and
the wear figures are unchanged. It changes what a reader is told the numbers
mean, which is the part that was misleading.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter reports all nine drives now, so the dashboard shows what it
measures: a worst-wear stat in the status row, wear per drive over time, and a
table of the failure-now signals — available spare, media errors, the critical
warning bitfield, power-on time and written bytes.
Every expression was run against the live TSDB and the label names read off real
series rather than a README. That mattered: the exporter labels devices `nvme0`
and carries model and serial on a separate `smartctl_device` info metric, so
each query joins on (node, device) to pull the model in. A legend on the device
name alone would compare different hardware across machines, which is the same
trap the block-layer panels already avoid.
Wear and damage are presented as different things because they are. The wear
percentage is a budget the drive spends and says nothing about tomorrow; spare
exhaustion, media errors and the critical warning mean something is wrong now.
The caveats panel says so, and no longer claims endurance is missing — it was
missing when that text was written and is not any more.
Also fixes the OSD stat, which still rendered two identical numbers side by
side. The commit that fixed it did not make it into #668 before that PR merged,
so the panel went out in the state that prompted the complaint. It now shows
OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about
it separately.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third attempt, and the first two were wrong for the same reason: I kept looking
for the gate in the capability and syscall layers, and it was never there.
Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything
else dropped. It found all three drives per node and could open none of them.
Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the
default action for a syscall an unlisted profile blocks and that the kubelet
runs seccompDefault: true. /proc/1/status confirmed both capabilities effective
and a filter loaded, so the theory was at least testable — and removing the
filter changed nothing.
What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1`
inside the container also fails with EPERM. Reading a block device involves no
ioctl, no capability and nothing a seccomp profile filters. The denial is the
device cgroup, which containerd applies to every non-privileged container
regardless of whether /dev is mounted. Mounting /dev makes the nodes visible;
it does not make them usable.
Kubernetes has no per-device allowlist in a Pod spec — no equivalent of
`docker --device`. So the honest position is that `privileged: true` here is
not a lazy shortcut standing in for a narrower setting; it is the only
mechanism the API offers, short of writing a device plugin. The manifest now
says that, and records what was ruled out and how, so the next reader does not
spend the same two rounds in the capability layer.
The /dev mount loses its readOnly flag with it. smartmontools opens NVMe
devices read-write on some paths, and once the container is privileged that
flag suggests a boundary that is not there.
A device plugin advertising the drives as a schedulable resource is the
genuinely narrower answer and is noted as such. It is worth building when more
than one workload needs raw device access; for a single read-only metrics
exporter it is a component to write, deploy and maintain in exchange for the
same access.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter deployed cleanly, ran 3/3 with no restarts, answered its liveness
probe, and produced no drive metrics at all. Only smartctl_devices and
smartctl_exporter_build_info reached Prometheus — a green DaemonSet measuring
nothing, which is the failure mode the dashboard's own caveats panel warns
about.
The logs said why: it found all three drives per node and could open none of
them, with 'Smartctl open device: /dev/nvme0 failed: Operation not permitted'.
EPERM rather than EACCES is the tell — it is the default seccomp action for a
syscall a profile does not list. Rather than guess between that and the
capabilities, I read /proc/1/status in the running container: CapEff was
0x220000, so SYS_ADMIN and SYS_RAWIO were both effective, while Seccomp was 2
with one filter loaded. The capabilities were never the problem. The kubelet
runs with seccompDefault: true, so every pod that does not say otherwise gets
RuntimeDefault.
SELinux was ruled out the same way: /sys/fs/selinux/enforce reads 0.
This stays narrower than privileged: true, which would additionally grant every
capability, all host devices and write access to /sys. The tighter option is a
custom seccomp profile allowing exactly the ioctl smartctl issues; that is worth
building if this set of exceptions ever grows, and is not worth it for one pod.
Refs #549
Drive wear has been invisible in every signal this cluster has. `ceph -s` reads
active+clean at any wear level, no Ceph health check fires until a device
actually fails, and the block-layer metrics show throughput and latency but
never endurance. That blind spot is how a set of OSD drives reached 81-87%
before anyone looked (#549, #434).
This closes it for every drive, and it costs a deliberate exception to the
no-privileged-containers rule. The owner made that call explicitly after being
shown the alternative.
The alternative was real and is worth recording. Ceph's devicehealth module
already archives complete smartctl output daily — percentage_used,
power_on_hours, media_errors, critical_warning, available_spare — collected
without any elevated privileges, because the OSD pods hold the device access
already. Exporting that needs no exception at all. It also covers OSDs and
nothing else: three of nine drives here, and the six it misses include the ones
retired from Ceph for wear that now carry EPHEMERAL. Monitoring the healthy
drives while leaving the worn ones dark was not worth a new service.
The requirement comes from the kernel, not from the tool. NVMe health is read
through an admin ioctl on the character device, which is gated on CAP_SYS_ADMIN;
sysfs publishes the controller temperature and nothing more. Swapping
smartctl_exporter for another exporter would not change that, so the header says
so — the thing to re-check one day is whether the kernel grew a sysfs interface,
not whether some other tool asks for less.
The exception is kept to the smallest form that works: no `privileged: true`, no
host namespaces, no hostPID. Root plus CAP_SYS_ADMIN and CAP_SYS_RAWIO, all
other capabilities dropped, a read-only root filesystem, and /dev as the only
host mount — required because Kubernetes cannot hand individual character
devices to a pod without a device plugin. The admission policy that forbids
privileged pods is bound to the woodpecker-ci namespace only, so nothing here
circumvents it; monitoring already enforces pod-security `privileged`.
Two details that would otherwise have made the metrics useless:
The ServiceMonitor relabels the pod's node onto every series. Without it each
sample carries only the DaemonSet pod's IP, which says nothing about which
machine the drive is in and changes on every reschedule — an unattributable wear
figure is most of the value gone.
No --smartctl.device list. The exporter scans and rescans on its own, and a
written-down device list would quietly stop covering a drive the next time one
is replaced. Device names are not stable across these machines in the first
place.
The alert rules deliberately separate wear from damage: a wear percentage is a
budget being spent and warns at 80, while spare exhaustion, a critical-warning
flag or new media errors mean the drive is failing now. Media errors alert on
the increase rather than the total, so a drive that recorded errors long ago
does not alert forever while one that starts failing today does.
The completeness check compares the number of reporting drives against
node-exporter's own NVMe inventory rather than a written-down fleet size, so
adding or removing a drive moves both sides together. A rule that alerts on
nothing looks exactly like a healthy cluster, which is the failure mode worth
guarding.
Dashboard panels follow separately, once the metrics are live and the label
names can be read off the real series rather than guessed from a README.
Closes#549
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The descriptions I wrote listed the drives fitted today: which model holds which
role, that nvme0 is the system disk on two named nodes and something else on the
third, that a count below three means a drive is gone. All true this afternoon,
none of it true after the next swap — and prose does not get re-read when
hardware changes. That is the same failure this session already found in the
velero schedule comment, in the ci-claude consumer list, and in an instruction
pointing at a deleted file. I built a fresh instance of it while writing about
the others.
The text now describes mechanisms that outlive any particular set of drives, and
points at the inventory table for anything specific. Roles are assigned by serial
in Talos and by by-id path in Rook; a broad `disk.transport == 'nvme'` selector
matches every drive in the box; a drive retired from Ceph is not necessarily out
of service. None of that changes when the hardware does.
The OSD-count panel had the same problem in a worse form, since it was a
threshold rather than a sentence: green at exactly 3. It now reads OSDs down and
out as a difference against ceph_osd_metadata, so growing or shrinking the
cluster does not turn the panel red.
Fixes a claim that was simply false: the inventory description said the OSD
column came from ceph_disk_occupation, and the query never joined it. There was
no such column. It exists now, as a PromQL outer join — the matched half via
group_left, the unmatched half via `unless`, unioned — so drives with no OSD
still appear. Verified against the live TSDB: nine rows, three carrying a
ceph_daemon.
A Grafana `merge` transformation would have been the shorter route and would not
have worked. Merge only combines rows whose shared fields all agree, and the two
queries disagree on `instance` and `job` — node-exporter reports an IP and port
where ceph-mgr reports a node name — so it would have produced twelve rows
instead of nine, silently. Doing the join in PromQL also made it testable before
commit rather than after deploy.
Temperature thresholds are now labelled as generic NVMe limits rather than any
drive's rating, since they are not derived from a datasheet.
Refs #549
The first version graphed throughput, latency and temperature but never answered
"is anything wrong". Establishing that still meant querying Prometheus and Ceph
by hand — which is exactly the work a dashboard exists to remove, and which the
next person would have had to repeat.
A status row now leads: Ceph health, OSDs up and in, Ceph capacity used, and the
fullest EPHEMERAL volume across the three nodes. Each expression was run against
the live TSDB before being written in.
EPHEMERAL fill level gets its own graph too. It was the one number checked by
hand that had no home here, and it is the failure that has no safety net: Ceph
rebalances away from a full OSD, nothing rebalances away from a full /var — the
node simply stops.
The caveats panel gains three things found while answering the question:
Ceph does hold wear for the OSDs, in its devicehealth module and not in
Prometheus. The panel names the command rather than the current value, because a
number written into a dashboard is stale the moment it is written. It also
records that the retired Crucial drives were purged from Ceph's device registry,
so their history is not recoverable either — verified by asking for it and
getting ENOENT.
Talos boots with nvme_core.io_timeout=4294967295, so NVMe commands never time
out. A dying drive stalls instead of returning errors, and the latency panel
goes flat rather than spiking. That inverts what a reader would otherwise
conclude from a quiet graph.
Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and
a systemd-journal from elsewhere. I found this by searching Loki for NVMe errors,
getting nothing, and checking whether the collector existed before believing the
zero; it did not. talosctl dmesg is the only source, it is a ring buffer, and
this morning's reboots emptied it.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Throughput, IOPS, mean operation latency, utilisation, queue depth and
controller temperature for all nine NVMe drives, plus Ceph's own OSD latency for
comparison against the block layer.
Every panel joins node_disk_* to node_nvme_info and node_uname_info rather than
plotting the device name. That is not decoration: nvme0 is the Kingston system
disk on k8s-1 and k8s-3 and the Crucial on k8s-2, so a legend keyed on the
device compares different hardware across nodes and does so silently. The
inventory table resolves the OSD column through ceph_disk_occupation for the
same reason — it follows Rook instead of restating a device list that goes stale
the next time a drive is swapped.
Every expression was run against Prometheus before being written into the JSON,
which is also how the caveats in the panel descriptions were found. Latency is
accumulated-time over completed-operations, so it is a mean with no tail and it
vanishes while a device is idle: absence of traffic, not absence of a problem.
Utilisation near 100% on NVMe does not mean saturation the way it does on a
spindle, which is why queue depth shares that panel.
Temperature is the only SMART-adjacent value present, because it is the only one
the kernel publishes through sysfs. Wear, written-bytes totals, power-on hours
and media errors need an NVMe admin ioctl and no exporter here produces them
(#549). The dashboard says so in a panel rather than letting a green screen imply
a healthy drive.
That gap is worse than #549 describes, and the note records why. The Crucial P3
drives were retired from Ceph because they were worn, and now carry the
EPHEMERAL volume — so the drives with the least endurance left are the ones
holding container state, and they are the ones nothing watches. The measured
write latency already separates them from the Samsung OSDs by more than an order
of magnitude.
Also fixes the add-a-dashboard instructions above the generator, whose third
step told the reader to tick a row off ../DASHBOARD-MIGRATION.md. That file does
not exist anywhere in the repo, so the step could not be followed.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
review-bot found that this bump reached five of seven consumers: e2e.yml and
ci-scripts-test.yml stayed on the old version. Verified — both still pinned the
previous tag on the bump branch while the other five moved.
The cause is worth more than the symptom. ci-claude-refresh.yml carried the
consumer list twice, hardcoded, under the instruction "keep this list in sync
with the consumers". It was not in sync, and nothing could ever notice: the two
missing files were invisible to the bump that would have raised them and to the
guard that checks the pins agree. They were not drifting slowly, they were
unreachable — no future run would have touched them either.
So both copies are now derived by grepping .woodpecker for a version-pinned
reference. A list that must be maintained by hand is the defect; deriving it
removes the failure mode rather than correcting one instance of it.
The pattern deliberately matches a digit after the tag, which is what keeps this
file out of its own consumer set — its steps run ci-claude:latest, which never
needs rewriting. That makes the exclusion load-bearing, so it is spelled out in
a comment: a literal version anywhere in this file, prose included, would add it
to the set and let the sed rewrite it. The tag-scheme note at the top now says
<claude-version> for exactly that reason.
An empty result is a hard stop. Without it the following grep would be invoked
with no file arguments, read stdin, and hang rather than fail — a pipeline that
never finishes is worse than one that exits non-zero.
The generated PR body no longer claims a fixed "5 pipelines"; it counts.
Both files are also bumped here rather than left for a follow-up, and the order
matters: adding them to the consumer set without bumping them would make the
next refresh see two disagreeing tags and abort with "refusing to guess". That
guard is well built — it simply never saw these two files.
Refs #666
Pipeline 6861 answered the question the staged first step was there to ask.
build-alpine-jq ran on the fork's v1.25.18, unpacked the rootfs, executed the
RUN, and pushed both destinations to the Gitea registry at digest a430c9c6 —
while the seven untouched kaniko steps stayed green in the same run. The
executor behaves the same and the registry auth path is unaffected, so the rest
can follow.
This moves the other seven steps here plus the one in ci-claude-refresh.yml.
Nothing but the image: lines change; no flags, no commands, no credentials.
That leaves zero gcr.io/kaniko-project references in the pipelines.
The per-step note from the first commit is gone. It described a migration state
that no longer exists, and a comment that survives its own situation is how the
velero schedule ended up claiming there was no CNPG cluster in a namespace that
had one. The durable half moved to the file header, where it explains why the
kaniko image is not Google's and why the pin is past the version their repo
ended on.
Also corrects the admission policy's comment, which cited
gcr.io/kaniko-project/executor as the image its exemption was verified against.
Worth being precise about what did not change: the policy matches on the buildx
plugin image only and never on kaniko, so the swap needed no policy edit — which
is exactly why the stale image reference in the prose was easy to miss.
Closes#552
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#552 concluded that kaniko is terminal and that the nine build steps need
migrating to buildah or BuildKit. The premise is wrong. Google archived
GoogleContainerTools/kaniko on 2025-06-03, but the original authors forked it to
chainguard-dev/kaniko and kept shipping: releases run to v1.25.18, well past the
v1.24.0 the Google repo ended on. Upstream moved, it did not die.
So this is an image swap, not a migration. Same /kaniko/executor, same flags,
nothing else in the step changes. The fork publishes no binaries of its own, so
the image comes from gitlab-ci-utils, which builds it from that fork for amd64
and arm64 with versioned tags — the generic :debug tag exists too and is
deliberately not used, since every other image reference here is pinned.
Verified by pulling the image and running it rather than by reading its README:
/kaniko/executor is at the same path and reports v1.25.18, /bin/sh is a symlink
into busybox, and printf, base64, tr, mkdir and date are all present — the
utilities the auth and tag lines depend on. Anonymous pull works, so no registry
credential is needed for it.
Only this one step moves. A change to this file rebuilds every step regardless,
so a single run puts the new executor on alpine-jq beside seven unchanged kaniko
builds and shows both in the same log. alpine-jq is the smallest context here,
one FROM and one RUN, and the remaining seven plus ci-claude-refresh.yml follow
once this is green.
Two counts in #552 are also wrong and worth correcting there: the pin was
v1.23.2-debug everywhere, not v1.24.0-debug — #551 bumped alpine, not kaniko —
and there are eight steps in this file plus one in ci-claude-refresh.yml, so
nine invocations rather than ten.
Refs #552
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first draft ended on an open question and proposed a two-VM sandbox to
answer it. That was unnecessary: the question is answerable from the Cilium
source and the agent flag reference, and the answer is negative.
installDirectRoute takes its next-hop as a parameter, and nodeUpdate supplies
GetNodeIP(false), which prefers NodeInternalIP. NodeInternalIP comes from the
Kubernetes Node object; --ipv4-node looks like an override but initFromK8s()
runs after initFromConfig() and overwrites it. --direct-routing-device selects
a device, not an address. The same value feeds the VXLAN endpoint in tunnel
mode.
So cluster-internal traffic is welded to the kubelet InternalIP, and the routing
mode does not matter — tunnel sources the VXLAN from it, native uses it as the
next-hop. That deletes the one disruptive, non-rolling step the first draft
contained. Native routing buys nothing for this problem.
Since cluster-internal cannot be moved, the separation has to come from moving
management services off the node IP, and those do not share a mechanism, so each
is checked separately: etcd moves via cluster.etcd.advertisedSubnets, the
apiserver endpoint is already a VIP whose Layer2VIPConfig carries a link field,
the Talos API turned out to answer on every node address, and kubelet cannot
move because it is the node IP.
The result is four VLANs, but the boundary does not fall where the class names
suggest — kubelet stays with cluster-internal. Stated plainly rather than filed
under management and hoped over.
Recommends leaving the node IP where it is and relabelling VLAN 40 as the
cluster VLAN, rather than moving every node's identity to a new VLAN. Same end
state, and the address the cluster is administered through never changes.
Also records that the Ceph pattern is the only mechanism that fully detaches a
workload from the node IP, and that it is too expensive to generalise to
east-west traffic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue #576 framed this as "give node management its own NIC, and Ceph public
its own VLAN". That is two of four classes, and one of them turned out to be
solved already — Ceph client I/O riding the pod network is correct, since that
traffic is cluster-internal.
The classes are Ceph replication (done, VLAN 49), cluster-internal,
cluster-external and management. Three of the four share mgmt0, which was never
a decision: until the bare-metal conversion there was one NIC.
Each class gets a VLAN; which interface a VLAN lands on stays a separate and
changeable decision. That ordering means the separation survives recabling and
does not need one port per class.
Three of the four are controllable with documented levers. Management is the
InternalIP and does not move — everything else moves away from it, which is
also the direction that never changes the address the cluster is administered
through. Ingress is CiliumBGPNodeConfigOverride's per-peer localAddress, on a
BGP session that has been running for months. Egress follows the output device
under eBPF masquerading.
Cluster-internal is the one that does not resolve. In tunnel mode VXLAN sources
from the node IP with no documented alternative. Native routing is the way out
and its prerequisite is already met — BGP has been advertising the pod CIDRs to
the UDM all along — but whether nodePort.directRoutingDevice moves the next-hop
address or only the interface is undocumented across the three Cilium pages
checked.
So the proposal ends on a question rather than a plan, and the next step is a
two-interface Talos sandbox, not a cable. If the answer is unfavourable the
design changes, not the wiring.
Records what was measured rather than assumed: the BGP session state, that
Cilium's direct-routing device follows the InternalIP (adding a second address
elsewhere left it on mgmt0), that the override CRD exposes localAddress, and
where Ceph's two address families actually sit.
Also states what the split does not buy. Management shares a 10 Gb link and is
not contended; nothing observed is slow because of this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rewrite earlier today stated, at the top and again in known gaps, that the
rebuild cannot be driven remotely: management is a tagged VLAN, a maintenance
mode node cannot configure VLANs, so it comes up unreachable and every machine
needs a keyboard.
That is wrong. A maintenance-mode node DHCPs untagged and is reachable at that
address — it is not on management, but it answers, and that is precisely the
address `apply-config --insecure` targets in Step 2. All three bare-metal
conversions were driven that way.
The claim came from #576 and was carried into this document without being
checked against what had actually happened days earlier in this repo. It is the
worst place to put an unverified constraint: a disaster-recovery runbook is read
when there is no capacity to discover that a stated prerequisite is fiction, and
"you need physical access to three machines" is exactly the kind of statement
that changes whether someone attempts a recovery at all.
What survives is narrower and true: MS-01 has no IPMI and nothing here does
PXE, so booting from USB is physical. Everything after the boot is remote.
Also removes the follow-on error from the first correction, which claimed the
switch port change on 2026-08-27 had fixed a reachability problem. There was no
reachability problem to fix. That change is real and useful — a maintenance mode
node now lands in management rather than the fallback network — but it did not
turn an impossible rebuild into a possible one.
#576 still carries the original claim and is flagged in known gaps as wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 was "tofu: provision the VMs" and there are no VMs. The OpenTofu stack
has no node definitions left — only the uninstantiated proxmox_vm module beside
DNS, object storage and UniFi — so `tofu apply` provisions nothing and Step 2
then has no cluster to form. The document was half-updated: it knew k8s-1 was
bare metal but still treated k8s-2 and k8s-3 as Proxmox VMs, which stopped
being true at the neptune conversion.
That matters more than an ordinary stale doc, because it is the document
someone reaches for when everything is gone — exactly when there is no capacity
to discover that step one is fiction.
Step 1 splits into two unrelated halves: a tofu apply for platform resources
that do still exist, and physically booting three machines from install media.
Step 2 stops calling `bootstrap.sh talos`, which cannot generate a config
(#581), and carries the procedure that actually works — the one
RUNBOOK-baremetal-conversion.md uses. Both traps it hides are now stated: the
whole cilium-cni.yaml cannot be passed as a patch, and the generated
HostnameConfig collides with the per-node hostname. So is the two-stage apply:
a config referencing an interface that does not exist yet hangs the install at
STAGE: Installing with nothing timing out.
Scoped that claim rather than repeating "bootstrap.sh is dead" wholesale — the
flux and wait-apps phases are still used in Steps 3, 5 and 6 and work. Only
config generation is broken, and saying otherwise would have left the runbook
contradicting itself three sections later.
Adds the constraint that decides whether this is even attemptable remotely, at
the top and again in known gaps: management is a tagged VLAN, a maintenance
mode node cannot configure VLANs, and MS-01 has no IPMI. Every node needs a
keyboard and a monitor until #576 puts VLAN 40 native on the I226-V port. That
was true before today and undocumented.
Also records what changed under the runbook recently: the schematic now carries
i915, Ceph's cluster network is a Multus attachment rather than host
networking, and OpenBao runs three replicas that each need unsealing
individually — a sealed server is neither a voter nor a standby, so doing one
leaves the cluster short of quorum.
Disk verification before install is now explicit. Talos enumerates NVMe in an
order that is not stable across boots, the per-node configs select by serial for
that reason, and the failure mode is the installer taking the Ceph OSD drive.
Closes#660.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At 30h a single missed cycle pages critical, and a single missed cycle is the
normal cost of touching the nodes. Velero moves data through pods, so draining
a node kills whatever upload is in flight and the schedule is lost for that day
— it is not retried. Backup windows run 03:15 to 08:45 UTC, so any rolling node
work overlapping them produced a guaranteed critical alert per affected
schedule.
That is noise about planned work, not signal about backups. It fired for eleven
schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no
longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailed fires on the first failure regardless, so the first cycle is
still visible — only the escalation to critical waits for a second cycle to
confirm it.
Calibration against the current state: the four stale schedules sit at 46.8 to
48.5 hours since their last success, so the new threshold leaves roughly twelve
hours — exactly until today's runs. If those fail too, the alert fires, which
is the intended behaviour rather than a gap.
Verified: the rule passes a server-side dry run, the expression returns nothing
at 60h, and the same query at a lower bound returns the four affected schedules
with their ages — so the selector is known to match rather than merely silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes run v1.12.11, which is the last 1.12 patch — v1.12.12 does not exist.
v1.13.9 is current.
The upgrade path is the documented one rather than a shortcut. Migration is
only tested between adjacent minor releases, and the recommendation is to go
to the latest patch of each intermediate minor. 1.12.11 -> 1.13.9 satisfies
both: one minor step, taken from the last patch of the version being left. No
intermediate hop is needed.
Kubernetes is not forced along. Talos 1.13's support matrix lists 1.36, 1.35,
1.34, 1.33, 1.32 and 1.31, and the cluster runs 1.35.2 — so it stays there
until `talosctl upgrade-k8s` moves it as a separate decision. This was the one
open question before the jump and it is answered from the matrix rather than
assumed.
Deprecations in 1.13 were checked against this cluster, not just read:
.machine.env -> EnvironmentConfig not used
.machine.network.kubespan -> KubeSpanConfig not used
EtcdConfigs/KubeletConfigs protobuf change no consumer in this repo
legacy upgrade API -> LifecycleService still functional; `talosctl
upgrade` keeps working, this
only matters once it is removed
Both the five machine-config files here and the live config on k8s-1 were
checked for the first two, because the running config is generated from a base
this repo only patches and could carry fields the patches never mention.
Neither appears in either.
The removed interactive installer and the dropped `talosctl install` do not
apply — installation here is `talosctl upgrade --image` and Image Factory
media.
Schematic is unchanged; the factory image was verified to exist for the new
tag before committing. The cold-rebuild ISO URL moves with it.
Not applied to any node. Rolling it out is drain -> upgrade -> wait for Ceph
to return to 217 PGs active+clean, one node at a time, as in the 1.12.11 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).
Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.
Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.
mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.
kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.
Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was .4-.6 because the nodes held .1-.3 for as long as Ceph needed
host networking to reach VLAN 49 — three addresses for three OSDs, with no
spare, during the one phase where things were actually moving.
The nodes have since given those addresses up: the link is only a macvlan
parent now and does not need one of its own. The reservation has nothing left
to avoid, so the pool takes .1 through .6.
Six for three is the headroom the migration could not afford. It leaves room
for a leaked allocation to sit until the whereabouts reconciler collects it —
the case where a node is lost hard enough that the CNI DEL never runs — and
room to add a second OSD per node without touching this file.
Keeping the pool inside the /29 rather than moving it somewhere roomier on the
segment stays deliberate. There is no router here, so an address outside the
/29 is unreachable from anything using one inside it, which is what would
break a future rolling change where migrated and un-migrated OSDs still have
to talk to each other. That was nearly the shape of this attachment and it
would have taken every pool down at the first OSD.
Inert for the running cluster: a NetworkAttachmentDefinition is read when a
pod is created, so the three existing OSDs keep .4, .5 and .6 until they next
restart for some other reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.
enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.
That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.
This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.
How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.
Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:
mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged
fd00:: addresses 2 -> 0
IPv6 default routes 4 -> 2 (mgmt0, lo)
failing blackbox probes 5 -> 0
Ceph 217 PGs active+clean, no OSD restart
All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no OpenBao alerting of any kind — no rule, no ServiceMonitor. That
is why today's seal went unnoticed for two hours: the k8s-1 Talos upgrade
sealed the single replica, the ClusterSecretStore went InvalidProviderConfig,
and nothing said so.
Three replicas make this more important rather than less. Before, a seal was a
visible outage that announced itself. Now the service survives one, so a
sealed replica is a quiet loss of redundancy that nothing surfaces until the
next drain takes quorum with it.
Watches pod readiness rather than a seal metric, because there is no seal
metric to watch. OpenBao is not scraped: no ServiceMonitor exists, and its
metrics endpoint needs a token or an explicit unauthenticated_metrics_access
telemetry setting, neither configured. Verified against the running
Prometheus — no bao_* or vault_* series at all. External Secrets exports
nothing usable either, so the ClusterSecretStore going NotReady is invisible.
Readiness carries the signal for a structural reason, not by luck: a sealed
server fails its readiness probe because it can neither serve requests nor act
as a raft standby. Sealed is therefore always not-Ready. The converse does not
hold — a crash or stuck scheduling fires the same alert — but those want a
human too.
Three rules:
OpenBaoReplicaNotReady < 3 ready for 10m, warning. The duration only
ignores ordinary rescheduling; a sealed replica
never recovers on its own.
OpenBaoQuorumLost < 2 ready for 5m, critical. Raft quorum of three
is two, so below that the store is down.
OpenBaoMetricsMissing absent() for 15m, warning. Both rules above are
threshold comparisons, so if the series disappears
they stop evaluating and the silence is
indistinguishable from health.
Verified against the running Prometheus rather than assumed: the selector
matches (the same query with `< 4` returns the series at 3), and the alerting
thresholds correctly return nothing at 3/3 ready. Testing only that it returns
nothing would not have caught a typo in the selector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenBao ran a single replica. Every node drain took the whole secret store
down with it: today the k8s-1 Talos upgrade sealed it, the ClusterSecretStore
went InvalidProviderConfig, and it stayed that way for two hours because
nothing was watching.
Three, not two. Raft needs a quorum and the quorum of two is two, so a
two-replica cluster tolerates no failures — it would stop the moment one pod
is sealed or rescheduled, which is worse than the single replica it replaced.
Three tolerates one.
What this buys: a drain seals the replica on that node, and raft elects a new
leader from the remaining standbys within seconds. The service does not go
down.
What it does not buy, stated plainly because it is easy to assume otherwise: a
sealed server can neither serve requests nor act as a standby, so that replica
stays out of the cluster until someone unseals it. Scaling removes the outage,
not the manual unseal. For rolling node maintenance the consequence matters —
a Talos upgrade drains all three nodes in turn, so unseal each replica as it
comes back, or the third drain takes the last unsealed one and quorum is lost
anyway.
retry_join is what makes a scaled replica join the existing cluster rather
than come up isolated and uninitialized; without it each new pod would need a
manual `bao operator raft join`. All three peers are listed on every node —
the stanza is a list of possible leaders, a node ignores its own entry, and
listing all three keeps the config identical across replicas regardless of
which pod currently leads. Addressed through the headless service so each
entry resolves to one specific pod.
The snapshot job moves from the plain service to openbao-active. With one
replica the plain service could only ever hit the one pod; with three it
selects any of them, and a raft snapshot has to be taken against the leader.
No affinity was added: the chart already applies a required podAntiAffinity on
kubernetes.io/hostname, so the replicas spread across the three nodes on their
own — verified against the running StatefulSet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link
required CephCluster.spec.network.provider: host, which is not per-daemon and
put every Ceph daemon on host networking. Rook now attaches the link to the
OSD pods through a Multus macvlan, and a macvlan parent only needs to be up —
it does not need an address of its own for its children to work.
By the rule this segment is built on — Ceph-internal traffic and nothing else,
no router, no other participants — the node has no business holding an address
there either. This is stricter than before, not looser.
The link keeps up: true and mtu: 9000. Both still matter: the parent must be
up for the macvlan children to exist, and the MTU is what the children
inherit.
Not in this commit, deliberately: widening the attachment's whereabouts pool
from .4-.6 to the full /29. That must not land before these addresses are
actually gone from the running nodes, or whereabouts could hand .1 to an OSD
while a node still holds it. Follow-up once these configs are applied and
verified.
Also records the one-time step that the manifests cannot express. Removing
addressRanges did not clear public_network and cluster_network from Ceph's
config database — the same trap the bluestore comment in the same file
describes. The first OSD of the phase A rollout came up on the pod network,
still had public_network 10.2.40.0/24 in the config database, found no
interface in it and crash-looped on "Failed to pick public address". Both keys
were removed by hand. A rebuild from scratch will not hit this because the
config database starts empty; another provider change on a live cluster will.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting
every Ceph daemon on host networking to get there.
The cluster network is an OSD concern. Per the Ceph documentation it carries
OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on
the public network". Reaching it through addressRanges required
provider: host, which is not per-daemon — the mons went onto host networking
as a side effect, making a mon's address the node's DHCP lease.
That failure is documented rather than hypothetical. When a lease moved, the
mon could not bind ("Cannot assign requested address") and crash-looped. Rook
detected it and began a failover, but with three mons on three nodes each
holding the mon host ports the replacement could not be scheduled anywhere —
the canary sat Pending on "didn't have free ports". Two of three nodes hit it
during the Talos v1.12.11 upgrade; both needed hand intervention.
Selecting only `cluster` leaves `public` on the pod network, so mons, MDS,
MGR and the CSI pods keep Rook-managed identities and hold no host ports.
That removes the failure mode rather than working around it.
Isolation is not weakened. Rook's Multus prerequisites about hosts routing to
the Multus network apply to a Multus *public* network; selecting only
`cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic
and nothing else, with no router and no other participants.
MUST NOT be applied before phase A. Verified rather than assumed — kubectl
diff against the current cluster refuses it:
spec.network.provider: Invalid value: "multus": network provider must be
disabled (reverted to empty string) before a new provider is enabled
Second of the two rolling OSD restarts, bounded by the same two settings: one
OSD at a time, only once PGs are clean.
Follow-up, not in this commit: once this settles, the node addresses on VLAN
49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos
configs, after which the attachment's pool can take the whole /29 instead of
the three addresses it reserves today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Intermediate state, not a destination. san-network-attachment.yaml is already
in the cluster and stays unused until phase B sets provider: multus.
The detour is forced by Rook's own CRD validation, not chosen:
rule: self == '' || oldSelf == '' || self == oldSelf
message: network provider must be disabled (reverted to empty string)
before a new provider is enabled
host -> multus is rejected outright. It has to pass through empty, and an
empty provider means the default CNI — so addressRanges, which only takes
effect under provider: host, goes with it.
What this costs while it is in place: there is no cluster network. OSD
heartbeat, replication and recovery move onto the pod network and share it
with everything else instead of the dedicated X710 SFP+ link. Slower, and
every OSD restart pays that price in backfill. Not a correctness problem —
the OSDs can still reach one another throughout, which is the property that
matters, and the one a separate subnet would have broken.
The mons do not move here. Rook leaves existing mons on the network they were
created with; they change only on failover, which is deliberate and separate.
Bounded by the two settings that landed ahead of this: one OSD restarts at a
time, and only once PGs are clean again. Expect three sequential OSD restarts
with full recovery between them, and the same again in phase B.
kubectl diff against the live cluster shows provider host -> "" and the
generation bump. The removal of addressRanges does not appear there, because
this resource carries no managedFields and client-side diff cannot see a
field deletion. It is inert either way — Rook does not read addressRanges
without provider: host — but worth confirming after the apply rather than
assuming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rook's defaults are wrong for a cluster this size, and nothing here had
overridden them:
storage.osdMaxUpdatesInParallel default 20
upgradeOSDRequiresHealthyPGs default false
Twenty parallel OSD restarts is a small fraction of a large cluster. Here it
is all three, at once, with no gate on PG recovery in between. Every PG goes
inactive and every pool — RBD volumes and both CephFS mounts — becomes
unreachable for the duration. Not a degraded cluster, a stopped one.
With three OSDs at size 3 there is no spare copy to absorb the second
restart, so the health gate between one OSD returning and the next going down
is the only thing separating a rolling change from an outage.
This binds any change that rewrites the OSD deployment spec, not only Ceph
version upgrades. The immediate reason is the pending network provider
switch, which is exactly such a change — but the exposure predates it and
would have applied to any of them.
Lands inert: `kubectl diff` against the live cluster shows the two added
fields and the generation bump, nothing else. No OSD deployment spec changes,
so nothing restarts.
Deliberately separate from the provider switch rather than bundled with it,
so the bound is in place and observed before the operation it exists to
bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was 10.99.1.0/24, a separate subnet on the same segment, chosen to
avoid carving up a /29 with only three addresses left. That choice would have
broken the migration it exists for.
OSDs move one at a time, so the cluster runs mixed: an OSD already on the
attachment has to reach an OSD still using its node's host address on
10.99.0.x. There is no router on this segment — by design, it carries
Ceph-internal traffic and nothing else — so a pool outside the /29 is
unreachable from the un-migrated side. The two OSDs would mark each other
down, during precisely the window that is supposed to be the safe one.
Moves the pool into 10.99.0.0/29 with range_start/range_end reserving .4-.6,
leaving .1-.3 to the nodes. That is one address per OSD with nothing spare,
which is sufficient rather than merely tolerable: osdMaxUpdatesInParallel
bounds the migration to one OSD at a time, so allocations grow 1, 2, 3 and
never overlap. Headroom returns after the switch is confirmed and the node
addresses come out of the Talos configs — then the pool takes the whole /29.
Verified: the embedded CNI JSON parses with the intended range and bounds;
the NAD passes a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of moving Ceph's cluster network off host networking. Inert on its
own: a NetworkAttachmentDefinition does nothing until a pod references it,
and nothing will until cephcluster.yaml names it under
network.selectors.cluster.
Why this exists at all: `addressRanges` only takes effect under
`provider: host`, so carrying replication over the dedicated 10G link meant
setting a provider that is not per-daemon. It puts mons on host networking
too, which makes a mon's address the node's DHCP lease — and when that lease
moves the mon cannot bind. Three mons on three nodes then each hold the mon
host ports, so Rook cannot schedule the replacement it just decided to create
and the failover stalls. Both happened during the v1.12.11 upgrade, on two of
three nodes.
None of that serves the goal. Per the Ceph documentation the cluster network
carries OSD heartbeat, replication and recovery only — "Ceph Monitors always
operate on the public network". Attaching it here gives the OSDs the link and
leaves every other daemon on the pod network.
Its own subnet rather than the existing /29. The nodes hold 10.99.0.1-.3 for
as long as Ceph still uses host networking, which leaves exactly three
addresses for three OSDs — no headroom during the one phase where things
actually move, and a leaked allocation from a hard node loss would mean an
OSD without an address. OSDs never need to reach the node addresses, since
the cluster network is OSD-to-OSD, so the pool sits in its own range on the
same segment instead. A /24 for three addresses costs nothing: nothing else
can ever live on this segment, and it takes address count off the table when
OSDs are added.
After the provider switch is confirmed, the node addresses on VLAN 49 have no
remaining purpose and can come out of the Talos configs.
No gateway and no routes, because there is no router on this segment and
there must not be one. That also means no `sbr` chaining, unlike the v30iot
attachments — there is no DHCP-supplied default route to confine here.
whereabouts for IPAM: dhcp needs a server that must not exist on the SAN, and
host-local keeps state per node with no cluster-wide coordination, so two
nodes would hand out the same address on shared L2.
MTU 9000 matches the link. macvlan inherits it, but stating it means a change
to the parent cannot silently halve replication throughput with nothing in
the tree recording the expectation.
Verified: whereabouts from step 1 is running on all three nodes with its
binary installed next to macvlan; the NAD passes a server-side dry run; the
embedded CNI JSON parses and carries the intended master, mtu, ipam type and
range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for moving Ceph's cluster network off host networking. Nothing
references whereabouts yet, so this lands inert: no existing attachment
changes, and the DaemonSet only installs a CNI binary that nothing invokes
until a NetworkAttachmentDefinition names it.
The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by
design. Neither IPAM already available covers it:
dhcp cni-dhcp-daemon works where a DHCP server is on the wire.
There is none on the SAN and there must not be one.
host-local installed by cni-plugins-installer, but it keeps allocation
state per node with no cluster-wide coordination. On a shared
L2 segment two nodes hand out the same address.
Whereabouts coordinates through the Kubernetes API — allocations live in the
ippools and overlappingrangeipreservations CRDs. Nothing about assigning an
address touches the segment being addressed, which is the property that makes
it usable on an isolated SAN.
The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and
it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and
cni-plugins-installer.yaml already do: the pod writes a CNI binary and its
config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it
has to work before the attachment it serves exists. It is the fourth instance
of an existing exception, not a new one.
The reconciler is split out deliberately. It needs no host access — only the
API, where the allocations are — so it runs as an ordinary pod-network
Deployment. It matters more than it looks: the normal release path is the CNI
DEL when a pod goes away, which does not run when a node is lost hard, and on
a /29 a handful of leaked addresses is the difference between an OSD getting
one and not.
Vendored from upstream v0.9.4 with the image tag pinned — upstream ships
`:latest`. Tag rather than digest, matching the other images in this
directory. The three CRDs are vendored unmodified so a future upstream diff
stays readable. nodeslicepools is included because the ClusterRole references
it, though the node-slice controller itself is not deployed.
Verified: kustomize builds; all nine resources pass a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes run v1.12.6; v1.12.11 is the last patch of the 1.12 series. Upstream
tests migration only between adjacent minors and recommends going to the latest
patch of each intermediate minor, so 1.12.11 is the required stop before 1.13,
not an optional one.
Adds siderolabs/i915 while the image is being rebuilt anyway. Nothing requests
/dev/dri today — every manifest was checked, the only devices any workload asks
for are Ceph RBD and the NVMe disks by-id. It is included for the reason
thunderbolt already is: the extension set is fixed at install and upgrade time,
so adding it later would mean another rolling upgrade of all three nodes rather
than a config change. The MS-01's Alder Lake-P Iris Xe is present in hardware
either way.
New schematic 0f6fb8bd…, identical to a21cec8f… plus i915. Registered with the
Image Factory and the resulting installer image verified to exist for v1.12.11.
The absences are deliberate and now written down, so they are not re-derived as
gaps later: the X710 (i40e) and I226 (igc) NICs need no firmware extension —
intel-ice-firmware covers E810/ice only; mei is a prerequisite for Intel Arc
discrete GPUs and there are none; iscsi-tools, nvme-cli and util-linux-tools
have no consumer, since storage is Ceph RBD and device health comes from Ceph
rather than a node-level exporter.
Both runbooks move to the new schematic and version.
Note for whoever applies this: talosctl upgrade --image does NOT write
.machine.install.image, so this config has to be applied separately or the node
keeps pointing at whatever it was installed with. That is how the running nodes
ended up on ghcr.io/siderolabs/installer:v1.12.6 despite this file naming a
Factory image — which is also why talosctl get extensions is empty on all
three.
Kubernetes is untouched; talosctl upgrade does not move it. It stays on v1.35.2
and is a separate upgrade-k8s step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sockets bound the wildcard, so they accepted datagrams on eth0 as well as
net1. The namespace CNP grants every pod in home-assistant unrestricted access
to every other pod on every port, so any sibling pod could reach :5353 over the
cluster network and have records forwarded into HA's zeroconf cache. The
header comment described only the IoT-VLAN side of that.
so-bindtodevice=net1 confines delivery to datagrams that actually arrived on
the IoT VLAN interface. Binding to net1's address is still not possible — it is
DHCP-leased and unknown at exec time — but binding to the device gets the same
confinement without needing the address, and needs no capability: verified that
SO_BINDTODEVICE succeeds as uid 65534 with all capabilities dropped. Also
verified it costs no reception: 6 packets in 12s with the option, 6 in 12s
without.
Scope of what this fixes, stated exactly: the relay is not what makes HA's
zeroconf reachable from inside the cluster. HA binds :5353 on its pod address
(v4) and the wildcard (v6) — confirmed in /proc/net/udp and /proc/net/udp6 —
and the namespace CNP already permits any sibling pod to reach it directly.
This removes a second route to that listener, not the first.
Adds ipv6-v6only=1 on the v6 socket. The two containers share the pod's network
namespace, so a dual-stack v6 socket overlaps the v4 container's port. Measured
behaviour without it was no contention — both bound cleanly and the v4 receiver
kept taking packets throughout the v6 container's lifetime — so this is not a
latent bug being fixed, but the outcome depended on the kernel's bindv6only
default rather than on anything the manifest said. The roles are now explicit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The relay joined 224.0.0.251 only. mDNS is two protocols, and a socket joined
to one group hears nothing of the other, so every IPv6 announcement was
dropped — and a device that announces only over v6 would have stayed exactly
as invisible as before the relay existed.
Not hypothetical for this cluster: HA's zeroconf holds ff02::fb on eth0
alongside the v4 group, verified in /proc/net/igmp6. The listener was already
waiting for traffic the relay was never going to send it.
Adds a second socat container for the v6 group. Two processes rather than one
because socat carries a single address pair per invocation and the families
need different receive and send addresses. They fail independently and the
kubelet restarts each on its own.
Joining ff02::fb needs no address configuration on net1. The group is
link-local, so the kernel's automatic link-local address on the macvlan is
enough to receive on it; whether v30iot hands out global v6 addresses does not
affect reception.
The delivery Service becomes dual-stack (PreferDualStack with both families
listed, matching mosquitto-iot in this namespace), so each family is forwarded
over its own transport instead of folding v6 announcements into a v4 hop.
UDP4-SENDTO takes the A record and UDP6-SENDTO the AAAA from the same name.
Verified: the v6 socat address syntax parses and joins; a server-side dry run
allocates both families to the Service; all three manifests still apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HA's zeroconf listener joins 224.0.0.251 on eth0 and hears nothing. The HA
pod has received 0 multicast frames against 1.9M packets total. Two
independent reasons: multicast is link-local, so IoT-VLAN announcements never
reach the pod CIDR; and Cilium runs tunnel mode (VXLAN) with no
`enable-multicast`, so it does not replicate multicast between pods either.
The visible cost is that the ESPHome integration dials each device at the
address cached in its config entry and has no way to learn when that address
changes. A rotated lease produces `[Errno 113] Connect call failed` until
someone repoints the entry by hand. All eight ESPHome config entries were
created by `source=zeroconf` and now carry `discovery_keys: dhcp` — both
paths are inert today, so the addresses are frozen at whatever they were when
last discovered.
An avahi-style reflector cannot fix this. It receives on one interface and
re-emits as multicast on the other, and the second half is exactly what the
CNI drops. The published recipes for this all put avahi on the node, which
Talos does not permit.
So the relay bridges the last hop as unicast, which is the one direction the
CNI carries. It holds the multicast membership on v30iot through its own
macvlan and forwards each datagram to HA's :5353.
This works because python-zeroconf does not filter on how a packet arrived —
verified against _listener.py: datagram_received checks size, deduplicates,
and parses, with no test on destination address, source address, or source
port. Unicast mDNS is RFC 6762's QU mechanism, so this stays inside the
protocol.
Scope of the fix, stated plainly: passive reception only. RFC 6762 requires a
responder to re-announce when its address changes, which is the failure this
addresses, and new devices announcing themselves are seen too. Queries HA
sends still leave on eth0 and die there, so nothing that depends on
soliciting a response is restored.
No macvlan on the HA pod itself — that stays off-table, and the outcome since
it was removed supports keeping it that way. A separate relay was already the
recorded intent; this is that relay.
Shape follows shelly-manager-iot, which is the working precedent for L2
presence on v30iot: macvlan with DHCP IPAM, chained with sbr so the leased
default gateway cannot displace eth0's route. The forward path runs over eth0,
so that route has to survive.
Delivery target is a dedicated ClusterIP rather than the pod. socat resolves
its target once at start and cannot chase pod IPs; a ClusterIP is stable and
Cilium re-DNATs to the current backend. Kept separate from the
`home-assistant` Service, which fronts the nginx sidecar and is the XFF trust
anchor.
Exposure, since it is not zero: socat binds :5353 on all interfaces including
net1, because net1's address is DHCP-leased and unknown at exec time. Any host
on the IoT VLAN can send it a datagram that reaches HA's zeroconf cache — the
same trust already extended to real mDNS from that segment, which any device
there can equally forge. macvlan traffic bypasses Cilium policy, so the
namespace CNP does not constrain that interface. The container runs socat and
nothing else.
No CNP change needed: home-assistant-ns.yaml already allows intra-namespace
traffic on all ports.
Verified before committing: the socat address syntax joins the group and reads
in a loop without forking per datagram (UDP4-RECV, not UDP4-RECVFROM); it runs
as uid 65534 with no privileged operation; the image digest is pinned from the
registry; kustomize builds; all three manifests pass a server-side dry run.
Not yet verified, and only observable once deployed: that HA acts on the
forwarded records. The check is `discovery_keys` flipping back to zeroconf, or
a device surviving a lease change without hand-editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotated turn_secret and relay_secret in production; both are correct and stay
rotated. proxy_token was in the same set and should never have been: rotating
it locked every proxy out of the management server.
The misjudgement was specific. proxy_token appears in proxy.env on the server
AND in proxy.env on each proxy, both Ansible templates, which looked exactly
like the symmetry test the other two pass. It is not: the management server
never reads it from a file Ansible writes. The token is registered in the
management DATABASE, so rotating it changed only what the proxies present.
management: proxy auth failed: rpc error: code = ResourceExhausted
desc = too many failed authentication attempts (~75/min, steady)
proxy: http: TLS handshake error ... unknown domain "home.renner.uno"
The second follows from the first — a proxy that cannot authenticate never
receives its domain configuration, so it rejects the domains it holds valid
certificates for.
Recovered by restoring the original proxy_token from the pre-rotation vault and
re-applying both roles. Auth failures went to zero and the domain errors
stopped. turn_secret and relay_secret keep their new values; both appear in
management.json.j2, so both ends really are Ansible's.
vault_netbird_proxy_token moves back into vault.yml. It was in
netbird-rotatable.yml, which this play rewrites wholesale — the next run would
have silently dropped it. Verified by re-running both deploys: changed=0.
Adds the test that would have caught this, in the play's own words: a secret is
safe to rotate here only if it appears in management.json.j2. Templating both
sides of a wire is not the same as owning both ends of a trust relationship.
Also corrects the shadow-key guard. vault.yml carried BOTH
vault_netbird_turn_secret and a bare netbird_turn_secret; the role default maps
one to the other, but a group var outranks a role default, so the bare key won
and rotating only the prefixed one would have changed nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Anything holding a long-lived TCP session to a device outside the cluster keeps
a half-open socket when the node under it moves. The peer drops its side; the
pod never learns, because no RST arrives and no keepalive fires.
This cost real time after the conversion. Home Assistant held seven ESTABLISHED
sockets to ESPHome devices that were themselves reporting "not connected". It
logged nothing and retried nothing, because as far as it knew the connections
were fine — so the search went through logs that were empty by construction.
The one device Home Assistant did log was the only one genuinely offline.
Records the check that settles it in one command, reading /proc/net/tcp in the
pod, and notes that reloading the integration is the lighter fix than
restarting the pod.
Placed in "Verify and restore" rather than the known-stale list: it is something
to check before calling a conversion finished, not a trap in the tooling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
common.yml applied monitoring-agent (Alloy) to every inventory host, pointing at
loki.srv.renner.uno:3100 and prometheus.srv.renner.uno:9090. Both moved into the
cluster and neither name resolves any more, so nas-1, claude-ws, netbird and the
proxies have been shipping logs and metrics nowhere.
Repointing them was the obvious fix and the wrong one. The cluster endpoints are
authenticated — loki.renner.uno returns 401 and prometheus.renner.uno sits behind
an Authentik forward-auth gate — while monitoring-agent has no auth support at
all. Making it work would mean per-host credentials, htpasswd entries in the
cluster secret, and some answer for remote-write past the gate. Nobody reads
those metrics: edge-1 is the host that matters, and edge-2 when it exists.
edge hosts are unaffected. install-edge-host.yml is self-contained and
deliberately never carried monitoring-agent; they ship via edge-log-shipper,
which already does this correctly — authenticated push to the cluster ingress
with a per-host credential.
install-netbird-monitoring.yml goes too; applying monitoring-agent to
netbird_server was its only content.
Comments referring to the role are corrected in install-edge-host.yml,
install-claude-workstation.yml, the claude-workstation role and its RUNBOOK.
Three more described the old srv Loki as "firewalled off the IoT VLAN" — it is
not firewalled, it is gone. The reason edge hosts use the authenticated ingress
is unchanged, but the stated reason was no longer true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/vault.yml held 35 keys. Nine are referenced by the surviving
tree, all NetBird:
netbird_oidc_client_id vault_netbird_auth_client_secret
netbird_oidc_client_secret vault_netbird_datastore_key
netbird_oidc_issuer vault_netbird_proxy_token
netbird_turn_secret vault_netbird_relay_secret
vault_netbird_turn_secret
The other 26 belonged to Proxmox, n8n, owncloud, the grafana and gitea OIDC
pairs, woodpecker, SNMP, cloudflared, the UniFi Poller account and both Home
Assistant tokens — services that now run in-cluster with their own credentials,
or no longer exist at all.
Edited YAML-aware rather than line-based: at least one value was a multi-line
block, which a line delete would have corrupted.
Verified the nine survivors are byte-identical, by comparing a normalised hash
of just those keys before and after, and again after re-encrypting:
e9fbbcef4127b84289ebf32e3b99e59a68e96e275af3c8c17d8ad0414a08e790
Same hash all three times. Decryption happened in a 0600 file outside the repo,
shredded afterwards; no plaintext touched the working tree.
Removing a key here does NOT revoke the credential. Anything still live in its
target system needs revoking there — that is a separate exercise, and the Home
Assistant tokens in particular are unreferenced here but valid in HA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Missed in the Proxmox removal. I built that list from a `hosts:` listing I had
truncated with `head -40`, so anything past the fortieth line was never
considered — a self-inflicted blind spot rather than a judgement call.
install-postgres-monitoring.yml authentik-db-1.srv.renner.uno and
prometheus.srv.renner.uno, both NXDOMAIN
configure-audiobookshelf.yml audiobookshelf.srv.renner.uno is NXDOMAIN;
audiobookshelf runs in-cluster
The postgres-exporter role goes with them — install-postgres-monitoring.yml was
its only consumer.
Found by cross-referencing every vault key against the surviving tree rather
than by looking at plays again: vault_postgres_exporter_password came back as
"still referenced", which was only true because a dead play still referenced it.
That check now leaves the group_vars/all vault holding nine keys anything uses,
all NetBird, and twenty-six nothing does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The play generated secrets with community.general.random_string. The CI image
(ci/images/ansible/Containerfile) installs only community.proxmox and
community.docker, so the play failed at its first task when run through the
project's own tooling. It resolved on a workstation that happened to have the
collection, which is exactly why verifying it there proved nothing.
Replaced with ansible.builtin.password, which needs no collection and produces
the same thing: 48 alphanumerics, generated without persisting (path /dev/null).
Also hardens the shadow-key guard. It piped `ansible-vault view` into `grep -c`
with failed_when: false, so a failed decrypt — wrong password, moved file —
sent empty input to grep, which reported zero matches. Indistinguishable from a
clean vault, meaning the guard would have waved the rotation through precisely
when it knew least. A preceding task now proves the vault is readable, and the
assertion requires both that proof and the zero count.
Corrects the openbao rotation path in the header comment; it is under
kubernetes/.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.woodpecker/update-all.yml ran `ansible-playbook -i inventory/ plays/update.yml`
on a cron. This branch deletes that playbook, because every playbook it
imported targeted Proxmox hosts or guests that no longer exist, so the next
scheduled run would have failed outright.
Removing the pipeline rather than repointing it: there is nothing left for it
to update. Its imports were update-vm, update-lxc, update-authentik,
install-n8n, install-grist, install-music-assistant, update-docker and
update-pve-rolling — all removed here, all targeting hosts that no longer
resolve.
Swept every pipeline for references to deleted plays. The five still named
(install-claude-workstation, install-garage-synology, install-netbird-client,
install-netbird-proxy, install-netbird-server) all survive.
Also from the review:
common.yml `all:!pbs` was a no-op I introduced when simplifying the
old exclusion list — no pbs group exists. Now plain
`all`.
qemu-guest-agent dropped the `pve_cluster`/`proxmox_nodes`/
`proxmox_all_lxc` group guards; those groups are gone,
so the conditions were always true.
Caught by the review-bot on #625.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/<group>/ files merge alphabetically and the last one wins.
netbird-rotatable.yml sorts before vault.yml, which still defines
vault_netbird_turn_secret, vault_netbird_relay_secret and
vault_netbird_proxy_token — verified, all three are present.
The rotation itself would have worked, since the plays pass the new values
explicitly and play vars outrank group vars. The damage came later: the next
ordinary run of install-netbird-server.yml or install-netbird-proxy.yml would
read the stale value from vault.yml and push it back out, breaking
server<->proxy auth. Those deploys run unattended on changes to the roles, so
it would have surfaced on something unrelated to secrets.
Adds a pre-flight check that counts — never prints — matching keys in vault.yml
and refuses to rotate while any remain, plus the one-time migration step in the
header.
Caught by the review-bot on #626.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotates the three shared secrets where Ansible controls both ends, so the two
sides cannot disagree:
vault_netbird_turn_secret turnserver.conf (coturn) + management.json, one
role, one host
vault_netbird_relay_secret management.json on the server + relay.env on the
proxies
vault_netbird_proxy_token proxy.env on both
Opt-in, because rotation is not idempotent and briefly interrupts the mesh:
ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
New values are written to a dedicated encrypted file rather than edited into
the main vault. The plaintext is rendered at 0600 with a .new suffix, encrypted,
and only then moved into place, so a failure at any step cannot leave readable
secrets where the inventory would load them. The play asserts the installed
file carries the $ANSIBLE_VAULT header before proceeding.
Two are deliberately excluded and the play says why:
vault_netbird_datastore_key encrypts the datastore at rest; changing it
without dump/restore makes existing data
unreadable
vault_netbird_auth_client_secret its other half is in Authentik, so rotating
only the consumer breaks login
These are Ansible's own credentials, not copies of the cluster's. OpenBao holds
separate credentials for some of the same systems, deliberately different values
— one consumer, one credential, independently revocable. This play does not
touch OpenBao and is not meant to.
Proxies roll one at a time so a relay stays reachable. The server is
reconfigured first, so there is a short window where a not-yet-updated proxy
fails to authenticate; clients reconnect and peer-to-peer tunnels are not torn
down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing here had a target left. Every removal is backed by a probe rather than
an assumption:
pve_cluster (jupiter/neptune/saturn) run Talos; no SSH at all
authentik-db-1/2, authentik-app-1/2 NXDOMAIN
grafana.srv, vault.srv, paperless.srv gone (grafana/paperless resolve but
answer neither ping nor SSH)
n8n.srv, grist.srv, wiki.srv NXDOMAIN
owncloud.srv, ical-proxy.srv, loki.srv NXDOMAIN
proxmox_* dynamic groups api.proxmox.yaml queried
jupiter.renner.uno:8006
24 plays, 14 roles and the Proxmox dynamic inventory go. update.yml goes with
them: every playbook it imported is in that list, so the aggregate updater had
no fleet left to update.
Kept, because their hosts answer: netbird server and proxies, nas-1, edge-1 and
its capability groups, claude-ws. common.yml and install-node-exporter.yml lose
exclusion clauses naming groups that no longer exist.
NTP IS NO LONGER HIGHLY AVAILABLE, and this change records that rather than
causing it. ntp_cluster was jupiter (MASTER, prio 100), saturn (90), neptune
(80) and edge-1 (BACKUP, 70); the conversion took the first three. A single
VRRP member holds the address regardless of configured state, so fd40::123 is
still served — verified answering — but by one thin client with nothing behind
it. install-ntp-ha.yml still carries the four-host priority map, now describing
hosts that do not exist. Restoring redundancy needs a second chrony host that
is not a Talos node.
The gatus role is removed entirely, which supersedes the narrower edit in the
DNS cleanup that dropped its Proxmox DNS check.
Left alone: plays/archived/ still references pve_cluster, which is what an
archive is for. paperless-migration stays because an archived play uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The removal has been applied:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed, 1 forgotten.
`tofu state list` now returns no Proxmox resources at all, and a full plan with
refresh enabled produces no Proxmox errors — the "no route to host" against
10.2.40.201:8006 that failed every infrastructure plan since the conversion is
gone.
The block was always one-shot. Left in place it refers to a state entry that no
longer exists, which is harmless but misleading: it reads as pending work.
Worth recording for the next time this shape comes up, because it cost a failed
merge to learn. A `removed` block alone does not fix an unreachable provider:
OpenTofu still refreshes the resource before it can forget it, and that refresh
is exactly what cannot succeed. It takes one run with `-refresh=false` to
process. `-target` does not help — targeting does not reach `removed` blocks, so
a targeted plan reports "No changes" while the state entry sits there untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Proxmox DNS HA VIP (10.2.40.53) was decommissioned after it dead-ended
non-Ingress resolution and broke ~1374 Home Assistant entities on a pod restart
(2026-07-14). With Proxmox itself now gone, the implementation and everything
pointing at it are dead weight — and one piece was actively dangerous.
Removed:
dns-server/ the Go implementation, its systemd unit,
and a 7.4 MB checked-in binary
opentofu/scripts/remove_unifi_clients.sh
the "Proxmox DNS (VIP)" Gatus check in the ansible role
remove_unifi_clients.sh is the one worth calling out. It instructs the reader
to deploy module.proxmox_dns and point UniFi DNS at 10.2.40.53 — precisely the
configuration that caused the outage. It also works by `tofu state rm`, which
orphans resources rather than removing them. It was already marked deprecated
in its own header; anyone who ignored that and followed it would have
reintroduced a known failure.
The Gatus check queried k8s-1.srv.renner.uno against 10.2.40.53 every 30s. It
belongs to the ansible gatus role, which targets a Proxmox LXC that no longer
exists, so nothing was running it — but the check itself asserts the decommis-
sioned server should answer, which is the opposite of the intended end state.
Deliberately kept: the three comments in bootstrap.sh, coredns-config and
k8s-gateway that record 10.2.40.53 as DECOMMISSIONED and say not to reintroduce
it. They carry the outage date and blast radius, and are the reason this keeps
not happening again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every infrastructure plan has failed since neptune was wiped. module.k8s_3 still
declared a Proxmox VM, so the bpg/proxmox provider tried to reach a hypervisor
that no longer exists:
Error: failed to get resources list of type ("vm") for cluster:
dial tcp 10.2.40.201:8006: connect: no route to host
with module.k8s_3.proxmox_virtual_environment_vm.this
Deleting vm_152_k8s_3.tf alone does not fix that. Removing a resource from
configuration makes OpenTofu plan a destroy, and planning a destroy refreshes
the resource through the same unreachable provider.
A `removed` block with `lifecycle { destroy = false }` forgets the state entry
without contacting Proxmox. Nothing is orphaned: the guest, its disks and the
hypervisor underneath were all destroyed during the conversion. This is the
declarative equivalent of a state removal, reviewable in git rather than run by
hand against production state.
talos_image.tf goes with it. Its locals named the nocloud image for VM import
and had no consumer other than the VM module.
Deliberately kept for a follow-up: the proxmox provider block, the
proxmox_vm/proxmox_lxc/bpg_proxmox_container modules and sync_state_from_pve.py.
A provider with no resources is never contacted, so leaving it costs nothing,
and removing it should wait until a clean plan proves state holds no other
Proxmox resources. That check cannot be made from here — the S3 credentials
used for local runs have been revoked.
Verified with `tofu validate` and `tofu fmt -check` offline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
machine.install.image was never set, so every node installed from the stock
ghcr.io/siderolabs/installer that talosctl gen config defaults to. Extensions
live in the installer image, which makes the schematic a property of the boot
media alone — nothing lands on disk. talosctl get extensions returns empty on
all three nodes.
Two extensions are missing as a result, and the cold-rebuild runbook explains
why each was chosen for bare metal:
siderolabs/intel-ucode CPU microcode. On a VM the hypervisor supplied it;
on bare metal nothing does except the BIOS.
siderolabs/thunderbolt Inert until machine.kernel.modules asks for it.
Present so enabling the MS-01 mesh later needs
neither a new image nor a reinstall — precisely the
property installing without it destroys.
Set in the shared cluster patch rather than per node: all three are bare metal
now and want the same image. Schematic a21cec8f is the bare-metal one from
RUNBOOK-cold-rebuild.md — same net.ifnames=1 as the Proxmox schematic, minus
qemu-guest-agent.
Config-only. install.image is read at install and upgrade time, so no running
node changes until it is upgraded; the three nodes stay without extensions
until a deliberate rolling upgrade is scheduled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cilium reads datapath options once, at agent start, and the chart puts no
config checksum on the pod template. A changed ConfigMap therefore leaves every
agent running the previous datapath until something restarts it, and nothing
does — the DaemonSet is not rolled by a ConfigMap-only Helm change.
This already cost an outage. vlan-bpf-bypass landed in cilium-config and every
v30iot macvlan workload stayed broken until the agents were deleted by hand,
one node at a time (issue #592). The merge looked successful: HelmRelease
Ready, release upgraded, key present in the ConfigMap, and nothing working.
policyMapMax carries the same exposure. Its comment says "takes effect on agent
restart", which is documentation standing in for automation — as does the
equivalent note in the two bootstrap copies.
Reloader is already deployed and used across the repo for the same purpose.
Named per-ConfigMap rather than blanket, matching the existing convention.
The trade, stated plainly: any cilium-config change now rolls the CNI
cluster-wide, node by node. That is the same disruption a datapath change
already required; the difference is that it happens on its own rather than
being forgotten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step pushed a nocloud Talos image onto each Proxmox node so the k8s VMs
could boot from it. No Proxmox node remains — neptune was the last one, and all
three control-plane nodes now boot the metal image from their own installed
disk.
Left in place the step would call the Proxmox storage API for a host that no
longer answers it, then sit through its 60x10s download-wait before giving up,
on every infrastructure apply.
The block's own comment already scheduled its removal: "the whole step goes away
once neptune is converted too."
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every v30iot macvlan workload on a bare-metal node has been stuck in sandbox
creation since its node was converted. Cilium's datapath drops VLAN-tagged
frames unless bpf.vlanBypass lists the tag, and cilium-config carried no
vlan-bpf-bypass key at all.
Inside the Proxmox guests ens19 was an untagged virtio NIC on the v30iot
bridge, so nothing was tagged by the time Cilium saw it. On bare metal ens19 is
a VLAN 30 sub-interface of enp2s0f0np0, and the macvlan children's DHCP
exchange never reaches the wire.
Measured rather than inferred: cilium_drop_count_total on the affected node
reports ~123k INGRESS and ~1.1k EGRESS drops with reason "VLAN traffic
disallowed by VLAN filter", against none on the node that was still a VM, and
node_network_receive_packets_total for ens19 reads 0 there against 10 pkt/s on
the VM. Full analysis in issue #592.
The symptom hides its own cause. A tc-layer drop surfaces to an AF_PACKET
sender as ENOBUFS, so the CNI dhcp plugin reports "no buffer space available"
and "network is down" instead of a DHCP timeout.
VLAN 40 is deliberately not listed. mgmt0 rides the same parent and carries the
node IP, etcd and the API, and it passes today without a bypass entry, so
adding it would change a working control-plane datapath to fix an IoT one.
All three copies of the key move together — the HelmRelease, kubernetes/talos/
cilium.yaml, and the inline bootstrap manifest in patches/cilium-cni.yaml — so
a cold rebuild does not come up without it.
The runbook predicted this exact failure and is updated to record that it
happened, along with how to recognise it from the drop counters rather than the
misleading CNI error. music-assistant's comment claiming ens19 is "the v30iot
NIC" on every node is corrected; it is a NIC on the edge worker and a VLAN
sub-interface on the control-plane nodes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.
The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.
The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.
cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.
cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three nodes are converted and the cluster is back to three OSDs across
three zones with 217 PGs active+clean, so the conversion window that justified
2/1 on this pool is over.
The min_size override goes with it. It existed only because Ceph does not
lower min_size when size is reduced, so the pool would have been 2/2 while a
host was out. At size 3 the default min_size of 2 is what the pool wants.
The comment loses its TEMPORARY banner and now states the standing rationale
and the condition under which the pool degrades to two copies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is converted and back in the cluster, so k8s-3 returns to the storage
fleet on its new PM983. All three nodes now name their OSD by /dev/disk/by-id,
and the storage fleet is whole again.
The comment generalises to all three nodes and records neptune's serials.
k8s-3 is the clearest evidence for by-id addressing: it enumerated the Kingston
as nvme0n1 on one boot and the Crucial as nvme0n1 on the very next one, so a
kernel-name reference had no stable meaning on that machine at all.
The old osd.1 lived on the Crucial that is now EPHEMERAL. That drive was wiped
during the conversion, so the OSD is unrecoverable and is purged from the CRUSH
map separately; this change only creates its replacement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune now runs Talos directly, so k8s-3 moves from the VM-era layout to the
multi-document form k8s-1 and k8s-2 already use. The old config selected
interfaces by busPath against virtio NICs and installed to /dev/vda; neither
exists on bare metal.
Disks are selected by serial rather than kernel name. That is not defensive
styling: Talos enumerated this machine's drives differently than Proxmox did,
and differently again between two consecutive boots — nvme0n1 was the Kingston
on one boot and the Crucial on the next. A device-name reference would have
installed onto the wrong drive.
nvme 50026B7382B48B90 Kingston system disk
nvme 2326E6E8E67D Crucial EPHEMERAL, 83% of rated endurance
nvme S437NF0M501176 PM983 OSD, 3% used
Serials, interface names and drive health were read from the machine itself —
the Kingston and Crucial off the running Proxmox host before the wipe, and
confirmed again from Talos maintenance mode.
k8s-2.yaml loses the sentence claiming k8s-3 is still a Proxmox VM. No Proxmox
remains in the homelab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
neptune is the last node to convert to bare metal, and its OSD drive is swapped
in the same window. Removing k8s-3 from the explicit node list first means Rook
stops managing that OSD before the drive it lives on goes away, rather than
recreating it mid-reconcile and backfilling onto a disk that is about to be
pulled.
The device comment for k8s-3 goes with it: it described the PCI-passed-through
drive that node saw as a Proxmox guest, which stops being true here.
Ceph runs on two OSDs and size-3 pools stay degraded until the node rejoins.
cephfs-metadata is already at 2/1 for exactly this window.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is back on bare metal with the PM983 in place, so k8s-2 returns to the
storage fleet. Like k8s-1 it names the device by /dev/disk/by-id: the machine
has three NVMe drives and Talos enumerates them differently than Proxmox did,
so nvme0n1 there is the Kingston system disk.
The runbook prescribed applying the full per-node config in maintenance mode,
which hangs the install. A document referencing an interface that does not
exist yet leaves the node at STAGE: Installing with KUBELET n/a, the console
repeating
k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
which reads as a Kubernetes fault but is an install that never completed. On
saturn it was the ens19 VLAN document and the sysctl naming it. Three cold
installs hung; the identical config applied cleanly to the running node.
Wiping disks changes nothing.
Install now runs in two stages: a minimal network config plus EPHEMERAL, which
is one-shot and cannot be deferred, then the full config applied to the running
node behind --mode=try so a bad network config reverts itself.
Also corrects the claim that maintenance mode cannot be inspected. talosctl get
works there; --insecure is a subcommand flag and has to follow `get`, otherwise
cobra resolves the legacy `talosctl disks` command and rejects the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VM 605 is pinned to saturn, so it cannot survive the conversion, and it has to
go before the node leaves the Proxmox cluster — afterwards the provider cannot
reach it and every plan errors.
Nothing uses it. Its purpose was cloning new Proxmox VMs, Proxmox is being
retired, and the one remaining VM definition sets template_name = null
explicitly rather than cloning from it.
The module default is changed to null for the same reason: it named a template
that no longer exists, which would only surface as an apply-time failure for
whoever created the next VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saturn is being converted to bare metal, so its VM definition goes and it
leaves the list of Proxmox nodes the pipeline seeds the nocloud image onto.
The apply must run while saturn is still a Proxmox cluster member. Once it
leaves, the provider cannot reach the node, the plan errors, and the only way
out is tofu state rm.
module.k8s_2 is referenced nowhere else in opentofu/, so this destroys exactly
one resource.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Structurally identical to k8s-1.yaml; only serials, MACs and addresses differ.
Written from saturn's live hardware rather than assumed: the X710 port 1 MAC
came from the guest that currently owns it, since the host cannot see a
passed-through NIC.
Kingston 50026B7382DA5F6E system disk
Crucial 2332E86B74BC EPHEMERAL, retiring from osd.0 at 81% wear
X710 p0 58:47:ca:76:6f:73 trunk, VLAN 40 + VLAN 30
X710 p1 58:47:ca:76:6f:74 Ceph cluster network, 10.99.0.2/29
Links are addressed by kernel name rather than LinkAliasConfig, and both disks
by serial. Talos and Proxmox do not agree on NVMe enumeration order, and a
network document that fails to resolve takes the whole machine's DHCP with it.
Not applied by anything on merge — talos patches are applied by hand. Landing
it before the conversion so the file under review is the file that gets
installed.
Verified with talosctl validate --mode metal against v1.12.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the node from storage.nodes before purging its OSD is what makes the
purge stick. Under useAllNodes the operator rebuilt a purged OSD within one
reconcile and began backfilling onto the drive being replaced; listing nodes
explicitly exists so a node can be taken out first.
Re-add after the conversion naming the device by /dev/disk/by-id, as k8s-1
does. A bare device name is not safe on bare metal: Talos enumerates the three
NVMe drives in a different order than Proxmox, and nvme0n1 is the system disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pvecm expected 1 is the official recovery, documented verbatim in the storage
replication chapter of the Proxmox admin guide. Its caveat is about restoring
the missing node instead; here the missing node is deliberately gone for good,
so enforcing quorum is the correct branch rather than a workaround.
Also records that it must be run only once the node is alone -- corosync
rejects a value below the members it can see, which is why the attempt on the
current two-node cluster returned CS_ERR_INVALID_PARAM -- and that it is
runtime only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is converted and its OSD rebuilt, so the capacity constraint that took
immich down is gone with all three nodes back.
cephfs-metadata deliberately stays at size 2 / min_size 1. It has to be
degraded again for saturn and then neptune, and each transition re-replicates
the pool, so toggling it per node buys nothing but churn. Its comment said to
restore it once k8s-1 was back, which is now misleading about the plan; it
restores after the last conversion instead.
The immich comments lose their jupiter-specific framing — taking both
Deployments down is a pre-flight step for any single-node maintenance now, and
the runbook carries it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written from doing jupiter, with saturn and neptune still to go. The conversion
itself was straightforward; every failure came from state that outlived the VM,
so the runbook is weighted towards those rather than towards the happy path.
The three that cost real time:
A rebuilt node gets a new MAC and a new lease, and Ceph keeps its mon pinned to
the old address in the monmap, the mon-endpoints ConfigMap and the deployment's
--public-addr. The mon then cannot bind, the operator's next step wants to
restart a healthy mon, Ceph refuses because that would drop quorum, and it
retries every minute forever without reaching the storage reconcile. Fixing the
ConfigMap before the monmap and the deployment matters, or the operator
recreates the mon from the stale entry.
Talos stops running default DHCP for the whole machine as soon as one new-style
network document exists, and a LinkAliasConfig selector that matches nothing is
skipped silently. Together those turn a small config mistake into a node with no
address on any interface.
EPHEMERAL placement is honoured only on first provision, so it has to be right
in the first apply-config; afterwards it is ignored and relocating needs a wipe.
Also records the Proxmox quorum step that starts mattering at the second
conversion, the Secure Boot blocker, and that Talos and Proxmox enumerate the
same three NVMe drives in different orders — which is why the OSD is named by
by-id path rather than device name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter is back as a bare-metal Talos node, so its OSD returns to the storage
fleet on the new drive.
The device is named by /dev/disk/by-id rather than by bare device name, because
the bare name would be wrong here. Talos enumerates this machine's three NVMe
drives in a different order than Proxmox did, and nvme0n1 on k8s-1 is the
Kingston system disk:
nvme0n1 50026B7382B48B86 Kingston Talos system disk
nvme1n1 2332E86B749A Crucial Talos EPHEMERAL (/var)
nvme2n1 S437NX0M400426 PM983 this OSD
Rook would most likely have declined the system disk for carrying partitions,
but relying on that is luck rather than intent. The by-id path derives from
model and serial and does not depend on enumeration order.
k8s-2 and k8s-3 keep bare names: each guest sees exactly one block device,
the drive passed through from its host, so there is nothing to disambiguate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first install of this node came up with no address on any interface and had
to be recovered from the console. This is the configuration that replaced it,
now running on jupiter.
The alias layer was the cause, but not for the reason it first appeared. Reading
the v1.12.6 source: aliases ARE supported as VLAN parents — link_config.go
resolves parent links through the alias map, and the field's own doc comment
says so. The structure was fine. What is not fine is the failure behaviour.
LinkAliasConfigController skips a selector that matches zero links with no log
of any kind, contradicting the doc comment on the type which claims an error is
returned. And permanent_addr comes from the ETHTOOL_GPERMADDR ioctl with no
fallback to the current address, so a driver that does not answer it yields an
empty string, an unmatched glob, and silence.
That alone would have been survivable. What made it total is that Talos stops
running default DHCP operators for the WHOLE machine as soon as a single
new-style link, VLAN or DHCP document exists — the gate is presence of a
document, not success of resolution. So the silent miss removed the only
configured path while the safety net was already disabled. There is no fail-open
option.
Aliases are worth having and should come back. The condition is that they are
verified rather than assumed: apply with `--mode=try`, which rolls back on its
own, and check that linkaliasspec exists and that permanentAddr is actually
populated before trusting it. A LinkConfig on the real kernel name alongside the
alias resolves regardless of whether the alias landed, and costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three NVMe drives are present, not two: the Crucial stays in the machine as
EPHEMERAL rather than being pulled.
The second I226 is enp90s0, not enp89s0. Adding that third drive renumbered
the PCI buses and renamed it, which is a concrete demonstration of why every
NIC in this file is matched by permanent MAC rather than by name or bus path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Crucial P3 that was this node's Ceph OSD stays in the machine rather than
being pulled, and takes over /var. At 87% of rated endurance it is finished as
an OSD, but everything EPHEMERAL holds — containerd images and snapshots, pod
logs, the Ceph mon store — is reconstructible, so the drive is consumable in
that role.
It also removes the constraint that keeps biting elsewhere: /var becomes
~931 GB instead of the 64 GB the VM-era zvol gave it. k8s-3 tripped
DiskPressure twice today the moment it absorbed another node's pods, with
image GC already at its floor and ~20 GB of unreclaimable overlayfs snapshots.
This node will not have that problem, and the same treatment is available to
saturn and neptune when they convert.
Talos only honours volume configuration when the volume does not yet exist, so
this must ship in the first apply-config; applied later it is silently ignored
and relocating EPHEMERAL then needs a disk wipe and a boot from install media.
The selector matches an exact serial. There are three NVMe drives in this
machine now and `disk.transport == 'nvme' && !system_disk` would match the
PM983 as well, which would hand Talos the OSD drive.
Verified with talosctl validate --mode metal against v1.12.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With one of three nodes out for the bare-metal conversion, the remaining pair
cannot hold the workload: cluster CPU requests exceed two nodes' capacity by
roughly 200m once DaemonSets and unplaceable Ceph pods are discounted, and
k8s-3 sits ~5.8 GiB above the kubelet's eviction threshold with image GC
already at its floor.
immich is the right thing to stop rather than squeeze. machine-learning
reserves 500m against a 2m idle draw, but inference bursts hard, so cutting
its request would misrepresent what it needs the moment someone uploads
photos. Stopping it removes the reservation honestly, and takes its pods and
image pulls off k8s-3 for the duration.
The Postgres cluster stays up, so recovery is these two replica counts and
nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
useAllNodes makes it impossible to retire an OSD ahead of pulling its drive.
Purging osd.2 and deleting its deployment held for one reconcile: the operator
saw a node it was told to use, carrying a device with a BlueStore signature,
and rebuilt the OSD. Ceph then began backfilling onto the very drive being
replaced.
Listing nodes explicitly makes membership of the storage fleet a property of
this file rather than of which nodes happen to exist, so a node can be removed
before its disk is touched.
k8s-1 is out while jupiter's OSD is swapped for the PM983 and the node is
converted to bare metal. The cluster runs on two OSDs until it is re-added;
all pools are size 2 / min_size 1 and stay fully redundant across the pair.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.
crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.
Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jupiter's Crucial P3 Plus OSD is at 87% endurance used, and reaching the
MS-01's only U.2-capable slot means moving the boot drive anyway. Doing the
bare-metal conversion in the same window avoids opening the machine twice and
throws away no work: the osd-ssd PCI mapping that a Proxmox-side swap would
need to be re-pointed at ceases to exist.
Networking uses the v1.12 multi-document config rather than the deprecated
machine.network.interfaces. The docs do not define what happens when one link
is configured by both schemas, so k8s-1 keeps every link in one of them. This
also buys VLANConfig.name, which is free-form: naming the VLAN 30 link ens19
keeps the four v30iot macvlan NADs working unchanged across VM and bare-metal
nodes, so the cluster can stay mixed without pinning those workloads.
Physical NICs are matched by permanent MAC, not driver — both X710 ports bind
i40e here, unlike the single passed-through port k8s-2/k8s-3 see.
cephfs-metadata drops to size 2 / min_size 1 for the duration. At size 3 on
two remaining hosts every PG sits undersized with min_size 2, so one further
OSD loss would block IO on every CephFS mount cluster-wide. This must go back
to size 3 once k8s-1 rejoins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sweep restores every discovered Velero schedule, immich's ~180 GiB
library included. At size 2 that is ~360 GiB raw, ~120 GiB per OSD, and
measured on 2026-08-25 it was ~350 of the ~740 GB each drive wrote per
week — ~47% of the endurance budget, against a ~65 GB/day weekday
baseline.
That baseline is itself the result of the 2026-08-18..21 write-reduction
work (#518, #519, #520, CNPG checkpoint tuning, RAM-backed emptyDirs,
paperless data to RBD, CI workspaces to cephfs-ci), which cut per-drive
writes ~70% from ~195 GB/day. With the application tier down, the restore
test is now the single largest consumer of remaining QLC endurance; the
three OSD drives are at 81/82/87% used.
Monthly still exercises every restore path and cuts the sweep's share of
drive wear ~4x, moving per-drive weekly writes from ~740 GB to ~530 GB.
Cadence-only change:
- 0 8 * * 0 -> 0 8 1 * *, keeping the 08:00 UTC slot that clears the
03:15-07:15 daily backup window (see the comment in the manifest).
- ttlSecondsAfterFinished 14 d -> 40 d, so the previous run is still
inspectable when the next one fires.
- Comments and docs that state the cadence updated to match, including
the BLUESTORE_SLOW_OP_ALERT runbook text, which tells the reader to
check whether a stall fell inside the restore-test window.
No day-of-week form is available: Kubernetes parses cron with
robfig/cron's standard parser, whose day-of-week field allows only
`* / , - ?` (no Quartz `#` nth-weekday), and restricting both day fields
ORs them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Wetterstation now carries a non-default reporting configuration: min
interval 5s on all ten reportable attributes, and tighter reportable-change
deadbands on temperature, humidity and pressure. None of it is expressible
in configuration.yaml - Zigbee2MQTT has no settings-schema field for
reporting configuration - so it lives only on the device and in Z2M's
database.db. This documents it and how to restore it.
min was 10s against an 8.8s transmit interval. Since min is a hold-off, every
change-triggered report was held past a transmit cycle and landed on the
next, so attributes reported at best every 17.6s and drifted out of phase.
The published stream showed no 8.8s periodicity at all. At min 5 each
attribute reports on its own transmit and the stream carries one publish per
cycle, measured at a mean gap of 8.69s. Publish volume fell from ~631/h to
~445/h because each publish is now a whole cycle rather than a fragment.
The deadbands come from the shared modernExtend helpers, which the WS90
definition composes without overrides. They are anti-chatter values sized for
battery-powered indoor sensors. Temperature was 1.00 C, giving a staircase
that moved about twice an hour; it is now 0.1 C, matching the resolution the
device actually reports. Pressure was 5 kPa - 50 hPa, which real barometric
pressure never traverses, so it only ever updated on the hourly heartbeat
that pressure_trend and weather_condition derive from. Note the same library
configures the identical ZCL attribute with a 5 hPa deadband in its older
reporting.ts helper, so the two code paths disagree by 10x.
Anything that makes Z2M re-run configure on the device reverts all of it, and
a fresh PVC will not have it. The restore commands are written out literally
rather than generated by a shell helper, because the nested quoting through
`kubectl exec -- sh -c` is where this goes wrong; each was executed and
verified as written, including the read-back, and re-applying is a no-op.
No behaviour change from this commit itself - the device was reconfigured at
runtime, this only records it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#560 claimed a merged episode "surfaces as a single all-zero frame". It
does not, and the difference decides whether a downstream filter is even
possible.
The zeros propagate attribute by attribute through the burst, so the
merged frame carries whatever subset had zeroed when that burst ended.
Across three episodes captured at nanosecond resolution the propagation
happened to complete, which is where the wrong generalisation came from.
Nothing guarantees it. A burst ending mid-propagation publishes a frame
with temperature at zero while humidity and pressure still read normally,
which is a physically consistent measurement - Z2M recomputes dew_point
from the false temperature - and is not separable from a genuine winter
reading.
The zeros also begin inside an otherwise normal transmit cycle, roughly
20ms after a valid reading in the same burst (measured: 27.7ms, 19.8ms,
22.0ms). They occupy a regular reporting slot, so no timing property
distinguishes them either.
Since the merge is last-wins and the zeros arrive at the tail of the
burst, it resolves that burst to the zeros and discards the valid reading
that led it. So debounce trades six bad samples per episode for one, at
the cost of one good sample. That is a bound on the corruption, not a
filter, and the comment now says so.
No behaviour change; the debounce value and device stanza are untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The value is unchanged and correct; the comment committed alongside it in
#559 was neither accurate nor the right place for most of what it said.
Accuracy. It cited the device's ZCL reporting configuration as the bound,
reading minRepIntval 10 as "the station updates every 10s". minRepIntval
is not a cadence: with maxRepIntval 3600 and a reportable change set, the
ZCL reporting model gives periodic reports at the MAXIMUM interval — once
an hour — plus change-driven reports each held back so it lands no sooner
than 10s after the previous. It is a hold-off on change-triggered reports
and says nothing about the gap between reports of different attributes,
which is what a debounce window sees.
The bound that applies is the documented transmit interval — readings
every 2s, transmitted every 8.8s — against Z2M's documented rule not to
set debounce above the sensor's update interval. Shelly's knowledge base
rounds the same figure to 9s.
It also claimed the window had to exceed 1s because "the recovery burst
straddles a second boundary", inferred from log lines stamped 11:42:11
and 11:42:12. Those are second-resolution timestamps and cannot
distinguish 10ms from 900ms. No lower bound was ever established.
Length. The rest was commit-message material sitting in a config file:
justification for the chosen value, gap-distribution and run-length
measurements, and notes reconciling my own earlier misreadings. Comments
state facts about the system, not the history of arriving at them. What
remains is what the device does, what debounce does to it, and the
constraint on the value — 41 lines down to 13.
Verified before writing: post-debounce frames arrive complete (all 22
fields, no partials) at a settled ~10s cadence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes to the Wetterstation (Ecowitt WS90, 0xc02cedfffe958f40), both
aimed at the station's recurring zero-drop glitch.
2.13.0 carries zigbee-herdsman-converters#12748, which discards the ZCL
non-value markers (0xffff / 0xff / 0xffffff) the WS90 emits when a reading
is unavailable. Without it those markers are scaled and published as
measurements — the 6553 m/s gust readings in Koenkk/zigbee2mqtt#31048. The
release also recovers the rain rate after a counter reset, which is the
same lifetime-counter dip that packages/wetterstation.yaml compensates for
on the Home Assistant side. Its only breaking change is dropping Node.js 20
for bare-metal installs; the container ships its own runtime. The amd64
manifest was verified pullable before this commit.
debounce: 2 addresses the part 2.13.0 does not fix. Koenkk/zigbee2mqtt#31048
is still open, and the author of #12748 states explicitly that it does not
cover the zero drops: "a zero can be a real reading for these attributes,
only the all-bits-set markers are provably invalid."
The station reports as a burst of seven per-cluster ZCL frames and Z2M
publishes its full cumulative state once per frame, so one device report
becomes seven MQTT messages. During a glitch the zeros appear progressively
across those seven, which is what makes the corruption unrecognisable
downstream: an intermediate frame carrying {"temperature":0,"humidity":92,
"pressure":96.9} is internally consistent (Z2M recomputes dew_point from the
false temperature) and cannot be told apart from a legitimate winter
reading. Merging the burst yields a single frame with the all-zero
signature, which a consumer can reject.
The window is bounded on both sides. Above: the device's own reporting
configuration sets minRepIntval 10 / maxRepIntval 3600 on all seven WS90
clusters (1024/1026/1027/1029 and 0xfc01/0xfc02/0xfc03), so no attribute
repeats inside 10s. Below: the recovery burst straddles a second boundary,
so 1s splits it and emits a partial frame.
Staggering was checked rather than assumed — the 10s floor is per attribute,
and nine attributes report across those clusters, so a trailing-edge timer
could in principle keep resetting and never flush. Over 6h the longest
unbroken chain of sub-2s gaps was 5, bounding worst-case deferral under
~10s.
The device is also absent from the seeded device list, which predates it
joining, so this adds the friendly_name too.
NOTE: this file is seed-only. A running Z2M owns its configuration.yaml on
the PVC, so this does not change the live instance — it only makes a fresh
PVC come up correctly. debounce is requiresRestart, so applying it live
needs the option set on the running instance and a restart; the image bump
supplies the restart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From the #426 audit. Tag verified present before bumping.
The kaniko bump that was originally in this branch is dropped. It would
have pinned a new `-debug` image, and debug variants are not to run in
the production cluster. kaniko is archived upstream, so the fix is
migration rather than a version bump; tracked in #552 along with the
`-debug` requirement.
Comments should say what the setting does and what constrains it. The
blocks added in #542, #544, #545 and #547 had grown into incident
write-ups: measurement runs, prior values, how a problem was found,
what a check used to be. That belongs in commit messages and the wiki,
both of which already carry it.
Trimmed, no behaviour change:
cephcluster.yaml 22 lines -> 6 (cephx rotation)
cephfs-ci.yaml 26 lines -> 7 (why size 2)
prometheusrule-ceph.yaml (CephHealthWarn, 43 -> 13;
CephPoolLostRedundancy, 15 -> 7)
The operative facts are kept: which checks are excluded and why they
cannot clear, that POOL_NO_REDUNDANCY reports blocked maintenance and
not only data risk, and that size 1 needs dedicated OSDs.
The unreplicated CI pools introduced in #542 make every OSD on the
cluster permanently un-restartable.
`ceph osd ok-to-stop` refuses while stopping an OSD would take PGs
offline. A size-1 pool puts every PG on exactly one OSD, so the check
fails for all three:
Error EBUSY: unsafe to stop osd(s) at this time
(20 PGs are or would become offline)
bad_become_inactive: 13.6 13.9 ... 14.1c 14.1d
Pools 13 and 14 are cephfs-ci-metadata and cephfs-ci-data0. The size-2
pools are not implicated — they run min_size 1 and appear under
ok_become_degraded, which is allowed.
Rook gates OSD restarts on that check, so this blocks CephX key
rotation, Ceph upgrades, node drains and reboots, and drive
replacement. It surfaced when #545 deadlocked: Rook retried
ok-to-stop against all three OSDs several times a second and could
never succeed, while `ceph -s` reported 217 pgs active+clean
throughout. Nothing is unhealthy; the restarts are simply refused.
The reasoning in #542 was that unreplicated CI data only risks
in-flight pipelines, since every byte is a fresh git clone. That part
holds. What it missed is that the cost is not scoped to the pool: one
unreplicated pool freezes maintenance for the entire cluster.
Most of #542's saving came from keeping CI churn off `cephfs`, whose
metadataPool is size 3 — that survives this change. Only the second-
order gain from dropping the replica is given up, and cephfs-ci-data0
stores 0 B with cephfs-ci-metadata at 182 MiB, so the backfill is
trivial.
size 1 becomes viable again once these pools sit on OSDs of their own,
where the permanently un-stoppable OSD holds nothing but throwaway
data. That is planned with the retired QLC drives.
Alerting follows the same change:
- POOL_NO_REDUNDANCY is no longer excluded from CephHealthWarn. It was
excluded because cephfs-ci was expected to keep it permanently
active; no pool is unreplicated now, so the exclusion would only
hide a real one losing its replicas.
- CephPoolLostRedundancy drops its cephfs-ci exemption for the same
reason, and its description now states the maintenance consequence,
not just the data-loss one.
- The AUTH_INSECURE_* exclusion is reworded. Calling it
"known-accepted" reads as a decision that the risk is fine; it is a
kernel-blocked remediation (aes256k over the CSI mount path needs
kernel 7.0+, nodes run 6.18).
Refs #542, #545, #527
Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.
The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.
Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.
Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.
CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.
Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.
Refs #527, #528
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CephHealthWarn alerted on `ceph_health_status == 1`. It has been FIRING since
at least 2026-08-21T00:26Z and cannot clear: four AUTH_INSECURE_* checks are
permanently active from CVE-2025-30156 (#527) and stay that way until Rook
>=1.20.5 lands. A permanently-firing alert reports nothing -- every Ceph
warning since then has been invisible behind it.
This is a pre-existing gap, not a consequence of the cephfs-ci work, though
that work would have added a second permanent cause (POOL_NO_REDUNDANCY, since
the CI pools are size 1 on purpose).
ceph_health_detail exposes each check as its own series, so the alert now fires
on any check that is NOT known-accepted, and names it. Verified against live
data: 5 checks active, 0 unexpected, so it is quiet now and fires the moment
something genuinely new appears.
Excluding POOL_NO_REDUNDANCY wholesale would give something up: that check is
cluster-wide and does not say WHICH pool, so accepting it for cephfs-ci would
equally hide a production pool dropping to one replica. CephPoolLostRedundancy
restores that. There is no pool-size metric, but bytes_used/stored IS the
replication factor -- verified live as replicapool 2, cephfs-data0 2,
cephfs-metadata 3, cephfs-ci-metadata 1.
Both alerts were tested against live data before committing, in BOTH
directions. The positive control matters as much as the negative one: an early
draft used a >1e9 floor which silently excluded the CI pools entirely, so the
expression could not have matched anything. It looked correct precisely because
it returned no results. The control caught it; the floor is now 1e6.
The floor is not what suppresses the near-empty rgw pools. Their ratios are
9-36 because per-object overhead dominates, so they never match `< 1.5`.
6h on CephHealthWarn is unchanged -- Ceph warns for self-clearing conditions
and paging on those trains people to ignore the alert.
Repoints WOODPECKER_BACKEND_K8S_STORAGE_CLASS at ceph-filesystem-ci, the
dedicated filesystem added in #542 whose pools are both size 1.
Measured 2026-08-21, polling both CephFS pools every 15s:
CI active 1.5 min (6 intervals): data0 143.2 MB metadata 17.2 MB
CI idle 16.0 min (63 intervals): data0 0.0 MB metadata 11.0 MB
CI share of writes: data0 100%, metadata 61%
data0 takes zero bytes across 16 minutes without a pipeline, so its entire
write volume is CI. The 15s cadence is what made this visible: a pipeline
creates, uses and deletes its PVC inside ~2 minutes, so a 5-minute sample
reported no CI pods during bursts CI was causing.
Per pipeline this moves the drive cost from 71.6x2 + 8.2x3 = ~168 MB to
~80 MB, since neither pool is replicated on the new filesystem.
RWX is unchanged and still required: pipeline steps share the workspace PVC
and the k8s backend can schedule them on different nodes, so an RWO volume
fails to mount on the second node. Only the filesystem underneath changes.
Also corrects the justification written into #542 for leaving
ceph-filesystem-ephemeral alone. That commit claimed restore-test.sh handles
"REAL data which must not land on unreplicated pools". That is wrong: its
probe PVCs are as throwaway as a CI workspace -- the authoritative copy is the
backup on Hetzner, and losing a probe just means re-running the test.
The actual reason it cannot move is technical: those probes are snapshot
CLONES, and `ceph fs subvolume snapshot clone` operates within a single
filesystem, so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
Landing them unreplicated would require a second size-1 data pool inside
`cephfs`; restore-test runs weekly against CI's ~96 pipelines/day, so that is
not worth the complexity.
cephfs-ci verified healthy before this change: PHASE Ready, MDS Running, both
pools reporting size 1.
Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
checkout plus build output, and deletes it when the pipeline ends. Those
workspaces currently live on the shared `cephfs`, so throwaway data is
replicated as if it mattered: 2x in cephfs-data0 and 3x in cephfs-metadata.
Measured 2026-08-21 by polling both pools every 15s. The interval matters --
at 5 minutes a whole pipeline is invisible, because its PVC is created, used
and deleted inside ~2 minutes. Sampling that slowly reported zero CI pods
during bursts CI was causing, which sent this investigation down three wrong
paths before the cadence was fixed.
one pipeline, a single 15s interval: data0 +71.6 MB metadata +8.2 MB
every interval with no CI pod: data0 ~0 metadata ~0
So CI accounts for essentially all CephFS write volume; between pipelines the
metadata pool is quiet at ~4 MB/day. On the shared filesystem one pipeline
costs 71.6x2 + 8.2x3 = ~168 MB of drive writes. On this one it costs ~80 MB.
size 1 is correct here even though it was rejected for `cephfs`. There,
metadataPool stays at 3 because losing it destroys the entire filesystem
rather than the objects on one OSD, and 550 GiB of real data depends on it.
Here every byte is a fresh git clone: total loss means in-flight pipelines
fail and get re-run. Replicating throwaway data was the cost, so it is what
gets removed.
`ceph-filesystem-ephemeral` is deliberately NOT repointed. It has a second
consumer -- velero/restore-test.sh restores REAL data through it for
verification, which must not land on unreplicated pools. A new class also
sidesteps StorageClass.parameters being immutable.
Nothing uses ceph-filesystem-ci yet, so this commit is inert: it provisions
the filesystem and its MDS and changes no existing workload. Repointing the
Woodpecker agent is a separate change, once this is confirmed healthy.
Known consequence: size 1 raises POOL_NO_REDUNDANCY, so cluster health will
warn until that is addressed. Silencing it via mon_warn_on_pool_no_redundancy
is cluster-wide and would also hide a production pool dropping to 1, so it
wants a targeted alert instead -- deliberately left out of this commit rather
than switching the warning off blind.
The MDS is 512Mi/1Gi with no standby, not the 4Gi active+standby that `cephfs`
runs. That figure is driven by immich's cached inode count; a CI workspace is
a few thousand files that vanish with the pipeline, and a second daemon to
protect throwaway work is not worth its memory.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.
gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.
Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.
ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.
ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.
Also corrects two comments in ha-pg-cluster.yaml that this work falsified.
wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.
checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.
Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.
This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.
Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.
The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.
archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
`wal_compression: "on"` resolves to pglz, PostgreSQL's original 2016-era
algorithm. Every CNPG cluster here was set that way, so none of them were
getting the compression actually available to them: all seven builds accept
lz4 and zstd (verified via pg_settings.enumvals on each, including immich's
custom pgvector image and trmnl's PostgreSQL 18).
This matters because of what dominates recorder-style write volume. A page
is written to WAL in full -- all 8 KiB -- on its first modification after a
checkpoint, for torn-page protection. Measured on ha-pg, those full-page
images are 62.4% of WAL bytes. Compressing them is therefore the single
largest lever available on WAL volume, and pglz was the weakest option for
the job.
ha-pg additionally moves checkpoint_timeout 15min -> 30min. Full-page images
are emitted once per page per checkpoint interval, so doubling the interval
roughly halves how often each hot page pays the 8 KiB toll. max_wal_size
stays at 2GB deliberately: it must remain well above what one interval
generates, or checkpoints would trigger on size and the interval change
would do nothing. Recovery replays at most one interval of WAL -- under
200MB at the measured rate, i.e. seconds.
The other six clusters keep checkpoint_timeout at 15min. Their measured
write rates do not justify trading recovery time: gf-pg generates 4.5 MB/day
of WAL and wp-pg 62 MB/day, against ak-pg's 604 MB/day.
full_page_writes stays on everywhere. Disabling it would remove the
overhead entirely but RBD can tear a page across an object boundary, and
the trade is write volume against an unrecoverable database.
Both parameters are sighup-reloadable, so this does not restart any cluster.
Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
An image reference is not a secret. Encoding it as base64 `data` had two real
costs: no reviewer sees a version change in a diff, and no $imagepolicy marker
can attach to a base64 blob.
The result was a two-minor version skew in the component that performs every
base backup and WAL archive for seven Postgres clusters (#535):
controller plugin-barman-cloud:v0.14.0 <- auto-bumped by Flux
sidecar plugin-barman-cloud-sidecar:v0.12.0 <- invisible, never moved
Not stale pods: ha-pg-1 was created hours ago during the recorder migration and
still received v0.12.0.
Three changes:
1. The Secret now uses stringData with the tag in plaintext, carrying an
$imagepolicy marker. The Secret KIND is kept because the controller reads it
via secretKeyRef; switching to a ConfigMap would need a matching edit to that
reference and diverge further from upstream. stringData is accepted
identically by the API server.
2. ImageRepository + ImagePolicy added for the sidecar image, with the same
filter and 0.x range as the controller, so the two cannot drift apart again.
Only the controller was tracked before.
3. Tag aligned to v0.14.0 to match the controller.
The Secret name carries a kustomize hash suffix from upstream's secretGenerator
but is static here, so editing the value does not change the name and the
deployment's secretKeyRef stays valid — verified in the render, which still shows
the name twice (Secret + reference).
Verified the tag EXISTS before pinning it: pulling
plugin-barman-cloud-sidecar:v0.14.0 succeeds (the pod fails later on a missing
`true` binary in that minimal image, which is the container command, not the
pull). Pinning a non-existent tag would have put all seven Postgres clusters into
ImagePullBackOff.
EXPECTED IMPACT: bumping the sidecar changes every Postgres pod spec, so all
seven CNPG clusters roll. CNPG handles this as a normal rolling restart. Backups
should be confirmed working afterwards — 113 Backup CRs exist today, newest ~3h
old and completed.
Note for future re-imports: upstream ships this Secret as base64 `data` with no
marker, so a wholesale re-import silently reintroduces the skew. The manifest now
says so inline.
Refs #535
Co-Authored-By: Claude Bot <noreply@anthropic.com>
Switches the Deployment's `data` volume from paperless-data (RWX,
ceph-filesystem) to paperless-data-block (RWO, ceph-block). `consume` and `media`
stay on CephFS -- the ftp-scanner writes to those from another node, so they
genuinely need RWX.
The copy is already done and verified byte-for-byte with paperless stopped:
src files: 75
dst files: 75
IDENTICAL: name, size and mtime match for every file
index segments: src 3, dst 3
mtime parity matters specifically because the Whoosh index segments carry their
own consistency expectations; `cp -a` preserved them. (The "can't preserve" lines
in the copy log were for the mount root /dst/. itself, not the files.)
Why: only this Deployment ever mounted the volume, and its contents are
single-writer state. Being on CephFS meant every access crossed the MDS --
measured with zero CI running, 22 requests/sec sustained and ~15 journal
events/sec against 2 directory commits per 180s, i.e. cap/inode churn rather
than file writes. That accounted for most of the constant 7-8 GB/day on
cephfs-metadata, which is a size 3 pool.
cephfs-metadata stays at size 3 deliberately: losing it destroys the entire
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
depends on it.
paperless-data is retained (prune: disabled) as the rollback. It must NOT be
deleted until search has been confirmed returning results -- that is what proves
the index survived, not merely that paperless starts.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.
It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:
paperless-data client request_load_avg 22 (all other sessions 0-1)
MDS over 180s 4,052 client requests = 22.5/s
2,718 journal events = 15/s
2 dir commits <- not file writes
Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.
cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.
PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).
Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten Postgres instances across seven CNPG clusters are ~48 GB/day of the
cluster's ~114 GB/day of client writes, and only ha-pg carried any
write-reduction tuning. The rest ran Postgres defaults, which favour throughput
over storage that wears out.
MEASURED, not assumed. ak-pg since 2026-05-17:
checkpoints_timed 27,624 = 291/day, i.e. one every 5 min (the default)
buffers_checkpoint 25,801,628 = ~206 GB flushed
database size 1,046 MB
That is the whole database rewritten roughly twice a day. The pathology is
specific to idle databases: sessions, tokens and job state re-dirty the same few
pages constantly, and checkpoint_timeout=5min forces each of them to disk 288
times a day instead of letting them coalesce. At 15min they are written a third
as often.
max_wal_size is raised alongside it. Without that, checkpoints trigger on size
instead and the timeout change does nothing. It also bounds crash recovery to at
most 2GB of WAL replay.
Applied to ak-pg, pprl-pg, tr-pg, wp-pg, im-pg, gf-pg. ha-pg already had it.
synchronous_commit is deliberately NOT changed here. It is off for ha-pg because
a state recorder is telemetry; these clusters hold credentials, documents and job
state, where losing the last few hundred ms of commits on an unclean shutdown is
not an acceptable trade for write volume.
Separately, ha-pg shared_buffers 256MB -> 512MB. After 15h on Postgres its
buffers_backend was 629,704 of 939,025 total page writes -- 67% written by
backends rather than the checkpointer or bgwriter, which means the working set
does not fit in the buffer pool and pages are written out repeatedly instead of
coalescing. Memory limit raised 1Gi -> 1536Mi to keep the same headroom (measured
383Mi in use, of which 256MB was shared_buffers).
Verified before commit by rendering every cluster and asserting the tuning landed
under spec.postgresql.parameters with the barman plugin block untouched. That
check exists because the first attempt at this matched the wrong `parameters:`
key -- the barman plugin's, at 6-space indent under plugins: -- and corrupted
wp-pg's barmanObjectName. Reverted and redone with an indent-aware match.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.
Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.
HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:
/var/lib/code-server 8.0K
/mosquitto/auth 8.0K
/state (z2m guards) 8.0K
So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.
Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.
Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.
DELIBERATELY NOT CONVERTED:
- kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
and generated from the chart; a hand edit would be lost on upgrade.
- trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.
Cost: this rolls the affected pods once.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The renderer was the single largest writer in the cluster -- more than every
Ceph OSD and every database combined -- and none of it needed to touch disk.
Measured 2026-08-20 from the container's own cgroup io.stat:
253:0 (node /dev/vda5) 249,092,697,088 bytes over 7.3 days = ~34 GB/day
252:48 (its own PVC) 340 MB total = negligible
So the writes were not going to its volume at all; they were going to the
node's filesystem via a disk-backed emptyDir. `emptyDir: {}` without
`medium: Memory` is node storage, which is easy to read as "ephemeral, so
free".
The cause is cadence rather than volume. schedules.template.json renders on
"* * * * *", so Chrome is launched 1440 times a day and each launch creates,
fills and deletes a /tmp/puppeteer_dev_chrome_profile-* directory. At rest /tmp
holds under 8 MB -- it is pure churn for data that never outlives the render.
tmpfs pages count against the container's memory limit, so the 512Mi cap is
kept deliberately: measured usage is 129Mi against a 2Gi limit, leaving ample
headroom even if the cap were fully consumed.
Expected effect: ~34 GB/day off the OS disks, which measured 215 GB/day over a
4h window. This is a bigger cluster-wide reduction than the Home Assistant
recorder migration achieved, for a one-line change.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Net -44 lines. The previous version assembled the connection URI from
username/password/dbname, which forced percent-encoding, YAML-safe quoting via
json.dumps, a YAML round-trip validation, and python3 from HA's image.
None of that was necessary: CNPG already publishes a finished, percent-encoded
connection string in the app secret as `uri` / `fqdn-uri`. Reading that key
removes the whole class of problem the earlier review correctly flagged --
rather than solving it more carefully.
Verified against the live secret before simplifying: fqdn-uri contains only
[A-Za-z0-9:/@._%-], so naive double-quoting is provably safe and the YAML
validation was guarding against an input that cannot occur.
What is kept, because it is the part that actually matters: secrets.yaml is PVC
state holding every other HA secret (influxdb_token and friends), so the file is
filtered-and-appended rather than rewritten, moved atomically, and guarded
against shrinking. Tested in HA's own busybox shell -- comment and pre-existing
keys preserved, idempotent across runs, no duplicate key.
Also records why an env var cannot be used at all: HA removed `!env_var` from
its YAML loader, so `!secret` is the only mechanism for recorder.db_url, and
config-sync deliberately does not manage secrets.yaml.
Back to busybox; HA's image is no longer needed for this.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2b is done: 12.77M statistics rows copied SQLite -> Postgres with HA
running and no downtime (read-only mount; SQLite WAL gives readers a consistent
snapshot).
statistics_meta 1449 OK
statistics 10799904 OK 153.9s
statistics_short_term 1969804 OK 27.0s
span in Postgres: 2024-03-13 -> 2026-08-19
1449 entities with long-term history, 2627859 rows carrying sum
database size 2252 MB against a 20Gi PVC
Records the high-water marks the cutover top-up must resume from, and two schema
details that cost a failed first attempt:
- has_mean/has_sum are INTEGER in SQLite, BOOLEAN in Postgres, and Postgres
will not coerce them.
- the legacy created/start/last_reset DATETIME columns are 100% NULL in both
bulk tables, which is what makes a tab-separated COPY safe.
Also documents why the schema was built from HA's own models with schema_changes
left EMPTY: HA's _initialize_database detects a new database via the
events.time_fired_ts index and writes SCHEMA_VERSION itself, so pre-stamping that
row would bypass HA's own initialisation.
ANALYZE was run on all three tables. After a bulk COPY the planner has no
statistics and HA seq-scans 10.8M rows on every history/energy view.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #524, and it was a genuine blocker.
The first version built "postgresql://${PGUSER}:${PGPASSWORD}@host/db" in
busybox sh with no encoding. Two defects:
1. The CNPG-generated password is random and may contain @ : / ? #, each of
which changes how the URL parses. This repo already hit this exact issue
with the same CNPG password-generation mechanism -- see
kubernetes/apps/woodpecker/server.yaml, which switched to a lib/pq
keyword DSN specifically to avoid escaping generated passwords. That
option does not exist here: HA's recorder.db_url must be a SQLAlchemy URL.
So the fix is percent-encoding.
2. Worse, and the reason this was worth blocking: a quote in the password
makes the YAML line malformed, so secrets.yaml stops parsing entirely and
HA loses EVERY secret, not just this one. The line-count guard could never
catch that -- the count is unchanged; the file is simply invalid.
Rewritten to run python3 from HA's own image (already pulled for the main
container, so no extra image) instead of busybox:
- urllib.parse.quote(..., safe="") on user, password and dbname.
- json.dumps for the YAML value: YAML is a JSON superset, so this emits a
correctly escaped double-quoted scalar even with quotes or backslashes.
- The file is still rewritten LINE-WISE rather than round-tripped through the
YAML parser, so hand-maintained comments and key order survive.
- The guard now validates the RESULT: parse the new content, assert no key
from the old file disappeared, and assert recorder_db_url round-trips to
exactly the URL that was built. That catches corruption the line count
cannot.
Verified in HA's image against a deliberately hostile password
(p@ss:w/o"rd#?&\x=1) before commit:
new version -> parses, comment preserved, idempotent, password decodes back
byte-exact, host and dbname parse correctly
naive version -> yaml.safe_load raises ParserError
i.e. the approach this replaces would have produced an unparseable
secrets.yaml.
Also updates MIGRATION-recorder-postgres.md for this step landing, including
why the ordering (init container before the configuration.yaml change) is
mandatory.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2a of the recorder move off SQLite (MIGRATION-recorder-postgres.md).
Adds an init container that writes `recorder_db_url` into /config/secrets.yaml
from the CNPG-generated ha-pg-app Secret.
INERT until the recorder is repointed. It only adds a key to secrets.yaml;
nothing reads it until `db_url: !secret recorder_db_url` lands in
configuration.yaml in the homeassistant repo. That ordering is REQUIRED in this
direction — HA refuses to start when a referenced secret is missing, so the
config change must not go first.
Why an init container rather than a hand-added line: the password is generated
by CNPG and can be rotated, so a hand-written entry goes stale silently and HA
then fails to start on a bad password. Re-rendering on every pod start is also
what makes a rotation take effect.
secrets.yaml is PVC runtime state that config-sync deliberately does not manage
(it holds influxdb_token and others), so this must EDIT the existing file, never
replace it. The script filters out any existing recorder_db_url line, appends
the current one, and guards the result: if the rewrite would drop any
non-recorder line it aborts rather than truncating the file holding every other
secret HA needs. Written to a temp path and moved atomically, chmod 600.
Verified in HA's own Alpine/busybox shell against a fixed sample before commit
-- pre-existing keys preserved, guard reports old=3 new=3, and a second run
produces a byte-identical 4-line file (idempotent, no duplicate key).
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven Postgres clusters back up to Hetzner and nothing alerted if that stopped.
The only related rule was HetznerBackupEndpointDown, which probes the S3
endpoint — it proves the bucket is reachable, not that anything was written to
it. Same failure shape as the WordPress backup that failed nightly for three
days against a bucket declared in tofu but never applied: the status surface
looked fine because nothing checked the thing that mattered.
CNPGWalArchiveBacklog is the load-bearing rule. `ready` counts WAL segments
waiting in pg_wal/archive_status; steady state is 0 on every cluster. A
sustained backlog means the archiver is failing, so point-in-time recovery is
silently broken from that moment and pg_wal grows until it fills the PVC and
takes Postgres down. Threshold >5 sustained for 30m: a segment is 16 MiB, so
that is ~80 MiB of un-archived WAL — well below anything that threatens the
volume, far above idle noise. It reads only the instance-level collector, so it
works despite #522.
CNPGBackupFailed fires on a base backup failure within 24h, guarded on
timestamp > 0 so the epoch-zero default is not read as "failed in 1970".
DELIBERATELY NOT ADDED: the backup-age alert, which is the one you would most
want. `(time() - cnpg_collector_last_available_backup_timestamp) > 48h` cannot
be used today because that metric reads exactly 0 on every instance in the
cluster (verified across gf-pg, ak-pg-1/2, pprl-pg-2/3, im-pg, wp-pg, ha-pg).
CNPG has no record of any successful backup anywhere, because the Backup CRs it
derives that from are deleted within a minute of completing — #522. Adding it
now would fire permanently for all seven clusters and be silenced within a day,
which is worse than not having it.
The backups themselves DO work; verified independently by listing the bucket
with barman-cloud-backup-list and restoring a WAL segment back out of S3. The
gap is in the record, not the data.
Both expressions were validated against live Prometheus before commit rather
than written from the metric names: the WAL selector matches 8 real series (a
zero match would mean a dead rule, the defect fixed in
prometheusrule-woodpecker-rotation.yaml), and neither expression fires
currently, which is correct since archiving is healthy and the only recorded
backup failure is ~91 days old.
Refs #522
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recorder's SQLite database is the largest client writer in the cluster.
Measured 2026-08-19 via cgroup io.stat on the home-assistant container,
corroborated by kernel diskstats and Ceph pool counters: 40 GB/day to its RBD
volume, ~26% of all client writes, against only ~5 GB/day of actual row data.
The 8x gap is the write PATTERN, not the volume:
~30 state changes/sec (107k rows/hour, 2.57M/day)
x2 SQLite WAL + checkpoint -- every dirtied page written twice
x4 ext4 journal + commit, since synchronous=FULL fsyncs every commit
(measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
= ~15 KB written per recorded state change
Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
This attacks the multiplier rather than trimming what gets recorded, which would
hide the problem instead of fixing it.
PHASE 1 ONLY. The cluster comes up EMPTY and nothing consumes it -- HA stays on
SQLite. Deploying it is inert, so there is nothing to roll back if it misbehaves.
Phase 2 migrates the statistics tables and repoints recorder.db_url in the
homeassistant repo; MIGRATION-recorder-postgres.md has the sequence.
What phase 2 must not get wrong, recorded now while it is fresh: HA does NOT
migrate data between database engines. `statistics` holds 10,793,712 rows
spanning 2024-03-13 to today -- 2.4 years of energy history that is not in
InfluxDB and cannot be reconstructed. Repointing db_url at an empty database
silently discards it. `states` (18M rows) is disposable at purge_keep_days: 7.
Postgres is tuned to minimise write volume, which is the point of the move:
synchronous_commit off -- the recorder is telemetry, not transactional
record-keeping. Postgres stays crash consistent;
only the last few hundred ms of commits are at risk
on unclean shutdown. In exchange, commits stop
forcing an fsync -- the exact cost that makes SQLite
expensive here.
wal_compression on -- recorder writes are small inserts scattered across
pages, so post-checkpoint full-page writes dominate
WAL volume.
checkpoint_timeout 15min / max_wal_size 2GB
-- fewer checkpoints means fewer repeat full-page
images for the hot rows the recorder rewrites.
full_page_writes -- deliberately LEFT ON. RBD can tear a page across a
4 MiB object boundary; trading that for write volume
risks an unrecoverable database.
Also included:
- ObjectStore on its own s3 prefix (two clusters sharing a destinationPath
interleave WAL streams and corrupt both).
- ScheduledBackup at 02:50 UTC, offset from the other CNPG clusters.
- PodMonitor added WITH the cluster rather than after it, so the migration has
metrics from the start instead of discovering the gap later (#433).
- Cilium policies as DEDICATED policies with their own endpointSelector, not
rules on the namespace-wide default-deny -- that policy is endpointSelector:{}
so any rule there applies to every pod in the namespace, and :8000 is served
by more than the CNPG instance manager. Includes the ICMP allowance that
attaching L4 rules to the ha-pg endpoints makes necessary for PMTUD.
The hetzner-creds ExternalSecret reuses seed/paperless/hetzner-creds, the same
coupling #397 tracks for woodpecker. Noted in the file with the exact bao command
to fix it; the mount is `homelab`, so #397's recorded 403 on `bao kv list seed`
was a wrong-path error rather than missing permissions.
Refs #434, #397
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three Ceph monitors were writing 240 GB/day between them for a ~106 MB
database. That was ~62% of all OS-disk writes on the nodes and roughly a
quarter of everything the cluster writes -- more than every client workload
combined.
Measured 2026-08-19:
paxos transactions ~99,000/day (1.15/s, sampled over 60 s)
paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime)
ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three
(cgroup io.stat: 2.15 TB / 27 d)
=> ~98x write amplification
Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix
(including paxos) when we trim its old states". With paxos_trim_min at 250 and
~99k transactions/day, that is hundreds of forced range compactions per day,
each rewriting the store.
The churn driving it is not cluster activity. 99% of paxos versions are cluster
log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook
operator poll (mgr stat / versions / status / report / osd dump; 23 audit
entries per minute per mon) becomes a cluster-log entry, hence a paxos
transaction, replicated to all three mons. Reducing that polling is the
complementary fix and is tracked separately; this change removes the
amplification regardless of the rate.
This does NOT disable compaction. RocksDB's own levelled compaction still runs
and still reclaims trimmed keys; what stops is the extra forced compact_range
on every trim. Two backstops if that assumption is wrong:
- mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn
in monitoring-rules already alerts on.
- mon_compact_on_start is set to true here (upstream default false), so every
mon restart -- upgrade, node reboot, reschedule -- does a full compaction.
Cheap on a ~106 MB store.
Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z:
wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z.
Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly
double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212,
and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank
writers; every figure above comes from cgroup io.stat or kernel diskstats, which
reconcile with each other and with the drives' SMART counters.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With 3 OSDs on 3 hosts, size=3 meant every write landed on every drive in the
cluster. Per-drive write volume equalled TOTAL cluster write volume, so all
three drives wore at an identical rate and would have reached end-of-rated-life
together: measured 2026-08-19 from the drives' own SMART counters, 80/82/87%
endurance used, advancing ~0.13 points/day, exhausted between Nov 2026 and
Jan 2027 (#434). Replication gives no protection there — the wear-out is
correlated by construction, not independent.
size 2 cuts per-drive write volume by a third and desynchronises the wear.
min_size 1 is required, not incidental. Ceph does NOT lower min_size when size
is reduced; it was 2, and leaving it would make replicapool 2/2, where a single
OSD down blocks IO for its PGs. That would hang every RBD volume in the cluster
on every rolling node reboot and every OSD restart.
Accepted trade, stated explicitly: while degraded (one OSD down), a second
drive failure loses data, and at 80%+ endurance that window is not
hypothetical. What limits the blast radius is that CNPG barman and Velero both
ship offsite to Hetzner, so exposure is the RPO of those backups rather than
total loss. Node reboots must stay serialised with HEALTH_OK between them.
CephFS metadataPool deliberately stays at size 3: 467 MiB against data0's
275 GiB, so it contributes nothing measurable to wear, while losing it destroys
the entire filesystem rather than the objects on one OSD. The wear saving comes
from data0. The RGW pools in objectstore.yaml were already size 2.
One-time cost on apply: Ceph deletes the third replica of everything, which is
background IO, and raw used drops from ~1.4 TiB to ~0.95 TiB.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bluestore_prefer_deferred_size_ssd was set to 65536 and
bluestore_deferred_batch_ops_ssd to 64 as a QLC throughput tune. Ceph's SSD
defaults are 0 and 16. Reverted to the defaults, set explicitly so the stored
`ceph config` value is overwritten rather than merely un-declared.
prefer_deferred_size defers every write SMALLER than the threshold: it lands in
RocksDB's WAL first and is applied to the object store afterwards. That is a
double write, and at 64 KiB it caught essentially every write this cluster
makes -- SQLite pages are 4 KiB, Postgres pages 8 KiB. Ceph disables deferral on
SSD by default because solid state does not need the random-write batching that
helps spinning media.
Why the original tune was the wrong trade:
- The benchmark justifying it cannot have exercised it. The 2026-05-05
baseline used 4 MiB writes, 64x above the 64 KiB cutoff, so none of those
writes were ever deferred. The setting was validated by a workload it does
not affect.
- It doubles writes onto QLC, the lowest-endurance NAND type. Measured
2026-08-19 from the drives' own SMART counters: all three OSD devices at
80/82/87% endurance used, advancing ~0.13 points/day, i.e. rated life
exhausted between Nov 2026 and Jan 2027. Because size=3 on 3 OSDs, every
drive absorbs 100% of cluster writes, so they wear in lockstep and expire
together (see #434).
Measured amplification before this change: 640 GB/day written to the three NVMe
devices against 117 GB/day at the RBD client layer -- 5.5x, of which 3x is
replication and ~1.8x BlueStore. Removing the double write should take the
BlueStore factor toward ~1.1x.
The cluster is write-light, so throughput was never the binding constraint;
endurance is. Both parameters are runtime-updatable, so no OSD restart.
Also corrects the drive model in the comment: the OSD devices are
CT1000P3PSSD8 (Crucial P3 Plus), not the P2 named previously.
bdev_enable_discard / bdev_async_discard are unrelated and kept as-is.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the noise-log dedup, both of which silently mute real alerts.
1. Prose-mentioned fingerprints became suppressions.
The extractor grepped the WHOLE issue body for 16-hex tokens. Triage reasons
routinely cite another alert's fingerprint in prose ("see the issue filed for
fingerprint <fp>"), so those citations were read as suppression entries.
Verified against the live noise log: the old extractor yields 22 fingerprints
for 21 real entries. The extra one is 2ddc41edb689fc33 -- the crowdsec alert
that HAD been filed as a genuine issue, permanently muted purely because two
other entries mention it. Now only lines in the exact shape the write-back step
emits count:
- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>
Re-verified with the new parser: 21, and 2ddc41edb689fc33 is no longer among
them.
2. Suppression never expired.
The fingerprint is sha256(alertname|scope) and is deliberately stable across
recurrences, so a single blip verdict muted that alertname+scope pair forever.
21 pairs were permanently muted, including openbao's KubePodNotReady,
KubeStatefulSetReplicasMismatch and KubePdbNotEnoughHealthyPods -- so a real
openbao outage would have filed no issue at all.
A noise verdict means "this occurrence was benign", not "this alert can never be
interesting again". Entries now expire after NOISE_TTL_DAYS (default 30) counted
from the date already present in each line. Re-triaging a recurrence weeks later
is cheap; missing an outage because a July blip muted it is not. If it is still
noise, triage re-adds it automatically.
Rollout is gentle by construction: every current entry is dated 2026-07-21 or
later, so at a 30d TTL the cutoff is 2026-07-20 and NOTHING un-mutes on merge --
the oldest entries lapse a day later, and only re-triage if they actually fire
again.
Unparseable lines are dropped, i.e. they stop suppressing: this fails toward
more alerting, never less.
Verified end-to-end by running the shipped snippet inside the real
ci-claude:2.1.218 image against the live noise log: cutoff 2026-07-20,
21 suppressions, prose-only fingerprint absent. Note that image ships BusyBox
date, not GNU -- `date -d '30 days ago'` errors there, so the cutoff uses epoch
arithmetic with `date -d @<epoch>`, which was confirmed working in that image.
Refs #326
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #516: the restart-triggering mosquitto_pub was unbounded,
unlike the sibling call in zigbee2mqtt-interview-watch.sh.
Neither client bounds a TCP connect that HANGS rather than being refused, and a
stalled connect is most likely precisely when this guard runs -- around a broker
restart. Unbounded, the loop would block forever and stop logging, which is the
silent-failure mode the guard exists to prevent.
Applied to the reads as well, not just the publish: mosquitto_sub -W bounds the
wait for a message once CONNECTED, so it does not cover the connect phase
either. The timeout wrapper is set above -W so the protocol-level wait still
wins in the normal case.
Not mosquitto_pub -W: that flag does not exist for mosquitto_pub and makes every
publish fail on a usage error.
The cooldown stamp was already written only on a successful publish, so a failed
restart request still retries on the next cycle rather than burning the cooldown.
`timeout` is confirmed present in eclipse-mosquitto:2.0.22 (already used by
interview-watch, and exercised directly in this image).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mosquitto runs `persistence false` with no PVC (deliberate and settled), so
every broker restart wipes the retained store. Zigbee2MQTT publishes its HA
discovery configs retained and has no way to restore them itself, so a broker
restart not followed by a Z2M restart leaves every Z2M entity unavailable in HA
indefinitely -- ~14 h on 2026-07-23.
Adds a discovery-guard sidecar to the Z2M pod that detects the wiped state and
asks Z2M to restart, which re-runs discover().
Detection. bridge/state and bridge/{info,devices,groups,definitions} are all
retained, but only bridge/state is republished on reconnect (mqtt.ts onConnect).
So "bridge/state present + bridge/groups absent" means Z2M is talking to this
broker but nothing has re-run start() since the store was wiped.
bridge/groups specifically, because the absent-half partner must not be
re-publishable by ordinary runtime events or it self-heals and the guard goes
blind while HA is still broken. bridge/devices, bridge/definitions and
bridge/info all reach inbound radio traffic via devicesChanged (receive.ts hands
deviceExposesChanged to every fromZigbee converter); bridge/health self-heals on
its own 10 min timer. bridge/groups is only republished by deliberate admin
actions, and publishGroups() always publishes, emitting `[]` when no groups
exist -- confirmed live: it currently holds a 2-byte payload, so presence is
testable.
Not AND-ed with bridge/info being absent. That would cut false positives at the
cost of false negatives, and the costs are asymmetric: a false positive is one
~60s Z2M restart, a false negative is every Zigbee entity dead until a human
notices. Tuned for sensitivity.
Spurious positives are handled by a 90s confirm re-check, which covers both the
startup race (bridge/state is published at mqtt.connect(), bridge/groups only at
the end of BridgeExtension.start()) and a dropped publish (these topics go out at
QoS 0). A 30 min cooldown keeps a Z2M that will not come up healthy from being
restarted in a loop.
Action is bridge/request/restart, which is a full in-process restart -- index.js
restart() -> stop(true) tears down every extension, then start() builds a new
Controller and re-runs HomeAssistant.start(). discover() then republishes only
missing or changed configs, which after a wipe is all of them.
Reuses the existing zigbee2mqtt broker credential (ACL `readwrite
zigbee2mqtt/#`), so no new broker user and no read grant on homeassistant/#,
which the ACL denies on purpose.
Verified in the eclipse-mosquitto:2.0.22 image the sidecar actually runs:
mosquitto_sub -C/-W, mosquitto_pub and `date -u +%s` all present; `sh -n` clean;
startup run logs correctly and enters the loop. Detection was exercised against
the live broker and correctly reports healthy (bridge/state online,
bridge/groups present) with no action.
NOTE: this rolls mosquitto, because the corrected ACL comment lives inside the
container command. That roll wipes the retained store, which the guard then
repairs within ~6 min if the Z2M roll does not already do so.
Closes#381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four comments across the broker manifests asserted that retained state is
re-populated by all clients on reconnect, and specifically that Zigbee2MQTT
re-publishes HA discovery when it sees the homeassistant/status birth message.
Both are false, and the second is the premise that kept issue #381 open.
Verified against Zigbee2MQTT 2.12.1 source:
- discover() runs only from HomeAssistant.start(); it is never called from
the reconnect path.
- lib/mqtt.ts onConnect() republishes bridge/state and resubscribes, nothing
more. republishRetainedTimer is armed only inside connect() and is never
re-armed on reconnect.
- lib/extension/homeassistant.ts onMQTTMessage(), on status_topic == "online",
republishes bridge/state plus cached device states after a hardcoded 30s
delay -- but never the discovery configs.
- Discovery is published retain:true, and the only knob is the global
mqtt.force_disable_retain, which upstream documents as breaking the HA
integration. Broker retention is the sole mechanism Z2M relies on.
- No bridge/request endpoint forces a rediscovery; upstream declined to add
one (Koenkk/zigbee2mqtt #27458, #22977, #16950, all closed NOT_PLANNED).
The mosquitto ACL was never the blocker: `read homeassistant/status` is already
granted. That grant is still correct -- the cached-state republish is what makes
HA show current values after an HA restart -- so it is kept and the comment now
says what it actually buys.
Also records that a RollingUpdate roll is equivalent to a broker restart for
retention: maxUnavailable:0/maxSurge:1 preserves connectivity, but the new pod
starts with an empty retained store.
Comment-only. The two deployment comments are plain YAML comments outside the
container command, so the rendered Deployment is byte-identical and the broker
does not roll; the ConfigMap carries no reloader annotation, so it updates in
place without restarting mosquitto.
Refs #381
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stage 2 of the decommission the shim was written for. Deleting the file makes
the next apply destroy the three containers.
The containers were taken out of service months ago — they are stopped
(pve_up = 0 for all three) and rustfs-{1,2,3}.srv.renner.uno no longer resolve
— but the LXC objects and their disks still exist on saturn/jupiter/neptune.
Two earlier destroy attempts (#270) died instantly with "context deadline
exceeded", because the bpg provider reads timeout_delete from STATE and the
imported resources had stored 0.
That is now fixed: apply 5675 landed timeout_delete = 1800 and the rest of the
timeouts in state, in place, with 0 destroyed. The destroy has a real deadline
to work with for the first time.
Checked before removing, since the containers are tagged `state-backend` and
`terraform-backend`:
- The tofu backend is Hetzner S3 (renner-uno-tfstate at
fsn1.your-objectstorage.com), not rustfs. Destroying these cannot pull the
state out from under the apply. The tags are historical, from before that
migration.
- No Kubernetes or Ansible resource references rustfs.
- scripts/restore-authentik-db.sh mentions rustfs only as an `mc` alias name;
its endpoint is s3.renner.uno:9000, the Ceph RGW `migration` store.
- DNS for the three names is already gone.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Both of these were learned the expensive way today and existed only in commit
messages and PR descriptions, which is not where anyone looks.
providers.tf carried a warning that a UniFi PUT may require _id in the body,
which restapi does not send, and that a 400 should be answered by tainting the
object so CI recreates it. That is wrong on this controller. Tested against a
live rule with exactly the body tofu sends: HTTP 200, object returned intact,
rule byte-identical before and after, 60 rules before and after. Confirmed at
scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and
immich.renner.uno still 200.
That wrong warning had a cost. A note saying an apply might break firewall
rules is one nobody argues with, so the apply did not get run, and the
renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
while the WordPress backup failed nightly with "The specified bucket does not
exist". The warning is replaced with the evidence that disproves it.
unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and
cosmetic, and lists the four attempts to remove it with the plan number that
disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662),
restapi 3.x ignore_server_additions (5667), and create_returns_object (5677).
The first three cannot work in principle — `data` is a configured attribute, so
Terraform core diffs it against state before provider-side ignore logic runs.
The fourth was the better theory, that PUT responses re-polluted `data`, and it
is wrong too: the diff returned on the plan immediately after the apply. That
means the response is written into `data` on the read path, and settling it
needs the provider source rather than more guesses at its schema.
No behaviour change. Documentation only, so the next person inherits the
findings instead of the dead ends.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Root cause of the 60 permanently-dirty hairpin NAT rules.
The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.
No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:
ignore_changes_to = ["_id"] plan 5661 — 60 remained
ignore_all_server_changes = true plan 5662 — 60 remained
restapi 3.x + ignore_server_additions = true plan 5667 — 60 remained,
plus a new api_data diff
create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.
This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Plan 5657 rejected the previous commit:
Error: Missing required argument
on lxc_800_rustfs_decommission.tf line 98, in resource
"proxmox_virtual_environment_container" "rustfs_decommission":
The argument "template_file_id" is required, but no definition was found.
The plan's own state dump showed only `type = "debian"` inside
operating_system, so mirroring it looked complete. It is not:
template_file_id is create-time-only, PVE does not report the original
template for an existing container, and nothing else in opentofu/ declares
one to copy — these three are the only container resources in the repo.
Guessing a template would be a lie that stays invisible until something
recreates these containers, which is precisely the event this file exists to
prevent. So operating_system is left undeclared and ignored via lifecycle
instead, which stops it planning as a removal without asserting a value that
is not known.
The timeouts remain outside ignore_changes on purpose — they are the one
thing this shim needs to actually apply.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The v2 shim's own instructions say the plan MUST show an in-place update and
no replace before applying. It does not. Pipeline 5656 planned all three
containers as `must be replaced`, which would destroy and recreate LXCs
tagged critical / state-backend / terraform-backend instead of nudging a
timeout value into state.
Cause: v2 declared only node_name, vm_id, started, unprivileged and the
timeouts, on the stated assumption that "the other attributes mirror the real
containers so the plan stays in-place". They were not declared at all, so
every absent attribute planned as a removal — and two of those removals force
replacement:
- mount_point { ... } # forces replacement (size, volume)
- operating_system { ... } # forces replacement
Declaring the full attribute set, mirrored from the values the plan itself
reported out of state: cpu, memory, disk, mount_point, network_interface,
operating_system and initialization. hostname and mac_address differ per
container, so the local map carries them per key rather than being repeated
or guessed.
tags is declared for a separate reason: it was planning as
["critical","rustfs","state-backend","terraform-backend"] -> null, which does
not force replacement but would have stripped the tags off the live
containers on apply.
This does not unblock the WordPress backup on its own. That needs
renner-uno-k8s-mariadb, which the same plan wants to create — but the bucket
cannot be applied while the plan also carries three container replacements,
and a -target apply is not an acceptable way around it: it diverges state
from config and leaves the real problem in place.
Verification is the file's own gate — a plan showing in-place only.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Rebased onto master now that #511 landed the WOODPECKER_URL fix for both
rotators — that part is dropped from here rather than duplicated. What remains
is the half nothing else covered.
All four rules in prometheusrule-woodpecker-rotation.yaml were selecting
series that do not exist, so the whole group was silent:
- WoodpeckerServerNotReady queried kube_statefulset_status_replicas_ready.
The server became a Deployment when the sqlite->CNPG migration dropped its
PVC, so that metric has no series at all — a `critical` "all CI is down"
alert that could never fire, and had been dead well before the namespace
move. Now kube_deployment_status_replicas_ready in the woodpecker namespace.
- WoodpeckerAgentNotReady selected namespace="apps". Broken by the namespace
migration; mine.
- The two woodpecker-secret-rotate rules selected namespace="apps", but that
rotation runs from the openbao namespace. Broken since that move, which the
woodpecker kustomization documents.
Verified against live Prometheus instead of reasoned about: server ready=1,
agent available=1, cronjob 1 series. The originals returned nothing, which is
exactly why they never fired rather than firing wrongly.
Two stale claims in the same file corrected while here: it pointed at
kubernetes/apps/woodpecker/rotate-agent-secret.yaml, which no longer exists,
and asserted the CronJob/Job series "stay empty until the rotation feature is
merged" — that rotation has since landed in openbao.
Also finishes the netpol comments #511 left mid-update, which still credited
woodpecker-allow-rotators to `apps/*`. gitea.apps.svc:3000 in review-bot.yaml
is deliberately untouched: gitea really is still in apps.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
From the update audit (#426), which flags this as the one item warranting
prompt action. Confirmed live: woodpecker-server runs v3.17.0 while the lint
step still pinned the v3.16.0 CLI.
That gap is the whole point of the step. The CLI compiles pipeline configs
against its own schema, so one behind the server can pass a config the server
then rejects — the gate stops predicting what the server will do, which is the
only thing it exists to do.
Verified the bump does not trade one problem for another: pulled
v3.17.0-alpine and ran the step's own lint loop over all 18 files in
.woodpecker/. Every one passes, so v3.17's schema does not reject anything
currently committed.
The drift is structural, not an oversight, and the comment now says so instead
of implying the two versions track each other. The server is a Kubernetes
manifest and image-automation bumps it; this pipeline file is plain YAML with
no apiVersion/kind, which Flux cannot patch (fluxcd/flux2#2826, recorded in
infrastructure/image-automation/README.md). So the server always moves first
and this always trails until someone bumps it by hand. Caught here by the
audit rather than by CI, which is the arrangement working as designed.
Also drops two stale "v3.16.0" references in the surrounding commentary that
would have aged into the same confusion.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#433.
Deletes the four apps-namespace manifests, promotes the migration-* set onto
their names, and gives the surviving Cluster the prune: disabled data guard it
was deliberately denied while a rollback still existed. After this the
kustomization renders nothing into `apps` at all.
Merging this deletes nothing. The old resources carry prune: disabled, so
dropping them from git only stops Flux managing them — they keep running and
keep their PVC. The teardown is a separate, deliberate act, and the runbook
carries the commands plus the precondition to check first.
That precondition is already satisfied: the new cluster took its own base
backup to the wp-pg-woodpecker prefix (wp-pg-daily-20260818205831, completed)
and reports Initialized, ConsistentSystemID, Ready, ContinuousArchiving and
LastBackupSucceeded all True. Recovery no longer depends on the retired
cluster's data.
Two things kept rather than cleaned up, both deliberate:
- bootstrap.recovery and its externalCluster stay on the Cluster. CNPG honours
bootstrap only when PGDATA is absent, so it is inert on a live cluster — and
keeping it means a cold rebuild restores from backup instead of coming up
empty, which is the gap RECOVERY.md exists to close for the other clusters.
- wp-pg-store-source stays, pointing read-only at the old prefix. Nothing
writes to it, and it is a second independent restore point predating the
move — worth having until the new prefix builds real backup history. It is
also the actual safety net for the teardown, rather than the running cluster.
The hetzner-creds comment now records the openbao ClusterSecretStore allowlist
requirement that cost stage 1 a failed restore (#505), so the next namespace
move does not rediscover it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The stage-1 verification section said row counts "will not match exactly if
builds ran after the backup the restore used — that is expected, and stage 2
quiesces the server before the final cutover".
Stage 2 does not quiesce. Flux prunes the apps workloads and creates the
woodpecker ones; nothing stops writes to the old database in between. The
sentence described an intent that was never implemented, which is worse than
saying nothing.
Measured on the live cutover rather than estimated: the restore captured
through pipeline 9488; 9491, 9492, 9494 and 9496 landed in apps/wp-pg before
the swap; the new cluster resumed at 9521. Four pipeline records and their log
entries exist only in the old cluster. No ID collision and no functional
breakage — cluster.yaml itself calls this database small and non-critical with
largely reconstructable state — but it is real history loss, and it should be a
decision rather than something discovered later from a row count.
Documents how to avoid it next time (suspend the KS, scale the old server and
agent to zero, re-run the restore) and notes the rows stay recoverable while
the old cluster keeps its prune guard, which is until stage 3.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Follow-up to #506, which merged before this landed. Addresses the review's
stale-comment suggestion, and a real over-grant found while checking it.
The comment above woodpecker-allow-rotators still explained the rule in terms
of the `apps` baseline and allow-ingress.yaml. Chasing that down showed
allow-ingress.yaml is hard-scoped to `namespace: apps` and matches traefik by
label — so it does not apply in the new namespace, and the traefik admit I put
in the woodpecker baseline instead was broader than the thing it replaced:
endpointSelector:{} plus a whole-namespace source, where the `ingress`
namespace also holds netbird-ingress. That let those pods reach wp-pg and the
agent, not just the server.
Replaced with woodpecker-allow-traefik, scoped to traefik and to the server
pod, matching how the rotator and CI-step rules are already written.
It lives in network-policies/woodpecker-ns.yaml rather than beside the other
workload policies, deliberately: the baseline admit is withdrawn in the same
file, so both halves ship in the infrastructure-crds Kustomization and swap in
one reconcile. Split across Kustomizations they would not — infrastructure-crds
reconciles before the `woodpecker` KS, so the admit would be withdrawn minutes
before its replacement arrived and ci.renner.uno would 503 in the gap.
Confirmed safe against certificate renewal: letsencrypt-cluster solves DNS-01
through Cloudflare, so there is no in-cluster ACME solver pod needing ingress.
With an HTTP-01 issuer this scoping would have broken renewals silently, 90
days later.
Runbook: records that the stage-2 outage outlasts pods becoming Ready, because
certificates are namespace-scoped and cert-manager must reissue woodpecker-tls.
Observed during the live cutover as a TLS SNI failure with no HTTP status at
all, which reads like a routing fault and is not one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. This is the downtime window: Flux prunes the apps copies and
creates the woodpecker ones, so ci.renner.uno is unavailable in between and
in-flight pipelines die with their agent.
Moved into `woodpecker`: server, agent, service, ingress, pdb, the agent
ServiceAccount, the woodpecker-secret ExternalSecret, and the rotator ingress
policy. The server's DSN now points at wp-pg-rw.woodpecker.svc and the agent's
gRPC at woodpecker.woodpecker.svc:9000.
Stage 1's restore is verified before doing this, not assumed: the new cluster
reports Ready=True and ConsistentSystemID=True (both False on the apps copy for
three weeks), its row counts match the live database exactly — 6 repos, 6125
pipelines, 2 users, 659 MB — and cnpg_collector_up now has a woodpecker series
for the first time.
Cross-namespace edges that had to move with it:
- rbac.yaml's RoleBinding subject. The Role and RoleBinding stay in
woodpecker-ci, because that is where they grant pod create/delete; only the
subject namespace changes.
- The openbao rotators' egress, which targeted `namespace: apps` + name
woodpecker on :8000.
New: woodpecker-allow-ci-steps grants woodpecker-ci → server :8000, scoped to
the SERVER pod. Stage 1 left this out on purpose after review found the same
rule on the namespace baseline would have handed arbitrary build steps a route
to wp-pg, since the CNPG instance manager serves its admin API on :8000 too.
Post-stage-1 the datapath confirms step-pod identities reach wp-pg on ICMP types
only; this rule must not change that.
Deliberately NOT touched: cluster.yaml, objectstore.yaml, scheduled-backup.yaml
and hetzner-creds-externalsecret.yaml still describe the original apps cluster.
They are the rollback, and they keep their prune guards. Stage 3 retires them
and renames the migration-* set onto their names.
Also corrects the runbook. It told the next person to verify policy semantics
with `cilium-dbg policy trace`, which does not exist in Cilium 1.20 — the
subcommand was removed, leaving only get/selectors/subject-selectors/wait.
Replaced with reading the realized BPF policy map via `bpf policy get
<endpoint-id>`, which is better evidence anyway: it is the datapath as
programmed rather than a simulation, and it carries live byte/packet counters.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Stage 1 of #433 is failing in the cluster. The wp-pg restore job errors
with:
can't get credentials for cluster wp-pg: while getting secret
hetzner-s3-creds: secrets "hetzner-s3-creds" not found
and the ExternalSecret behind it reports SecretSyncedError:
using cluster store "openbao" is not allowed from namespace
"woodpecker": denied by spec.condition
The ClusterSecretStore carries an explicit namespace allowlist, and the new
namespace was not on it. Its own header says so — "adding an ExternalSecret
in a NEW namespace requires adding that namespace here first, or ESO
reports 'not allowed to reference ClusterSecretStore' and the secret never
syncs" — so the requirement was documented and simply missed.
The reference sweep for #504 looked for everything that referenced
woodpecker or resolved *.apps.svc. It never asked the other question a
namespace move raises: what allowlists must the NEW namespace join? This
allowlist is exactly that shape, and it is the kind of thing kustomize
build and dry-run cannot surface, because the manifests are all valid — the
denial happens at ESO's admission of a cross-namespace reference.
Added in sorted position, keeping the list's stated ordering.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
From the fourth review on #504 — no blocking issues, but this one is a real
gap rather than a style point.
The woodpecker baseline was the only one of the repo's namespace policies
without an ICMP allow. That matters here specifically: wp-pg-allow-cnpg-operator
attaches an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP
to an endpoint once any layer 4 policy selects it unless the traffic is
CT-related. So PMTUD Destination Unreachable and Packet Too Big toward wp-pg
would have been dropped silently, surfacing as stalled transfers rather than
as a policy denial.
Uses the 16-type set the other baselines now carry after #495 trimmed them
from 39, rather than reintroducing the old list.
Granting this namespace-wide does not contradict the caution comment above it.
The :8000 case had to be per-workload because that port is shared between the
woodpecker server and the CNPG instance manager's admin API, so a blanket rule
handed CI step pods a route to the database. ICMP echo and path-MTU messages
expose no API surface, so a namespace-wide grant concedes nothing a scoped one
would withhold — which is why every other baseline does it this way too.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the third blocking review finding on #504.
The PR claimed the namespace move fixes wp-pg's absent metrics. It does not,
on its own. Prometheus requires the `release: kube-prometheus-stack` label
that CNPG's own enablePodMonitor omits, so every scraped cluster has a
hand-written PodMonitor in apps/monitoring-rules/cnpg-podmonitors.yaml —
ak-pg, im-pg, gf-pg, pprl-pg. There has never been one for wp-pg.
So wp-pg was missing both halves: no scrape target configured, and no
monitoring ingress allow in the apps baseline. Fixing the policy half was
necessary but not sufficient, and the runbook's cnpg_collector_up check
would have returned empty either way — unable to confirm the thing it
existed to confirm. Adds the PodMonitor in the woodpecker namespace.
Also from the review:
- Added a ScheduledBackup for the restored cluster. Every other CNPG
cluster here pairs isWALArchiver with one. Without it the new
wp-pg-woodpecker prefix takes continuous WAL but never a base backup, so
it has no independent recovery point and wp-pg-store's 30d retention has
nothing to prune against — the WAL chain grows unbounded for as long as
stage 1 sits before cutover. Scheduled 02:40, clear of the apps cluster's
02:35 and the other clusters' 02:30.
- Documented that the `woodpecker` KS goes not-Ready while the restore runs.
Its healthCheckExprs matches Cluster by GVK with no name qualifier, so
Flux evaluates it against both the live apps/wp-pg and the restoring
woodpecker/wp-pg, and wait: true makes the KS that owns production
Woodpecker report unhealthy. Expected, does not cascade since nothing
dependsOn it, but it would read as an outage without a note.
- Dropped the app.kubernetes.io/name label from the new Namespace; no other
namespace in that file carries one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the second blocking review finding on #504.
bootstrap.recovery set only `source`. The live CRD documents the defaults:
`database` defaults to `app`, and `owner` defaults to whatever `database`
is. Neither name exists in the physically restored data, which carries
woodpecker/woodpecker from the backup.
So CNPG would have restored the real data, then provisioned a separate
empty `app` database and pointed the generated wp-pg-app Secret and the
-app Service at that instead — a cluster reporting healthy, holding the
right bytes, with the credentials aimed at the wrong database. The runbook's
own verification (psql -U woodpecker -d woodpecker using the wp-pg-app
password) would not have authenticated, so the failure would have surfaced
as a confusing auth error rather than as what it was.
kubernetes/RECOVERY.md sets both fields on all three recovery examples it
documents (ak-pg, im-pg, pprl-pg) for exactly this reason; this one did not
follow the pattern.
Added a runbook step to read dbname/username back out of the wp-pg-app
Secret before the row-count checks, so the mismatch is caught directly
rather than inferred from a psql failure.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #504.
The "step pods call the server back on :8000" rule was attached to the
namespace baseline, which uses endpointSelector: {}. A toPorts block
narrows the port, not the destination workload — that is governed solely by
the enclosing policy's endpointSelector. So the rule granted woodpecker-ci
access to :8000 on every pod in the namespace, and the CNPG instance
manager serves its status API on :8000 (server.yaml:94 uses the same port).
Arbitrary repo-supplied build steps would have had a route to wp-pg's
instance manager.
That is the same endpointSelector:{}-means-blanket-grant failure this PR
exists to escape, reproduced inside the fix.
Removed rather than rescoped: there is no woodpecker server in this
namespace until stage 2, so the rule was premature as well as wrong. It
lands there as its own policy selecting the server pod, the way
woodpecker-allow-rotators already scopes its :8000 rule.
The baseline now carries no port-scoped rules at all, so nothing in it can
be mistaken for workload scoping, and a comment warns why that must stay
true.
Also from the review:
- Corrected the KS attribution. The network policies ship with
infrastructure-crds (infrastructure/post-crds includes
../network-policies/), not the top-level infrastructure KS. The ordering
conclusion is unchanged — infrastructure-crds dependsOn infrastructure,
and woodpecker dependsOn both — but the comments named the wrong one.
- Documented why the baseline has no egress section, instead of leaving it
to read as an oversight. Writing that ruleset in stage 2, with the server
and agent present, beats guessing it now and breaking the WAL archiving
this stage exists to verify.
- Dropped the monitoring -> :9187 rule from wp-pg-allow-cnpg-operator. The
baseline already allows the monitoring namespace unrestricted, so it added
no coverage while implying scoping it did not provide.
- Added a cilium-dbg policy trace step to the runbook. dry-run validates
manifest shape, not Cilium semantics, and would not have caught this bug.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. No cutover, no CI impact: this stands up the `woodpecker`
namespace and restores wp-pg into it from barman, running in parallel with
the live apps/wp-pg. Nothing points at it yet.
Why a namespace move rather than a policy fix. wp-pg has been Ready=False
with "Instance Status Extraction Error" for three weeks because the CNPG
operator in `database` cannot reach https://<pod-ip>:8000/pg/status. The
apps default-deny-ingress uses endpointSelector:{}, so the only
namespace-level remedy is to allow the whole database namespace in — which
would expose gitea, gatus, external-dns and ical-proxy along with wp-pg, to
fix one pod on one port. The same gap explains why cnpg_collector_up has
series for authentik, immich, monitoring and paperless but not wp-pg: apps
never allowed the monitoring namespace either.
woodpecker-ci is deliberately not the target. It enforces pod-security:
privileged and exists for ephemeral step pods running repo-supplied code;
the database has no business sharing it.
Two ObjectStores, deliberately. wp-pg-store-source reads the existing
s3://renner-uno-k8s-cnpg/wp-pg prefix; wp-pg-store archives to a new
wp-pg-woodpecker prefix. Two clusters writing one destinationPath would
interleave their WAL streams and corrupt both, and splitting them keeps the
pre-migration backup intact as rollback.
The Namespace is declared in infrastructure/namespaces rather than beside
these workloads: its network policies ship with the `infrastructure` KS,
which the `woodpecker` KS dependsOn, so creating it in the workload KS would
invert that order and deadlock a cold rebuild.
The restored cluster deliberately carries no prune: disabled guard. The
original has one because it holds the only copy of the data; this one is
reconstructable, and a prune guard on a migration artifact is how orphans
outlive an abandoned migration. It gains the guard in stage 3.
Barman creds still resolve from seed/paperless/hetzner-creds. #397 wants a
woodpecker-owned path and this was the natural moment, but creating it needs
OpenBao write access that is not available here (bao kv list seed -> 403).
Pointing at a path that may not exist would leave the restored cluster
unable to archive WAL, which is what this stage exists to prove.
Runbook, including how to verify the restore holds real data rather than
merely coming up healthy: MIGRATION-namespace.md.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Ceph's 30% default assumes the mon owns its filesystem, where 30% free is
a real reserve. Our mons sit on dataDirHostPath /var/lib/rook — the node's
/var, shared with ~29 GB of containerd storage — and the mon DB is ~106 MB.
The default therefore measures container churn rather than Ceph state, and
k8s-2 trips MON_DISK_LOW whenever its image cache fills.
Tuning image GC does not help: measured on k8s-2, dropping the GC high
threshold to 60 deleted 28 images and moved free space by 0.02 percentage
points, because most of /var/lib/containerd is overlayfs snapshots held by
running containers, which image GC cannot reclaim.
20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database.
mon_data_avail_crit is untouched at 5%, and CephMonDiskLow watches /var
directly for the real risk — the node filling, not the mon growing.
Set through cephConfig so it is reconciled from git rather than applied as
a live `ceph config set`.
The comment records that this override should be revisited, and dropped,
when mons get their own volume at bare-metal migration. That is not
possible on the current cluster: every StorageClass is Ceph-backed, so a
mon PVC would be circular, and Rook applies mon.volumeClaimTemplate only
to newly created mons.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The 60/50 change in #500 was based on a wrong premise. Reverting the
values and correcting the reasoning recorded around them.
Applied to k8s-2 alone and measured. GC did fire: 28 images deleted,
content store 9.93 -> 8.62 GB. Free space moved 31.49% -> 31.47% — no
effect.
The premise was that ~19 GB of overlayfs snapshots would be reclaimed
along with their images, at roughly 2.9x the compressed size. That is
wrong. /var/lib/containerd is 20.5 GB of snapshots against 8.6 GB of
content, and those snapshots are unpacked layers shared with running
containers. Image GC cannot touch them, so deleting unused image
references frees only their unshared blobs. Remaining reclaimable
content is now ~0.7 GB; further sweeps free nothing.
70/60 stands on its own merits as a bound on the image cache — it is
simply not a lever on MON_DISK_LOW, and the comment no longer claims it
is. The mon's own data is ~106 MB, so a 30%-free threshold on a 60 GB
filesystem it merely shares says nothing about Ceph.
The CephMonDiskLow runbook text carried the same wrong claim and is
corrected to point at finding the real consumer instead.
k8s-2's live kubelet is already back to 70/60 via talosctl patch; node
Ready, Ceph HEALTH_OK with mon quorum a,b,c intact. k8s-1 and k8s-3 were
never changed.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Gap 2 — no admin path. There is still no rook-ceph-tools deployment, so
ceph status, ceph health detail, ceph osd df tree, ceph osd perf and ceph
device ls are all unreachable during an incident: mons, OSDs and mgr
mount no admin keyring and the operator's /etc/ceph is empty. Adds the
upstream toolbox from rook v1.20.4 (matching the running operator image)
with the image pinned to the cluster's own quay.io/ceph/ceph:v19.2.5 and
carrying the same imagepolicy marker as cephcluster.yaml, so the client
cannot drift ahead of the mons. Prerequisites verified present: the
rook-ceph-default ServiceAccount, the rook-ceph-mon-endpoints ConfigMap,
and ceph-secret/ceph-username in the rook-ceph-mon secret.
Image GC. The thresholds were already lowered once from the kubelet
defaults, to 70/60, explicitly to stay under Ceph's mon_data_avail_warn.
But that line is <=30% free and a high of 70 fires at exactly 30% free —
on the threshold, not below it. So the cache fills to precisely the point
where the mon warns, and MON_DISK_LOW trips before kubelet reclaims
anything. Observed on k8s-2: 29.81% free and HEALTH_WARN MON_DISK_LOW,
clearing to 31.64% once GC ran. 60/50 fires at 40% free and reclaims to
50%, leaving 10 points of margin.
There is plenty to reclaim, so the lower threshold will bite: containerd
holds ~29 GB of k8s-2's 60 GB /var and caches 179 images against 60 in
use, some dating to May. (The ~309 GB talosctl attributes to
/var/lib/kubelet is mounted Ceph PVCs counted through mount points, not
local disk.)
Alerting, the follow-up #434 parked until ceph_* series existed — they do
now, via #432. CephHealthWarn deliberately waits 6h: Ceph warns for
self-clearing conditions and paging on those teaches people to ignore it,
so what is worth waking for is a warning that does not clear.
CephHealthError fires at 5m. CephMonDiskLow watches /var free directly
rather than going through Ceph, because this fires for a full image cache
rather than for Ceph growth, and naming that keeps the next person from
debugging the wrong subsystem.
NOTE: cluster.yaml is a bootstrap patch. Merging changes nothing on
running nodes — the live kubelet value needs talosctl patch machineconfig.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The earlier commits in this branch raised policyMapMax and added an alert.
Neither addresses why the map fills. This does.
Measured on k8s-1 with cilium-dbg: 8 endpoint policy maps sit at exactly
16384 entries — full — and the rest cluster at ~8,500. The split tracks
policy direction, so one direction costs ~8,500 entries. Dumping velero's
endpoint 2328 (8,499 entries, ingress only) shows 217 identities x 39
port/proto values, of which 38 are ICMP types. The map is ~99% ICMP, and
3,744 of its entries (44%) belong to woodpecker-ci-step identities.
Two multiplying causes, both fixed here.
The ICMP rule in every default-deny-ingress allowed 39 enumerated types
from `namespace: Exists` — every identity in the cluster. Cilium expands
that to one entry per (identity, type), per endpoint, per direction, in
19 namespaces. The rule's own comment records it was widened "from the
error class to the full standard type set"; that widening is the cost.
Trimmed to 16: the error types that motivated the rule (v4 3/11/12, v6
1/2/3/4 — PMTUD is the documented need), echo, and the IPv6 ND/RA/MLDv2
types observed in flight via Hubble (133, 135, 143). Dropped types are
deprecated (v4 4 Source Quench, RFC 6633), router-only (v4 5, v6 137),
obsolete (v4 13-18, 30), superseded (v6 130-132 by 143), or never
deployed (v6 138, 141/142, 148/149, 151-153). openbao-ns.yaml already
runs an error-only set of 7 in production, so 16 is the conservative side
of existing practice.
Cilium had no `labels` filter, so identity came from all pod labels.
Woodpecker stamps each step pod with woodpecker-ci.org/task-uuid and
step=<name>, making every build step its own security identity — 96 of
them minted in the last hour, against zero pods in the namespace. The GC
is healthy and simply outpaced. Excluding those two label patterns
collapses 78 current identities to one. No CiliumNetworkPolicy selects on
them, so they bought churn and nothing else.
Together these take a direction from ~8,500 entries to roughly 140
identities x 17 values ~= 2,400 — about 4,800 for both directions, well
inside even the old 16384 ceiling. policyMapMax stays raised as headroom,
not as the fix.
Per Cilium's docs, existing identities are not recomputed by a `labels`
change; new ones use the filter and the old ones age out through normal
GC, so the map drains over roughly an hour rather than instantly.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
renner.uno and www.renner.uno were pinned to 159.69.42.237, the VM that hosted
WordPress before it moved into the cluster. Neither record is live on the UDM
any more (verified: a LAN query for www.renner.uno returns the public answer,
proxy.renner.uno), so the file was drift waiting to be re-applied — a tofu
apply would have pointed the house at a host that is being decommissioned,
while the rest of the world already reaches the cluster.
Their absence is the correct end state, not an omission: www.renner.uno
carries dns.renner.uno/public, so external-dns-unifi writes no UDM record by
design and LAN clients take the public path and hairpin back in. The apex
stays with Cloudflare, which now redirects renner.uno/* to www.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Two optional points from the second review on #495.
The PrometheusRule alerts on cilium_bpf_map_pressure without filtering on
map_name, so it covers conntrack, NAT and LB maps as well as the policy
map. That is intended — running out of any of them degrades the datapath
— but the PR is framed around the policy map, so a reader could take the
narrow reading. Say so in the file.
The cross-references between the three copies of bpf-policy-map-max were
one-directional: helmrelease.yaml and patches/cilium-cni.yaml each named
only one other site, while talos/cilium.yaml named both. Each now names
the other two.
Not taken: the suggestion to quote policyMapMax as "65536". The chart
declares it `type: [null, integer]` in its values schema
(helm show values cilium --version 1.20.0), so an integer is correct and
a quoted string would be the schema violation.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #495.
Raising bpf.policyMapMax only in the HelmRelease left the bootstrap path
on the old value. kubernetes/bootstrap.sh:126 passes
kubernetes/talos/patches/cilium-cni.yaml to talosctl apply-config for
every node, and that manifest hardcoded bpf-policy-map-max: "16384" — so
a cold rebuild would install Cilium with the map size this PR exists to
fix, and stay there until Flux reconciled the HelmRelease.
The sibling bpf-lb-map-max is "65536" in both files, matching the live
agent config, so keeping these in sync is the repo's existing practice.
kubernetes/talos/cilium.yaml carries the same key and is updated too.
Nothing in the repo references that file — only patches/cilium-cni.yaml
is used by bootstrap.sh — so it looks like an orphaned rendered snapshot.
Syncing it rather than deleting it: whether it should exist at all is a
separate question from this fix.
Each of the three sites now names the others, so the next change to this
value has the lockstep written down where someone will hit it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#451. Closes#454.
Both issues are the same fault: a pod's dedicated cilium_policy_v3_<id>
BPF map ran out of space, endpoint regeneration failed with "no space
left on device", the CNI ADD call never returned, and the pod sat in
sandbox creation. For paperless — replicas: 1 with strategy Recreate —
that was a 41-minute outage on 2026-07-30.
It is not over. Prometheus (metrics landed via #452) puts
cilium_bpf_map_pressure{map_name="cilium_policy_v3_*"} at 0.867 on all
three nodes right now, and max_over_time(...[30d]) at 1.03 — the map has
been driven past capacity. Loki has the overflow recurring as recently
as 2026-08-12 17:46Z, hitting authentik and esphome endpoints.
Three changes:
- bpf.policyMapMax 16384 (chart default) -> 65536, matching the cluster's
existing bpf-lb-map-max. Takes steady-state pressure from ~87% to ~22%.
The map is sized at creation, so this only takes effect as agents
restart.
- The chart range ">=1.19.0" is what let Flux take 1.19.6 -> 1.20.0
unattended 45 minutes before the first outage. Narrowed to
">=1.20.0 <1.21.0" so a CNI minor is a reviewed diff, not a surprise.
- A PrometheusRule on cilium_bpf_map_pressure: warning over 80% for 15m,
critical at 100%. The metric was already scraped but nothing alerted on
it, so the first signal was a downstream replica mismatch.
Not addressed: why the busiest endpoint needs ~14k entries. Only 178
CiliumIdentities exist and no CiliumNetworkPolicy declares more than 9
ports, so the entry count is not obviously explained by policy shape.
Establishing that needs `cilium-dbg bpf policy get` on an agent.
MERGING THIS ROLLS THE CNI. The DaemonSet uses
updateStrategy.rollingUpdate.maxUnavailable: 2, so two of three agents
restart at once.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Verified against the deployed cluster: GET /wp-admin/ correctly 302s to
https://www.renner.uno/outpost.goauthentik.io/start?rd=... — and that URL
returned WordPress's 404 page, because /outpost.goauthentik.io fell through to
the public '/' prefix router.
The effect was that /wp-admin was unreachable rather than protected. An
attacker gains nothing, but neither does the editor the gate exists for.
paperless needs the same route and solves it with a separate Ingress on
auth.renner.uno (apps/authentik/paperless-gate-ingress.yaml). Here the whole
flow stays on www.renner.uno, so the path belongs in this rule set.
These endpoints are the login flow itself and are served by the outpost, so
routing them to it is not an authentication hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Closes#461.
Five pipeline steps pinned ci-k8s at two stale tags: e2e-test.yml,
e2e.yml, alert-monitor.yml and issue-fix.yml at 2026-05-01, and
ci-manifests-test.yml at 2026-07-30. 2026-08-10 is the newest dated tag
in gitea_admin/ci-k8s and shares a digest with :latest
(sha256:0019f264...), so all five now run the same image the build
pipeline last produced.
A stale pin eventually points at a garbage-collected tag, and the split
had no deliberate reason behind it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The first deploy returned 500 on every request:
PHP Fatal error: Uncaught Error: Call to undefined function add_filter()
in /var/www/html/wp-config.php(128) : eval()'d code
WORDPRESS_CONFIG_EXTRA is eval'd inside wp-config.php, which runs long before
wp-settings.php loads the plugin API. add_filter() does not exist yet at that
point, so the call was fatal on every request.
Removing it exposed the second, worse half: the comment justified the filter by
claiming xmlrpc.php was 'blocked at the ingress'. It was not. The public router
matches '/' as a prefix, so xmlrpc.php was served like any other path — the
claim was written without being checked.
An Ingress has no deny verb, so xmlrpc.php now routes to the Authentik gate
alongside /wp-admin and /wp-login.php: anonymous callers meet Authentik instead
of the XML-RPC endpoint. Nothing on this site uses it — no Jetpack, no mobile
app, no pingbacks worth keeping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot
issue-fix-bot