The name says the bootstrap depends on a registry mirror. Reading the patch list
in the runbook, that is the obvious conclusion — and it would be a serious
problem, because a cold rebuild has no mirror: an in-cluster one cannot exist
before the cluster does.
The file configures no mirror. It writes one containerd setting,
discard_unpacked_layers = false, so that pulled layers stay in the content store
and Spegel can advertise them to peers later. Spegel writes the per-registry
hosts.toml itself at runtime; the file's own comment says a
machine.registries.mirrors block is not needed, and there is none anywhere in
talos/patches. Image pulls at bootstrap go straight to the real registries.
So the alarm the name raises is false, but it is the name doing it, and the cost
of that is someone deciding a rebuild is blocked when it is not.
containerd-layer-retention.yaml describes the single thing inside.
Renamed with git mv and the three references updated. Verified `talosctl gen
config` still produces a config with the renamed patch in the list.
The Cilium install had become a block of shell in a runbook for a human to
paste. That is the same rot as a pinned version — nobody runs it between
rebuilds, so nothing tells you when it stopped working. It belongs in the script
that already drives the rest of the bootstrap.
The talos phase was unusable since #581: it passed cilium-cni.yaml whole, and
that file's `$patch: delete` document fails against a config with no
inlineManifests. The file is gone, so the failure is gone with it — reviving the
phase was mostly deleting two lines that referenced it.
It now installs Cilium between `talosctl bootstrap` and the node-Ready wait,
which is the only place it can go: with `cni: none` the kubelet stays NotReady
until a CNI exists, so the wait below would otherwise time out against a cluster
that was never going to converge.
Chart repo, version constraint and values are read out of helmrelease.yaml at
run time, so the script cannot install something different from what Flux
reconciles minutes later. helm takes the semver range verbatim. There is a guard
on the extracted values — the HelmRelease is a two-document file and selecting
the wrong one yields an empty values set that helm silently renders as chart
defaults, which is a failure mode worth catching loudly rather than discovering
in a datapath.
`--wait` because the next step needs a working network, and `upgrade --install`
so a re-run after a partial bootstrap converges instead of erroring.
Verified as far as it can be without wiping a cluster: `talosctl gen config`
with exactly the patch list the script now passes produces a config carrying
cni: none and proxy: disabled, no Cilium manifests, and it validates for metal
mode. The apply-and-bootstrap half is untested since the rewrite, and the
runbook says so — it keeps the manual procedure as the thing to read first and
to fall back on.
The install command in Step 2b hardcoded --version 1.20.1 and the chart repo
URL. That is a copy of the HelmRelease in a text file nobody runs between
rebuilds, so it would have gone stale unnoticed and the next rebuild would have
installed something Flux immediately fought — the same failure the deleted
bootstrap seed produced, in a new place.
Everything now comes out of helmrelease.yaml at the moment the command runs:
the repo URL from the HelmRepository, the values from .spec.values, and
--version from .spec.chart.spec.version verbatim. Helm accepts the semver range
and resolves it exactly as Flux does, so the two cannot pick different releases.
Verified by running the block as written:
--version ">=1.20.0 <1.21.0" -> cilium:v1.20.1, operator-generic:v1.20.1
enable-bgp-control-plane: "true" bpf-lb-sock: "true"
bpf-policy-map-max: "65536" ipv4-native-routing-cidr: 10.0.0.0/8
ServiceMonitor objects: 0
which is the config the cluster is running.
The only literal left is --set prometheus.serviceMonitor.enabled=false, and it
is a bootstrap constraint rather than a value: the CRD arrives with the
monitoring stack long afterwards. Because nothing else is overridden,
helm-controller adopts this release instead of replacing it.
This does not make the step unnecessary. Flux's controllers are pods and cannot
reconcile a CNI they need in order to run; something has to seed it. What it
does mean is that the step has nothing left in it to maintain.
Both blocking findings from the review on #709, and both were real.
The deleted cilium-cni.yaml was the only place in the repo that set
cluster.network.cni.name: none and cluster.proxy.disabled: true. They sat in its
second document next to the inline manifest, so removing the file removed them
too — a fresh `talosctl gen config` from this repo's patches would have produced
a config with Talos's default CNI and kube-proxy running, against a Cilium
release configured with kubeProxyReplacement: true. Two runbooks meanwhile
asserted that cluster.yaml supplied both, which it did not.
RUNBOOK-baremetal-conversion.md also still passed `@<cilium additive half>` in
its gen-config example, three lines above prose saying there is no CNI patch any
more. The equivalent block in the cold-rebuild runbook had been fixed; this one
was missed.
Verified the way it should have been in the first place — by running the
runbook's own command rather than inspecting the live cluster:
talosctl gen config homelab https://k8s.renner.uno:6443 \
--output-types controlplane \
--config-patch @talos/patches/cluster.yaml \
--config-patch @talos/patches/registry-mirror.yaml \
--config-patch @talos/patches/k8s-1.yaml
-> cni=none proxy.disabled=true
-> 0 cilium manifests
The live cluster could never have caught this. It has carried both settings
since it was built, and the node configs were patched by hand rather than
regenerated, so nothing in the verification touched the path a rebuild actually
uses.
Final step of PROPOSAL-cilium-single-owner.md. Talos no longer carries Cilium in
any form, and the HelmRelease is the only place it is configured.
The cluster side is already done and verified:
- the cilium inline manifest is gone from all three machine configs, applied
node by node with no reboot and no effect on the running CNI
- Kubernetes went 1.35.2 -> 1.36.3 in the same operation, with
--manifests-no-prune so the 22 objects Talos claimed were not deleted
- Talos's bootstrap-manifest inventory went from 41 entries to 13, and the 28
cilium/hubble entries are gone. A prune-enabled `upgrade-k8s --dry-run`
reported 28 deletions before and reports 0 now
That last number is the point of the whole exercise. A bare `upgrade-k8s` used
to destroy the CNI and now does nothing to it.
The inventory entries had to be removed by hand. --manifests-no-prune skips the
prune AND the inventory rewrite, so the run that saved the objects also left
Talos still claiming them — the mine was deferred, not cleared. Talos offers no
way to drop a stale claim without the prune that deletes, so the ConfigMap was
edited directly. It is derived bookkeeping, not configuration, and the result is
verifiable in both directions, which is why this was acceptable where hand-
editing configuration would not be.
Repo side, this commit:
- deletes talos/patches/cilium-cni.yaml and talos/cilium.yaml
- adds Step 2b to RUNBOOK-cold-rebuild.md: install Cilium from the HelmRelease
values with helm, after bootstrap and before Flux. The nodes stay NotReady
until then, which the runbook now says out loud rather than leaving someone
to wonder
- drops the "keep in lockstep" and "THREE COPIES" comments, which described a
duplication that no longer exists, and replaces them with one note saying
where Cilium is configured
- corrects RUNBOOK-baremetal-conversion.md, which told the reader to extract
half of a file that is gone
bootstrap.sh is left alone. Its talos phase was already dead and is now dead for
a second reason; the runbook says so.
The cost, stated plainly: the cluster no longer forms itself from the machine
config. A cold rebuild needs one documented helm install that it did not need
before. That is the price of Cilium existing in exactly one place.
Step 1 of the ownership transition in PROPOSAL-cilium-single-owner.md. Nothing
about ownership changes yet — this only moves three live settings out of nowhere
and into git.
enable-bgp-control-plane and both native-routing CIDRs existed in no file at
all. They were applied by hand and are held by the `kubectl-patch` field manager
on cilium-config; server-side apply left them alone, so nothing broke and
nothing reported them. The peering itself was always in git as CRDs in
post-crds/cilium-bgp.yaml, which is what made the gap invisible: BGP looked
configured because its policy was version-controlled, while the switch that
starts the control plane was not. A cold rebuild installs only what git
describes, so it would have come up with two established-looking peers
configured and no BGP at all.
The CIDRs go with it because they are the same decision. They are a masquerading
exception, not a routing statement: pod traffic to 10.0.0.0/8 and fd00::/8
leaves unmasqueraded so LAN services see real pod IPs, and that only works
because BGP advertises the pod CIDRs and gives the UDM a route back. Tracking
one without the other would leave a half-decision in git.
This is not purely a change of ownership, which is why it is its own step and
not folded into a later one. Enabling bgpControlPlane through the chart emits
five companion keys the hand-patch never set, so Cilium has been running on its
own internal defaults for router-id allocation among others. They should agree
with the chart's; if they do not, the sessions re-establish. Verify the peers
after Flux reconciles before moving to step 2.
The proposal called the tunnel redundant because the underlay already has
routes to the pod CIDRs. That reads the two as alternatives, and they are not:
VXLAN carries pod-to-pod directly between nodes, while the BGP-advertised pod
CIDRs exist so traffic from the LAN can reach pods at all, which necessarily
goes through the UDM.
Removing the encapsulation would therefore not remove a layer. It would move
inter-node pod traffic off a direct node-to-node path and onto the underlay,
through the router.
The observation worth keeping is narrower: routingMode is set nowhere, so the
current behaviour is an unexamined chart default rather than a recorded
decision. It happens to be the right default.
Cilium is currently owned twice. The 22 objects that make up the whole
installation — both DaemonSets, the operator, both ConfigMaps, the RBAC, the
cilium-secrets namespace — are listed in Talos's bootstrap-manifest inventory
and simultaneously carry Helm's release annotations. cilium-config has three
field managers.
That is not a tidiness problem. It produced a bootstrap seed twelve days behind
the chart, where a Kubernetes upgrade would have downgraded the CNI and switched
off socket load balancing, and it left the BGP control-plane switch and both
native-routing CIDRs existing in no file at all, so a cold rebuild would have
come up without BGP while the peering CRDs in git said otherwise.
This records the decision — Flux owns Cilium, Talos provides only cni: none and
proxy: disabled — and the ordered transition, including why the order matters:
Talos prunes what its inventory used to list and no longer does, so removing the
manifest naively deletes the CNI.
Two claims in it are verified rather than assumed. An OS upgrade does not
reconcile bootstrap manifests: Talos went to 1.13.9 four days ago and Cilium
stayed at 1.20.1 despite the seed saying 1.19.2. And the prune can be tested
before it is run, because NoPrune is honoured by the dry run as well as the
apply.
No code changes here. PR #706 stays WIP until this is agreed.
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.
What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.
So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".
Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.
Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.
BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.
The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.
BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.
Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.
Both were unmonitored. Neither is a leaf service: Woodpecker runs every cron
and deploy in this repo, and every ExternalSecret in the cluster resolves
through OpenBao.
OpenBao is probed at /v1/sys/health rather than at the root, because a sealed
instance still serves its UI. Root would answer 307 -> /ui/ -> 200 while every
secret lookup failed — the exact green-while-broken shape probes-blackbox.yaml
already warns about for the gated monitoring UIs.
standbyok=true is required, not cosmetic: the endpoint returns 200 only on the
active node and 429 on a standby, and the external route currently lands on a
standby. Measured — the bare path returns 429 right now, so an http_2xx probe
without the parameter would have been red from the first scrape. With it, 200
means initialized, unsealed and serving; 503 sealed and 501 uninitialized stay
red.
The alert matches OpenBao by instance prefix instead of the full target string,
because that target carries a query parameter and an exact match would fail
silently if the label ever encodes it differently.
Both alerts are new rules rather than additions to an existing file. Worth
noting while here: there is no generic blackbox-down rule. Only fsn1, netbird
and the two relays have alerts, so gitea, immich, auth, home, prometheus,
alerts and dash are probed and graphed but nothing fires if they go down. Not
addressed here — it needs a decision about whether to write one generic rule
and retire the per-instance ones.
The scan loop swallowed every Trivy failure with `|| true`, so a step that
scanned 21 of 22 images reported exactly the same as one that scanned all 22.
The gap was found by the fix agent noticing a missing file in its own input,
not by the pipeline.
The loop still continues past a bad image — one unpullable reference should not
cost the scan of everything after it — but now counts what failed, names it,
and exits non-zero:
--- scanned 21 of 22 images ---
NOT SCANNED: proxmox/proxmox-backup-client:latest
`failure: ignore` stays, which per Woodpecker's semantics reports the step as
failed while letting the workflow continue. That is deliberate: coverage is
reported, not enforced. The step goes red and the fix agent still runs on what
was scanned. Making the whole pipeline fail would mean one unscannable image
blocks all remediation.
The image that prompted this is gone with the nas-1 removal in the previous
commit, so the next run should report 20 of 20. The accounting is what matters
— the next dead reference will announce itself instead of being absorbed.
Refs #694
nas-1 has been off for months. Everything scoped to it went with it:
inventory/host_vars/nas-1.renner.uno/ vars and vault
[synology_nas] in pve_hosts.yml the group's only member
roles/pbs-client/ PBS backup client
plays/install-pbs-client.yml role-based, hosts: synology_nas
plays/install-pbs-client-synology.yml raw-module variant, same host
plays/install-garage-synology.yml Garage S3, replaced by Hetzner
.woodpecker/install-garage-synology.yml the pipeline that ran it
The two PBS plays were already inconsistent with each other before the host
went away: the role pulled aterfax/pbs-client, while the standalone play
pulled proxmox/proxmox-backup-client, which has never existed on Docker Hub
(404). Nothing referenced the standalone play at all. Both are moot now.
pve_hosts.yml keeps its "gone from this file, and why" list, so synology_nas is
recorded there alongside pve_cluster and the in-cluster migrations rather than
vanishing without trace.
Checked: `ansible-inventory --graph` resolves, and grep finds no remaining
reference to synology_nas, nas-1, pbs-client or the garage play outside that
one note. plays/archived/ is left alone — it is history, not live config.
file.renner.uno served HTTP 502. The Service is an ExternalName to
nas.renner.uno:7001, a CNAME to nas-1, and nas-1 has been off for months. The
Ingress and Service had been in the cluster for 110 days pointing at nothing.
Nothing caught it because file.renner.uno is not in the blackbox probe list, so
a publicly resolvable hostname returned 502 for months without a single alert.
The insecure-backend ServersTransport goes with it. It existed to let Traefik
re-encrypt to the NAS web UI's self-signed certificate and apps/file was its
only consumer, which its own header comment documents.
Rolls the security-scan pilot out to the other thirteen. Every workflow-root
gate of the form
- event: manual
evaluate: 'PIPELINE == "x"'
becomes
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "x"'
The target rides in pipeline metadata rather than in the manual-run variable
map, so it reaches the workflow-root filter on Woodpecker 3.18 where PIPELINE
does not. Verified on the pilot: pipeline 7004 selected the security-scan
workflow and nothing else.
Two files also carry step-level manual gates, and those had to move with the
root or the pipelines would have converted into something that runs but cannot
do its job:
deploy-infrastructure the apply step gates on TOFU_APPLY. Left on
`event: manual` it would never fire again, so the
pipeline would plan and silently never apply.
TOFU_APPLY now arrives via `--param`, which lands in
the same non-plugin environment step-level `when`
evaluates against. No step here is a plugin — nothing
in .woodpecker uses `settings:` — so that path holds.
build-ci-images eight per-image steps gate on manual-or-push to decide
whether to rebuild. All eight follow the root.
Cron gates are untouched. They already matched by name everywhere except
security-scan, which the pilot fixed.
Also corrects the trigger documentation, including a claim in update-audit.yml
that the other pipelines gate on "an injected PIPELINE var set at cron-create
time in the UI". Crons cannot carry variables at all — `repo cron add` takes
name, branch, schedule and enabled. That sentence described a mechanism that
never existed and was the same misunderstanding that left the daily security
scan silently not running.
The `timeout: 30` lint warning on the four deploy-* files predates this and is
left alone.
Two separate bugs in one `when` block, only one of which is #582.
The cron gate has never matched. It asked for `PIPELINE == "security-scan"` on
`event: cron`, and crons cannot carry variables — `repo cron add` takes name,
branch, schedule and enabled, and nothing else. So the variable is undefined on
every cron run and the workflow is filtered out. The `daily-security-scan` cron
does fire, and pipeline 6877 shows what actually ran under it: the `e2e`
workflow alone, which opts into that cron by name. The Trivy scan and the
security-fix agent have not been running on schedule. This predates 3.18.0 and
would have survived any fix to it.
The manual gate is the #582 case. Woodpecker 3.18 stopped merging manual-run
variables into the environment the workflow-root filter evaluates against
(builder.go:102-111 builds it from metadata plus global envs; AdditionalEnvs is
passed to the compiler at line 223 but never into this map), so
`PIPELINE == "security-scan"` cannot be true there either.
Both now use selectors the root filter can actually see:
cron -> matched by the cron's NAME, the way alert-monitor and e2e
already do it. First-class filter, no variable involved.
deployment -> CI_PIPELINE_DEPLOY_TARGET, which comes from pipeline metadata
(metadata/environment.go:74) and therefore reaches the root
filter on 3.18.
Dispatch becomes `woodpecker-cli pipeline deploy k.renner/homelab <n>
security-scan` — a named deployment target rather than a magic variable that
exists only to pick which of fourteen files is allowed to run.
Pilot for the remaining thirteen. Nothing else is converted until this one is
confirmed working end to end.
The comment blamed #522 — Backup CRs deleted within a minute of completing, so
CNPG has no record to derive the metric from. That is not what is happening.
The CRs are retained: 31 completed backups for ak-pg, im-pg and pprl-pg, 19 for
tr-pg, 11 for wp-pg, 10 for ha-pg, the newest of each from this morning.
The metric still reads 0 everywhere. The actual cause is one level up: the
collector derives it from Cluster.status.lastSuccessfulBackup, and that field
is absent on every Cluster because they all back up with method: plugin and the
barman-cloud plugin path never writes it back. lastFailedBackup is populated on
three of them, which is what makes the absence look like "no backups" rather
than "no reporting".
The conclusion does not change — the alert still cannot be built on that metric
— but the stated blocker does, and #522 is not it. Whoever picks this up next
would have gone looking at CR retention and found nothing wrong.
Also drops the incident retellings and the dated "verified on" stamps from this
file's comments, which is the cleanup already applied elsewhere in the repo.
Chasing a stale comment turned up a live duplicate. Since serviceMonitorSelector
was loosened to {}, Prometheus discovers the goauthentik.io operator's own
ServiceMonitors — which is why it was loosened. The hand-written copies for ldap
and paperless-gate, added back when the strict selector was dropping the
operator's, now scrape the same endpoints a second time.
Measured, not assumed: 10.244.4.113:9300 and 10.244.4.76:9300 were each
producing 40 samples per 10 minutes against 20 for every other outpost, and
ldap picked up a third target because the hand-written selector matched the
plain Service as well as the -metrics one. No duplicate-timestamp errors,
because the two pools scrape at different offsets — the cost is doubled
ingestion and a target list that lies about how many outposts there are.
Radius stays, and the comment now says why it is not the same case: the
operator's ServiceMonitor for it selects on managed-by=goauthentik.io together
with service-type=metrics, and no Service carries both — the operator's has the
first, the one defined here has the second. It is the only thing scraping
radius, so deleting it would have been a silent loss.
Also corrects the comments that claimed the release label is required for
discovery. It is not, for ServiceMonitors and PodMonitors: those two selectors
are empty. It very much is for Probes, PrometheusRules and ScrapeConfigs, whose
selectors still match on it, so probes-blackbox.yaml now says so explicitly —
otherwise the corrected comments elsewhere read as license to drop the label
there too.
Live selector state, for the record:
serviceMonitorSelector {} podMonitorSelector {}
probeSelector / ruleSelector / scrapeConfigSelector release=kube-prometheus-stack
Deleting the wear warning outright went too far. Wear that is high AND still
climbing is exactly the thing worth knowing about; only the level on its own
is noise. This replaces the flat 80% threshold with the combination.
The rule now projects: at the write rate of the last seven days, when does the
drive reach 95%? Under six months, and above 80% already, it alerts. The
projection clears when writes slow, which the level never could.
The rate is self-calibrating. Each drive reports both lifetime bytes written
and its own wear percentage, so bytes-per-percent is derived from the drive
rather than from a rated-endurance table that would have to be kept in sync
with the hardware. That matters here because the fleet is three different
models with three different endurance ratings sharing one rule.
It also has to work that way. percentage_used is a whole-number percent and
advances roughly once every six weeks on these drives, so any rate taken from
it directly is either zero or a single step — far too coarse to project from,
and the reason a delta- or predict_linear-based rule would have been useless.
bytes_written moves on every scrape.
The >= 80 gate is numerical rather than editorial: bytes-per-percent divides
by an integer percentage, so on a lightly worn drive half a percent of
rounding swings the projection by tens of percent. Idle drives divide by a
zero rate, yield +Inf, and stay silent without needing a special case.
Measured against live Prometheus before committing: the expression returns
nothing today, and with the horizon widened it selects exactly the three
Crucial drives at 81/83/87%, projecting 7822, 6513 and 3544 days to 95%. High,
flat, and roughly a decade of headroom — which is why the old rule was firing
every fifteen minutes about nothing.
The 95% critical alert stays a flat threshold. Past rated endurance the drive
is outside what the manufacturer stands behind however fast it got there, and
that alert clears by replacing the drive.
Closes#686
NvmeDriveWearHigh has produced four identical issues since yesterday — #673,
#675, #682, #686 — and would have kept producing them every fifteen minutes.
The alert-monitor deduplicates against OPEN issues, so closing one just makes
room for the next.
That is not a bug in the pipeline. It is a bug in the rule I wrote yesterday:
wear only ever increases, so a `>= 80` threshold starts firing the day a drive
crosses it and never stops. It is a status dressed as an alert, and no
suppression mechanism fixes a rule that is permanently true by construction.
The second half is worse. The drives it fired on are the EPHEMERAL volumes,
worn deliberately — they hold /var precisely because their remaining endurance
does not matter there. The alert was reporting the design as a fault, three
times over, for drives at 81/83/87% that are doing exactly what they were put
there to do.
So the warning goes and the critical threshold stays. 95% is a decision — the
drive is outside what the manufacturer stands behind — and a decision is what
an alert is for. The wear figure itself lives on the Disk Health dashboard,
per drive and with the role attached, which is where a slowly-moving budget
belongs.
The comment left behind says all of this, because the obvious reaction to
"there is no wear warning" is to add one back.
Not done here, deliberately: the noise log (#326) would also have silenced
this, and it is the designed path for an alert that fires continuously without
being repo-fixable. It is the wrong tool here — the alert was fixable, and
suppressing a bad rule leaves the bad rule.
Closes#686
559 -> 509 comment lines. A far smaller cut than the other files, and that is
the honest result: most of this file is per-entry rationale for a declarative
blueprint, where the comment is the only place the constraint is written down.
Everything load-bearing stays, and stays emphasised. Every ORDER MATTERS note
survives — Application before Provider, Application before Provider before
Outpost — because Application.provider is on_delete=PROTECT and getting the
order wrong leaves the blueprint failing to apply with no obvious cause. So do
both "git owns membership, add and remove people HERE not in the UI" notes,
since blueprint apply silently reverts UI edits to those groups. So does the
reason each gate exists at all: code-server runs --auth=none as root over
plaintext HA secrets, and shelly-manager has no authentication whatsoever.
Cut: the derivations. Why the dual-OAuth2 setup was collapsed into one
provider, the full W3C-spec account of credentials:'omit', a paragraph
reconstructing how the music-assistant gate failed. The instruction that came
out of that last one is kept and sharpened into "do not reintroduce a gate in
front of a PWA".
Two stale things went with it. The paperless proxy comment said the ingress
controller "only routes paperless.renner.uno to the outpost service" and named
ingress-nginx as that controller — there is no ingress-nginx left, Traefik is
the only IngressClass. And a leftover instruction to "delete via the Authentik
UI once everything's confirmed working" from a session long finished.
Verified beyond the usual: these comments live INSIDE the embedded blueprint
string, so a broken indent would render fine and only fail when Authentik
parses it. Extracted the rendered ConfigMap's k8s-outposts.yaml and parsed it
separately — valid, and still 74 entries.
security-bot
k.renner
claude-bot