Cleans up the duplication behind the version drift rather than only the drift.
talos/cilium.yaml is deleted. It was a second, un-indented copy of the same
~2000 lines already embedded in the patch, and nothing read it — the only
references anywhere were two comments in the HelmRelease. It existed to be
readable next to the patch and cost a whole file that could go stale on its
own, which is precisely what happened.
The seed's header now records how to reproduce it: the exact yq and helm
template invocation, and why the two overrides exist (the ServiceMonitor CRD is
absent at bootstrap; the chart's default cert method would render private keys
into git, #188). None of that was written down, which is why reconstructing the
original value set today took a diff-until-the-object-kinds-match exercise
rather than reading a line.
Three settings move from nowhere into the HelmRelease: bgpControlPlane.enabled
and the two native-routing CIDRs. They were live but tracked in no file at all,
applied by hand and held by the kubectl-patch field manager on cilium-config.
Server-side apply left them alone so nothing ever broke — but a cold rebuild
installs only the seed, and would have brought the cluster up with the BGP
control plane off and no session to the UDM, while BGP looked configured
because the peering CRDs are in git.
That is not a free change of ownership. Enabling bgpControlPlane through the
chart emits five companion keys the hand-patch never set, so Cilium has been
running on its own internal defaults for router-id allocation and friends. The
comment says which ones and what to watch.
The per-key "THREE COPIES of this key exist" warnings are gone, replaced by one
note at the top of the values block: these values feed the seed, and nothing
enforces agreement. Two copies now, not three, and the surviving one says how
it is made.
Deliberately not done: a generator script and a CI check to diff the committed
seed against a fresh render. That answers duplication with more machinery and
still leaves a 2000-line generated artifact in the tree. If this should stop
being hand-maintained the real question is whether Talos should carry the CNI
at all, which is a rebuild-path decision, not a scripting one.
bootstrap.sh is untouched. Its talos phase remains dead.
Its header claimed the delete document is a no-op against an empty initial
config, so "same file therefore serves both paths". #581 disproved exactly
that: the delete fails against a base with no inlineManifests, which is why
`bootstrap.sh talos` cannot generate a config and why RUNBOOK-cold-rebuild.md
tells a rebuild to extract only the second, additive document by hand.
The issue was closed and both runbooks were updated. The claim it disproved was
left in the file, where anyone reading the patch is still told it works for
bootstrap.
The inline manifest that seeds the CNI at bootstrap still carried Cilium
v1.19.2 while the Helm release has been on 1.20.x since 2026-08-16. That was my
omission: 8f45e290 moved the chart constraint to >=1.20.0 <1.21.0 and never
carried the version into the seed, on the same day 621e4e25 was editing this
very file for policyMapMax.
It matters because talosctl upgrade-k8s reconciles bootstrap manifests. Its dry
run against the live cluster shows what a Kubernetes upgrade would have done
before this change:
image: quay.io/cilium/operator-generic:v1.20.1 -> v1.19.2
bpf-lb-external-clusterip: "true" -> "false"
bpf-lb-sock: "true" -> "false"
k8s-require-ipv6-pod-cidr: "false" -> "true"
A CNI downgrade across a minor, socket load balancing switched off, and
external-ClusterIP handling switched off, on a running cluster.
The seed is regenerated by rendering the chart at 1.20.1 with the same values
the HelmRelease uses, minus prometheus.serviceMonitor — its CRD does not exist
at bootstrap, which is why the seed never carried one. Everything else now
matches, so the reconcile is materially a no-op: verified key by key against
the live cilium-config, including the three flags above.
The rendered operator digest is identical to the one the live cluster runs,
which is the strongest available check that the values were reconstructed
correctly rather than merely plausibly.
Not changed, and worth recording because it looks like drift and is not: the
live ConfigMap has three field managers (helm-controller, talos, kubectl-patch)
and carries enable-bgp-control-plane, the native-routing CIDRs and the
identity-label filter. The dry run shows all of them as unchanged context —
server-side apply leaves fields owned by another manager alone. They are not at
risk here.
The coupling this exposes has no guard: nothing fails when the seed and the
HelmRelease drift apart, and the gap is only visible in an upgrade-k8s dry run
that nobody runs routinely. A CI check comparing the two would have caught this
in August.
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.
What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.
So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".
Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.
Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-bot
k.renner