The name says the bootstrap depends on a registry mirror. Reading the patch list
in the runbook, that is the obvious conclusion — and it would be a serious
problem, because a cold rebuild has no mirror: an in-cluster one cannot exist
before the cluster does.
The file configures no mirror. It writes one containerd setting,
discard_unpacked_layers = false, so that pulled layers stay in the content store
and Spegel can advertise them to peers later. Spegel writes the per-registry
hosts.toml itself at runtime; the file's own comment says a
machine.registries.mirrors block is not needed, and there is none anywhere in
talos/patches. Image pulls at bootstrap go straight to the real registries.
So the alarm the name raises is false, but it is the name doing it, and the cost
of that is someone deciding a rebuild is blocked when it is not.
containerd-layer-retention.yaml describes the single thing inside.
Renamed with git mv and the three references updated. Verified `talosctl gen
config` still produces a config with the renamed patch in the list.
The Cilium install had become a block of shell in a runbook for a human to
paste. That is the same rot as a pinned version — nobody runs it between
rebuilds, so nothing tells you when it stopped working. It belongs in the script
that already drives the rest of the bootstrap.
The talos phase was unusable since #581: it passed cilium-cni.yaml whole, and
that file's `$patch: delete` document fails against a config with no
inlineManifests. The file is gone, so the failure is gone with it — reviving the
phase was mostly deleting two lines that referenced it.
It now installs Cilium between `talosctl bootstrap` and the node-Ready wait,
which is the only place it can go: with `cni: none` the kubelet stays NotReady
until a CNI exists, so the wait below would otherwise time out against a cluster
that was never going to converge.
Chart repo, version constraint and values are read out of helmrelease.yaml at
run time, so the script cannot install something different from what Flux
reconciles minutes later. helm takes the semver range verbatim. There is a guard
on the extracted values — the HelmRelease is a two-document file and selecting
the wrong one yields an empty values set that helm silently renders as chart
defaults, which is a failure mode worth catching loudly rather than discovering
in a datapath.
`--wait` because the next step needs a working network, and `upgrade --install`
so a re-run after a partial bootstrap converges instead of erroring.
Verified as far as it can be without wiping a cluster: `talosctl gen config`
with exactly the patch list the script now passes produces a config carrying
cni: none and proxy: disabled, no Cilium manifests, and it validates for metal
mode. The apply-and-bootstrap half is untested since the rewrite, and the
runbook says so — it keeps the manual procedure as the thing to read first and
to fall back on.
The install command in Step 2b hardcoded --version 1.20.1 and the chart repo
URL. That is a copy of the HelmRelease in a text file nobody runs between
rebuilds, so it would have gone stale unnoticed and the next rebuild would have
installed something Flux immediately fought — the same failure the deleted
bootstrap seed produced, in a new place.
Everything now comes out of helmrelease.yaml at the moment the command runs:
the repo URL from the HelmRepository, the values from .spec.values, and
--version from .spec.chart.spec.version verbatim. Helm accepts the semver range
and resolves it exactly as Flux does, so the two cannot pick different releases.
Verified by running the block as written:
--version ">=1.20.0 <1.21.0" -> cilium:v1.20.1, operator-generic:v1.20.1
enable-bgp-control-plane: "true" bpf-lb-sock: "true"
bpf-policy-map-max: "65536" ipv4-native-routing-cidr: 10.0.0.0/8
ServiceMonitor objects: 0
which is the config the cluster is running.
The only literal left is --set prometheus.serviceMonitor.enabled=false, and it
is a bootstrap constraint rather than a value: the CRD arrives with the
monitoring stack long afterwards. Because nothing else is overridden,
helm-controller adopts this release instead of replacing it.
This does not make the step unnecessary. Flux's controllers are pods and cannot
reconcile a CNI they need in order to run; something has to seed it. What it
does mean is that the step has nothing left in it to maintain.
Both blocking findings from the review on #709, and both were real.
The deleted cilium-cni.yaml was the only place in the repo that set
cluster.network.cni.name: none and cluster.proxy.disabled: true. They sat in its
second document next to the inline manifest, so removing the file removed them
too — a fresh `talosctl gen config` from this repo's patches would have produced
a config with Talos's default CNI and kube-proxy running, against a Cilium
release configured with kubeProxyReplacement: true. Two runbooks meanwhile
asserted that cluster.yaml supplied both, which it did not.
RUNBOOK-baremetal-conversion.md also still passed `@<cilium additive half>` in
its gen-config example, three lines above prose saying there is no CNI patch any
more. The equivalent block in the cold-rebuild runbook had been fixed; this one
was missed.
Verified the way it should have been in the first place — by running the
runbook's own command rather than inspecting the live cluster:
talosctl gen config homelab https://k8s.renner.uno:6443 \
--output-types controlplane \
--config-patch @talos/patches/cluster.yaml \
--config-patch @talos/patches/registry-mirror.yaml \
--config-patch @talos/patches/k8s-1.yaml
-> cni=none proxy.disabled=true
-> 0 cilium manifests
The live cluster could never have caught this. It has carried both settings
since it was built, and the node configs were patched by hand rather than
regenerated, so nothing in the verification touched the path a rebuild actually
uses.
Final step of PROPOSAL-cilium-single-owner.md. Talos no longer carries Cilium in
any form, and the HelmRelease is the only place it is configured.
The cluster side is already done and verified:
- the cilium inline manifest is gone from all three machine configs, applied
node by node with no reboot and no effect on the running CNI
- Kubernetes went 1.35.2 -> 1.36.3 in the same operation, with
--manifests-no-prune so the 22 objects Talos claimed were not deleted
- Talos's bootstrap-manifest inventory went from 41 entries to 13, and the 28
cilium/hubble entries are gone. A prune-enabled `upgrade-k8s --dry-run`
reported 28 deletions before and reports 0 now
That last number is the point of the whole exercise. A bare `upgrade-k8s` used
to destroy the CNI and now does nothing to it.
The inventory entries had to be removed by hand. --manifests-no-prune skips the
prune AND the inventory rewrite, so the run that saved the objects also left
Talos still claiming them — the mine was deferred, not cleared. Talos offers no
way to drop a stale claim without the prune that deletes, so the ConfigMap was
edited directly. It is derived bookkeeping, not configuration, and the result is
verifiable in both directions, which is why this was acceptable where hand-
editing configuration would not be.
Repo side, this commit:
- deletes talos/patches/cilium-cni.yaml and talos/cilium.yaml
- adds Step 2b to RUNBOOK-cold-rebuild.md: install Cilium from the HelmRelease
values with helm, after bootstrap and before Flux. The nodes stay NotReady
until then, which the runbook now says out loud rather than leaving someone
to wonder
- drops the "keep in lockstep" and "THREE COPIES" comments, which described a
duplication that no longer exists, and replaces them with one note saying
where Cilium is configured
- corrects RUNBOOK-baremetal-conversion.md, which told the reader to extract
half of a file that is gone
bootstrap.sh is left alone. Its talos phase was already dead and is now dead for
a second reason; the runbook says so.
The cost, stated plainly: the cluster no longer forms itself from the machine
config. A cold rebuild needs one documented helm install that it did not need
before. That is the price of Cilium existing in exactly one place.
Step 1 of the ownership transition in PROPOSAL-cilium-single-owner.md. Nothing
about ownership changes yet — this only moves three live settings out of nowhere
and into git.
enable-bgp-control-plane and both native-routing CIDRs existed in no file at
all. They were applied by hand and are held by the `kubectl-patch` field manager
on cilium-config; server-side apply left them alone, so nothing broke and
nothing reported them. The peering itself was always in git as CRDs in
post-crds/cilium-bgp.yaml, which is what made the gap invisible: BGP looked
configured because its policy was version-controlled, while the switch that
starts the control plane was not. A cold rebuild installs only what git
describes, so it would have come up with two established-looking peers
configured and no BGP at all.
The CIDRs go with it because they are the same decision. They are a masquerading
exception, not a routing statement: pod traffic to 10.0.0.0/8 and fd00::/8
leaves unmasqueraded so LAN services see real pod IPs, and that only works
because BGP advertises the pod CIDRs and gives the UDM a route back. Tracking
one without the other would leave a half-decision in git.
This is not purely a change of ownership, which is why it is its own step and
not folded into a later one. Enabling bgpControlPlane through the chart emits
five companion keys the hand-patch never set, so Cilium has been running on its
own internal defaults for router-id allocation among others. They should agree
with the chart's; if they do not, the sessions re-establish. Verify the peers
after Flux reconciles before moving to step 2.
The proposal called the tunnel redundant because the underlay already has
routes to the pod CIDRs. That reads the two as alternatives, and they are not:
VXLAN carries pod-to-pod directly between nodes, while the BGP-advertised pod
CIDRs exist so traffic from the LAN can reach pods at all, which necessarily
goes through the UDM.
Removing the encapsulation would therefore not remove a layer. It would move
inter-node pod traffic off a direct node-to-node path and onto the underlay,
through the router.
The observation worth keeping is narrower: routingMode is set nowhere, so the
current behaviour is an unexamined chart default rather than a recorded
decision. It happens to be the right default.
Cilium is currently owned twice. The 22 objects that make up the whole
installation — both DaemonSets, the operator, both ConfigMaps, the RBAC, the
cilium-secrets namespace — are listed in Talos's bootstrap-manifest inventory
and simultaneously carry Helm's release annotations. cilium-config has three
field managers.
That is not a tidiness problem. It produced a bootstrap seed twelve days behind
the chart, where a Kubernetes upgrade would have downgraded the CNI and switched
off socket load balancing, and it left the BGP control-plane switch and both
native-routing CIDRs existing in no file at all, so a cold rebuild would have
come up without BGP while the peering CRDs in git said otherwise.
This records the decision — Flux owns Cilium, Talos provides only cni: none and
proxy: disabled — and the ordered transition, including why the order matters:
Talos prunes what its inventory used to list and no longer does, so removing the
manifest naively deletes the CNI.
Two claims in it are verified rather than assumed. An OS upgrade does not
reconcile bootstrap manifests: Talos went to 1.13.9 four days ago and Cilium
stayed at 1.20.1 despite the seed saying 1.19.2. And the prune can be tested
before it is run, because NoPrune is honoured by the dry run as well as the
apply.
No code changes here. PR #706 stays WIP until this is agreed.
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.
What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.
So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".
Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.
Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
security-bot
k.renner
claude-bot