Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.
Not every one of them got a probe, and the ones that did not are the more
interesting half.
Added:
whisper, openwakeword tcpSocket on the wyoming port. These speak a raw
speech-to-phrase TCP protocol with no HTTP surface and no health
verb, so accepting a connection is all that can
be asserted from outside the process. Weak, but
the previous state was nothing at all, and a dead
server kept being routed speech.
speech-to-phrase gets a startupProbe rather than
a long liveness delay: it runs --retrain-on-start
and retraining takes as long as it takes. The
gate allows 10 minutes and only then hands over,
so a slow retrain is not mistaken for a hang
while a real hang is still caught in a minute.
ftp-scanner tcpSocket on the FTP control port. This is the
failure that hides best: if vsftpd dies, the
scanner cannot deposit anything and paperless
just sees an empty consume directory, which looks
exactly like nobody having scanned.
code-server httpGet /healthz. Answers 200 whenever the server
is up; its body reports session heartbeat, which
is not health — an idle editor reports "expired"
and is fine.
external-secrets livenessProbe/readinessProbe enabled through the
chart's own values, which is also what starts its
health server: without them the controller
exposes nothing but a metrics port. A wedged
reconciler stayed Ready while every ExternalSecret
silently stopped refreshing.
Deliberately left alone:
protonmail-bridge Already carries a rationale: the bridge can be
offline while the user re-authenticates, and
paperless retries mail fetches itself. A liveness
probe would restart it mid-re-auth. That is a
considered decision, not an omission.
bridge-tls-proxy socat binds 127.0.0.1:11143. Probes are dialled
by the kubelet against the pod IP, so there is
nothing for it to reach.
mariadb-operator Its chart exposes no probe values and no health
bind address. Forcing them through a postRenderer
would break on the next chart bump, so it stays
unprobed and this records why.
ak-outpost-* (11) Pod specs are owned by the goauthentik.io
operator; nothing here can add to them.
Validated with Flux's field manager against the live cluster.
Under the split where gatus watches from outside and Kubernetes watches the
pods, paperless was the weakest link on the Kubernetes side: both probes were
tcpSocket, which only proves gunicorn accepted a connection. A worker deadlock
or a broken Django config would keep the pod Ready.
The existing comment explained why: HTTP probes against /api/ flood
`paperless.auth`, because ENABLE_HTTP_REMOTE_USER makes the remote-user
middleware log every unauthenticated API request. That was a real constraint
and the TCP fallback was a deliberate trade, not an oversight.
It is also narrower than the comment assumed. The middleware only covers the
API surface, and /accounts/login/ is a plain Django view. Measured against the
running pod:
4 requests to /accounts/login/ -> 0 new paperless.auth lines
4 requests to /api/ -> 4 new paperless.auth lines
So the noise is avoidable without giving up an HTTP probe. /accounts/login/
renders a template, which means URL routing, settings and the template engine
all have to be working. It does not prove Postgres is reachable — nothing cheap
on this surface does, and the comment now says so rather than implying more.
ALLOWED_HOSTS was the other way this could have failed: the kubelet sends the
pod IP as Host, and a restrictive value would answer 400 and crashloop the pod.
PAPERLESS_ALLOWED_HOSTS is "*", so it is safe today, and the comment records
what narrowing it would require.
Timings are unchanged — the 120s liveness delay is there for paperless's slow
start and nothing here makes it start faster.
Validated with Flux's own field manager. Applying as any other manager reports
the deployment invalid with two handler types, because tcpSocket stays owned by
kustomize-controller and is not cleared by a foreign apply; that is an artifact
of the check, not of the manifest.
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.
BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.
The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.
BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.
Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.
# The chart ships these disabled, so the controller ran with no probe of
# any kind: a wedged reconciler stayed Ready and every ExternalSecret would
# simply stop refreshing, which surfaces only when a consumer needs a
# credential that has since rotated.
#
# Enabling them also starts the health server — the chart derives
# --health-probe-bind-address from these values, and without them the
# controller exposes nothing but its metrics port.
livenessProbe:
enabled:true
readinessProbe:
enabled:true
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-bot
k.renner