feat: give the remaining unprobed workloads a health check

Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.

Not every one of them got a probe, and the ones that did not are the more
interesting half.

Added:

  whisper, openwakeword       tcpSocket on the wyoming port. These speak a raw
  speech-to-phrase            TCP protocol with no HTTP surface and no health
                              verb, so accepting a connection is all that can
                              be asserted from outside the process. Weak, but
                              the previous state was nothing at all, and a dead
                              server kept being routed speech.

                              speech-to-phrase gets a startupProbe rather than
                              a long liveness delay: it runs --retrain-on-start
                              and retraining takes as long as it takes. The
                              gate allows 10 minutes and only then hands over,
                              so a slow retrain is not mistaken for a hang
                              while a real hang is still caught in a minute.

  ftp-scanner                 tcpSocket on the FTP control port. This is the
                              failure that hides best: if vsftpd dies, the
                              scanner cannot deposit anything and paperless
                              just sees an empty consume directory, which looks
                              exactly like nobody having scanned.

  code-server                 httpGet /healthz. Answers 200 whenever the server
                              is up; its body reports session heartbeat, which
                              is not health — an idle editor reports "expired"
                              and is fine.

  external-secrets            livenessProbe/readinessProbe enabled through the
                              chart's own values, which is also what starts its
                              health server: without them the controller
                              exposes nothing but a metrics port. A wedged
                              reconciler stayed Ready while every ExternalSecret
                              silently stopped refreshing.

Deliberately left alone:

  protonmail-bridge           Already carries a rationale: the bridge can be
                              offline while the user re-authenticates, and
                              paperless retries mail fetches itself. A liveness
                              probe would restart it mid-re-auth. That is a
                              considered decision, not an omission.

  bridge-tls-proxy            socat binds 127.0.0.1:11143. Probes are dialled
                              by the kubelet against the pod IP, so there is
                              nothing for it to reach.

  mariadb-operator            Its chart exposes no probe values and no health
                              bind address. Forcing them through a postRenderer
                              would break on the next chart bump, so it stays
                              unprobed and this records why.

  ak-outpost-* (11)           Pod specs are owned by the goauthentik.io
                              operator; nothing here can add to them.

Validated with Flux's field manager against the live cluster.

kubernetes/apps/gatus/configmap.yaml

@@ -18,15 +18,18 @@ metadata:
 #
 # Opaque-label map (maintainer reference only — NOT served publicly):
 #   services group                   infrastructure group
-#     Web Service 1 = Authentik        Infrastructure 1 = PBS 2 (pbs2.renner.uno)
+#     Web Service 1 = Authentik        Infrastructure 1 = retired (was PBS 2)
 #     Web Service 2 = Gitea            Infrastructure 2 = retired (was Proxmox VE)
 #     Web Service 3 = Immich           Infrastructure 3 = UDM DNS (10.2.40.1)
 #     Web Service 4 = Woodpecker CI    Infrastructure 4 = Mosquitto MQTT broker
 #     Web Service 5 = Grafana
 #     Web Service 6 = Home Assistant
-#     Web Service 7 = Paperless
+#     Web Service 7 = retired (was the NAS file UI; never Paperless, which
+#                     this map claimed and which nothing here checks)
 #     Web Service 8 = Audiobookshelf
 #     Web Service 9 = Music Assistant
+#     Web Service 10 = TRMNL
+#     Web Service 11 = Paperless (gate-level only, see its entry)
 data:
   config.yaml: |
     storage:
@@ -121,18 +124,10 @@ data:
         conditions:
           - "[STATUS] == any(200, 301, 302)"
 
-      - name: Web Service 7
-        group: services
-        url: https://file.renner.uno
-        interval: 60s
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: the NAS file UI (file.renner.uno). Its Ingress and
+      # ExternalName Service were deleted in #696 because they pointed at
+      # nas-1, which is powered off; this check outlived them and had been
+      # reporting failure on the public page since.
 
       - name: Web Service 8
         group: services
@@ -178,6 +173,29 @@ data:
         conditions:
           - "[STATUS] == any(200, 301, 302)"
 
+      # Paperless sits entirely behind the Authentik gate — every path,
+      # including /api/, answers 302 to unauthenticated callers, so there is no
+      # endpoint that proves the app rather than the gate. This check is
+      # therefore reachability only: DNS, TLS, Traefik and the outpost. It
+      # stays green if paperless itself is dead behind a live gate.
+      #
+      # Closing that blind spot needs an in-cluster probe of the paperless
+      # ClusterIP, which crosses a namespace boundary and so needs a
+      # CiliumNetworkPolicy first — the monitoring probes get away without one
+      # only because their targets share a namespace with the prober.
+      - name: Web Service 11
+        group: services
+        url: https://paperless.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
       # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
       # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
       # no longer deployed. Re-add when/if they come back.
@@ -191,20 +209,11 @@ data:
       # NoData). Monitor the offsite PBS from the offsite side or via a
       # check that doesn't depend on this cluster's DNS path.
 
-      - name: Infrastructure 1
-        group: infrastructure
-        url: https://pbs2.renner.uno:8007
-        interval: 60s
-        client:
-          insecure-skip-verify: true
-        ui:
-          hide-hostname: true
-          hide-url: true
-          hide-port: true
-          hide-conditions: true
-          hide-errors: true
-        conditions:
-          - "[STATUS] == any(200, 301, 302)"
+      # Removed 2026-08-28: PBS 2 (pbs2.renner.uno:8007) — it has no clients
+      # left. Its two were the PVE cluster, retired to bare-metal Talos, and
+      # nas-1, powered off and dropped from the repo. Checking the liveness of
+      # a backup server that nothing backs up to reports on nothing. Re-add
+      # alongside the first host that is declared to back up to it.
 
       - name: Infrastructure 3
         group: infrastructure

kubernetes/apps/gatus/deployment.yaml

@@ -5,6 +5,12 @@ metadata:
   namespace: apps
   labels:
     app: gatus
+  annotations:
+    # Gatus reads its config once at startup and never re-reads the file, so
+    # without this a ConfigMap change reconciles cleanly and then does nothing:
+    # git and the running checks drift apart silently, with no failed resource
+    # to notice. Adding a check would not take effect either.
+    configmap.reloader.stakater.com/reload: gatus-config
 spec:
   replicas: 1
   # RWO PVC (`gatus-data`, ceph-block) — RollingUpdate's default 25% surge
@@ -22,9 +28,30 @@ spec:
       labels:
         app: gatus
     spec:
+      # The upstream image is FROM scratch with no USER, so without this the
+      # pod runs as root and trips PodSecurity restricted on four counts.
+      #
+      # The uid is numeric because a scratch image carries no /etc/passwd for
+      # the kubelet to resolve a name against. fsGroup is what makes the switch
+      # survivable: the only write path is the sqlite store on the RWO volume
+      # (/data/gatus.db, see the storage stanza in the ConfigMap), and fsGroup
+      # both chowns the existing volume and leaves the directory group-writable
+      # so sqlite can create its -wal and -journal siblings next to the db.
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
       containers:
         - name: gatus
           image: twinproduction/gatus:v5.36.0 # {"$imagepolicy": "flux-system:gatus"}
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
           ports:
             - containerPort: 8080
               name: http

kubernetes/apps/home-assistant/deployment.yaml

@@ -210,6 +210,28 @@ spec:
             - name: vscode
               containerPort: 8080
               protocol: TCP
+          # /healthz answers 200 whenever the HTTP server is up; its body
+          # reports session heartbeat state, which is not health — an idle
+          # editor reports "expired" and is perfectly fine. So this asserts
+          # that code-server is serving, not that anyone is using it.
+          #
+          # Liveness matters here more than elsewhere in this pod: code-server
+          # shares the Home Assistant config volume, and a wedged editor is
+          # otherwise invisible until someone tries to open it.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: config
               mountPath: /config

kubernetes/apps/monitoring-rules/probes-blackbox.yaml

@@ -66,6 +66,11 @@ spec:
         # parameter, 200 means initialized, unsealed and serving, on whichever
         # node answers; 503 sealed and 501 uninitialized both stay red.
         - https://openbao.renner.uno/v1/sys/health?standbyok=true
+        # ical-proxy serves an unauthenticated /health that answers 200 with
+        # {"status":"healthy"}. Unlike the gated hostnames this is real app
+        # health, not reachability — the root path returns 404 by design, since
+        # the service only answers on calendar paths.
+        - https://ical-proxy.renner.uno/health
         # APP-HEALTH probes for the gated monitoring UIs, from the correct
         # in-cluster vantage (monitoring-vantage-point-rule): the gate is
         # only on the ingress, so the ClusterIP /-/healthy endpoints are
@@ -78,6 +83,11 @@ spec:
         # external reachability probe above; it is not the health vantage.)
         - http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/-/healthy
         - http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/-/healthy
+        # Loki has no external vantage at all: its ingress requires auth on
+        # every path, so an outside probe can only ever report 401. In-cluster
+        # /ready answers 200 unauthenticated and is in this namespace, so the
+        # same intra-namespace allowance the two above rely on covers it.
+        - http://loki.monitoring.svc.cluster.local:3100/ready
 # WAN-path probes of the public edge BY PROXY IP. The name-based
 # home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
 # and never leaves the LAN — the vantage-point blind spot that let the

kubernetes/apps/openwakeword/deployment.yaml

@@ -24,6 +24,22 @@ spec:
             - name: wyoming
               containerPort: 10400
               protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process. Weaker than an HTTP probe, but the previous state was none
+          # at all: a dead server stayed Ready and kept being routed to.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 15
+            periodSeconds: 10
+            timeoutSeconds: 5
           resources:
             requests:
               cpu: 100m

kubernetes/apps/paperless/ftp-scanner.yaml

@@ -70,6 +70,27 @@ spec:
             - name: consume
               mountPath: /home/scanner
               subPath: scanner
+          # FTP has no health verb, so the control port accepting a connection
+          # is what can be asserted. It is enough for the failure that matters:
+          # if vsftpd dies the scanner cannot deposit anything and nothing else
+          # in the chain notices, because paperless only ever sees an empty
+          # consume directory — which is indistinguishable from nobody having
+          # scanned.
+          #
+          # Only the control port is probed. The passive range is negotiated
+          # per transfer and is not listening at rest.
+          livenessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 5
           resources:
             requests:
               cpu: 20m

kubernetes/apps/paperless/paperless.yaml

@@ -155,21 +155,33 @@ spec:
             - name: bridge-cert
               mountPath: /shared
               readOnly: true
-          # TCP probes — HTTP probes against /api/ flooded the
-          # `paperless.auth` logger every 10 s with "No authentication
-          # provided from private IP …". With ENABLE_HTTP_REMOTE_USER
-          # the middleware logs every unauth API request, and there's
-          # no public health endpoint to probe. TCP confirms gunicorn
-          # is bound; that's the same liveness signal we'd get from
-          # /api/ minus the noise.
+          # /accounts/login/ rather than /api/ or a TCP connect.
+          #
+          # /api/ is unusable as a probe target: with ENABLE_HTTP_REMOTE_USER
+          # the remote-user middleware logs "No authentication provided from
+          # private IP …" for every unauthenticated API request, so probing it
+          # floods `paperless.auth`. Measured: four requests to /api/ produce
+          # four log lines, four to /accounts/login/ produce none — the
+          # middleware only covers the API surface.
+          #
+          # A TCP connect avoids the noise but only proves gunicorn accepted a
+          # socket. /accounts/login/ renders a real Django view, so URL
+          # routing, settings and templating all have to work. It does not
+          # prove Postgres is healthy; nothing cheap here does.
+          #
+          # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
+          # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
+          # means giving these probes an explicit Host header.
           livenessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
               port: http
             initialDelaySeconds: 120
             periodSeconds: 30
             timeoutSeconds: 10
           readinessProbe:
-            tcpSocket:
+            httpGet:
+              path: /accounts/login/
               port: http
             initialDelaySeconds: 30
             periodSeconds: 10

kubernetes/apps/speech-to-phrase/deployment.yaml

@@ -34,6 +34,32 @@ spec:
             - name: wyoming
               containerPort: 10300
               protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process — weaker than an HTTP probe, but the previous state was
+          # none at all.
+          #
+          # A startupProbe rather than a long liveness delay: this container
+          # runs with --retrain-on-start, and retraining against the Home
+          # Assistant entity list takes as long as it takes. The startup gate
+          # allows up to 10 minutes (60 x 10s) and only then hands over to
+          # liveness, so a slow retrain cannot be mistaken for a hang, and a
+          # genuine hang after startup is still caught within a minute.
+          startupProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: models
               mountPath: /models

kubernetes/apps/whisper/deployment.yaml

@@ -29,6 +29,27 @@ spec:
             - name: wyoming
               containerPort: 10300
               protocol: TCP
+          # Wyoming is a raw TCP protocol with no HTTP surface and no health
+          # verb, so accepting a connection is the only thing that can be
+          # asserted from outside the process. That is weaker than an HTTP
+          # probe but strictly better than the previous state of none at all,
+          # where a dead server stayed Ready and Home Assistant kept routing
+          # speech to it.
+          #
+          # The liveness delay is generous because the model is loaded at
+          # startup from the data volume.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 90
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: data
               mountPath: /data

kubernetes/infrastructure/external-secrets/helmrelease.yaml

@@ -33,3 +33,16 @@ spec:
     crds: CreateReplace
     remediation:
       retries: 3
+  values:
+    # The chart ships these disabled, so the controller ran with no probe of
+    # any kind: a wedged reconciler stayed Ready and every ExternalSecret would
+    # simply stop refreshing, which surfaces only when a consumer needs a
+    # credential that has since rotated.
+    #
+    # Enabling them also starts the health server — the chart derives
+    # --health-probe-bind-address from these values, and without them the
+    # controller exposes nothing but its metrics port.
+    livenessProbe:
+      enabled: true
+    readinessProbe:
+      enabled: true