feat: give the remaining unprobed workloads a health check

Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.

Not every one of them got a probe, and the ones that did not are the more
interesting half.

Added:

  whisper, openwakeword       tcpSocket on the wyoming port. These speak a raw
  speech-to-phrase            TCP protocol with no HTTP surface and no health
                              verb, so accepting a connection is all that can
                              be asserted from outside the process. Weak, but
                              the previous state was nothing at all, and a dead
                              server kept being routed speech.

                              speech-to-phrase gets a startupProbe rather than
                              a long liveness delay: it runs --retrain-on-start
                              and retraining takes as long as it takes. The
                              gate allows 10 minutes and only then hands over,
                              so a slow retrain is not mistaken for a hang
                              while a real hang is still caught in a minute.

  ftp-scanner                 tcpSocket on the FTP control port. This is the
                              failure that hides best: if vsftpd dies, the
                              scanner cannot deposit anything and paperless
                              just sees an empty consume directory, which looks
                              exactly like nobody having scanned.

  code-server                 httpGet /healthz. Answers 200 whenever the server
                              is up; its body reports session heartbeat, which
                              is not health — an idle editor reports "expired"
                              and is fine.

  external-secrets            livenessProbe/readinessProbe enabled through the
                              chart's own values, which is also what starts its
                              health server: without them the controller
                              exposes nothing but a metrics port. A wedged
                              reconciler stayed Ready while every ExternalSecret
                              silently stopped refreshing.

Deliberately left alone:

  protonmail-bridge           Already carries a rationale: the bridge can be
                              offline while the user re-authenticates, and
                              paperless retries mail fetches itself. A liveness
                              probe would restart it mid-re-auth. That is a
                              considered decision, not an omission.

  bridge-tls-proxy            socat binds 127.0.0.1:11143. Probes are dialled
                              by the kubelet against the pod IP, so there is
                              nothing for it to reach.

  mariadb-operator            Its chart exposes no probe values and no health
                              bind address. Forcing them through a postRenderer
                              would break on the next chart bump, so it stays
                              unprobed and this records why.

  ak-outpost-* (11)           Pod specs are owned by the goauthentik.io
                              operator; nothing here can add to them.

Validated with Flux's field manager against the live cluster.

kubernetes/apps/home-assistant/deployment.yaml

@@ -210,6 +210,28 @@ spec:
             - name: vscode
               containerPort: 8080
               protocol: TCP
+          # /healthz answers 200 whenever the HTTP server is up; its body
+          # reports session heartbeat state, which is not health — an idle
+          # editor reports "expired" and is perfectly fine. So this asserts
+          # that code-server is serving, not that anyone is using it.
+          #
+          # Liveness matters here more than elsewhere in this pod: code-server
+          # shares the Home Assistant config volume, and a wedged editor is
+          # otherwise invisible until someone tries to open it.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: vscode
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: config
               mountPath: /config

kubernetes/apps/openwakeword/deployment.yaml

@@ -24,6 +24,22 @@ spec:
             - name: wyoming
               containerPort: 10400
               protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process. Weaker than an HTTP probe, but the previous state was none
+          # at all: a dead server stayed Ready and kept being routed to.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 15
+            periodSeconds: 10
+            timeoutSeconds: 5
           resources:
             requests:
               cpu: 100m

kubernetes/apps/paperless/ftp-scanner.yaml

@@ -70,6 +70,27 @@ spec:
             - name: consume
               mountPath: /home/scanner
               subPath: scanner
+          # FTP has no health verb, so the control port accepting a connection
+          # is what can be asserted. It is enough for the failure that matters:
+          # if vsftpd dies the scanner cannot deposit anything and nothing else
+          # in the chain notices, because paperless only ever sees an empty
+          # consume directory — which is indistinguishable from nobody having
+          # scanned.
+          #
+          # Only the control port is probed. The passive range is negotiated
+          # per transfer and is not listening at rest.
+          livenessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: ftp-ctrl
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 5
           resources:
             requests:
               cpu: 20m

kubernetes/apps/speech-to-phrase/deployment.yaml

@@ -34,6 +34,32 @@ spec:
             - name: wyoming
               containerPort: 10300
               protocol: TCP
+          # Wyoming is raw TCP with no HTTP surface and no health verb, so
+          # accepting a connection is all that can be asserted from outside the
+          # process — weaker than an HTTP probe, but the previous state was
+          # none at all.
+          #
+          # A startupProbe rather than a long liveness delay: this container
+          # runs with --retrain-on-start, and retraining against the Home
+          # Assistant entity list takes as long as it takes. The startup gate
+          # allows up to 10 minutes (60 x 10s) and only then hands over to
+          # liveness, so a slow retrain cannot be mistaken for a hang, and a
+          # genuine hang after startup is still caught within a minute.
+          startupProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: models
               mountPath: /models

kubernetes/apps/whisper/deployment.yaml

@@ -29,6 +29,27 @@ spec:
             - name: wyoming
               containerPort: 10300
               protocol: TCP
+          # Wyoming is a raw TCP protocol with no HTTP surface and no health
+          # verb, so accepting a connection is the only thing that can be
+          # asserted from outside the process. That is weaker than an HTTP
+          # probe but strictly better than the previous state of none at all,
+          # where a dead server stayed Ready and Home Assistant kept routing
+          # speech to it.
+          #
+          # The liveness delay is generous because the model is loaded at
+          # startup from the data volume.
+          livenessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 90
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: wyoming
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            timeoutSeconds: 5
           volumeMounts:
             - name: data
               mountPath: /data

kubernetes/infrastructure/external-secrets/helmrelease.yaml

@@ -33,3 +33,16 @@ spec:
     crds: CreateReplace
     remediation:
       retries: 3
+  values:
+    # The chart ships these disabled, so the controller ran with no probe of
+    # any kind: a wedged reconciler stayed Ready and every ExternalSecret would
+    # simply stop refreshing, which surfaces only when a consumer needs a
+    # credential that has since rotated.
+    #
+    # Enabling them also starts the health server — the chart derives
+    # --health-probe-bind-address from these values, and without them the
+    # controller exposes nothing but its metrics port.
+    livenessProbe:
+      enabled: true
+    readinessProbe:
+      enabled: true