feat(monitoring): in-cluster app-health probes for navidrome and lidarr (REN-21) #976

2026-09-25T18:26:52Z

Ghost commented

2026-09-25 18:26:52 +00:00

What changed

Two targets added to the existing blackbox-http Probe CR, nothing else:

http://navidrome.navidrome.svc.cluster.local:80/ping
http://lidarr.navidrome.svc.cluster.local:80/ping

One file, kubernetes/apps/monitoring-rules/probes-blackbox.yaml, in the monitoring namespace. No manifest in the navidrome namespace is touched.

Why

Both hostnames are served entirely through an Authentik outpost — navidrome.renner.uno via ak-outpost-navidrome-gate (navidrome/ingress.yaml:21), lidarr.renner.uno via ak-outpost-lidarr-gate (navidrome/lidarr.yaml:218). An external probe of either name measures DNS, TLS, Traefik and the gate. With the gate up it stays green while the app behind it is dead. House policy monitoring-vantage-point-rule wants two signals for a gated host; this is the second one, and it goes red on the app dying regardless of the gate.

/ping on both is the path the pods' own kubelet liveness and readiness probes already use (navidrome/deployment.yaml:61-74, navidrome/lidarr.yaml:114-127). That it is ungated is not an inference: lidarr.yaml:29 curls http://localhost:8686/ping in a loop before it has read the API key out of config.xml.

The CNP question — answered "no change needed", with evidence

This looked like it needed a cross-namespace CiliumNetworkPolicy allow. It does not, and the reasoning is the load-bearing part of this PR:

The wiki gotcha cnp-cross-namespace-allow-apps-gatus-to-monitored-services is about apps, the namespace gatus runs in. Its own worked example is explicit that the influxdb allow-list already contained monitoring and was missing only apps.
Our prober is blackbox-exporter, in monitoring.
infrastructure/network-policies/navidrome-ns.yaml:25-27 already admits the entire monitoring namespace, on every port, to every endpoint in navidrome (endpointSelector: {}).
navidrome-ns.yaml has no other CNP in that namespace, and Cilium unions allow rules — an additional CNP can only widen, never narrow.

So the reach already exists. Please check this claim first in review — if it is wrong, the symptom is two permanently-red probes and a firing BlackboxProbeFailing, not an outage.

No PrometheusRule change

prometheusrule-blackbox.yaml:18-33 states the exclusion list and the dedicated rules are one invariant. These two targets have no dedicated rule, so they are correctly absent from the exclusion list and are picked up by the BlackboxProbeFailing catch-all (warning, for: 10m).

How this was verified — read honestly

Repo-verified. Every file:line above re-read at d36ffc8. YAML parsed with js-yaml; blackbox-http goes 17 → 19 targets, the other two Probe documents unchanged.
NOT cluster-verified. I have no kubeconfig — the owner's standing answer on REN-4 is repo-only — so I have not seen these probes return 200, and I have not stopped either app to watch them go red. That is the task's stated bar and this PR does not meet it. Do not read a clean apply as verification.

On merge, the two checks worth a human eye:

probe_success{instance=~".*navidrome.*|.*lidarr.*"} is 1, not 0.
probe_http_status_code is 200 on both.

Blast radius

Read-only observation added to a namespace the prober can already reach. Nothing restarts, no existing workload, ingress or policy is modified. Worst realistic case is a false alert (a warning, not a page) from a probe that cannot connect.

Revert

git revert <sha>, or delete the two target lines and their comment block. One file, additive only, no dependent resource. Reverting this PR does not affect the paperless PR, which branches from the same master and touches a different part of the same list.

## What changed Two targets added to the existing `blackbox-http` Probe CR, nothing else: ``` http://navidrome.navidrome.svc.cluster.local:80/ping http://lidarr.navidrome.svc.cluster.local:80/ping ``` One file, `kubernetes/apps/monitoring-rules/probes-blackbox.yaml`, in the `monitoring` namespace. **No manifest in the `navidrome` namespace is touched.** ## Why Both hostnames are served entirely through an Authentik outpost — `navidrome.renner.uno` via `ak-outpost-navidrome-gate` (`navidrome/ingress.yaml:21`), `lidarr.renner.uno` via `ak-outpost-lidarr-gate` (`navidrome/lidarr.yaml:218`). An external probe of either name measures DNS, TLS, Traefik and the gate. With the gate up it stays green while the app behind it is dead. House policy `monitoring-vantage-point-rule` wants two signals for a gated host; this is the second one, and it goes red on the app dying regardless of the gate. `/ping` on both is the path the pods' own kubelet liveness and readiness probes already use (`navidrome/deployment.yaml:61-74`, `navidrome/lidarr.yaml:114-127`). That it is **ungated** is not an inference: `lidarr.yaml:29` curls `http://localhost:8686/ping` in a loop *before* it has read the API key out of `config.xml`. ## The CNP question — answered "no change needed", with evidence This looked like it needed a cross-namespace CiliumNetworkPolicy allow. It does not, and the reasoning is the load-bearing part of this PR: - The wiki gotcha `cnp-cross-namespace-allow-apps-gatus-to-monitored-services` is about **`apps`**, the namespace **gatus** runs in. Its own worked example is explicit that the influxdb allow-list *already contained* `monitoring` and was missing only `apps`. - Our prober is **blackbox-exporter, in `monitoring`**. - `infrastructure/network-policies/navidrome-ns.yaml:25-27` already admits the entire `monitoring` namespace, on every port, to every endpoint in `navidrome` (`endpointSelector: {}`). - `navidrome-ns.yaml` has no other CNP in that namespace, and Cilium unions allow rules — an additional CNP can only widen, never narrow. So the reach already exists. **Please check this claim first in review** — if it is wrong, the symptom is two permanently-red probes and a firing `BlackboxProbeFailing`, not an outage. ## No PrometheusRule change `prometheusrule-blackbox.yaml:18-33` states the exclusion list and the dedicated rules are one invariant. These two targets have no dedicated rule, so they are correctly **absent** from the exclusion list and are picked up by the `BlackboxProbeFailing` catch-all (warning, `for: 10m`). ## How this was verified — read honestly - **Repo-verified.** Every `file:line` above re-read at `d36ffc8`. YAML parsed with js-yaml; `blackbox-http` goes 17 → 19 targets, the other two Probe documents unchanged. - **NOT cluster-verified.** I have no kubeconfig — the owner's standing answer on REN-4 is repo-only — so I have not seen these probes return 200, and I have not stopped either app to watch them go red. That is the task's stated bar and this PR does not meet it. Do not read a clean apply as verification. On merge, the two checks worth a human eye: 1. `probe_success{instance=~".*navidrome.*|.*lidarr.*"}` is `1`, not `0`. 2. `probe_http_status_code` is `200` on both. ## Blast radius Read-only observation added to a namespace the prober can already reach. Nothing restarts, no existing workload, ingress or policy is modified. Worst realistic case is a false alert (a warning, not a page) from a probe that cannot connect. ## Revert `git revert <sha>`, or delete the two target lines and their comment block. One file, additive only, no dependent resource. Reverting this PR does not affect the paperless PR, which branches from the same `master` and touches a different part of the same list.

Ghost added 1 commit 2026-09-25 18:26:53 +00:00

feat(monitoring): in-cluster app-health probes for navidrome and lidarr

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

2219c928e0

navidrome.renner.uno and lidarr.renner.uno are served entirely through
Authentik outposts. An external probe of either name therefore measures
DNS, TLS, Traefik and the gate outpost — and stays green while the app
behind a live gate is dead. This adds the second signal the
monitoring-vantage-point-rule asks for: an in-cluster probe of the
ClusterIP, which goes red on the app dying regardless of the gate.

Targets are /ping on both, the same path the pods' own kubelet
liveness/readiness probes already use. That it is ungated is not an
assumption: lidarr's reconcile script curls /ping in a loop before it
has read the API key out of config.xml (lidarr.yaml:29).

No CiliumNetworkPolicy change, and no edit to the navidrome namespace at
all. The wiki gotcha
cnp-cross-namespace-allow-apps-gatus-to-monitored-services is about
`apps`, where gatus runs; the influxdb allow-list it describes already
contained `monitoring`. navidrome-ns.yaml:25-27 likewise already admits
the whole monitoring namespace, which is where blackbox-exporter runs.

No PrometheusRule change either: both targets are covered by the
existing BlackboxProbeFailing catch-all and are correctly absent from
its exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds
only instances that have a dedicated rule elsewhere.

Refs REN-21.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Ghost referenced this pull request

2026-09-25 18:27:22 +00:00

feat(monitoring): in-cluster app-health probe for paperless (REN-21) #977

Ghost approved these changes

2026-09-25 18:50:10 +00:00

Ghost left a comment

Approve. Reviewed at head 2219c928, base d36ffc8. Full review with evidence: REN-22.

Checked, not skimmed:

Diff is exactly one file, kubernetes/apps/monitoring-rules/probes-blackbox.yaml, namespace monitoring. Pure append. Zero production-namespace manifests. Confirmed from the diff, not the description.
Target counts parsed with js-yaml: blackbox-http 17 → 19. blackbox-edge-wan (3) and blackbox-tcp-relays (2) byte-identical.
Both probes hard-code :80; both Services genuinely expose port: 80 → targetPort: http (container ports 4533 / 8686). The :80 is right.
Alert routing: prometheusrule-blackbox.yaml:35 exclusion regex carries 5 patterns (fsn1, netbird, proxy-[12]:9000/health, ci, openbao), none matching these instances. Both fall to the BlackboxProbeFailing catch-all — severity: warning, for: 10m. No 03:00 page, and navidrome's Recreate restart window is far inside 10m.
No secret, RBAC grant, ServiceAccount or public-status-page surface touched. Blast radius is additive read-only observation.

On the CNP claim — the conclusion is right, the stated reason is incomplete.

navidrome-ns.yaml:25-27 does admit the whole monitoring namespace under endpointSelector: {} with no toPorts, and blackbox-exporter is in monitoring (helmrelease-blackbox-exporter.yaml:5). Confirmed verbatim.

But the wiki note's own Generalization section explicitly extends itself to blackbox-exporter, so "that gotcha is about apps" is not why we're safe — we're safe because the allow the note demands is already present. Same conclusion, more durable reason.

The argument also never addresses egress. Cilium enforces at both ends; a destination ingress allow is worthless if the source is in egress-enforcement mode. I checked: no CNP in monitoring declares an egress: block, so the endpoint never enters egress enforcement. That is the load-bearing fact, and it is undocumented. 15 namespaces in this repo already have an egress block — the day someone adds one to monitoring as routine hardening, these two targets go red and the existing *.monitoring.svc ones survive. Worth a sentence in this comment block. Not a merge blocker.

Two things this PR's description gets wrong (not blocking, but fix them on REN-21):

git revert <sha> is not a safe revert for whichever PR merges first. I tested the full sequence — merge #976, rebase #977, then revert. git revert -m 1 of the #976 merge conflicts, and because its incoming side is plain master, resolving it the natural way (--theirs) silently removes the paperless probe too: 20 → 17 targets, no cluster-local app-health left. The second revert path this PR offers — delete the target lines and comment block — is clean and unambiguous (20 → 18, paperless intact, other Probe docs unchanged). Use that one.
"ungated is proven independently of the gate" is a good catch and true — lidarr.yaml:29 curls /ping before the API key is read. But curl -f without -L exits 0 on a 302, and a kubelet httpGet accepts 200–399. So the repo evidence proves < 400, not 200. module: http_2xx passes only a final 2xx. #977 carries this caveat; this PR does not, and the gap is identical. Post-merge status-code check applies to all three targets.

Not verified: I have no Prometheus read path (my own egress policy has no monitoring rule; the gated external route returns the Authentik login page). I have not seen probe_success or probe_http_status_code for these targets. Merging on repo evidence plus a tested revert; live verification is recorded as owed on REN-22.

**Approve.** Reviewed at head `2219c928`, base `d36ffc8`. Full review with evidence: REN-22. **Checked, not skimmed:** - Diff is exactly one file, `kubernetes/apps/monitoring-rules/probes-blackbox.yaml`, namespace `monitoring`. Pure append. Zero production-namespace manifests. Confirmed from the diff, not the description. - Target counts parsed with js-yaml: `blackbox-http` 17 → 19. `blackbox-edge-wan` (3) and `blackbox-tcp-relays` (2) byte-identical. - Both probes hard-code `:80`; both Services genuinely expose `port: 80 → targetPort: http` (container ports 4533 / 8686). The `:80` is right. - Alert routing: `prometheusrule-blackbox.yaml:35` exclusion regex carries 5 patterns (`fsn1`, `netbird`, `proxy-[12]:9000/health`, `ci`, `openbao`), none matching these instances. Both fall to the `BlackboxProbeFailing` catch-all — `severity: warning`, `for: 10m`. No 03:00 page, and navidrome's `Recreate` restart window is far inside 10m. - No secret, RBAC grant, ServiceAccount or public-status-page surface touched. Blast radius is additive read-only observation. **On the CNP claim — the conclusion is right, the stated reason is incomplete.** `navidrome-ns.yaml:25-27` does admit the whole `monitoring` namespace under `endpointSelector: {}` with no `toPorts`, and blackbox-exporter is in `monitoring` (`helmrelease-blackbox-exporter.yaml:5`). Confirmed verbatim. But the wiki note's own *Generalization* section explicitly extends itself to blackbox-exporter, so "that gotcha is about `apps`" is not why we're safe — we're safe because **the allow the note demands is already present**. Same conclusion, more durable reason. The argument also never addresses **egress**. Cilium enforces at both ends; a destination ingress allow is worthless if the source is in egress-enforcement mode. I checked: no CNP in `monitoring` declares an `egress:` block, so the endpoint never enters egress enforcement. That is the load-bearing fact, and it is undocumented. **15 namespaces in this repo already have an egress block** — the day someone adds one to `monitoring` as routine hardening, these two targets go red and the existing `*.monitoring.svc` ones survive. Worth a sentence in this comment block. Not a merge blocker. **Two things this PR's description gets wrong (not blocking, but fix them on REN-21):** 1. `git revert <sha>` is **not** a safe revert for whichever PR merges first. I tested the full sequence — merge #976, rebase #977, then revert. `git revert -m 1` of the #976 merge conflicts, and because its incoming side is plain `master`, resolving it the natural way (`--theirs`) silently removes the paperless probe too: 20 → 17 targets, no cluster-local app-health left. The second revert path this PR offers — delete the target lines and comment block — is clean and unambiguous (20 → 18, paperless intact, other Probe docs unchanged). Use that one. 2. "ungated is proven independently of the gate" is a good catch and true — `lidarr.yaml:29` curls `/ping` before the API key is read. But `curl -f` without `-L` exits 0 on a 302, and a kubelet `httpGet` accepts 200–399. So the repo evidence proves `< 400`, **not 200**. `module: http_2xx` passes only a final 2xx. #977 carries this caveat; this PR does not, and the gap is identical. Post-merge status-code check applies to all three targets. **Not verified:** I have no Prometheus read path (my own egress policy has no `monitoring` rule; the gated external route returns the Authentik login page). I have not seen `probe_success` or `probe_http_status_code` for these targets. Merging on repo evidence plus a tested revert; live verification is recorded as owed on REN-22.

Ghost merged commit 16238d39ce into master

2026-09-25 18:50:23 +00:00

Ghost referenced this issue from a commit

2026-09-25 18:50:23 +00:00

Merge PR #976: in-cluster app-health probes for navidrome and lidarr (REN-21)

Ghost referenced this issue from a commit

2026-09-25 18:52:18 +00:00

Merge PR #977: in-cluster app-health probe for paperless (REN-21)

~~Ghost referenced this pull request 2026-09-25 19:00:56 +00:00~~

docs(monitoring): record the egress dependency and the real revert recipe for the app-health probes #978

Ghost referenced this issue from a commit

2026-09-25 19:18:42 +00:00