test(monitoring): temporary negative-test probes for REN-28 §7 (REN-140) (#1015)

Reviewed and merged by argus-bot (REN-149). Author atlas-bot; four-eyes split intact.

Verified live: no alerting rule can match job="blackbox-ren28-negative" (24 probe_/up/scrape_ rules enumerated; tcp_connect module timeout 5s < scrapeTimeout 10s keeps up==1); pod IPs 10.244.4.166/10.244.2.220 current; probeSelector strict so the release label is load-bearing and is set; monitoring has no egress enforcement.

Temporary. Delete probes-ren28-negative.yaml and its kustomization line once the REN-28 "after" reading is recorded.

Revert: git revert -m 1 <this sha>

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -58,3 +58,7 @@ resources:
   # kube-state-metrics instead. See the file.
   - flux-podmonitor.yaml
   - probes-blackbox.yaml
+  # TEMPORARY, REN-140. Negative-test probes for the REN-28 §7 bar, taking the
+  # "reachable before" reading that stops being takeable once the paperless PR
+  # merges. Delete this line and the file once the "after" reading is recorded.
+  - probes-ren28-negative.yaml

kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml

@@ -0,0 +1,98 @@
+# TEMPORARY — negative-test probes for REN-28 §7. DELETE AFTER THE "AFTER"
+# READING IS TAKEN. This file exists to make two of the four §7 negative tests
+# measurable without granting `pods/exec` in `monitoring`, which the owner
+# declined on REN-140 as a wider grant than the change it would verify.
+#
+# WHAT IT IS. blackbox-exporter is already an HTTP/TCP prober running in the
+# `monitoring` namespace. Pointing it at targets that the REN-28 paperless PR
+# is supposed to CLOSE turns "is this path still open?" into a number in
+# Prometheus, readable through the existing read-only REN-36 grant. Every
+# target below answers today and MUST stop answering after the PR lands. A
+# `probe_success` that stays 1 after the merge means the rule does not do what
+# §5 claims.
+#
+# WHY IT DOES NOT PAGE ANYONE. jobName is `blackbox-ren28-negative`, not
+# `blackbox-http`. Every rule in prometheusrule-blackbox.yaml is scoped to
+# job="blackbox-http" (BlackboxProbeFailing) or absent(...{job="blackbox-http"})
+# (BlackboxProbesAbsent), so neither can match these series — before or after.
+# TargetDown (general.rules) cannot match either: the scrape target of a Probe
+# is blackbox-exporter itself, which answers 200 with probe_success 0 on a
+# failed probe, so `up` stays 1 regardless of the result. Verified against the
+# live rule set on 2026-09-26. This is deliberate: a monitor that pages for an
+# expected failure is worse than no monitor.
+#
+# NO POLICY CHANGE IS NEEDED TO MAKE THESE WORK TODAY. Ingress: the paperless
+# baseline still admits the whole `monitoring` namespace on every port
+# (infrastructure/network-policies/paperless-ns.yaml:34-36) — that admission is
+# exactly what REN-28 removes. Egress: the `monitoring` namespace is in no
+# egress enforcement at all; its only two Cilium policies, monitoring-ns.yaml
+# and monitoring-alloy-syslog.yaml, are ingress-only. Same dependency the three
+# cross-namespace targets in probes-blackbox.yaml already rest on.
+#
+# TWO POD-IP TARGETS. pprl-pg's :9187 metrics port is on the POD, not on any
+# Service — that is why it is scraped by a PodMonitor and not a ServiceMonitor
+# (cnpg-podmonitors.yaml). There is no stable name for it, so the literal pod
+# IPs are used. They were read live on 2026-09-26; if either pod is rescheduled
+# the probe goes red for the wrong reason. BEFORE TRUSTING AN "AFTER" READING,
+# re-confirm the IP still belongs to the same pod:
+#
+#   kube_pod_info{namespace="paperless", pod=~"pprl-pg-.*"}   -> pod_ip
+#
+# The pprl-pg-r:5432 target below is the stable-name version of the same
+# statement and survives a reschedule; it is on the database port rather than
+# the metrics port, so the two are kept together on purpose.
+#
+# REVERT. Delete this file and its line in kustomization.yaml. Nothing else
+# references it; no workload is touched; Prometheus drops the job on the next
+# operator reconcile.
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: blackbox-ren28-negative
+  namespace: monitoring
+  labels:
+    # probeSelector is NOT loosened on this cluster (helmrelease-prometheus.yaml
+    # keeps ruleSelector/probeSelector/scrapeConfigSelector strict), so this
+    # label is load-bearing, not decorative. Without it the Probe is created and
+    # never discovered — and a test that silently does not run reads exactly
+    # like a test that passed.
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-ren28-negative
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: tcp_connect
+  interval: 60s
+  # 10s matches blackbox-tcp-relays. It is also the discriminator: a Cilium
+  # deny DROPS the SYN, so a refused path completes in ~1ms and a denied path
+  # burns the full timeout. probe_duration_seconds distinguishes "nothing is
+  # listening" from "policy stopped it" — probe_success alone does not.
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        # REN-28 §7 test 4 — "from a monitoring pod that is not Prometheus, a
+        # connection to pprl-pg :9187 must time out". blackbox-exporter is
+        # precisely that pod. Both CNPG instances, because the PR's
+        # pprl-pg-allow-prometheus rule selects on a label both carry: if the
+        # selector were wrong in a way that left one instance open, one target
+        # would tell us and the other would not.
+        - 10.244.4.166:9187 # pprl-pg-2, node k8s-2
+        - 10.244.2.220:9187 # pprl-pg-3, node k8s-3
+        # Same statement on a stable name and the port that actually matters:
+        # after the PR the only monitoring→pprl-pg allow is Prometheus on
+        # :9187, so a non-Prometheus monitoring pod must not reach Postgres.
+        - pprl-pg-r.paperless.svc.cluster.local:5432
+        # REN-28 §7 test 2, workload dimension — "from blackbox-exporter, a
+        # paperless target other than the one it is scoped to must time out".
+        # The test as written in §7 says "a paperless POD port other than
+        # 8000"; there is no such port. The paperless pod's only other listener
+        # is the bridge-tls-proxy socat, bound to 127.0.0.1:11143
+        # (apps/paperless/paperless.yaml:289), so every other port on that pod
+        # is closed today and answers RST — there is no "reachable before" to
+        # record and the port half of paperless-allow-blackbox (:8000) cannot
+        # be tested at all. gotenberg is the half that CAN be: it is one of the
+        # other eight workloads the endpointSelector:{} baseline currently
+        # exposes to all of `monitoring`, and the PR's per-workload selector is
+        # what closes it.
+        - gotenberg.paperless.svc.cluster.local:3000