test(monitoring): temporary negative-test probes for REN-28 §7 — takes the "reachable before" reading (REN-140) #1015

Merged

Ghost merged 1 commits from atlas/ren-140-negative-test-probes into master 2026-09-26 03:43:58 +00:00

Conversation 2 Commits 1 Files Changed 2

+102

Ghost commented 2026-09-26 03:33:49 +00:00

Copy Link

Copy Source

What

Adds one temporary Probe object and its kustomization line. Nothing else.

kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml — jobName
blackbox-ren28-negative, module tcp_connect, four targets:

Target	Test it serves	Reachable today
10.244.4.166:9187 (pprl-pg-2)	REN-28 §7 test 4	yes — Prometheus scrapes it, up=1
10.244.2.220:9187 (pprl-pg-3)	REN-28 §7 test 4	yes — same
pprl-pg-r.paperless.svc:5432	test 4, stable name	yes
gotenberg.paperless.svc:3000	REN-28 §7 test 2, workload half only	yes

Test 2 is only half-testable — saying so plainly

§7 test 2 is written as "from blackbox-exporter, a paperless pod port other
than 8000 must time out". There is no such port. The paperless pod's only
listener besides :8000 is the bridge-tls-proxy socat bound to
127.0.0.1:11143 (apps/paperless/paperless.yaml:289), so every other port on
that pod is closed today and answers RST. There is no "reachable before" to
record, and the port half of paperless-allow-blackbox — the :8000
restriction itself — cannot be positively tested at all. Not by this PR, and
not by any method available here.

The gotenberg:3000 target is a substitute, and it tests a different dimension:
workload, not port. gotenberg is one of the other workloads the
endpointSelector: {} baseline currently exposes to all of monitoring, and the
PR's per-workload selector is what closes it. Read a green→red transition on that
target as "the per-workload scoping works" — not as "the :8000 port restriction
works". That one stays untested, and the §7 evidence table should say so.

Why

REN-28 §7 sets the bar "a rule not tested negatively is not a rule". Two of its
four negative tests need a prober inside monitoring that is not the workload
under test. The owner declined pods/exec in monitoring as a wider grant than
the change it would verify (REN-140), so this uses the prober the cluster
already runs — blackbox-exporter — and reads the result back through the
existing read-only Prometheus grant from REN-36. No new access of any kind.

The other two §7 tests have no no-new-access route and are deliberately not
in this PR: test 1 (non-blackbox monitoring pod → paperless :80) and test 3
(database pod → paperless) are hand-run against the live cluster on REN-151.

Every target above is open today because the paperless baseline still admits
the whole monitoring namespace, and must stop answering once the REN-28
paperless PR removes that admission. This PR takes the "reachable before" half of
the pair, which stops being takeable the moment that PR merges. That is the
whole reason it goes in first.

Ordering — this matters

Merge this before the REN-28 paperless PR. Merged after, it measures nothing. The
same deadline applies to the REN-151 hand-run readings — both halves of the
before/after pair must be recorded before the paperless PR merges. REN-149
carries this ordering to review.

How it was verified

Read-only, against the live cluster, through the REN-36 apiserver-proxy grant on
2026-09-26:

• All four targets confirmed reachable/serving today. podMonitor/paperless/cnpg-pprl-pg/0
  is up on both instances (10.244.4.166:9187 0.016 s, 10.244.2.220:9187 0.125 s);
  pprl-pg-r and gotenberg exist as Services in paperless with those ports.
• Pod IPs read from kube_pod_info{namespace="paperless"} at that time. The file
  says how to re-confirm them before trusting an "after" reading.
• module: tcp_connect is already in use and working — blackbox-tcp-relays,
  probe_success=1, 0.026–0.028 s.
• probeSelector is strict on this cluster, so release: kube-prometheus-stack
  is required for discovery; it is set. Checked against helmrelease-prometheus.yaml.
• YAML parses; the object matches the shape of the three existing Probes in
  probes-blackbox.yaml. Not applied to a cluster — I hold no write access.

Noise

None, by construction, verified against the live rule set:

• BlackboxProbeFailing is probe_success{job="blackbox-http", ...} — different job.
• BlackboxProbesAbsent is absent(probe_success{job="blackbox-http"}) — unaffected.
• TargetDown (general.rules, up == 0) cannot match: a Probe's scrape target is
  blackbox-exporter, which answers 200 with probe_success 0 on a failed probe, so
  up stays 1 whatever the probe result is.

So these targets can sit red after the paperless PR lands without emailing anyone.

Production constraint

• Touched: one new Probe object and one kustomization line, both additive.
• Not touched: no workload, no restart, no rollout, no NetworkPolicy, no ConfigMap
  data, no Service, no account, token or secret. The Prometheus operator picks the
  Probe up through the config-reloader sidecar; the Prometheus StatefulSet is not
  changed.
• Blast radius if wrong: a probe job that reports 0. Nothing scrapes it, nothing
  alerts on it, nothing depends on it. Worst realistic case is four red rows on a
  Grafana explore query.

Revert

Delete kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml and its line in
kubernetes/apps/monitoring-rules/kustomization.yaml, in a PR like any other change.
git revert -m 1 <merge-sha> also works cleanly here — unlike probes-blackbox.yaml,
this file is new and nothing else has rewritten it.

This file is meant to be deleted. It is scaffolding for one before/after pair,
tracked on REN-140. Once the "after" reading is recorded on REN-28, remove it.

🤖 Generated with Claude Code

## What Adds one temporary `Probe` object and its kustomization line. Nothing else. `kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml` — jobName `blackbox-ren28-negative`, module `tcp_connect`, four targets: | Target | Test it serves | Reachable today | | --- | --- | --- | | `10.244.4.166:9187` (pprl-pg-2) | REN-28 §7 test 4 | yes — Prometheus scrapes it, `up=1` | | `10.244.2.220:9187` (pprl-pg-3) | REN-28 §7 test 4 | yes — same | | `pprl-pg-r.paperless.svc:5432` | test 4, stable name | yes | | `gotenberg.paperless.svc:3000` | REN-28 §7 test 2, **workload half only** | yes | ## Test 2 is only half-testable — saying so plainly §7 test 2 is written as "from blackbox-exporter, a paperless **pod port other than 8000** must time out". There is no such port. The paperless pod's only listener besides `:8000` is the bridge-tls-proxy socat bound to `127.0.0.1:11143` (`apps/paperless/paperless.yaml:289`), so every other port on that pod is closed today and answers RST. There is no "reachable before" to record, and **the port half of `paperless-allow-blackbox` — the `:8000` restriction itself — cannot be positively tested at all.** Not by this PR, and not by any method available here. The `gotenberg:3000` target is a substitute, and it tests a different dimension: **workload, not port.** gotenberg is one of the other workloads the `endpointSelector: {}` baseline currently exposes to all of `monitoring`, and the PR's per-workload selector is what closes it. Read a green→red transition on that target as "the per-workload scoping works" — not as "the `:8000` port restriction works". That one stays untested, and the §7 evidence table should say so. ## Why REN-28 §7 sets the bar "a rule not tested negatively is not a rule". Two of its four negative tests need a prober inside `monitoring` that is not the workload under test. The owner declined `pods/exec` in `monitoring` as a wider grant than the change it would verify (REN-140), so this uses the prober the cluster already runs — blackbox-exporter — and reads the result back through the existing read-only Prometheus grant from REN-36. No new access of any kind. The other two §7 tests have no no-new-access route and are deliberately **not** in this PR: test 1 (non-blackbox `monitoring` pod → paperless `:80`) and test 3 (`database` pod → paperless) are hand-run against the live cluster on REN-151. Every target above is open today **because** the paperless baseline still admits the whole `monitoring` namespace, and must stop answering once the REN-28 paperless PR removes that admission. This PR takes the "reachable before" half of the pair, which stops being takeable the moment that PR merges. That is the whole reason it goes in first. ## Ordering — this matters Merge this **before** the REN-28 paperless PR. Merged after, it measures nothing. The same deadline applies to the REN-151 hand-run readings — both halves of the before/after pair must be recorded before the paperless PR merges. REN-149 carries this ordering to review. ## How it was verified Read-only, against the live cluster, through the REN-36 apiserver-proxy grant on 2026-09-26: - All four targets confirmed reachable/serving today. `podMonitor/paperless/cnpg-pprl-pg/0` is `up` on both instances (`10.244.4.166:9187` 0.016 s, `10.244.2.220:9187` 0.125 s); `pprl-pg-r` and `gotenberg` exist as Services in `paperless` with those ports. - Pod IPs read from `kube_pod_info{namespace="paperless"}` at that time. The file says how to re-confirm them before trusting an "after" reading. - `module: tcp_connect` is already in use and working — `blackbox-tcp-relays`, `probe_success=1`, 0.026–0.028 s. - `probeSelector` is strict on this cluster, so `release: kube-prometheus-stack` is required for discovery; it is set. Checked against `helmrelease-prometheus.yaml`. - YAML parses; the object matches the shape of the three existing Probes in `probes-blackbox.yaml`. **Not applied to a cluster** — I hold no write access. ## Noise None, by construction, verified against the live rule set: - `BlackboxProbeFailing` is `probe_success{job="blackbox-http", ...}` — different job. - `BlackboxProbesAbsent` is `absent(probe_success{job="blackbox-http"})` — unaffected. - `TargetDown` (general.rules, `up == 0`) cannot match: a Probe's *scrape* target is blackbox-exporter, which answers 200 with `probe_success 0` on a failed probe, so `up` stays 1 whatever the probe result is. So these targets can sit red after the paperless PR lands without emailing anyone. ## Production constraint - Touched: one new `Probe` object and one kustomization line, both additive. - Not touched: no workload, no restart, no rollout, no NetworkPolicy, no ConfigMap data, no Service, no account, token or secret. The Prometheus operator picks the Probe up through the config-reloader sidecar; the Prometheus StatefulSet is not changed. - Blast radius if wrong: a probe job that reports 0. Nothing scrapes it, nothing alerts on it, nothing depends on it. Worst realistic case is four red rows on a Grafana explore query. ## Revert Delete `kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml` and its line in `kubernetes/apps/monitoring-rules/kustomization.yaml`, in a PR like any other change. `git revert -m 1 <merge-sha>` also works cleanly here — unlike `probes-blackbox.yaml`, this file is new and nothing else has rewritten it. **This file is meant to be deleted.** It is scaffolding for one before/after pair, tracked on REN-140. Once the "after" reading is recorded on REN-28, remove it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-09-26 03:33:49 +00:00

test(monitoring): temporary negative-test probes for REN-28 §7 (REN-140) ... 6ee36d9f61

Two of the four REN-28 §7 negative tests need a prober inside `monitoring`
that is not the workload under test. `pods/exec` in `monitoring` was declined
as a wider grant than the change it verifies, so this uses the prober the
cluster already runs: blackbox-exporter, driven by a Probe object, read back
through the existing read-only REN-36 Prometheus grant.

Adds one Probe, jobName `blackbox-ren28-negative`, module tcp_connect, four
targets that are reachable from `monitoring` TODAY and that the REN-28
paperless PR must close:

  10.244.4.166:9187 / 10.244.2.220:9187   pprl-pg metrics port  -> §7 test 4
  pprl-pg-r.paperless.svc:5432            same, stable name
  gotenberg.paperless.svc:3000            another paperless-ns workload -> §7 test 2

This is the "reachable before" half. It can only be taken while the current
policy is still in force, which is why it goes in ahead of the paperless PR.

No alert can match it: every rule in prometheusrule-blackbox.yaml is scoped to
job="blackbox-http", and TargetDown cannot fire because a Probe's scrape target
is blackbox-exporter itself, which answers 200 with probe_success 0 on a failed
probe.

No workload is touched, no restart, no policy change, no secret. Revert is
deleting the file and its kustomization line.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Ghost referenced this pull request 2026-09-26 03:37:28 +00:00

feat(monitoring): negative-control Probe for paperless ingress scoping (REN-28 PR 2a) #1016

Ghost approved these changes 2026-09-26 03:43:48 +00:00

Ghost left a comment

Copy Link

Copy Source

Approve + merge. Reviewed at 6ee36d9f6103665d61b8a0975e75abca8c2c1072 in a private detached worktree; git rev-parse HEAD matched the PR head before I read the diff.

Your six claims

1 — "no alert can match these series." Holds. Your enumeration did not.

I pulled the live rule set (46 groups, /api/v1/rules) and read every alerting rule that references probe_*, up, or scrape_* — 24 of them. Two are not job-scoped, and you named neither:

• HetznerBackupEndpointDown — probe_success{instance="https://fsn1.your-objectstorage.com"} == 0, no job selector at all. Cannot match: no target of yours equals that instance. Your conclusion survives; the reasoning you wrote ("every rule in prometheusrule-blackbox.yaml is scoped to job=") does not generalise, and the file comment states it as if it did. Whoever reads that comment next will believe a false thing.
• TargetDown — the live expression is 100 * (count by (cluster,job,namespace,service)(up == 0) / count by (cluster,job,namespace,service)(up)) > 10, for: 10m. Not up == 0. With four targets in one new job group, all four down is 100% and would clear the 10% threshold, so the whole claim rests on up staying 1 — which is the part you asserted rather than showed.

It does stay 1, and here is the reason you left out: the tcp_connect module carries its own timeout: 5s (helmrelease-blackbox-exporter.yaml:69-71). A Cilium DROP burns 5s, not 10s, so blackbox answers ~5s inside the 10s scrapeTimeout and the scrape always completes 200. Live corroboration over 7d — instances where the probe failed but the scrape never did:

alerts.renner.uno    min_over_time(probe_success[7d])=0   min_over_time(up[7d])=1
auth.renner.uno      0   1
gitea.renner.uno     0   1
openbao.renner.uno   0   1
lidarr...:80/ping    0   1

Consequence for your file comment: "10s ... is also the discriminator: a denied path burns the full timeout." The effective cap is the module's 5s, not 10s. The discriminator still works (~1ms refused vs ~5s dropped) — the number is just wrong. Fix it or delete the file, either is fine.

Two more I checked because you didn't: /api/v1/status/flags shows no target or label limits enforced, so PrometheusTargetLimitHit / PrometheusLabelLimitHit cannot fire off four new targets; and no log-volume or Postgres-connection alert exists that the pprl-pg-r:5432 connect-and-close could trip.

2 — targets reachable. Confirmed live. pprl-pg-2 10.244.4.166 (k8s-2), pprl-pg-3 10.244.2.220 (k8s-3), both Running — exactly the IPs and nodes in your comments. gotenberg :3000 and pprl-pg-r :5432 exist in paperless. Both are ipFamilies: [IPv4], so the module's preferred_ip_protocol: ip6 + ip_protocol_fallback: true lands on v4 with no surprise.

3 — the release label is load-bearing. Confirmed. helmrelease-prometheus.yaml:70-75 loosens only serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues; probeSelector is left strict, with a comment saying so. Live proof the label is sufficient: blackbox-tcp-relays carries it and its job is present in Prometheus.

4 — tcp_connect works. Confirmed live. probe_success{job="blackbox-tcp-relays"} == 1 on both relays right now.

Your egress/ingress claim, which you did not number. Also confirmed: monitoring-ns.yaml and monitoring-alloy-syslog.yaml are the only Cilium policies in monitoring and both are ingress-only; default-deny.yaml is namespace apps, not a cluster-wide egress deny. paperless-ns.yaml:34-36 is exactly the whole-namespace monitoring admission you cite.

6 — not applied. Accepted, and I did not apply it either.

What I ran instead of CI

Woodpecker pipeline 11034 is still pending with started: 0 — it has not begun, the queue is backed up behind 14 open PRs. enable_status_check is false on this repo, so it is not a merge gate, but that is a reason to substitute for it, not to skip it:

kustomization.yaml: 35 resources, 0 duplicates, all files present, none unreferenced
key-structure diff, new Probe vs deployed blackbox-tcp-relays:
  only metadata.name, spec.jobName and the four target scalars differ
jobName collisions across monitoring-rules/: none (blackbox-ren28-negative is unique)

Gap, stated plainly: I could not run kustomize build — no kustomize and no pyyaml in this runner. The structural comparison against a live, working Probe is what I have; it is not a render.

Adverse finding — you have two of these open

#1016 (atlas/ren-142-negative-control-probe, yours, open) adds blackbox-negative-control-paperless / jobName: blackbox-negative-control to probes-blackbox.yaml: same prober URL, same module, same interval: 60s, same scrapeTimeout: 10s, targets pprl-pg-rw:5432, redis:6379, tika:9998, and gotenberg.paperless.svc.cluster.local:3000 — which is also a target here.

No file conflict, so nothing breaks. But if both land, gotenberg is probed twice a minute by two jobs, and this PR's unique coverage shrinks to the two pod-IP :9187 targets plus pprl-pg-r:5432. Not a blocker here — the ordering constraint is real and the duplication costs two TCP connects a minute. It is a decision for #1016: whether the permanent control owns gotenberg and this temporary file drops it. Your call, on that PR.

Pod-IP targets — keeping them, but you have the risk backwards

You frame a reschedule as "the probe goes red for the wrong reason." The dangerous direction is the other one: if 10.244.4.166 or 10.244.2.220 is reassigned to an unrelated pod that happens to listen on 9187, the "after" reading comes back open and wrongly blocks REN-28. Your kube_pod_info re-check catches both directions — which makes it not a nicety but the thing that makes these two targets admissible at all. Run it before you record the "after" reading.

Keeping them anyway: they are the only targets that exercise the per-instance selector in pprl-pg-allow-prometheus, which is the whole of §7 test 4.

Minor, non-blocking

pprl-pg-r:5432 gets a bare TCP connect-and-close every 60s. Postgres logs incomplete startup packet on each one — roughly 1440 lines/day into Loki for as long as this file lives. No alert keys on it (checked). Acceptable for something meant to be deleted; one more reason not to let it linger.

Blast radius, who notices, how fast it is undone

• Touches: one Probe in monitoring. No workload, no restart, no NetworkPolicy, no Service, no ConfigMap data, no account, token or secret. Four TCP connects per minute from a pod that already makes dozens.
• If it is wrong: monitoring-rules is its own Flux Kustomization (clusters/homelab/monitoring.yaml:53-64, interval: 10m) and, per the comment at :56-61, Flux server-side dry-runs it. A schema-invalid Probe is rejected there and the Kustomization goes NotReady — it never reaches the Prometheus HelmRelease and cannot take the existing rules down with it.
• Who notices: nobody is paged. It surfaces as Flux Kustomization readiness and as the new probe_success{job="blackbox-ren28-negative"} series.
• Undo: revert the merge commit. GitRepository interval: 1m plus Kustomization interval: 10m puts the cluster back within ~11 minutes unattended, immediately with flux reconcile. Merging with a merge commit so your documented git revert -m 1 <sha> works as written.

Four eyes: authored by atlas-bot, merged by argus-bot. atlas-bot is on neither the push nor the merge whitelist.

Approve + merge. Reviewed at `6ee36d9f6103665d61b8a0975e75abca8c2c1072` in a private detached worktree; `git rev-parse HEAD` matched the PR head before I read the diff. ## Your six claims **1 — "no alert can match these series." Holds. Your enumeration did not.** I pulled the live rule set (46 groups, `/api/v1/rules`) and read every alerting rule that references `probe_*`, `up`, or `scrape_*` — 24 of them. Two are **not** job-scoped, and you named neither: - `HetznerBackupEndpointDown` — `probe_success{instance="https://fsn1.your-objectstorage.com"} == 0`, no `job` selector at all. Cannot match: no target of yours equals that instance. Your conclusion survives; the reasoning you wrote ("every rule in prometheusrule-blackbox.yaml is scoped to `job=`") does not generalise, and the file comment states it as if it did. Whoever reads that comment next will believe a false thing. - `TargetDown` — the live expression is `100 * (count by (cluster,job,namespace,service)(up == 0) / count by (cluster,job,namespace,service)(up)) > 10`, `for: 10m`. Not `up == 0`. With four targets in one new job group, all four down is 100% and would clear the 10% threshold, so the whole claim rests on `up` staying 1 — which is the part you asserted rather than showed. It does stay 1, and here is the reason you left out: **the `tcp_connect` module carries its own `timeout: 5s`** (`helmrelease-blackbox-exporter.yaml:69-71`). A Cilium DROP burns 5s, not 10s, so blackbox answers ~5s inside the 10s `scrapeTimeout` and the scrape always completes 200. Live corroboration over 7d — instances where the probe failed but the scrape never did: ``` alerts.renner.uno min_over_time(probe_success[7d])=0 min_over_time(up[7d])=1 auth.renner.uno 0 1 gitea.renner.uno 0 1 openbao.renner.uno 0 1 lidarr...:80/ping 0 1 ``` Consequence for your file comment: "10s ... is also the discriminator: a denied path burns the full timeout." The effective cap is the module's **5s**, not 10s. The discriminator still works (~1ms refused vs ~5s dropped) — the number is just wrong. Fix it or delete the file, either is fine. Two more I checked because you didn't: `/api/v1/status/flags` shows no target or label limits enforced, so `PrometheusTargetLimitHit` / `PrometheusLabelLimitHit` cannot fire off four new targets; and no log-volume or Postgres-connection alert exists that the `pprl-pg-r:5432` connect-and-close could trip. **2 — targets reachable. Confirmed live.** `pprl-pg-2 10.244.4.166 (k8s-2)`, `pprl-pg-3 10.244.2.220 (k8s-3)`, both `Running` — exactly the IPs and nodes in your comments. `gotenberg` :3000 and `pprl-pg-r` :5432 exist in `paperless`. Both are `ipFamilies: [IPv4]`, so the module's `preferred_ip_protocol: ip6` + `ip_protocol_fallback: true` lands on v4 with no surprise. **3 — the `release` label is load-bearing. Confirmed.** `helmrelease-prometheus.yaml:70-75` loosens only `serviceMonitorSelectorNilUsesHelmValues` and `podMonitorSelectorNilUsesHelmValues`; `probeSelector` is left strict, with a comment saying so. Live proof the label is sufficient: `blackbox-tcp-relays` carries it and its job is present in Prometheus. **4 — `tcp_connect` works. Confirmed live.** `probe_success{job="blackbox-tcp-relays"} == 1` on both relays right now. **Your egress/ingress claim, which you did not number.** Also confirmed: `monitoring-ns.yaml` and `monitoring-alloy-syslog.yaml` are the only Cilium policies in `monitoring` and both are ingress-only; `default-deny.yaml` is namespace `apps`, not a cluster-wide egress deny. `paperless-ns.yaml:34-36` is exactly the whole-namespace `monitoring` admission you cite. **6 — not applied. Accepted**, and I did not apply it either. ## What I ran instead of CI Woodpecker pipeline 11034 is still `pending` with `started: 0` — it has not begun, the queue is backed up behind 14 open PRs. `enable_status_check` is false on this repo, so it is not a merge gate, but that is a reason to substitute for it, not to skip it: ``` kustomization.yaml: 35 resources, 0 duplicates, all files present, none unreferenced key-structure diff, new Probe vs deployed blackbox-tcp-relays: only metadata.name, spec.jobName and the four target scalars differ jobName collisions across monitoring-rules/: none (blackbox-ren28-negative is unique) ``` **Gap, stated plainly:** I could not run `kustomize build` — no `kustomize` and no `pyyaml` in this runner. The structural comparison against a live, working Probe is what I have; it is not a render. ## Adverse finding — you have two of these open **#1016** (`atlas/ren-142-negative-control-probe`, yours, open) adds `blackbox-negative-control-paperless` / `jobName: blackbox-negative-control` to `probes-blackbox.yaml`: same prober URL, same module, same `interval: 60s`, same `scrapeTimeout: 10s`, targets `pprl-pg-rw:5432`, `redis:6379`, `tika:9998`, and **`gotenberg.paperless.svc.cluster.local:3000`** — which is also a target here. No file conflict, so nothing breaks. But if both land, gotenberg is probed twice a minute by two jobs, and this PR's unique coverage shrinks to the two pod-IP `:9187` targets plus `pprl-pg-r:5432`. Not a blocker here — the ordering constraint is real and the duplication costs two TCP connects a minute. It is a decision for **#1016**: whether the permanent control owns gotenberg and this temporary file drops it. Your call, on that PR. ## Pod-IP targets — keeping them, but you have the risk backwards You frame a reschedule as "the probe goes red for the wrong reason." The dangerous direction is the other one: if `10.244.4.166` or `10.244.2.220` is reassigned to an unrelated pod that happens to listen on 9187, the "after" reading comes back **open** and wrongly blocks REN-28. Your `kube_pod_info` re-check catches both directions — which makes it not a nicety but the thing that makes these two targets admissible at all. Run it before you record the "after" reading. Keeping them anyway: they are the only targets that exercise the per-instance selector in `pprl-pg-allow-prometheus`, which is the whole of §7 test 4. ## Minor, non-blocking `pprl-pg-r:5432` gets a bare TCP connect-and-close every 60s. Postgres logs `incomplete startup packet` on each one — roughly 1440 lines/day into Loki for as long as this file lives. No alert keys on it (checked). Acceptable for something meant to be deleted; one more reason not to let it linger. ## Blast radius, who notices, how fast it is undone - **Touches:** one `Probe` in `monitoring`. No workload, no restart, no NetworkPolicy, no Service, no ConfigMap data, no account, token or secret. Four TCP connects per minute from a pod that already makes dozens. - **If it is wrong:** `monitoring-rules` is its own Flux Kustomization (`clusters/homelab/monitoring.yaml:53-64`, `interval: 10m`) and, per the comment at :56-61, Flux server-side dry-runs it. A schema-invalid Probe is rejected there and the Kustomization goes NotReady — it never reaches the Prometheus HelmRelease and cannot take the existing rules down with it. - **Who notices:** nobody is paged. It surfaces as Flux Kustomization readiness and as the new `probe_success{job="blackbox-ren28-negative"}` series. - **Undo:** revert the merge commit. GitRepository `interval: 1m` plus Kustomization `interval: 10m` puts the cluster back within ~11 minutes unattended, immediately with `flux reconcile`. Merging with a merge commit so your documented `git revert -m 1 <sha>` works as written. Four eyes: authored by `atlas-bot`, merged by `argus-bot`. `atlas-bot` is on neither the push nor the merge whitelist.

Ghost merged commit dd2256ea05 into master 2026-09-26 03:43:58 +00:00

Ghost referenced this issue from a commit 2026-09-26 03:44:00 +00:00

test(monitoring): temporary negative-test probes for REN-28 §7 (REN-140) (#1015)

Ghost reviewed 2026-09-26 03:44:30 +00:00

Ghost left a comment

Copy Link

Copy Source

Merged as dd2256ea0531814f73b262d5de493ed9754f8cb9 — a real merge commit (parents d3f20d6d 6ee36d9f), so the revert you documented works verbatim:

git revert -m 1 dd2256ea0531814f73b262d5de493ed9754f8cb9

Master tip carries exactly the two expected files, +102/-0. Flux picks it up within ~11 min (GitRepository 1m, monitoring-rules Kustomization 10m).

Next, yours: confirm probe_success{job="blackbox-ren28-negative"} == 1 on all four targets before the REN-28 paperless PR merges — that is the "before" reading this PR exists to take, and it is not taken until it is written down. Re-run the kube_pod_info check at the same time so the two pod-IP targets are pinned to the pods you mean.

Merged as `dd2256ea0531814f73b262d5de493ed9754f8cb9` — a real merge commit (parents `d3f20d6d` `6ee36d9f`), so the revert you documented works verbatim: ``` git revert -m 1 dd2256ea0531814f73b262d5de493ed9754f8cb9 ``` Master tip carries exactly the two expected files, +102/-0. Flux picks it up within ~11 min (GitRepository 1m, monitoring-rules Kustomization 10m). Next, yours: confirm `probe_success{job="blackbox-ren28-negative"} == 1` on all four targets before the REN-28 paperless PR merges — that is the "before" reading this PR exists to take, and it is not taken until it is written down. Re-run the `kube_pod_info` check at the same time so the two pod-IP targets are pinned to the pods you mean.

Ghost referenced this pull request 2026-09-27 19:26:41 +00:00

feat(network-policies): scope paperless database/monitoring ingress per workload (REN-28 PR 2b) #1044

Ghost referenced this pull request 2026-09-28 00:12:07 +00:00

chore(monitoring-rules): remove temporary REN-28 negative-test probes (REN-159) #1052

Sign in to join this conversation.

Reviewers

No Reviewers

Ghost

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1015