TargetDown for blackbox-tcp-relays is not a netcup firewall or a dead
relay container — it is a broken monitoring config. The blackbox-tcp-relays
Probe added in #415 asks the blackbox-exporter for module=tcp_connect, but
that module is not configured on the exporter, so every scrape gets an instant 400 Bad Request and the target is marked down before a TCP connection is ever
attempted. The HelmRelease carried a comment asserting the chart's stock modules
"cover http_2xx, http_post_2xx, tcp_connect, icmp, dns" — that premise is wrong,
and it is why the Probe was merged referencing a module that does not exist. This
PR declares tcp_connect in the exporter's config.modules and corrects the
misleading comment so the next Probe author doesn't repeat the mistake.
Changes
kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml — add a tcp_connect module (prober: tcp, timeout: 5s) so the existing blackbox-tcp-relays Probe resolves. Address family is pinned explicitly to preferred_ip_protocol: ip6 + ip_protocol_fallback: true (blackbox's own tcp
default, written out so an upstream default flip can't silently break it),
because the two relay members are mixed-family — see Verification. Also replace
the incorrect "default modules cover …" comment with what the chart actually
ships (only http_2xx) and what the failure mode looks like.
Verification
Everything below was observed live this run via ./.ci-diag/kubectl.
The alert is real and current — /api/v1/query?query=up{job="blackbox-tcp-relays"}
returns up=0 for both proxy-1.renner.uno:33080 and proxy-2.renner.uno:33080.
The root cause — Prometheus target lastError (via the targets API, scrapePool=probe/monitoring/blackbox-tcp-relays), both targets:
"lastError": "server returned HTTP status 400 Bad Request",
"lastScrapeDuration": 0.001230785,
A 1.2 ms failure is an immediate rejection by the exporter. A filtered/dropped
SYN — the issue's hypothesis — would have produced a ~10 s scrape hitting scrapeTimeout, not a sub-millisecond 400.
Direct confirmation the module is missing — kubectl get configmap -n monitoring blackbox-exporter-prometheus-blackbox-exporter -o jsonpath='{.data}'
renders exactly two modules: http_2xx and http_2xx_wan_edge. There is no tcp_connect. (This also proves Helm is deep-merging correctly — the chart's http_2xx survives alongside the repo's addition; the chart simply never
shipped tcp_connect.)
Both relays are alive and WAN-reachable, so hypothesis 2 in the issue is
refuted: probe_success{instance=~"http://proxy-.\.renner\.uno:9000/health"}
returns 1 for both members.
Mixed address families confirmed (this is why the module must not pin a
single family): probe_ip_protocol on those same two health probes returns 6 for proxy-1 and 4 for proxy-2 — proxy-1 is v6-only, matching the
inline note at kubernetes/apps/monitoring-rules/probes-blackbox.yaml:88.
After Flux reconciles, the HelmRelease re-renders the ConfigMap and the
exporter reloads. To confirm: re-run the targets query and expect lastError to
be empty with health: "up", and probe_success{job="blackbox-tcp-relays"} — which
currently returns no data at all — to start producing samples. TargetDown
should resolve within a scrape interval or two.
Notes
The issue's diagnosis was wrong, and its own evidence contained the tell it
didn't follow up on: it noted probe_success returns "no data at all" and read
that as a scrape timeout. Absent probe_* metrics is the signature of the
exporter rejecting the request outright — on a timeout blackbox still returns probe_success 0. One targets-API lastError read settles it. No netcup
provider-firewall change and no SSH to proxy-1/proxy-2 is needed.
This fix restores the signal; it does not by itself prove the port is open.
Once the probe actually runs, probe_success becomes meaningful for the first
time. If :33080 really is blocked or unbound, probe_success will be 0 and NetbirdRelayMemberDown (prometheusrule-netbird-availability.yaml:59) will
fire after 15m — and that would be a genuine relay/firewall problem worth the
SSH investigation the issue proposed. Until now that question was unanswerable,
because the probe never ran. Reviewers should check probe_success after
reconcile rather than assuming green.
No threshold was raised and no rule was silenced; the alert was correctly
reporting a down target, the target was just down for a config reason.
I audited the other two Probes in probes-blackbox.yaml for the same class of
bug: blackbox-http uses http_2xx (chart default, present) and blackbox-edge-wan uses http_2xx_wan_edge (defined in this HelmRelease). Both
are fine; only tcp_connect was dangling.
Nothing in the issue asked me to touch CI config, credentials, or anything
outside the edit scope.
## What this PR does
`TargetDown` for `blackbox-tcp-relays` is **not** a netcup firewall or a dead
relay container — it is a broken monitoring config. The `blackbox-tcp-relays`
Probe added in #415 asks the blackbox-exporter for `module=tcp_connect`, but
that module is not configured on the exporter, so every scrape gets an instant
`400 Bad Request` and the target is marked down before a TCP connection is ever
attempted. The HelmRelease carried a comment asserting the chart's stock modules
"cover http_2xx, http_post_2xx, tcp_connect, icmp, dns" — that premise is wrong,
and it is why the Probe was merged referencing a module that does not exist. This
PR declares `tcp_connect` in the exporter's `config.modules` and corrects the
misleading comment so the next Probe author doesn't repeat the mistake.
## Changes
- `kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml` — add a
`tcp_connect` module (`prober: tcp`, `timeout: 5s`) so the existing
`blackbox-tcp-relays` Probe resolves. Address family is pinned explicitly to
`preferred_ip_protocol: ip6` + `ip_protocol_fallback: true` (blackbox's own tcp
default, written out so an upstream default flip can't silently break it),
because the two relay members are mixed-family — see Verification. Also replace
the incorrect "default modules cover …" comment with what the chart actually
ships (only `http_2xx`) and what the failure mode looks like.
## Verification
Everything below was observed live this run via `./.ci-diag/kubectl`.
1. **The alert is real and current** — `/api/v1/query?query=up{job="blackbox-tcp-relays"}`
returns `up=0` for both `proxy-1.renner.uno:33080` and `proxy-2.renner.uno:33080`.
2. **The root cause — Prometheus target `lastError`** (via the targets API,
`scrapePool=probe/monitoring/blackbox-tcp-relays`), both targets:
```
"lastError": "server returned HTTP status 400 Bad Request",
"lastScrapeDuration": 0.001230785,
```
A 1.2 ms failure is an immediate rejection by the exporter. A filtered/dropped
SYN — the issue's hypothesis — would have produced a ~10 s scrape hitting
`scrapeTimeout`, not a sub-millisecond 400.
3. **Direct confirmation the module is missing** —
`kubectl get configmap -n monitoring blackbox-exporter-prometheus-blackbox-exporter -o jsonpath='{.data}'`
renders exactly two modules: `http_2xx` and `http_2xx_wan_edge`. There is no
`tcp_connect`. (This also proves Helm *is* deep-merging correctly — the chart's
`http_2xx` survives alongside the repo's addition; the chart simply never
shipped `tcp_connect`.)
4. **Both relays are alive and WAN-reachable**, so hypothesis 2 in the issue is
refuted: `probe_success{instance=~"http://proxy-.\.renner\.uno:9000/health"}`
returns `1` for both members.
5. **Mixed address families confirmed** (this is why the module must not pin a
single family): `probe_ip_protocol` on those same two health probes returns
`6` for proxy-1 and `4` for proxy-2 — proxy-1 is v6-only, matching the
inline note at `kubernetes/apps/monitoring-rules/probes-blackbox.yaml:88`.
**After Flux reconciles**, the HelmRelease re-renders the ConfigMap and the
exporter reloads. To confirm: re-run the targets query and expect `lastError` to
be empty with `health: "up"`, and `probe_success{job="blackbox-tcp-relays"}` — which
currently returns *no data at all* — to start producing samples. `TargetDown`
should resolve within a scrape interval or two.
## Notes
- **The issue's diagnosis was wrong**, and its own evidence contained the tell it
didn't follow up on: it noted `probe_success` returns "no data at all" and read
that as a scrape timeout. Absent `probe_*` metrics is the signature of the
exporter rejecting the request outright — on a timeout blackbox still returns
`probe_success 0`. One targets-API `lastError` read settles it. No netcup
provider-firewall change and no SSH to proxy-1/proxy-2 is needed.
- **This fix restores the signal; it does not by itself prove the port is open.**
Once the probe actually runs, `probe_success` becomes meaningful for the first
time. If `:33080` really is blocked or unbound, `probe_success` will be `0` and
`NetbirdRelayMemberDown` (`prometheusrule-netbird-availability.yaml:59`) will
fire after 15m — and *that* would be a genuine relay/firewall problem worth the
SSH investigation the issue proposed. Until now that question was unanswerable,
because the probe never ran. Reviewers should check `probe_success` after
reconcile rather than assuming green.
- No threshold was raised and no rule was silenced; the alert was correctly
reporting a down target, the target was just down for a config reason.
- I audited the other two Probes in `probes-blackbox.yaml` for the same class of
bug: `blackbox-http` uses `http_2xx` (chart default, present) and
`blackbox-edge-wan` uses `http_2xx_wan_edge` (defined in this HelmRelease). Both
are fine; only `tcp_connect` was dangling.
- Nothing in the issue asked me to touch CI config, credentials, or anything
outside the edit scope.
Closes #419
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 4448a7c96db67de2262c783764ed61f67413101c.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR fixes the TargetDown alert for blackbox-tcp-relays by adding the missing tcp_connect module to the blackbox-exporter HelmRelease configuration. The module was referenced by a Probe but never defined, causing 400 Bad Request errors on every scrape. The PR includes an accurate correction to the misleading comment about which modules the chart provides.
🟢 Nits
The comment at helmrelease-blackbox-exporter.yaml:31-38 is thorough and well-justified. The distinction between chart-provided modules and example-only modules (in upstream's example config) is now clear.
Tests
This is infrastructure-as-code configuration for a Kubernetes HelmRelease. The PR author has provided extensive live verification:
Direct Prometheus evidence — queried the targets API and confirmed lastError: "server returned HTTP status 400 Bad Request" with 1.2ms scrape duration (immediate rejection, not a timeout).
ConfigMap inspection — verified the module is actually missing: kubectl get configmap ... blackbox-exporter-prometheus-blackbox-exporter shows only http_2xx and http_2xx_wan_edge.
Topology confirmation — live probe inspection confirms the mixed address-family topology (probe_ip_protocol is 6 for proxy-1, 4 for proxy-2), justifying the ip6-with-fallback configuration.
Audit of related config — verified that other Probes (blackbox-http uses http_2xx, blackbox-edge-wan uses http_2xx_wan_edge) reference modules that actually exist.
The fix is validated and ready.
Recommendation
Looks reasonable — no blocking issues; the PR is ready to merge.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `4448a7c96db67de2262c783764ed61f67413101c`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR fixes the `TargetDown` alert for `blackbox-tcp-relays` by adding the missing `tcp_connect` module to the blackbox-exporter HelmRelease configuration. The module was referenced by a Probe but never defined, causing 400 Bad Request errors on every scrape. The PR includes an accurate correction to the misleading comment about which modules the chart provides.
## 🟢 Nits
- The comment at `helmrelease-blackbox-exporter.yaml:31-38` is thorough and well-justified. The distinction between chart-provided modules and example-only modules (in upstream's example config) is now clear.
## Tests
This is infrastructure-as-code configuration for a Kubernetes HelmRelease. The PR author has provided extensive live verification:
1. **Direct Prometheus evidence** — queried the targets API and confirmed `lastError: "server returned HTTP status 400 Bad Request"` with 1.2ms scrape duration (immediate rejection, not a timeout).
2. **ConfigMap inspection** — verified the module is actually missing: `kubectl get configmap ... blackbox-exporter-prometheus-blackbox-exporter` shows only `http_2xx` and `http_2xx_wan_edge`.
3. **Topology confirmation** — live probe inspection confirms the mixed address-family topology (`probe_ip_protocol` is 6 for proxy-1, 4 for proxy-2), justifying the `ip6`-with-fallback configuration.
4. **Audit of related config** — verified that other Probes (`blackbox-http` uses `http_2xx`, `blackbox-edge-wan` uses `http_2xx_wan_edge`) reference modules that actually exist.
The fix is validated and ready.
## Recommendation
Looks reasonable — no blocking issues; the PR is ready to merge.
<!-- pr-review:sha=4448a7c96db67de2262c783764ed61f67413101c -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
TargetDownforblackbox-tcp-relaysis not a netcup firewall or a deadrelay container — it is a broken monitoring config. The
blackbox-tcp-relaysProbe added in #415 asks the blackbox-exporter for
module=tcp_connect, butthat module is not configured on the exporter, so every scrape gets an instant
400 Bad Requestand the target is marked down before a TCP connection is everattempted. The HelmRelease carried a comment asserting the chart's stock modules
"cover http_2xx, http_post_2xx, tcp_connect, icmp, dns" — that premise is wrong,
and it is why the Probe was merged referencing a module that does not exist. This
PR declares
tcp_connectin the exporter'sconfig.modulesand corrects themisleading comment so the next Probe author doesn't repeat the mistake.
Changes
kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml— add atcp_connectmodule (prober: tcp,timeout: 5s) so the existingblackbox-tcp-relaysProbe resolves. Address family is pinned explicitly topreferred_ip_protocol: ip6+ip_protocol_fallback: true(blackbox's own tcpdefault, written out so an upstream default flip can't silently break it),
because the two relay members are mixed-family — see Verification. Also replace
the incorrect "default modules cover …" comment with what the chart actually
ships (only
http_2xx) and what the failure mode looks like.Verification
Everything below was observed live this run via
./.ci-diag/kubectl.The alert is real and current —
/api/v1/query?query=up{job="blackbox-tcp-relays"}returns
up=0for bothproxy-1.renner.uno:33080andproxy-2.renner.uno:33080.The root cause — Prometheus target
lastError(via the targets API,scrapePool=probe/monitoring/blackbox-tcp-relays), both targets:A 1.2 ms failure is an immediate rejection by the exporter. A filtered/dropped
SYN — the issue's hypothesis — would have produced a ~10 s scrape hitting
scrapeTimeout, not a sub-millisecond 400.Direct confirmation the module is missing —
kubectl get configmap -n monitoring blackbox-exporter-prometheus-blackbox-exporter -o jsonpath='{.data}'renders exactly two modules:
http_2xxandhttp_2xx_wan_edge. There is notcp_connect. (This also proves Helm is deep-merging correctly — the chart'shttp_2xxsurvives alongside the repo's addition; the chart simply nevershipped
tcp_connect.)Both relays are alive and WAN-reachable, so hypothesis 2 in the issue is
refuted:
probe_success{instance=~"http://proxy-.\.renner\.uno:9000/health"}returns
1for both members.Mixed address families confirmed (this is why the module must not pin a
single family):
probe_ip_protocolon those same two health probes returns6for proxy-1 and4for proxy-2 — proxy-1 is v6-only, matching theinline note at
kubernetes/apps/monitoring-rules/probes-blackbox.yaml:88.After Flux reconciles, the HelmRelease re-renders the ConfigMap and the
exporter reloads. To confirm: re-run the targets query and expect
lastErrortobe empty with
health: "up", andprobe_success{job="blackbox-tcp-relays"}— whichcurrently returns no data at all — to start producing samples.
TargetDownshould resolve within a scrape interval or two.
Notes
didn't follow up on: it noted
probe_successreturns "no data at all" and readthat as a scrape timeout. Absent
probe_*metrics is the signature of theexporter rejecting the request outright — on a timeout blackbox still returns
probe_success 0. One targets-APIlastErrorread settles it. No netcupprovider-firewall change and no SSH to proxy-1/proxy-2 is needed.
Once the probe actually runs,
probe_successbecomes meaningful for the firsttime. If
:33080really is blocked or unbound,probe_successwill be0andNetbirdRelayMemberDown(prometheusrule-netbird-availability.yaml:59) willfire after 15m — and that would be a genuine relay/firewall problem worth the
SSH investigation the issue proposed. Until now that question was unanswerable,
because the probe never ran. Reviewers should check
probe_successafterreconcile rather than assuming green.
reporting a down target, the target was just down for a config reason.
probes-blackbox.yamlfor the same class ofbug:
blackbox-httpuseshttp_2xx(chart default, present) andblackbox-edge-wanuseshttp_2xx_wan_edge(defined in this HelmRelease). Bothare fine; only
tcp_connectwas dangling.outside the edit scope.
Closes #419
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit4448a7c96db67de2262c783764ed61f67413101c.Summary
This PR fixes the
TargetDownalert forblackbox-tcp-relaysby adding the missingtcp_connectmodule to the blackbox-exporter HelmRelease configuration. The module was referenced by a Probe but never defined, causing 400 Bad Request errors on every scrape. The PR includes an accurate correction to the misleading comment about which modules the chart provides.🟢 Nits
helmrelease-blackbox-exporter.yaml:31-38is thorough and well-justified. The distinction between chart-provided modules and example-only modules (in upstream's example config) is now clear.Tests
This is infrastructure-as-code configuration for a Kubernetes HelmRelease. The PR author has provided extensive live verification:
lastError: "server returned HTTP status 400 Bad Request"with 1.2ms scrape duration (immediate rejection, not a timeout).kubectl get configmap ... blackbox-exporter-prometheus-blackbox-exportershows onlyhttp_2xxandhttp_2xx_wan_edge.probe_ip_protocolis 6 for proxy-1, 4 for proxy-2), justifying theip6-with-fallback configuration.blackbox-httpuseshttp_2xx,blackbox-edge-wanuseshttp_2xx_wan_edge) reference modules that actually exist.The fix is validated and ready.
Recommendation
Looks reasonable — no blocking issues; the PR is ready to merge.