konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] TargetDown (monitoring) #423

New Issue
Closed
opened 2026-07-26 09:05:18 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-07-26 09:05:18 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: TargetDown (severity warning, namespace monitoring, job blackbox-tcp-relays)
  • Type: blip — fired and already resolved by the time this ran (fingerprint 605d5e62eee78802)
  • Window (from live Prometheus history, up{job="blackbox-tcp-relays"}): condition began 2026-07-26 07:33:35 UTC, both instances recovered together at 08:53:35 UTC — roughly 80 minutes down.

Evidence

Both relay targets were down together, not just one:

up{job="blackbox-tcp-relays",instance="proxy-1.renner.uno:33080"} == 0   07:33:35–08:51:35 UTC
up{job="blackbox-tcp-relays",instance="proxy-2.renner.uno:33080"} == 0   07:33:35–08:51:35 UTC

(via kubectl get --raw .../api/v1/query_range?query=up{job="blackbox-tcp-relays"}). Both flip back to 1 at the same 08:53:35 sample.

Currently healthy — confirmed live via the Prometheus targets API: both proxy-1.renner.uno:33080 and proxy-2.renner.uno:33080 report "health":"up", "lastError":"" as of 09:00 UTC.

The recovery lines up with a blackbox-exporter pod replacement, not a probe fix:

  • helmrelease/blackbox-exporter history shows release v7 deployed at 07:31:29Z (configDigest 6f47ff72…) and v8 deployed at 08:51:25Z (configDigest bc2d74bf… — different digest, so the rendered config actually changed between the two).
  • ReplicaSet blackbox-exporter-prometheus-blackbox-exporter-54c69c9cd8 (the v7 pod, created ~07:31) is now scaled to 0; …-567f9bb8c7 (the v8 pod) took over at 08:51:25Z, matching the alert's recovery almost to the minute.
  • The old (v7) pod is gone (RS at 0/0/0), so its logs from the failure window are not recoverable now.

Ruled out as causes: CoreDNS pods (kube-system) show 0 restarts / 2d19h uptime through the whole window; all three nodes stayed Ready throughout; the monitoring namespace has no egress-restricting NetworkPolicy/CiliumNetworkPolicy (only two ingress-only Cilium policies exist cluster-wide).

Related but not a full explanation: issue #420 (NetbirdPublicEdgeMemberDown) is already open and documents proxy-1's reverse-proxy listener (port 443) failing starting at the same 07:33 UTC timestamp — but that issue explicitly confirms proxy-2 was healthy throughout on the probes it checked. Since this alert shows proxy-2's relay port also down for the same window, #420's proxy-1-only diagnosis does not by itself account for what happened here.

Root cause (unconfirmed — both are hypotheses)

  1. blackbox-exporter pod-level issue, isolated to the tcp_connect module, in Helm release v7 (07:31–08:51) — e.g. an IPv6-preference/fallback hiccup (kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml, module tcp_connect uses preferred_ip_protocol: ip6 + fallback) or a transient egress problem on that pod's node — incidentally fixed when the routine Flux reconcile replaced the pod. Can't be confirmed now; the old pod's logs are gone.
  2. A broader netcup-side/WAN event around 07:33 UTC affecting both relay hosts' :33080 listeners, coincident with but broader than the proxy-1-only symptom in #420.

Where to fix / what to do

Not clearly fixable in-repo from current evidence — this looks like either a transient exporter-pod issue or an external network/relay-host event, not a manifest bug. If TargetDown on blackbox-tcp-relays recurs:

  • Capture ./.ci-diag/kubectl -n monitoring logs deploy/blackbox-exporter-prometheus-blackbox-exporter --tail=50 and kubectl -n monitoring get pods -o wide while it is still failing (this run only caught it after the pod had already been replaced).
  • Cross-reference with #420 if proxy-1's port 443 is also down at the same time.
  • Relevant manifests: kubernetes/apps/monitoring-rules/probes-blackbox.yaml (the blackbox-tcp-relays Probe) and kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml (the tcp_connect module config).

Filed automatically by the alert-monitor pipeline (2026-07-26). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `TargetDown` (severity `warning`, namespace `monitoring`, job `blackbox-tcp-relays`) - **Type:** blip — fired and already resolved by the time this ran (fingerprint `605d5e62eee78802`) - **Window (from live Prometheus history, `up{job="blackbox-tcp-relays"}`):** condition began **2026-07-26 07:33:35 UTC**, both instances recovered together at **08:53:35 UTC** — roughly **80 minutes** down. ## Evidence **Both relay targets were down together, not just one:** ``` up{job="blackbox-tcp-relays",instance="proxy-1.renner.uno:33080"} == 0 07:33:35–08:51:35 UTC up{job="blackbox-tcp-relays",instance="proxy-2.renner.uno:33080"} == 0 07:33:35–08:51:35 UTC ``` (via `kubectl get --raw .../api/v1/query_range?query=up{job="blackbox-tcp-relays"}`). Both flip back to `1` at the same 08:53:35 sample. **Currently healthy** — confirmed live via the Prometheus targets API: both `proxy-1.renner.uno:33080` and `proxy-2.renner.uno:33080` report `"health":"up"`, `"lastError":""` as of 09:00 UTC. **The recovery lines up with a blackbox-exporter pod replacement, not a probe fix:** - `helmrelease/blackbox-exporter` history shows release v7 deployed at `07:31:29Z` (configDigest `6f47ff72…`) and v8 deployed at `08:51:25Z` (configDigest `bc2d74bf…` — different digest, so the rendered config actually changed between the two). - ReplicaSet `blackbox-exporter-prometheus-blackbox-exporter-54c69c9cd8` (the v7 pod, created ~07:31) is now scaled to 0; `…-567f9bb8c7` (the v8 pod) took over at `08:51:25Z`, matching the alert's recovery almost to the minute. - The old (v7) pod is gone (RS at 0/0/0), so its logs from the failure window are not recoverable now. **Ruled out as causes:** CoreDNS pods (`kube-system`) show 0 restarts / 2d19h uptime through the whole window; all three nodes stayed `Ready` throughout; the `monitoring` namespace has no egress-restricting NetworkPolicy/CiliumNetworkPolicy (only two ingress-only Cilium policies exist cluster-wide). **Related but not a full explanation:** issue #420 (`NetbirdPublicEdgeMemberDown`) is already open and documents proxy-1's reverse-proxy listener (port 443) failing starting at the *same* 07:33 UTC timestamp — but that issue explicitly confirms **proxy-2 was healthy** throughout on the probes it checked. Since this alert shows proxy-2's relay port also down for the same window, #420's proxy-1-only diagnosis does not by itself account for what happened here. ## Root cause (unconfirmed — both are hypotheses) 1. **blackbox-exporter pod-level issue**, isolated to the `tcp_connect` module, in Helm release v7 (07:31–08:51) — e.g. an IPv6-preference/fallback hiccup (`kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml`, module `tcp_connect` uses `preferred_ip_protocol: ip6` + fallback) or a transient egress problem on that pod's node — incidentally fixed when the routine Flux reconcile replaced the pod. Can't be confirmed now; the old pod's logs are gone. 2. **A broader netcup-side/WAN event** around 07:33 UTC affecting both relay hosts' `:33080` listeners, coincident with but broader than the proxy-1-only symptom in #420. ## Where to fix / what to do Not clearly fixable in-repo from current evidence — this looks like either a transient exporter-pod issue or an external network/relay-host event, not a manifest bug. If `TargetDown` on `blackbox-tcp-relays` recurs: - Capture `./.ci-diag/kubectl -n monitoring logs deploy/blackbox-exporter-prometheus-blackbox-exporter --tail=50` and `kubectl -n monitoring get pods -o wide` **while it is still failing** (this run only caught it after the pod had already been replaced). - Cross-reference with #420 if proxy-1's port 443 is also down at the same time. - Relevant manifests: `kubernetes/apps/monitoring-rules/probes-blackbox.yaml` (the `blackbox-tcp-relays` Probe) and `kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml` (the `tcp_connect` module config). --- <!-- alert-monitor:fingerprint=605d5e62eee78802 --> _Filed automatically by the alert-monitor pipeline (2026-07-26). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-07-26 10:46:26 +00:00
Member
Copy Link
Copy Source

Assessment: no repo change needed — already fixed by PR #421 at HEAD

TargetDown{job="blackbox-tcp-relays"} fired because the blackbox-tcp-relays
Probe was introduced referencing a tcp_connect module that was not declared
in the blackbox-exporter values at that moment. The exporter answers /probe for
an unknown module with 400 Bad Request, so Prometheus recorded up == 0 and
no probe_* series at all for both relay targets. The missing module was
added by the merge of PR #421 (fix for issue #419) at 08:49:25Z, which Flux
rolled out as HelmRelease revision 8 at 08:51:25Z — matching the recovery to
the sample. That commit (41676a8) is the parent of this branch, so the fix is
already in the repository. Making any edit here would be cosmetic.

Changes

None.

Verification (observed live this run)

  1. Chart identical across the two revisions — so this was a values change, not
    an upstream chart flip.
    ./.ci-diag/kubectl -n monitoring get helmrelease blackbox-exporter -o jsonpath=...

    rev=8 chart=11.15.1 digest=sha256:bc2d74bf… status=deployed   at=2026-07-26T08:51:25Z
rev=7 chart=11.15.1 digest=sha256:6f47ff72… status=superseded at=2026-07-26T07:31:29Z

    Same chartVersion, different configDigest.

  2. The failure signature is "module missing", not "TCP connect failed".
    query_range on probe_success{job="blackbox-tcp-relays"} over
    07:20Z–09:10Z (step 300s) returns samples only from 08:55:00Z onward —
    the series is entirely absent for the whole reported outage window. A real
    TCP failure would have produced probe_success == 0, not no series. Absent
    probe_* with up == 0 is exactly the 400-Bad-Request behaviour documented
    in the comment at kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml:31-38.

  3. Not a retention artifact. count(up{job="blackbox-http"}) over
    2026-07-25T00:00Z–2026-07-26T10:00Z (step 3h) returns a value at every
    step, so Prometheus does hold data across the window where
    blackbox-tcp-relays has none.

  4. The module is present live and matches the repo.
    ./.ci-diag/kubectl -n monitoring get configmap blackbox-exporter-prometheus-blackbox-exporter -o jsonpath='{.data.blackbox\.yaml}'
    renders http_2xx, http_2xx_wan_edge and tcp_connect
    (preferred_ip_protocol: ip6, ip_protocol_fallback: true) — byte-for-byte
    the intent of the file at HEAD.

  5. Both targets healthy now. Prometheus targets API, scrapePool
    probe/monitoring/blackbox-tcp-relays: proxy-1.renner.uno:33080 and
    proxy-2.renner.uno:33080 both "health":"up", "lastError":"", last
    scrapes 2026-07-26T10:44:22Z / 10:44:44Z.

  6. No other Probe references an undeclared module. Every module: key under
    kubernetes/ (probes-blackbox.yaml:14,80,116 → http_2xx,
    http_2xx_wan_edge, tcp_connect) is present in the rendered ConfigMap, so
    the same class of bug is not lurking on another probe.

For the reviewer: nothing to reconcile. TargetDown for this job should stay
resolved; confirm with the targets API command above, or
probe_success{job="blackbox-tcp-relays"} == 1.

Notes — where the issue's diagnosis was wrong

  • "roughly 80 minutes down" implies a regression from healthy — it wasn't.
    up{job="blackbox-tcp-relays"} has no samples at all before
    2026-07-26 07:33:35Z (checked back to 2026-07-25T00:00Z, with retention
    proven by point 3). The target was new at 07:31 and was born broken; it never
    had a good scrape until the fix. The alert was not a "blip" on a working probe.
  • Hypothesis 1 (IPv6-preference hiccup in the tcp_connect module) is refuted
    in mechanism    , though it was right to point at the exporter: an IP-family
    problem would still have emitted probe_success == 0. The complete absence of
    probe_* series rules that out.
  • Hypothesis 2 (broader netcup/WAN event hitting both relay hosts) is refuted.
    The failure was exporter-side, and independently: blackbox-edge-wan
    probe_success for proxy-2's v4 (45.157.177.26) and v6
    (2a03:4000:4b:c1e:…:8f85) targets was 1 at every 15-minute step from
    07:45Z to 10:00Z — proxy-2 was reachable over the WAN throughout, so
    nothing broad was happening at netcup.
  • The pod replacement was the fix, not a coincidence. The issue read the v7→v8
    ReplicaSet swap as a routine reconcile that incidentally healed things. Point 1
    shows the swap was caused by the config change from the PR #421 merge two
    minutes earlier.
  • Issue #420 is genuinely separate and still real. blackbox-edge-wan
    probe_success for proxy-1's v6 target (2a03:4000:28:f6e:…:9a29) was 0
    from 07:45Z through 08:30Z and 1 from 09:00Z — a real proxy-1 edge
    problem, but unrelated to this alert's cause, which was exporter-side and
    affected both relay targets identically. #423 can be closed as a duplicate of
    the #419/#421 incident; #420 should be judged on its own evidence.

Considered and rejected

  • Pinning the chart range. helmrelease-blackbox-exporter.yaml:11 uses a
    floating version: ">=9.0.0", which is a real supply-of-truth hazard worth
    raising separately — but it did not cause this (point 1: chart 11.15.1 on
    both revisions), and version pinning belongs to the update-audit process, not
    to an alert fix.
  • An absent(probe_success{...}) guard rule to catch "Probe references an
    undeclared module" earlier. It would have shortened this incident, but it is
    new alerting surface rather than a fix for the firing alert, and point 6 shows
    no probe is currently exposed. Worth a separate issue if the owner wants it.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment: no repo change needed — already fixed by PR #421 at HEAD `TargetDown{job="blackbox-tcp-relays"}` fired because the `blackbox-tcp-relays` Probe was introduced referencing a `tcp_connect` module that was **not declared** in the blackbox-exporter values at that moment. The exporter answers `/probe` for an unknown module with `400 Bad Request`, so Prometheus recorded `up == 0` and **no `probe_*` series at all** for both relay targets. The missing module was added by the merge of PR #421 (fix for issue #419) at `08:49:25Z`, which Flux rolled out as HelmRelease revision 8 at `08:51:25Z` — matching the recovery to the sample. That commit (`41676a8`) is the parent of this branch, so the fix is already in the repository. Making any edit here would be cosmetic. ## Changes None. ## Verification (observed live this run) 1. **Chart identical across the two revisions — so this was a values change, not an upstream chart flip.** `./.ci-diag/kubectl -n monitoring get helmrelease blackbox-exporter -o jsonpath=...` ``` rev=8 chart=11.15.1 digest=sha256:bc2d74bf… status=deployed at=2026-07-26T08:51:25Z rev=7 chart=11.15.1 digest=sha256:6f47ff72… status=superseded at=2026-07-26T07:31:29Z ``` Same `chartVersion`, different `configDigest`. 2. **The failure signature is "module missing", not "TCP connect failed".** `query_range` on `probe_success{job="blackbox-tcp-relays"}` over `07:20Z–09:10Z` (step 300s) returns samples **only** from `08:55:00Z` onward — the series is entirely *absent* for the whole reported outage window. A real TCP failure would have produced `probe_success == 0`, not no series. Absent `probe_*` with `up == 0` is exactly the 400-Bad-Request behaviour documented in the comment at `kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml:31-38`. 3. **Not a retention artifact.** `count(up{job="blackbox-http"})` over `2026-07-25T00:00Z–2026-07-26T10:00Z` (step 3h) returns a value at every step, so Prometheus does hold data across the window where `blackbox-tcp-relays` has none. 4. **The module is present live and matches the repo.** `./.ci-diag/kubectl -n monitoring get configmap blackbox-exporter-prometheus-blackbox-exporter -o jsonpath='{.data.blackbox\.yaml}'` renders `http_2xx`, `http_2xx_wan_edge` and `tcp_connect` (`preferred_ip_protocol: ip6`, `ip_protocol_fallback: true`) — byte-for-byte the intent of the file at HEAD. 5. **Both targets healthy now.** Prometheus targets API, scrapePool `probe/monitoring/blackbox-tcp-relays`: `proxy-1.renner.uno:33080` and `proxy-2.renner.uno:33080` both `"health":"up"`, `"lastError":""`, last scrapes `2026-07-26T10:44:22Z` / `10:44:44Z`. 6. **No other Probe references an undeclared module.** Every `module:` key under `kubernetes/` (`probes-blackbox.yaml:14,80,116` → `http_2xx`, `http_2xx_wan_edge`, `tcp_connect`) is present in the rendered ConfigMap, so the same class of bug is not lurking on another probe. **For the reviewer:** nothing to reconcile. `TargetDown` for this job should stay resolved; confirm with the targets API command above, or `probe_success{job="blackbox-tcp-relays"} == 1`. ## Notes — where the issue's diagnosis was wrong - **"roughly 80 minutes down" implies a regression from healthy — it wasn't.** `up{job="blackbox-tcp-relays"}` has **no samples at all** before `2026-07-26 07:33:35Z` (checked back to `2026-07-25T00:00Z`, with retention proven by point 3). The target was *new* at 07:31 and was born broken; it never had a good scrape until the fix. The alert was not a "blip" on a working probe. - **Hypothesis 1 (IPv6-preference hiccup in the `tcp_connect` module) is refuted in mechanism**, though it was right to point at the exporter: an IP-family problem would still have emitted `probe_success == 0`. The complete absence of `probe_*` series rules that out. - **Hypothesis 2 (broader netcup/WAN event hitting both relay hosts) is refuted.** The failure was exporter-side, and independently: `blackbox-edge-wan` `probe_success` for proxy-2's v4 (`45.157.177.26`) and v6 (`2a03:4000:4b:c1e:…:8f85`) targets was `1` at every 15-minute step from `07:45Z` to `10:00Z` — proxy-2 was reachable over the WAN throughout, so nothing broad was happening at netcup. - **The pod replacement was the fix, not a coincidence.** The issue read the v7→v8 ReplicaSet swap as a routine reconcile that incidentally healed things. Point 1 shows the swap was *caused by* the config change from the PR #421 merge two minutes earlier. - **Issue #420 is genuinely separate and still real.** `blackbox-edge-wan` `probe_success` for proxy-1's v6 target (`2a03:4000:28:f6e:…:9a29`) was `0` from `07:45Z` through `08:30Z` and `1` from `09:00Z` — a real proxy-1 edge problem, but unrelated to this alert's cause, which was exporter-side and affected both relay targets identically. #423 can be closed as a duplicate of the #419/#421 incident; #420 should be judged on its own evidence. ### Considered and rejected - **Pinning the chart range.** `helmrelease-blackbox-exporter.yaml:11` uses a floating `version: ">=9.0.0"`, which is a real supply-of-truth hazard worth raising separately — but it did **not** cause this (point 1: chart 11.15.1 on both revisions), and version pinning belongs to the update-audit process, not to an alert fix. - **An `absent(probe_success{...})` guard rule** to catch "Probe references an undeclared module" earlier. It would have shortened this incident, but it is new alerting surface rather than a fix for the firing alert, and point 6 shows no probe is currently exposed. Worth a separate issue if the owner wants it. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-07-27 19:22:51 +00:00
Member
Copy Link
Copy Source

Verified resolved during the 2026-07-27 issue round. Closing.

Both targets are currently up: up=1 and probe_success=1 for proxy-1.renner.uno:33080 and proxy-2.renner.uno:33080, probe_duration_seconds 0.030s / 0.023s, probe_ip_protocol=6, and min_over_time(up[24h])=1.

The root cause, more precisely than this issue framed it

v7 was not broken — it was incomplete, and the relays were never down.

The decisive discriminator: during the outage window up=0 while probe_success was absent entirely, not zero.

up:            proxy-1 zeros=40 (07:34:00Z .. 08:52:00Z), proxy-2 zeros=39
probe_success{job="blackbox-tcp-relays"} @1785053000  ->  []   (empty)

up=0 with no probe_* series at all means the scrape itself failed — the exporter answered /probe with HTTP 400. That is not a TCP-connect failure. And every up sample from the series' birth was 0, so the target was born down.

Why: PR #415 (8b1480f4, 06:57:01Z) added a Probe referencing module: tcp_connect, but that module was never declared. The chart ships exactly one module, http_2xx; tcp_connect exists only in upstream blackbox_exporter's example config, which this chart does not use. The v7 release secret confirms it:

sh.helm.release.v1.blackbox-exporter.v7 -> .config.modules = {http_2xx_wan_edge}              <- tcp_connect ABSENT
sh.helm.release.v1.blackbox-exporter.v8 -> .config.modules = {http_2xx_wan_edge, tcp_connect}

Fixed by PR #419 (4448a7c9, 08:44:03Z), which added the module; HelmRelease v8 landed 08:51:25Z — matching the recovery to the minute.

The relays were verifiably fine throughout. blackbox-http probes of proxy-1:9000/health and proxy-2:9000/health — same hosts, same WAN path, same exporter pod, different module — were probe_success=1 for every sample across 07:00–09:30Z. Only the tcp_connect probes 400'd.

This also supersedes the speculation in this issue that #420's proxy-1-only diagnosis couldn't account for proxy-2 also being down: both were down because neither could be probed at all, not because either host had a problem.

No repo change warranted

The tcp_connect module is sound: no query_response expect-string to regress, no TLS settings (correct — :33080 serves plaintext rel://), and preferred_ip_protocol: ip6 + ip_protocol_fallback: true is deliberate for a mixed v6-only/dual-stack pair. All three module: references in the repo resolve against the live ConfigMap — no dangling references remain.

Two things worth knowing (deliberately not filed as fixes here)

  1. This bug class is unguarded. remediation.retries: 3 cannot catch it — the config is valid YAML, the module is merely absent, and helm upgrade succeeded. There is no config-check init container, no helm test, and no manifest validation in CI. Structurally the Probe (monitoring-rules KS) and the module definition (monitoring KS) live in different Flux Kustomizations, so the cross-reference spans a reconcile boundary with no ordering guarantee. A Probe.module → declared-modules linter would close it — worth filing separately if wanted.
  2. NetbirdRelayMemberDown is structurally blind to this failure mode (prometheusrule-netbird-availability.yaml:54-67): its expr tests probe_success == 0, which cannot match an absent series. Generic TargetDown was the only thing that fired — and the pipeline worked end to end: alert → issue → fix in 73 minutes. Leaving it; adding absent() duplicates TargetDown's job.

Unrelated background noise, so it isn't misread later: four brief (1–3 min) dips after the fix (07-26 11:40Z; 07-27 01:22Z, 01:40Z, 06:30Z). At each instant all four proxy probes — both :33080 TCP and both :9000/health HTTP — read 0 together, i.e. a shared cluster-egress/WAN/DNS transient, not the relays. up stayed 1 and all are far under the 15m for:, so nothing fired.

Verified resolved during the 2026-07-27 issue round. Closing. **Both targets are currently up:** `up`=1 and `probe_success`=1 for `proxy-1.renner.uno:33080` and `proxy-2.renner.uno:33080`, `probe_duration_seconds` 0.030s / 0.023s, `probe_ip_protocol`=6, and `min_over_time(up[24h])`=1. ### The root cause, more precisely than this issue framed it v7 was not *broken* — it was **incomplete**, and the relays were never down. The decisive discriminator: during the outage window `up`=0 while `probe_success` was **absent entirely**, not zero. ``` up: proxy-1 zeros=40 (07:34:00Z .. 08:52:00Z), proxy-2 zeros=39 probe_success{job="blackbox-tcp-relays"} @1785053000 -> [] (empty) ``` `up`=0 with no `probe_*` series at all means the scrape itself failed — the exporter answered `/probe` with HTTP 400. That is not a TCP-connect failure. And every `up` sample from the series' birth was 0, so the target was **born down**. Why: PR #415 (`8b1480f4`, 06:57:01Z) added a Probe referencing `module: tcp_connect`, but that module was never declared. The chart ships exactly one module, `http_2xx`; `tcp_connect` exists only in upstream blackbox_exporter's *example* config, which this chart does not use. The v7 release secret confirms it: ``` sh.helm.release.v1.blackbox-exporter.v7 -> .config.modules = {http_2xx_wan_edge} <- tcp_connect ABSENT sh.helm.release.v1.blackbox-exporter.v8 -> .config.modules = {http_2xx_wan_edge, tcp_connect} ``` Fixed by PR #419 (`4448a7c9`, 08:44:03Z), which added the module; HelmRelease v8 landed 08:51:25Z — matching the recovery to the minute. **The relays were verifiably fine throughout.** `blackbox-http` probes of `proxy-1:9000/health` and `proxy-2:9000/health` — same hosts, same WAN path, same exporter pod, different module — were `probe_success=1` for every sample across 07:00–09:30Z. Only the `tcp_connect` probes 400'd. This also supersedes the speculation in this issue that #420's proxy-1-only diagnosis couldn't account for proxy-2 also being down: both were down because neither could be probed at all, not because either host had a problem. ### No repo change warranted The `tcp_connect` module is sound: no `query_response` expect-string to regress, no TLS settings (correct — `:33080` serves plaintext `rel://`), and `preferred_ip_protocol: ip6` + `ip_protocol_fallback: true` is deliberate for a mixed v6-only/dual-stack pair. All three `module:` references in the repo resolve against the live ConfigMap — no dangling references remain. ### Two things worth knowing (deliberately not filed as fixes here) 1. **This bug class is unguarded.** `remediation.retries: 3` cannot catch it — the config is valid YAML, the module is merely absent, and `helm upgrade` *succeeded*. There is no config-check init container, no `helm test`, and no manifest validation in CI. Structurally the Probe (`monitoring-rules` KS) and the module definition (`monitoring` KS) live in different Flux Kustomizations, so the cross-reference spans a reconcile boundary with no ordering guarantee. A `Probe.module` → declared-modules linter would close it — worth filing separately if wanted. 2. **`NetbirdRelayMemberDown` is structurally blind to this failure mode** (`prometheusrule-netbird-availability.yaml:54-67`): its expr tests `probe_success == 0`, which cannot match an *absent* series. Generic `TargetDown` was the only thing that fired — and the pipeline worked end to end: alert → issue → fix in 73 minutes. Leaving it; adding `absent()` duplicates TargetDown's job. **Unrelated background noise, so it isn't misread later:** four brief (1–3 min) dips after the fix (07-26 11:40Z; 07-27 01:22Z, 01:40Z, 06:30Z). At each instant all four proxy probes — both `:33080` TCP *and* both `:9000/health` HTTP — read 0 together, i.e. a shared cluster-egress/WAN/DNS transient, not the relays. `up` stayed 1 and all are far under the 15m `for:`, so nothing fired.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#423