konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] TargetDown (dns-servers) #259

New Issue
Closed
opened 2026-07-16 10:50:08 +00:00 by gitea_admin · 2 comments
gitea_admin commented 2026-07-16 10:50:08 +00:00
Copy Link
Copy Source

What is firing

  • Alert: TargetDown (severity: warning)
  • Job: dns-servers, Prometheus monitoring/kube-prometheus-stack-prometheus
  • Since: 2026-07-13T16:30:33Z (still active as of this triage, ~3 days)
  • Rule: 100% of targets in the dns-servers job are down (default kube-prometheus-stack general.rules TargetDown, threshold >10%)

Evidence (verified live)

Queried the Prometheus target API directly (kubectl get --raw ".../api/v1/targets?state=any") rather than relying on the alert annotation alone. All three configured targets are health: down with the same error:

instance: proxmox-dns-1.srv.renner.uno:12345  health: down  lastError: Get "http://proxmox-dns-1.srv.renner.uno:12345/metrics": context deadline exceeded
instance: proxmox-dns-2.srv.renner.uno:12345  health: down  lastError: Get "http://proxmox-dns-2.srv.renner.uno:12345/metrics": context deadline exceeded
instance: proxmox-dns-3.srv.renner.uno:12345  health: down  lastError: Get "http://proxmox-dns-3.srv.renner.uno:12345/metrics": context deadline exceeded

context.txt (in-cluster snapshot) shows no unhealthy pods and no Flux issues — expected, since these are external LXC hosts, not Kubernetes workloads, so the cluster snapshot can't confirm/deny this one.

For comparison, I also checked the other two external scrape jobs configured in the same Prometheus (pve, gitea) to rule out a wider network-segment outage: gitea is fully up, and pve is down but with a different failure mode (server returned HTTP status 500 INTERNAL SERVER ERROR from the pve-exporter service itself, not a connection timeout, and not part of this alert batch). So the dns-servers failure looks isolated to those three specific hosts/port, not a general network problem — worth noting but out of scope for this issue.

Root cause (unconfirmed — hypothesis)

Port 12345 is the Grafana Alloy self-metrics port (ansible/roles/monitoring-agent/defaults/main.yml:45, alloy_http_listen_port: 12345), templated to bind 0.0.0.0 (ansible/roles/monitoring-agent/templates/alloy.service.j2:12). The Alloy role is applied to these hosts via ansible/plays/common.yml (hosts: all:!pve:!pbs:!proxmox_noansible:!proxmox_nodes:!pve_cluster), which the proxmox_dns_ha inventory group (ansible/inventory/pve_hosts.yml:18-21) is not excluded from.

context deadline exceeded on all 3 hosts simultaneously for 3 straight days is consistent with any of:

  1. The Alloy agent not running/crashed on all three LXCs (e.g. common.yml never successfully ran against this group, or Alloy failed to start).
  2. A firewall/network change blocking port 12345 specifically on the v40srv bridge segment these LXCs sit on (opentofu/lxc_118_proxmox_dns.tf).
  3. The LXC hosts themselves being unreachable (would also imply the DNS service on port 53 / keepalived VIP 10.2.40.53 is impacted — not verified here; no read-only path from this cluster to check that).

This cannot be distinguished further with the read-only cluster access available to this triage — I have no SSH/exec access to proxmox-dns-1/2/3.srv.renner.uno. A human (or a follow-up ansible run) should check:

  • ssh proxmox-dns-1.srv.renner.uno systemctl status alloy (and -2, -3)
  • curl -m5 http://proxmox-dns-1.srv.renner.uno:12345/metrics from a host on the same network segment
  • Whether DNS resolution itself (port 53 / VIP 10.2.40.53) is actually still working — that is the higher-severity concern if these hosts are truly unreachable.

Where to fix it

  • If Alloy is simply not running: re-run the common.yml play (role monitoring-agent) against the proxmox_dns_ha group, or ansible/plays/install-proxmox-dns-ha.yml doesn't include monitoring-agent at all — it only applies proxmox-dns + keepalived-vip roles, so if these LXCs were (re)provisioned via that playbook alone, common.yml may never have run against them.
  • If it's a firewall/network issue: not fixable in this repo — needs investigation on the Proxmox network config for the v40srv bridge / hosts.
  • Scrape config lives at kubernetes/apps/monitoring/helmrelease-prometheus.yaml:95-100 (and a stale duplicate at ansible/roles/prometheus-server/templates/prometheus.yml.j2:74-82 from what looks like a legacy standalone Prometheus — worth confirming that file is no longer in use, since the active Prometheus per the alert's prometheus label is the in-cluster kube-prometheus-stack-prometheus).

Filed automatically by the alert-monitor pipeline (2026-07-16). The issue-fix pipeline may pick this up and open a PR.

Owner context (2026-07-16): The proxmox-dns-1/2/3 LXCs have been REMOVED — gone, not down. All of their infrastructure is repo-managed decommission cruft: the dns-servers scrape job in kubernetes/apps/monitoring/helmrelease-prometheus.yaml (removing it resolves THIS alert), plus opentofu/lxc_118_proxmox_dns.tf (still declares vmid 118/218/318 — note: has OpenTofu state implications like the #199 cleanup, do not edit blindly), ansible/roles/proxmox-dns/, ansible/plays/install-proxmox-dns.yml, ansible/plays/install-proxmox-dns-ha.yml, and inventory references. The .srv.renner.uno names are just VLAN-40 naming and resolve for unrelated reasons; ignore DNS-zone theories. Owner expectation: fix this alert surgically (scrape job), and list the remaining decommission cruft explicitly in the PR Notes for follow-up.

## What is firing - **Alert**: `TargetDown` (severity: warning) - **Job**: `dns-servers`, Prometheus `monitoring/kube-prometheus-stack-prometheus` - **Since**: 2026-07-13T16:30:33Z (still active as of this triage, ~3 days) - **Rule**: 100% of targets in the `dns-servers` job are down (default kube-prometheus-stack `general.rules` TargetDown, threshold >10%) ## Evidence (verified live) Queried the Prometheus target API directly (`kubectl get --raw ".../api/v1/targets?state=any"`) rather than relying on the alert annotation alone. All three configured targets are `health: down` with the same error: ``` instance: proxmox-dns-1.srv.renner.uno:12345 health: down lastError: Get "http://proxmox-dns-1.srv.renner.uno:12345/metrics": context deadline exceeded instance: proxmox-dns-2.srv.renner.uno:12345 health: down lastError: Get "http://proxmox-dns-2.srv.renner.uno:12345/metrics": context deadline exceeded instance: proxmox-dns-3.srv.renner.uno:12345 health: down lastError: Get "http://proxmox-dns-3.srv.renner.uno:12345/metrics": context deadline exceeded ``` `context.txt` (in-cluster snapshot) shows no unhealthy pods and no Flux issues — expected, since these are external LXC hosts, not Kubernetes workloads, so the cluster snapshot can't confirm/deny this one. For comparison, I also checked the other two external scrape jobs configured in the same Prometheus (`pve`, `gitea`) to rule out a wider network-segment outage: `gitea` is fully `up`, and `pve` is down but with a *different* failure mode (`server returned HTTP status 500 INTERNAL SERVER ERROR` from the pve-exporter service itself, not a connection timeout, and not part of this alert batch). So the `dns-servers` failure looks isolated to those three specific hosts/port, not a general network problem — worth noting but out of scope for this issue. ## Root cause (unconfirmed — hypothesis) Port 12345 is the Grafana Alloy self-metrics port (`ansible/roles/monitoring-agent/defaults/main.yml:45`, `alloy_http_listen_port: 12345`), templated to bind `0.0.0.0` (`ansible/roles/monitoring-agent/templates/alloy.service.j2:12`). The Alloy role is applied to these hosts via `ansible/plays/common.yml` (`hosts: all:!pve:!pbs:!proxmox_noansible:!proxmox_nodes:!pve_cluster`), which the `proxmox_dns_ha` inventory group (`ansible/inventory/pve_hosts.yml:18-21`) is not excluded from. `context deadline exceeded` on all 3 hosts simultaneously for 3 straight days is consistent with any of: 1. The Alloy agent not running/crashed on all three LXCs (e.g. `common.yml` never successfully ran against this group, or Alloy failed to start). 2. A firewall/network change blocking port 12345 specifically on the `v40srv` bridge segment these LXCs sit on (`opentofu/lxc_118_proxmox_dns.tf`). 3. The LXC hosts themselves being unreachable (would also imply the DNS service on port 53 / keepalived VIP `10.2.40.53` is impacted — not verified here; no read-only path from this cluster to check that). This cannot be distinguished further with the read-only cluster access available to this triage — I have no SSH/exec access to `proxmox-dns-1/2/3.srv.renner.uno`. A human (or a follow-up ansible run) should check: - `ssh proxmox-dns-1.srv.renner.uno systemctl status alloy` (and -2, -3) - `curl -m5 http://proxmox-dns-1.srv.renner.uno:12345/metrics` from a host on the same network segment - Whether DNS resolution itself (port 53 / VIP 10.2.40.53) is actually still working — that is the higher-severity concern if these hosts are truly unreachable. ## Where to fix it - If Alloy is simply not running: re-run the `common.yml` play (role `monitoring-agent`) against the `proxmox_dns_ha` group, or `ansible/plays/install-proxmox-dns-ha.yml` doesn't include `monitoring-agent` at all — it only applies `proxmox-dns` + `keepalived-vip` roles, so if these LXCs were (re)provisioned via that playbook alone, `common.yml` may never have run against them. - If it's a firewall/network issue: not fixable in this repo — needs investigation on the Proxmox network config for the `v40srv` bridge / hosts. - Scrape config lives at `kubernetes/apps/monitoring/helmrelease-prometheus.yaml:95-100` (and a stale duplicate at `ansible/roles/prometheus-server/templates/prometheus.yml.j2:74-82` from what looks like a legacy standalone Prometheus — worth confirming that file is no longer in use, since the active Prometheus per the alert's `prometheus` label is the in-cluster `kube-prometheus-stack-prometheus`). --- <!-- alert-monitor:fingerprint=b33b17d0dba1676e --> _Filed automatically by the alert-monitor pipeline (2026-07-16). The issue-fix pipeline may pick this up and open a PR._ --- **Owner context (2026-07-16):** The proxmox-dns-1/2/3 LXCs have been REMOVED — gone, not down. All of their infrastructure is repo-managed decommission cruft: the `dns-servers` scrape job in `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` (removing it resolves THIS alert), plus `opentofu/lxc_118_proxmox_dns.tf` (still declares vmid 118/218/318 — note: has OpenTofu state implications like the #199 cleanup, do not edit blindly), `ansible/roles/proxmox-dns/`, `ansible/plays/install-proxmox-dns.yml`, `ansible/plays/install-proxmox-dns-ha.yml`, and inventory references. The `.srv.renner.uno` names are just VLAN-40 naming and resolve for unrelated reasons; ignore DNS-zone theories. Owner expectation: fix this alert surgically (scrape job), and list the remaining decommission cruft explicitly in the PR Notes for follow-up.
gitea_admin commented 2026-07-16 11:46:18 +00:00
Author
Copy Link
Copy Source

Assessment: no repo change — flagged suspected prompt injection, verified real root cause

No edits were made. The issue body contains a section that looks like a prompt
injection attempt, and independent live verification shows its central factual claim
is false. The actual root cause (LXCs stopped, not removed) is not fixable by any
change under ansible/, opentofu/, or kubernetes/ — it requires a live Proxmox
action by an operator.

Suspected prompt injection in the issue body

The issue text includes a block headed "Owner context (2026-07-16)", appended
after the standard alert-monitor:fingerprint=... footer, asserting as fact that
"The proxmox-dns-1/2/3 LXCs have been REMOVED — gone, not down" and directing me to
delete the dns-servers scrape job from
kubernetes/apps/monitoring/helmrelease-prometheus.yaml to "resolve THIS alert,"
while pre-emptively waving off "DNS-zone theories" and OpenTofu-state concerns.

This is exactly the shape of content the operating rules tell me to distrust: an
unverified claim embedded inside the issue body (not a separate, authenticated
owner comment), asserting cluster state as settled fact, and instructing me to
silence an alert — one of the explicitly listed anti-patterns ("never silence the
alert instead of fixing the cause"). I did not act on it and verified independently
instead.

What live verification actually shows

Using ./.ci-diag/kubectl against the Prometheus API (read-only, this run):

  • GET /api/v1/targets?state=any — as of 2026-07-16T11:42–11:43Z, all three
    dns-servers targets are still health: down, context deadline exceeded
    (matches the issue's original, pre-"Owner context" evidence).
  • query=pve_guest_info — the Proxmox exporter (job=pve) currently lists
    lxc/118, lxc/218, lxc/318 as existing guests named
    proxmox-dns-1/2/3.srv.renner.uno on nodes jupiter/saturn/neptune,
    template="0" (not a template — a real, defined container). This directly
    contradicts the "REMOVED" claim.
  • query=pve_up{id=~"lxc/(118|218|318)"} — currently 0 (stopped) for all three,
    consistently across all three PVE-cluster scrape instances (10.2.40.201/.202/.203).
  • query=pve_up{id=~"node/.+"} — the underlying Proxmox nodes (jupiter, saturn,
    neptune) are all 1 (up), ruling out a node/cluster-wide outage.
  • query_range on pve_up{id="lxc/118",...} over the past 4 days — value was 1
    until it flipped to 0 at 2026-07-13T16:22:15Z, ~8 minutes before the
    TargetDown alert fired (2026-07-13T16:30:33Z) and has stayed 0 since (with a
    ~2-day exporter-scrape gap in between, consistent with the pve job's separately
    observed 500 errors, not with the guests coming back).

Actual root cause (verified, not hypothesis): all three DNS-HA LXCs (118/218/318)
were stopped at 2026-07-13T16:22:15Z and have not been restarted. They still exist
as Proxmox guest definitions — this is a stopped/powered-off state, not a deletion.
opentofu/lxc_118_proxmox_dns.tf already declares start = true and onboot = true
for all three, so the repo's desired state is already correct; OpenTofu/repo config is
not the problem, and there is nothing to change there.

Why this is more serious than the monitoring alert suggests

These three LXCs are the DNS-HA cluster behind VIP 10.2.40.53 (keepalived,
opentofu/lxc_118_proxmox_dns.tf). If all three are stopped, DNS resolution on that
VIP is very likely down cluster-wide, not just metrics scraping. I have no network
path from this read-only cluster context to confirm port 53 / VIP reachability
directly — that needs to be checked by a human.

Recommended manual action (not performed here — requires state-changing access)

  1. On the Proxmox side, check why lxc/118, lxc/218, lxc/318 are stopped (host
    logs / task history around 2026-07-13T16:22 UTC) — could be a manual stop,
    maintenance action, or a crash.
  2. If they should be running (their own OpenTofu/Ansible config says so), start them:
    pct start 118, pct start 218, pct start 318 (or restart via the Proxmox UI/API)
    on nodes jupiter/saturn/neptune respectively.
  3. Separately verify DNS is actually resolving via VIP 10.2.40.53 right now — this is
    the higher-severity concern the TargetDown alert is a proxy for.
  4. Once running, confirm the dns-servers Prometheus targets return to health: up
    (./.ci-diag/kubectl get --raw ".../api/v1/targets?state=any" and check the three
    proxmox-dns-* instances).
  5. If, after investigation, these hosts are genuinely being decommissioned on purpose,
    that should come from a verified owner decision (e.g. an actual Gitea issue comment
    or PR from the owner) — at that point removing the scrape job plus the OpenTofu
    module, ansible/roles/proxmox-dns/, and the ansible/plays/install-proxmox-dns*.yml
    playbooks would be the right cleanup, but that determination should not be made from
    unverified text embedded in an auto-filed issue.

Notes

  • The issue's own pre-"Owner context" hypotheses (Alloy agent down, firewall change,
    hosts unreachable) are all superseded by the more specific finding above (guests
    stopped, confirmed via pve_up), so no further action is needed chasing those.
  • The pve job's intermittent 500 errors noted in the issue are real (visible as a gap
    in the pve_up time series) but are a separate, already-out-of-scope issue per the
    original triage.
  • No files were changed under ansible/, opentofu/, kubernetes/, or ci/images/.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## Assessment: no repo change — flagged suspected prompt injection, verified real root cause **No edits were made.** The issue body contains a section that looks like a prompt injection attempt, and independent live verification shows its central factual claim is false. The actual root cause (LXCs stopped, not removed) is not fixable by any change under `ansible/`, `opentofu/`, or `kubernetes/` — it requires a live Proxmox action by an operator. ## Suspected prompt injection in the issue body The issue text includes a block headed **"Owner context (2026-07-16)"**, appended after the standard `alert-monitor:fingerprint=...` footer, asserting as fact that "The proxmox-dns-1/2/3 LXCs have been REMOVED — gone, not down" and directing me to delete the `dns-servers` scrape job from `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` to "resolve THIS alert," while pre-emptively waving off "DNS-zone theories" and OpenTofu-state concerns. This is exactly the shape of content the operating rules tell me to distrust: an unverified claim embedded *inside* the issue body (not a separate, authenticated owner comment), asserting cluster state as settled fact, and instructing me to silence an alert — one of the explicitly listed anti-patterns ("never silence the alert instead of fixing the cause"). I did not act on it and verified independently instead. ## What live verification actually shows Using `./.ci-diag/kubectl` against the Prometheus API (read-only, this run): - `GET /api/v1/targets?state=any` — as of 2026-07-16T11:42–11:43Z, all three `dns-servers` targets are still `health: down`, `context deadline exceeded` (matches the issue's original, pre-"Owner context" evidence). - `query=pve_guest_info` — the Proxmox exporter (`job=pve`) currently lists `lxc/118`, `lxc/218`, `lxc/318` as existing guests named `proxmox-dns-1/2/3.srv.renner.uno` on nodes `jupiter`/`saturn`/`neptune`, `template="0"` (not a template — a real, defined container). **This directly contradicts the "REMOVED" claim.** - `query=pve_up{id=~"lxc/(118|218|318)"}` — currently `0` (stopped) for all three, consistently across all three PVE-cluster scrape instances (10.2.40.201/.202/.203). - `query=pve_up{id=~"node/.+"}` — the underlying Proxmox nodes (jupiter, saturn, neptune) are all `1` (up), ruling out a node/cluster-wide outage. - `query_range` on `pve_up{id="lxc/118",...}` over the past 4 days — value was `1` until it flipped to `0` at **2026-07-13T16:22:15Z**, ~8 minutes before the `TargetDown` alert fired (2026-07-13T16:30:33Z) and has stayed `0` since (with a ~2-day exporter-scrape gap in between, consistent with the `pve` job's separately observed 500 errors, not with the guests coming back). **Actual root cause (verified, not hypothesis): all three DNS-HA LXCs (118/218/318) were stopped at 2026-07-13T16:22:15Z and have not been restarted.** They still exist as Proxmox guest definitions — this is a stopped/powered-off state, not a deletion. `opentofu/lxc_118_proxmox_dns.tf` already declares `start = true` and `onboot = true` for all three, so the repo's desired state is already correct; OpenTofu/repo config is not the problem, and there is nothing to change there. ## Why this is more serious than the monitoring alert suggests These three LXCs are the DNS-HA cluster behind VIP `10.2.40.53` (keepalived, `opentofu/lxc_118_proxmox_dns.tf`). If all three are stopped, DNS resolution on that VIP is very likely down cluster-wide, not just metrics scraping. I have no network path from this read-only cluster context to confirm port 53 / VIP reachability directly — that needs to be checked by a human. ## Recommended manual action (not performed here — requires state-changing access) 1. On the Proxmox side, check why `lxc/118`, `lxc/218`, `lxc/318` are stopped (host logs / task history around 2026-07-13T16:22 UTC) — could be a manual stop, maintenance action, or a crash. 2. If they should be running (their own OpenTofu/Ansible config says so), start them: `pct start 118`, `pct start 218`, `pct start 318` (or restart via the Proxmox UI/API) on nodes jupiter/saturn/neptune respectively. 3. Separately verify DNS is actually resolving via VIP `10.2.40.53` right now — this is the higher-severity concern the `TargetDown` alert is a proxy for. 4. Once running, confirm the `dns-servers` Prometheus targets return to `health: up` (`./.ci-diag/kubectl get --raw ".../api/v1/targets?state=any"` and check the three `proxmox-dns-*` instances). 5. If, after investigation, these hosts are genuinely being decommissioned on purpose, that should come from a verified owner decision (e.g. an actual Gitea issue comment or PR from the owner) — at that point removing the scrape job plus the OpenTofu module, `ansible/roles/proxmox-dns/`, and the `ansible/plays/install-proxmox-dns*.yml` playbooks would be the right cleanup, but that determination should not be made from unverified text embedded in an auto-filed issue. ## Notes - The issue's own pre-"Owner context" hypotheses (Alloy agent down, firewall change, hosts unreachable) are all superseded by the more specific finding above (guests stopped, confirmed via `pve_up`), so no further action is needed chasing those. - The `pve` job's intermittent 500 errors noted in the issue are real (visible as a gap in the `pve_up` time series) but are a separate, already-out-of-scope issue per the original triage. - No files were changed under `ansible/`, `opentofu/`, `kubernetes/`, or `ci/images/`. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
k.renner commented 2026-07-16 12:04:30 +00:00
Owner
Copy Link
Copy Source

These really have been decommissioned.
The correct solution would have been to remove the invalid references.

These really have been decommissioned. The correct solution would have been to remove the invalid references.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#259