- Nodes Online/Down used job="node" but the scrape job is "node-exporter":
both stats showed No data. Nodes Down additionally gets OR on() vector(0)
so it renders 0 instead of No data when all nodes are up, and green-at-0/
red-at-1 thresholds (was permanently red).
- Avg Disk Usage + Disk Usage by Node filtered mountpoint="/" which does not
exist on Talos (only /var is a real xfs filesystem): both panels were empty.
- Disk I/O summed all block devices including the ~12 Rook-Ceph rbd* mappings
per node (~438 kB/s vs 8-129 kB/s physical): filter to nvme.*|vd.*.
- Network I/O filter was written for a Docker host (veth/docker/br-) and let
~40 Cilium lxc* per-pod interfaces plus kernel pseudo-devices through,
double-counting pod traffic: filter to eth.* (bond0 is down, 0 slaves).
- Legends showed instance IP:port; join node_uname_info via
on(instance) group_left(nodename) so panels label k8s-1/2/3. Both join
sides are wrapped in max by (...) so historical duplicate series from
pod restarts / the pre-2026-07-16 duplicate-scrape residue cannot break
the join or rainbow-duplicate long-window views.
- Stat panels reduce lastNotNull instead of mean: show current value, not
a time-average over the selected window.
All queries verified against live Prometheus before applying.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1 of the dashboard triage.
Deleted (no current data source, no plan to wire one):
- unifi-overview (unpoller removed after UDM crash)
- caddy-overview (Caddy retired 2026-05-09)
- docker-overview (Docker host era done; workloads on K8s)
- log-explorer (legacy host log shipping not in scope)
- dns-overview (dns-servers job lives but a dedicated dashboard isn't worth keeping; Explore tab covers ad-hoc)
Repointed datasource from `prometheus-legacy` to `prometheus` (the
legacy LXC 227 Prom is retired):
- node-overview, node-metrics — show K8s nodes only (legacy host
node-exporters were never re-scraped post-migration)
- infrastructure-overview — generic up{} + node_* metrics work on K8s
- proxmox-cluster — pve-exporter is now in-cluster (commit 33b452a)
Also deletes DASHBOARD-MIGRATION.md (the migration cut-over completed
with VM 109 retirement on 2026-05-09).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copies every dashboard from `ansible/roles/grafana-observability/files/dashboards/`
into `kubernetes/apps/monitoring/dashboards/` and wires each as a
ConfigMap for the kube-prometheus-stack Grafana sidecar. UID rewrites
follow the buckets in DASHBOARD-MIGRATION.md:
- 11 k8s-native dashboards copied with no rewrite (UID stays
`prometheus`/`loki`/`influxdb`).
- 9 host-legacy + 3 mixed-resolved-as-legacy (blackbox, infrastructure,
synology-nas Prom panels) had every `"uid": "prometheus"` rewritten to
`"uid": "prometheus-legacy"` so they continue to render against LXC 227
until the global cut-over retires VM 109.
- log-explorer and netbird-overview are pure Loki, no rewrite.
- paperless-overview and unifi-overview kept on `prometheus` (paperless
per k8s-native intent; unifi because the K8s Prom also scrapes the same
unpoller exporter).
Validated with `kubectl kustomize kubernetes/apps/monitoring`: 27
ConfigMaps generated, all carrying the `grafana_dashboard: "1"` label
that the sidecar watches in the `monitoring` namespace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 4.8
k.rennerandClaude Opus 4.7