The single pve scrape job hit all three node IPs with cluster=1, and
pve-exporter returns the full cluster-wide resource set regardless of
which node answers - every node/guest series landed in the TSDB three
times (same id, differing instance). Dashboards papered over it with
max by (id); the cardinality cost stayed.
Split per upstream prometheus-pve-exporter guidance: cluster collectors
scrape one node only, node-local collectors keep scraping all three
under a new pve-node job. No dashboard queries filter on job=, and all
already aggregate by id, so this is query-compatible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every panel was dead: all 11 datasource refs pointed at the
prometheus-legacy datasource (retired 2026-05-09). Repoint to the
active prometheus datasource.
The alloy job also carries no 'host' label, so all legends and the
Agent Versions table rendered empty labels. Join via kube_pod_info
(pod -> node) so DaemonSet panels are labelled k8s-1/2/3, collapse
the eval-time histogram per pod before quantile, and tidy the
versions table to node/pod/version columns.
All queries verified against live Prometheus (3 agents, one clean
series per node).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 10 panels were dead: the dashboard queried datasource uid
prometheus-legacy (prometheus.srv.renner.uno:9090), which no longer
resolves (Grafana proxy: 502 dial tcp lookup i/o timeout). Gitea now
runs in-cluster and is scraped by the in-cluster Prometheus
(job="gitea", instance gitea.apps.svc.cluster.local:3000), so switch
every panel to uid prometheus.
Also wrap all targets in max(): the pre-migration instance
gitea.srv.renner.uno:3000 stopped 2026-07-12, and windows spanning
that date would otherwise render two identically-named series per
query. Stat panels additionally get instant:true so a dead exporter
shows No data instead of a stale last value.
- Healthy Hosts: job 'integrations/unix' does not exist here -> node-exporter;
thresholds now red/yellow/green(3) so a single down node shows yellow
- Unhealthy Services: 'count(up == 0)' returns empty vector when healthy ->
'or vector(0)' renders an honest 0; both stats now instant queries
- CPU: 'avg by(host)' collapsed all 3 nodes into one series (no host label) ->
avg by(instance) joined with node_uname_info for k8s-1/2/3 legend names
- Memory: legend showed raw label dumps ({{host}} does not exist) -> nodename join
- Disk: Talos has no '/' mountpoint in node_filesystem -> use /var (the
persistent data disk); retitle panel accordingly
- Current by Phase: regex /smart_meter.*current/ matched nothing (smart
meter exposes no current entities) -> use verified
e380_energy_meter_e380_current_l[123]; spanNulls for sparse updates
- Energy Consumption: measurement "kWh" does not exist in the bucket
-> plot verified Wh daily counters (grid import/export, PV production,
battery charge/discharge) with readable display names
- Active Lights: light-domain data is never recorded under measurement
"%" -> replace dead table with Lights/Standby/Switches power
timeseries from verified powercalc aggregates (all_*_power)
- Top 20 Devices: add group() so sort/limit work globally (was
alphabetical, unsorted, per-table); exclude grid meters, PV/battery,
forecasts, powercalc room/floor aggregates and _device_power twins;
fresh 30-min window instead of stale last() over the full range;
rowsToFields for clean labels (was "_value xyz_power")
- Grid semantics: retitle Total Power -> Grid Power (+Import/-Export),
blue threshold for export; unstack mixed-sign phase graph; short
L1/L2/L3 legend names; taller voltage/current panels so the legend
table is not cut off
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The template variable queried label_values(node_uname_info, host), but no
node-exporter metric in this Prometheus carries a 'host' label (that label
only exists on blackbox-probe metrics). The variable resolved to empty and
all five panels rendered blank.
- Replace $host with a visible $node variable (nodename: k8s-1/2/3) plus a
hidden chained $instance variable (nodename -> instance via node_uname_info)
- Filter Cilium pod interfaces (lxc*, cilium_*) and always-zero kernel
pseudo-devices (ip6tnl0, sit0, teql0, tunl0) out of Network I/O
- Filter Ceph RBD-mapped volumes (rbd0..11) out of Disk I/O so the panel
shows physical disks (nvme0n1, vda) only
- Exclude the rootfs bind mount (/etc/extensions.yaml) from Filesystem
Usage, leaving the meaningful /var xfs mount
All queries verified against live Prometheus before commit.
- All temperature/humidity panels: replace vendor regexes (shelly.*temp etc.)
that dragged in device-internal temperatures (pond pump 34C, power meter 60C,
relay 76C) with explicit allow-lists of verified room sensors
- Add missing sensors: shelly_blu_h_t_7203 (KG Schlafzimmer),
treppenhaus_eg_bewegungsmelder (EG Treppenhaus), vento humidity sensors,
Shelly Wall Display (DG Buero)
- OG & DG panel: regex had no og_ alternative and showed only device internals;
now honest DG-only series + description (OG deliberately has no sensor)
- Heating System / Hot Water panels were empty (domain climate/water_heater
does not exist in the C measurement) -> use vitocal_250_a_* sensors
- Current-value panels: fixed -6h window so dead sensors (EG Shelly H&T trio,
DG Meter Pro) disappear instead of showing stale values as live
- Legends: map entity_ids to room names in Flux + displayName, removes the
'_value xyz' prefix
- Align CO2 thresholds between gauge and history (800/1000/1500)
- Server node: sensor.keller_server_power is dead (UPS retired) -> Wave PM
Mini server outlet (kg_vorratskeller_wave_pm_mini_server_steckdose)
- Garden node: sensor.eg_garten_teichpumpe_power_2 is dead ->
sensor.shelly_teichpumpe_power_0 (live, peaks ~77 W)
- Home node/series: floor sums only cover submetered circuits (702 W shown
vs ~1.1 kW real); Home is now the meter balance PV + Grid + Battery, with
a new grey 'Rest' node for the non-submetered remainder
- Flow lines: Heat Pump is metered inside Keller (was drawn from EG);
Garden is on floor Aussenbereich, in no floor sum (was drawn from OG)
- Drop redundant batteryIn target (derived from the signed battery sensor);
update stale comments from the retired *_absolute sensors
- History: max by (entity) guards against duplicate-scrape residue that
doubled the summed Home series; sign hints in Battery/Grid legends;
grid query pinned to entity id instead of friendly_name
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Status panel: job label is home-assistant (hyphen), was homeassistant -> No data
- Doors & Windows / Motion: exporter has no device_class label; select curated
entity lists instead (one canonical is-open entity per physical contact,
physical motion/presence sensors without zone/magic_areas duplicates)
- Temperature: was all 61 temp series incl. device-internal electronics temps
(fridge 5C, PoE switch 75C, Shelly 58-75C); curated to room/ambient sensors
+ outdoor temp, incl. currently-dead EG H&T sensors so they reappear
- Power Consumption: exclude non-consumption series (Solcast forecasts,
battery max charge/discharge capability, derived night-load stats)
- Energy: cumulative lifetime counters (up to 80 MWh) told nothing; now
bargauge of top 12 increase() over 24h, forecast counters excluded,
max by () dedup against scrape residue
- Switches: 248 unreadable micro-tiles -> single 'Switches On' count stat
- Battery: bottomk(12) instant, battery_notes *_battery_plus twins excluded
- Entity counts: max by (entity) dedup, instant queries on all stats
- Unavailable Entities: neutral blue; ~1139 is the structural baseline
(verified against HA API), old yellow>=1/red>=5 thresholds were meaningless
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All queries selected raw series by entity label only; over a 90d window HA pod
restarts plus the (since fixed) duplicate proxy scrape produce 30 distinct
series per entity, rendering the Außentemperatur & Puffer panel as an
unreadable 30-entry legend jumble. Wrap every query in max() so exactly one
series per entity survives — this also keeps the xychart joinByField transform
correct once heating data arrives, since the join needs unique field names.
The stat panel now says 'keine Heizdaten im Zeitraum' instead of a giant
'No data': the logging sensor is intentionally unavailable outside active
heating loading, so an empty summer dashboard is by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All four panels queried raw HA metrics that still carry ~12 duplicate
series per entity in the 90d window (two scrape jobs x six pod
lifetimes; the duplicate-scrape bug was fixed 2026-07-16 but the
residue stays in range queries until it ages out). This broke the
scatter join (no points rendered), made the stat panel show 8
conflicting averages, and drew rainbow duplicates in both time series.
Wrapping every target in max by (entity) (...) collapses the residue
to one series per entity; values between the duplicate jobs were
near-identical, so max is lossless.
- Windows/Doors/Batteries tiles: literal ${__value.raw} never interpolates in
value mappings; replaced range mappings with suffix units. Counts now use
curated entity lists (one canonical 'is open' entity per physical sensor)
instead of friendly-name regexes that matched inverted *_is_closed sensors,
'Battery low' sensors and duplicate _2 entities.
- Alarm tile: old min(...==1) query could never show TRIGGERED; now
max(...{state="triggered"}) with 0=OK/1=TRIGGERED mapping.
- Power Usage: old regex summed phases A+B+C AND total_active_power AND
device_power -> ~2x real value; now reads total_active_power only.
- Floor temperature panels: select verified room sensors by entity id instead
of vendor regexes (which pulled 65-75C device-internal temps and missed the
I-9PSL sensors that live in the EG). Instant queries so dead sensors do not
render stale values. OG: no room sensor exists (owner-confirmed) -> honest
noValue; DG SwitchBot offline -> noValue 'Sensor offline'.
- Tables: dedup via max by (entity, friendly_name), drop k8s scrape label
columns, colour only the value column; battery list excludes battery_notes
*_battery_plus twin entities and no longer hides 0% batteries.
- Power Consumption: L1/L2/L3 legends, Total from total_active_power.
- Indoor Temperatures: room sensors only (y-axis no longer squashed to 80C).
- Heating row: Hot Water/Heating pointed at post-open3e-migration vitocal
sensors (old queries returned nothing resp. only the fridge); humidity/CO2
deduplicated; row renamed to Heating & Air Quality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The union-based timeseries series were named via set(_field, ...), which Grafana's InfluxDB frontend rendered as "_value {_start=...}" instead of the intended labels (and broke the byName color overrides). Data was correct; only the legend/colors were affected.
Switch to the keep(columns:[_time,_value]) |> rename(_value -> "Name") pattern (as used in heatpump-roi), which names each series cleanly. Verified in Grafana Explore. No data/logic change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
The 2026-07 ViCare->open3e/MQTT migration renamed the Vitocharge PV/battery entities (e3_vitocharge_03_* -> vitocharge_vx3_vx3_*) and changed some units. Both dashboards still queried the retired entities and showed No data.
power-flow (Prometheus, real-time): solar_power->pv_power, battery SoC->battery_soc; the two absolute battery-power sensors are replaced by a single signed battery_power (open3e: positive=discharge, negative=charge), so charge/discharge derive from its sign.
pv-system-roi (InfluxDB): repoint every query; fix units (PV/battery energy now Wh not kWh -> /1000; grid energy now kWh not Wh); grid power sign unchanged (negative=export). Timeseries panels UNION pre-migration e3_* with post-migration series (unit-normalized) so history back to 2023 stays unbroken. Reserved Energy recomputed as capacity x backup_discharge_limit. strompreis + the two input_number config entities survived and are kept.
Validated live via Grafana Explore: signed-battery charge/discharge union, unit normalization, pivot/value patterns (reused from merged heatpump-roi fix#271).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Both destroy applies (runs 1180/1181) died instantly with 'context
deadline exceeded' on the pre-destroy status GET while the PVE API
answers in <1s: the bpg provider reads timeout_delete from STATE at
destroy time, and the stage-1 import stored 0 for the timeout-less
shims — a zero-second context. Read paths carry their own timeout,
which is why import and plan were clean.
v2 declares all timeouts explicitly (one in-place, no-API update
writes them into state) and mirrors the real container attributes
(unprivileged=true is ForceNew — mirroring prevents a replace).
No ignore_changes this time: the timeout update must actually apply.
Flow: merge -> plan (must be in-place only) -> apply -> follow-up PR
deletes the file -> destroy apply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The ViCare cloud integration was replaced by open3e/MQTT (2026-07), renaming all heat-pump/PV entities (e3_vitocal_*/e3_vitocharge_03_* -> vitocal_250_a_vitocal_*/vitocharge_vx3_vx3_*). The dashboard still queried the retired IDs, and a duplicate Prometheus scrape broke the surviving-entity panels with many-to-many errors.
Rebuild every panel on InfluxDB (bucket suedstr28, authoritative + live) using current entity IDs. Where an entity was renamed, UNION the pre-migration e3_* series with the post-migration series (unit-normalized: PV Wh->kWh) so history stays unbroken. Drop all Prometheus targets. Add a grid_price variable (0.3271) replacing the retired live strompreis read.
Also rename the home-assistant-bgp Service port http->ha so the home-assistant ServiceMonitor no longer scrapes HA a second time via the LB (:8123), which duplicated homeassistant_sensor_* series cluster-wide.
Queries validated live via Grafana Explore: blended price 0.248 EUR/kWh, total consumption 14284 kWh, projected annual savings 4698 EUR, oil proration 11912 EUR, old-union-new grid-import continuous.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Stage 1 (#268) adopted the never-tofu-managed rustfs containers into
state (import run 1173: all three imported, plan 'No changes').
Removing the shims makes the next apply destroy the containers —
guest lifecycle fully through tofu, per owner rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The live 15-min snapshot structurally misses anything that fires and
resolves between ticks: on 2026-07-16 a 17-minute KubeHpaMaxedOut
(authentik) and last evening's four-alert home-assistant incident were
invisible to the monitor (owner noticed the blips; monitor did not).
The fetch step now also queries Prometheus's ALERTS history over a
35-minute window (~2 ticks + jitter). The dedup step synthesizes
pseudo-alerts for series not currently firing at all — checked against
the PRE-filter snapshot so live-but-null-routed alerts don't reappear
as blips — with a stable sha256-derived fingerprint so one noise
verdict suppresses a flappy alert permanently. Blips then flow through
the same title/noise dedup and triage as live alerts; the prompt
explains their semantics (self-resolved; judge the underlying cause;
null-routed per the repo route config = noise).
Verified against live data: currently-firing KubeCPUOvercommit
excluded, injected KubeHpaMaxedOut blip synthesized correctly;
sha256sum confirmed present in ci-claude; lookback query tested via
the service proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner rule: ALL guest lifecycle goes through tofu; no manual pct/qm
destroy, by anyone. The rustfs trio (800/801/802) was never
tofu-managed, so the #265 sweep could not reach it. Stage 1 adopts the
stopped containers into state (pipeline TOFU_IMPORT hook, commands in
the file header); stage 2 deletes this file and the apply destroys
them.
lifecycle ignore_changes=all + started=false keep the shims inert:
tofu must never reconcile or start these, only own their destruction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Enables the single-click HA onboarding shipped in shelly-manager PR #41:
shelly-manager registers a provisioned Shelly into HA's native integration
by driving HA's config-flow + device-registry API.
- shelly-manager-llat ExternalSecret projects the full-admin HA LLAT from
OpenBao (apps/home-assistant/shelly-manager-llat, key 'token').
- HomeAssistant__BaseUrl = http://home-assistant (in-namespace svc :80 → nginx
sidecar → HA :8123; nginx forwards the WS upgrade, so both the REST config-
flow and the device-registry websocket work). Reachability confirmed: same-
namespace ingress is allowed by the default-deny-ingress CNP's home-assistant
rule (probed :80 → 401 = reachable).
- HomeAssistant__Token from the new secret.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
k.renner
security-bot
claude-botandClaude Fable 5