Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 6b0617b4a7 chore(ci): bump CI tool pins to match the deployed cluster
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.

ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
  talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
  the upstream release assets (talos sha256sum.txt, flux checksums.txt)
  and against a local download.

.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
  already run; the lint step was one minor behind its own server.

Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:58:35 +02:00
k.renner 6b7736ea58 Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824) from fix/openbao-alerts-immediate into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#824
2026-09-08 06:02:32 +00:00
k.renner 7ff20e5b68 Merge pull request 'feat(openbao): auto-unseal with a static seal' (#825) from feat/openbao-static-seal into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#825
2026-09-08 05:54:54 +00:00
claude-botandClaude Fable 5.1 296accd1cb feat(openbao): auto-unseal with a static seal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.

OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.

This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.

Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:52:27 +02:00
claude-botandClaude Fable 5.1 2665e5ebf3 fix(alerts): page the moment OpenBao is below quorum
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.

OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.

Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:40:12 +02:00
k.renner 0fe512746b Merge pull request 'feat(monitoring): Intel iGPU dashboard' (#823) from feat/intel-gpu-dashboard into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#823
2026-09-08 05:24:35 +00:00
claude-botandClaude Fable 5.1 4e2303f455 feat(monitoring): Intel iGPU dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.

Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.

gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:22:46 +02:00
k.renner 483093a104 Merge pull request 'feat(monitoring): export Intel iGPU utilisation, frequency and power' (#822) from feat/intel-gpu-exporter into master
Reviewed-on: k.renner/homelab#822
2026-09-08 05:18:43 +00:00
claude-botandClaude Fable 5.1 8d348e7f88 fix(monitoring): let intel_gpu_top pick the GPU instead of hard-coding card0
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:15:04 +02:00
claude-botandClaude Fable 5.1 6f9ff9d624 feat(monitoring): export Intel iGPU utilisation, frequency and power
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.

Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.

End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.

XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.

Fixes #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:14:24 +02:00
k.renner 7bf65783be Merge pull request 'feat(intel-gpu-plugin): advertise the monitoring resource' (#820) from feat/intel-gpu-monitoring-resource into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#820
2026-09-08 04:54:15 +00:00
k.renner dd9fb01359 Merge pull request 'fix(reloader): restart audiobookshelf and immich-server on ConfigMap change' (#819) from fix/reloader-configmap-annotations into master
Reviewed-on: k.renner/homelab#819
2026-09-08 04:52:32 +00:00
claude-botandClaude Fable 5.1 da8a725b32 feat(intel-gpu-plugin): advertise the monitoring resource
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.

Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:51:43 +02:00
claude-botandClaude Fable 5.1 6008d59c9f fix(reloader): restart audiobookshelf and immich-server on ConfigMap change
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.

ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.

Reloader already watches all namespaces (kubernetes/infrastructure/reloader).

Fixes #742

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:46:12 +02:00
k.renner 5aa6c0609d Merge pull request 'fix(netbird-proxy-services): pin the digest the master build produced' (#818) from netbird-proxy-protocol-digest into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#818
2026-09-07 11:52:17 +00:00
claude-botandClaude Fable 5.1 0ebe113abb fix(netbird-proxy-services): pin the digest the master build produced
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.

The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:47:22 +02:00
k.renner 1723f73b5d Merge pull request 'feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)' (#817) from netbird-proxy-protocol-deploy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#817
2026-09-07 11:34:52 +00:00
k.renner 8d8df3b635 Merge pull request 'feat(netbird): carry the WAN client address through the passthrough edge (1/2)' (#816) from netbird-proxy-protocol into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#816
2026-09-07 11:34:43 +00:00
claude-botandClaude Fable 5.1 29c91769a7 feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.

Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:32:06 +02:00
15 changed files with 524 additions and 28 deletions
+1 -1
View File
@@ -158,7 +158,7 @@ steps:
# whenever it trails the deployed server. If you bump the server, bump here.
- name: woodpecker-lint
depends_on: [] # see note above `steps:`
image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
commands:
- |
set -eu
+5 -5
View File
@@ -1,9 +1,9 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TALOS_VERSION=v1.12.6
ARG TALOS_VERSION=v1.13.9
ARG VELERO_VERSION=v1.16.0
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KUBECTL_VERSION=v1.36.3
ARG FLUX_VERSION=2.9.5
RUN apk add --no-cache \
aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \
RUN curl -sfL -o /tmp/flux.tar.gz \
"https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
&& echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3 /tmp/flux.tar.gz" | sha256sum -c - \
&& echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473 /tmp/flux.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
&& chmod +x /usr/local/bin/flux \
&& rm -f /tmp/flux.tar.gz
@@ -21,6 +21,9 @@ metadata:
namespace: audiobookshelf
labels:
app.kubernetes.io/name: audiobookshelf
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: audiobookshelf-config
spec:
replicas: 1
strategy:
+3
View File
@@ -35,6 +35,9 @@ metadata:
namespace: immich
labels:
app.kubernetes.io/name: immich-server
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: immich-config
spec:
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -0,0 +1,20 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
relabelings:
# The series would otherwise carry only the exporter pod's IP.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -39,4 +39,5 @@ resources:
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- intel-gpu-exporter-servicemonitor.yaml
- probes-blackbox.yaml
@@ -34,7 +34,9 @@ spec:
# query parameter, and an exact match would break silently if the
# instance label ever encodes it differently.
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
for: 5m
# Two consecutive failed probes: enough to skip a single lost probe
# through Traefik, not enough to sit on a sealed store.
for: 1m
labels:
severity: critical
annotations:
@@ -31,7 +31,9 @@ spec:
# A replica that is sealed never recovers on its own, so the
# duration is only there to ignore ordinary rescheduling: a healthy
# pod moving between nodes is Ready again in well under a minute.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
# max by: kube-state-metrics stamps instance/pod on the series, so a
# rescheduled exporter starts a new series and resets the timer.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
for: 10m
labels:
severity: warning
@@ -53,11 +55,11 @@ spec:
# the ClusterSecretStore goes InvalidProviderConfig and no
# ExternalSecret can refresh.
#
# Shorter window than the warning above because this is an outage,
# not a degradation — but not zero, so a rolling restart that
# briefly dips does not page.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
for: 5m
# No window: below quorum the secret store is down and every
# ExternalSecret stalls with it. A sealed replica never recovers on
# its own, so waiting only delays the page.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
for: 0m
labels:
severity: critical
annotations:
@@ -0,0 +1,368 @@
{
"title": "Intel iGPU",
"uid": "intel-gpu",
"tags": ["gpu", "intel", "i915"],
"schemaVersion": 39,
"editable": true,
"timezone": "",
"refresh": "1m",
"time": { "from": "now-6h", "to": "now" },
"templating": {
"list": [
{
"name": "node",
"label": "Node",
"type": "query",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"query": {
"query": "label_values(gpumon_power, node)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"definition": "label_values(gpumon_power, node)",
"includeAll": true,
"multi": true,
"refresh": 2,
"sort": 1,
"current": { "selected": true, "text": ["All"], "value": ["$__all"] }
}
]
},
"panels": [
{
"type": "stat",
"title": "Render/3D busy now",
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"decimals": 0,
"min": 0,
"max": 100,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GPU power now",
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"decimals": 1,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GT frequency now",
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"decimals": 0,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Render/3D busy",
"description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Video busy",
"description": "Fixed-function decode/encode (QuickSync).",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "VideoEnhance busy",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Blitter busy",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GT frequency (actual)",
"description": "0 while the GT sits in RC6.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GPU power",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Package power",
"description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Memory controller bandwidth",
"description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "MiBs",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
{ "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_imc_read{node=~\"$node\"}",
"legendFormat": "{{node}} read",
"refId": "A"
},
{
"expr": "gpumon_imc_write{node=~\"$node\"}",
"legendFormat": "{{node}} write",
"refId": "B"
}
]
}
]
}
@@ -77,6 +77,9 @@ configMapGenerator:
- name: dashboard-infrastructure-overview
files:
- infrastructure-overview.json
- name: dashboard-intel-gpu
files:
- intel-gpu.json
- name: dashboard-netbird-overview
files:
- netbird-overview.json
@@ -0,0 +1,81 @@
# Intel iGPU utilisation, frequency and power for every node (#752).
#
# Not privileged: /dev/dri comes from the device plugin's monitoring resource
# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
# hostNetwork.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
template:
metadata:
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
# Every node is a control plane; without this the exporter covers nothing.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: intel-gpu-exporter
image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
args:
# No -device: intel_gpu_top picks the only GPU the plugin injects.
- -interval=5s
ports:
- name: http-metrics
containerPort: 9100
protocol: TCP
securityContext:
# Root so the PERFMON capability is effective without file
# capabilities on the binary.
runAsUser: 0
runAsNonRoot: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["PERFMON"]
resources:
requests:
cpu: 10m
memory: 32Mi
gpu.intel.com/monitoring: 1
limits:
memory: 128Mi
gpu.intel.com/monitoring: 1
livenessProbe:
httpGet:
path: /metrics
port: http-metrics
initialDelaySeconds: 15
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
clusterIP: None
selector:
app.kubernetes.io/name: intel-gpu-exporter
ports:
- name: http-metrics
port: 9100
targetPort: http-metrics
protocol: TCP
@@ -16,6 +16,7 @@ resources:
- prometheus-gate-ingress.yaml
- ci-diagnostics-rbac.yaml
- smartctl-exporter.yaml
- intel-gpu-exporter.yaml
- dashboards/
# prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
# separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -72,6 +72,9 @@ spec:
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
# Advertises gpu.intel.com/monitoring: one resource that grants a
# pod every GPU on the node instead of a slot. The exporter needs it.
- -enable-monitoring
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
@@ -76,7 +76,11 @@ spec:
type: RuntimeDefault
containers:
- name: controller
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07
# Pinned by digest: the date tag is rebuilt on every push of the
# image directory (branch and master alike) and the registry keeps
# only the newest manifest per tag, so the digest must come from
# the MASTER build of the code change.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
@@ -25,10 +25,10 @@ spec:
name: openbao
namespace: openbao
install:
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` + unseal
# (readiness probe reports sealed). helm --wait would time out and
# remediation would uninstall the release, so don't wait here; the
# init/unseal procedure is manual (see kubernetes/MIGRATION-sops-openbao.md).
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` (readiness
# reports sealed). helm --wait would time out and remediation would
# uninstall the release, so don't wait here; init is manual (see
# kubernetes/MIGRATION-sops-openbao.md). Unseal is automatic (static seal).
disableWait: true
remediation:
retries: 3
@@ -42,6 +42,11 @@ spec:
# the agent-injector mutating webhook is unused surface.
enabled: false
server:
# The static seal key, from the openbao-seal-key Secret (bootstrap.sh).
extraSecretEnvironmentVars:
- envName: BAO_SEAL_KEY
secretName: openbao-seal-key
secretKey: key
# Raft (not standalone/file) for two reasons: `bao operator raft
# snapshot` as a consistent backup primitive, and real HA across three
# nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
@@ -53,16 +58,8 @@ spec:
# the moment one pod is sealed or rescheduled, which is worse than
# the single replica it replaced. Three tolerates one.
#
# What this buys, and what it does not: a node drain seals the replica
# on it, and raft elects a new leader from the remaining standbys
# within seconds, so the service does not go down. It does NOT remove
# the manual unseal — a sealed server can neither serve requests nor
# act as standby, so that replica stays out until someone unseals it.
#
# The consequence for rolling node maintenance is worth stating: a
# Talos upgrade drains all three nodes in turn, sealing each replica
# as it goes. Unseal each one as it comes back, or the third drain
# takes the last unsealed replica with it and quorum is lost anyway.
# A restarted replica unseals itself from the static seal and rejoins
# raft, so a rolling reboot only ever costs one replica at a time.
replicas: 3
raft:
enabled: true
@@ -111,6 +108,14 @@ spec:
service_registration "kubernetes" {}
# Auto-unseal. The key is a 32-byte AES key in the openbao-seal-key
# Secret, created from pass at bootstrap before this release. The
# former shamir key is the recovery key (#805).
seal "static" {
current_key_id = "1"
current_key = "env://BAO_SEAL_KEY"
}
audit "file" "stdout" {
description = "audit trail to pod stdout -> Loki"
options {