Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 8d348e7f88 fix(monitoring): let intel_gpu_top pick the GPU instead of hard-coding card0
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:15:04 +02:00
claude-botandClaude Fable 5.1 6f9ff9d624 feat(monitoring): export Intel iGPU utilisation, frequency and power
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.

Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.

End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.

XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.

Fixes #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:14:24 +02:00
k.renner 7bf65783be Merge pull request 'feat(intel-gpu-plugin): advertise the monitoring resource' (#820) from feat/intel-gpu-monitoring-resource into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#820
2026-09-08 04:54:15 +00:00
k.renner dd9fb01359 Merge pull request 'fix(reloader): restart audiobookshelf and immich-server on ConfigMap change' (#819) from fix/reloader-configmap-annotations into master
Reviewed-on: k.renner/homelab#819
2026-09-08 04:52:32 +00:00
claude-botandClaude Fable 5.1 da8a725b32 feat(intel-gpu-plugin): advertise the monitoring resource
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.

Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:51:43 +02:00
claude-botandClaude Fable 5.1 6008d59c9f fix(reloader): restart audiobookshelf and immich-server on ConfigMap change
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.

ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.

Reloader already watches all namespaces (kubernetes/infrastructure/reloader).

Fixes #742

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:46:12 +02:00
k.renner 5aa6c0609d Merge pull request 'fix(netbird-proxy-services): pin the digest the master build produced' (#818) from netbird-proxy-protocol-digest into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#818
2026-09-07 11:52:17 +00:00
claude-botandClaude Fable 5.1 0ebe113abb fix(netbird-proxy-services): pin the digest the master build produced
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.

The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:47:22 +02:00
k.renner 1723f73b5d Merge pull request 'feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)' (#817) from netbird-proxy-protocol-deploy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#817
2026-09-07 11:34:52 +00:00
k.renner 8d8df3b635 Merge pull request 'feat(netbird): carry the WAN client address through the passthrough edge (1/2)' (#816) from netbird-proxy-protocol into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#816
2026-09-07 11:34:43 +00:00
8 changed files with 116 additions and 2 deletions
@@ -21,6 +21,9 @@ metadata:
namespace: audiobookshelf
labels:
app.kubernetes.io/name: audiobookshelf
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: audiobookshelf-config
spec:
replicas: 1
strategy:
+3
View File
@@ -35,6 +35,9 @@ metadata:
namespace: immich
labels:
app.kubernetes.io/name: immich-server
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: immich-config
spec:
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -0,0 +1,20 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
relabelings:
# The series would otherwise carry only the exporter pod's IP.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -39,4 +39,5 @@ resources:
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- intel-gpu-exporter-servicemonitor.yaml
- probes-blackbox.yaml
@@ -0,0 +1,81 @@
# Intel iGPU utilisation, frequency and power for every node (#752).
#
# Not privileged: /dev/dri comes from the device plugin's monitoring resource
# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
# hostNetwork.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
template:
metadata:
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
# Every node is a control plane; without this the exporter covers nothing.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: intel-gpu-exporter
image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
args:
# No -device: intel_gpu_top picks the only GPU the plugin injects.
- -interval=5s
ports:
- name: http-metrics
containerPort: 9100
protocol: TCP
securityContext:
# Root so the PERFMON capability is effective without file
# capabilities on the binary.
runAsUser: 0
runAsNonRoot: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["PERFMON"]
resources:
requests:
cpu: 10m
memory: 32Mi
gpu.intel.com/monitoring: 1
limits:
memory: 128Mi
gpu.intel.com/monitoring: 1
livenessProbe:
httpGet:
path: /metrics
port: http-metrics
initialDelaySeconds: 15
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
clusterIP: None
selector:
app.kubernetes.io/name: intel-gpu-exporter
ports:
- name: http-metrics
port: 9100
targetPort: http-metrics
protocol: TCP
@@ -16,6 +16,7 @@ resources:
- prometheus-gate-ingress.yaml
- ci-diagnostics-rbac.yaml
- smartctl-exporter.yaml
- intel-gpu-exporter.yaml
- dashboards/
# prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
# separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -72,6 +72,9 @@ spec:
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
# Advertises gpu.intel.com/monitoring: one resource that grants a
# pod every GPU on the node instead of a slot. The exporter needs it.
- -enable-monitoring
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
@@ -77,8 +77,10 @@ spec:
containers:
- name: controller
# Pinned by digest: the date tag is rebuilt on every push of the
# image directory, so the tag alone cannot express a new build.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:0d266b0288aec692c698ff5c05f76c826e247bfef365e0306a0b9a6541ebf180
# image directory (branch and master alike) and the registry keeps
# only the newest manifest per tag, so the digest must come from
# the MASTER build of the code change.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true