feat(gpu): iGPU workloads run only where a node declares the iGPU
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki behind them are non-essential and bound to the Intel iGPU. On a site without one (Hetzner, restore rehearsal runs 27/28) their pods stayed Pending, llama-server failed its health check and the Kustomizations held kustomize-controller workers; the GPU plugin and exporter DaemonSets sat on every node with nothing to serve. The node patches k8s-1..3 label their machines hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the three llama Deployments and the wiki select on it, and llama-server and wiki are no longer waited on. Elsewhere the DaemonSets have no pods and the Deployments stay unscheduled without blocking anything. Production rollout order: the label must be on the nodes (talosctl patch, no reboot) before these manifests reconcile, or the GPU plugin and the llama/wiki pods are removed or left Pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -44,6 +44,9 @@ spec:
|
||||
labels:
|
||||
app.kubernetes.io/name: llama-server
|
||||
spec:
|
||||
# Runs only where the node patch declares the Intel iGPU.
|
||||
nodeSelector:
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
@@ -34,6 +34,9 @@ spec:
|
||||
labels:
|
||||
app.kubernetes.io/name: llama-embed
|
||||
spec:
|
||||
# Runs only where the node patch declares the Intel iGPU.
|
||||
nodeSelector:
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
@@ -23,6 +23,9 @@ spec:
|
||||
labels:
|
||||
app.kubernetes.io/name: wiki-embed
|
||||
spec:
|
||||
# Runs only where the node patch declares the Intel iGPU.
|
||||
nodeSelector:
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
|
||||
@@ -25,6 +25,8 @@ spec:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
securityContext:
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
@@ -39,6 +39,10 @@ spec:
|
||||
labels:
|
||||
app.kubernetes.io/name: wiki
|
||||
spec:
|
||||
# Its embedding backend (wiki-embed) needs the Intel iGPU; without it
|
||||
# the wiki has nothing to serve, so it runs where the iGPU is declared.
|
||||
nodeSelector:
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
|
||||
@@ -15,7 +15,10 @@ spec:
|
||||
name: flux-system
|
||||
path: ./kubernetes/apps/llama-server
|
||||
prune: true
|
||||
wait: true
|
||||
# Non-essential, and its pods run only on nodes with the Intel iGPU
|
||||
# (hardware.renner.uno/intel-gpu); on a site without one they stay
|
||||
# Pending, which must not hold a reconcile worker or any dependent.
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: infrastructure
|
||||
- name: infrastructure-crds
|
||||
|
||||
@@ -13,7 +13,8 @@ spec:
|
||||
name: flux-system
|
||||
path: ./kubernetes/apps/wiki
|
||||
prune: true
|
||||
wait: true
|
||||
# Non-essential and iGPU-bound like llama-server: not waited on.
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: infrastructure
|
||||
- name: infrastructure-crds
|
||||
|
||||
@@ -119,3 +119,4 @@ spec:
|
||||
type: DirectoryOrCreate
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: amd64
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
|
||||
@@ -54,6 +54,9 @@ machine:
|
||||
# this label instead of listing hostnames, so which boxes carry storage is
|
||||
# stated once, here, by the machine that knows.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# This machine has the Intel iGPU. The GPU plugin and everything that needs
|
||||
# gpu.intel.com run only on nodes with this label.
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
# Publishes this node's pod0 addresses, declared below in its VLANConfig, so
|
||||
# the bird DaemonSet reads its peers from the node list rather than from a
|
||||
# copy. Annotations, not labels: a label value cannot contain a colon.
|
||||
|
||||
@@ -34,6 +34,8 @@ machine:
|
||||
topology.kubernetes.io/zone: saturn
|
||||
# Holds a Ceph OSD drive — see k8s-1.yaml.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# Has the Intel iGPU — see k8s-1.yaml.
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
# pod0 addresses for the bird peer list — see k8s-1.yaml.
|
||||
nodeAnnotations:
|
||||
net.renner.uno/pod0-v6: fd02:48::12
|
||||
|
||||
@@ -30,6 +30,8 @@ machine:
|
||||
topology.kubernetes.io/zone: neptune
|
||||
# Holds a Ceph OSD drive — see k8s-1.yaml.
|
||||
storage.renner.uno/ceph-osd: "true"
|
||||
# Has the Intel iGPU — see k8s-1.yaml.
|
||||
hardware.renner.uno/intel-gpu: "true"
|
||||
# pod0 addresses for the bird peer list — see k8s-1.yaml.
|
||||
nodeAnnotations:
|
||||
net.renner.uno/pod0-v6: fd02:48::13
|
||||
|
||||
Reference in New Issue
Block a user