claude-botandClaude Opus 5.5
5e115cde11
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki behind them are bound to the Intel iGPU. On a site without one (Hetzner, restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin and exporter DaemonSets sat on every node with nothing to serve. The node patches k8s-1..3 label their machines hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the three llama Deployments and the wiki now select on it. Elsewhere the DaemonSets have no pods and the Deployments stay unscheduled. Production rollout order: the label must be on the nodes (talosctl patch, no reboot) before these manifests reconcile, or the GPU plugin and the llama/wiki pods are removed or left Pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
174 lines
6.7 KiB
YAML
174 lines
6.7 KiB
YAML
---
|
|
# The embedding endpoint, for RAG and semantic search.
|
|
#
|
|
# Deliberately separate from llama-server and deliberately not behind the
|
|
# ingress: an embedding model has no generative training, and when it shares a
|
|
# port with the chat model the built-in WebUI lists it as a choice. Selecting it
|
|
# returns fluent nonsense rather than an error, because `--embeddings` does not
|
|
# actually refuse chat completions. Keeping it on its own ClusterIP-only Service
|
|
# is what makes it unselectable, since llama.cpp has no way to hide a model from
|
|
# the picker.
|
|
#
|
|
# Reachable in-cluster at http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings.
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: llama-embed
|
|
namespace: ai
|
|
labels:
|
|
app.kubernetes.io/name: llama-embed
|
|
spec:
|
|
replicas: 1
|
|
# Same reasoning as llama-server: the startupProbe budget (600s) must stay
|
|
# below progressDeadlineSeconds, whose default is also 600 — exactly equal, so
|
|
# any cold download that uses most of its budget trips
|
|
# KubeDeploymentRolloutStuck on a rollout that is still progressing normally.
|
|
progressDeadlineSeconds: 900
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: llama-embed
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: llama-embed
|
|
spec:
|
|
# Runs only where the node patch declares the Intel iGPU.
|
|
nodeSelector:
|
|
hardware.renner.uno/intel-gpu: "true"
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
fsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: llama-embed
|
|
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731@sha256:0262343f64f030e8efff15ec05ee8180b69b7f6745de2caa3af5148e0dcb8aba
|
|
args:
|
|
- --hf-repo
|
|
- gpustack/bge-m3-GGUF:Q8_0
|
|
# Serves POST /v1/embeddings.
|
|
- --embeddings
|
|
# A short, stable id for consumers. Without it the model is
|
|
# advertised as the whole --hf-repo string, which would end up
|
|
# verbatim in Paperless's config and break on the next model swap.
|
|
# One name only: llama-server keeps aliases in a sorted set and
|
|
# reports the first as the id, so a second name can steal it.
|
|
- --alias
|
|
- bge-m3
|
|
# bge-m3 is trained for CLS pooling — a different type from the
|
|
# Qwen family this replaces, which uses last-token. Left unspecified
|
|
# llama-server takes the GGUF default, which is not visible from
|
|
# outside and would silently produce vectors the model was not
|
|
# trained to emit. Read from the model's own 1_Pooling/config.json
|
|
# rather than assumed; of the candidates measured, three different
|
|
# pooling types were needed.
|
|
- --pooling
|
|
- cls
|
|
- --n-gpu-layers
|
|
- "99"
|
|
- --threads
|
|
- "2"
|
|
# Paperless chunks at 1024 tokens, but counts them with its own
|
|
# tokenizer while this server measures the batch in bge-m3's.
|
|
# Those disagree: XLM-RoBERTa splits German into more pieces than
|
|
# llama_index's counter does, so a chunk Paperless considers full
|
|
# at 1024 arrived here as 1069 and the request failed outright —
|
|
# "input (1069 tokens) is too large to process".
|
|
#
|
|
# The headroom therefore has to absorb a tokenizer ratio, not just
|
|
# a rounding error, and 1.5x covers far more than the 1.044 that
|
|
# broke it. The defaults are batch 2048 / ubatch 512, and with
|
|
# --embeddings llama-server clamps both to the smaller rather than
|
|
# run them mismatched, so both are set.
|
|
#
|
|
# 2048 stays avoided: that is where the chat model met
|
|
# ErrorDeviceLost on this Vulkan stack. That failure came under a
|
|
# 20 GB model's memory pressure and may not bind a 600 MB one, but
|
|
# this does not need to find out.
|
|
- --batch-size
|
|
- "1536"
|
|
- --ubatch-size
|
|
- "1536"
|
|
# Room for a full chunk plus overhead rather than exactly one.
|
|
- --ctx-size
|
|
- "4096"
|
|
# No browser ever talks to this Service, so the WebUI is dead weight
|
|
# and one more unauthenticated surface.
|
|
- --no-webui
|
|
env:
|
|
# Same cache as llama-server, so the two share one copy of anything
|
|
# they have in common and neither re-downloads the other's weights.
|
|
- name: LLAMA_CACHE
|
|
value: /models
|
|
ports:
|
|
- name: http
|
|
containerPort: 8080
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
# Measured, not estimated: embedding 400 real documents peaked at
|
|
# 1.6 GiB (Prometheus max_over_time on the working set). The limit is
|
|
# well above that because the footprint is dominated by the four slots
|
|
# at 4096 context and the Vulkan working set rather than by the
|
|
# weights — the Qwen model this replaces peaked at 10.6 GiB on the
|
|
# same settings for the same 600 MiB of weights, and several
|
|
# candidates were OOMKilled at limits that looked generous against
|
|
# their model size.
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 2Gi
|
|
gpu.intel.com/i915: "1"
|
|
limits:
|
|
cpu: "2"
|
|
memory: 8Gi
|
|
gpu.intel.com/i915: "1"
|
|
# ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
|
|
# model needs, but still sized for a cold download.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: http
|
|
periodSeconds: 10
|
|
failureThreshold: 60
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: http
|
|
periodSeconds: 10
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: http
|
|
periodSeconds: 30
|
|
timeoutSeconds: 10
|
|
failureThreshold: 6
|
|
volumeMounts:
|
|
- name: models
|
|
mountPath: /models
|
|
volumes:
|
|
- name: models
|
|
persistentVolumeClaim:
|
|
claimName: llama-server-models
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: llama-embed
|
|
namespace: ai
|
|
labels:
|
|
app.kubernetes.io/name: llama-embed
|
|
spec:
|
|
selector:
|
|
app.kubernetes.io/name: llama-embed
|
|
ports:
|
|
- name: http
|
|
port: 8080
|
|
targetPort: http
|