feat(llama-server): split embeddings off, move chat to Qwen3.6-35B-A3B

Two defects surfaced once the endpoint was live, and both trace back to router mode. The embedding model was selectable in the chat UI. Every model on the port appears in the picker, and choosing the embedding one returns fluent nonsense ("capital capital capital ...") rather than an error, because --embeddings does not actually refuse chat completions. llama.cpp offers no way to hide a model from the list: --tags is informational, --ui-config only sets UI preferences. So the embedding model moves to its own Deployment on a ClusterIP-only Service with --no-webui, which is what keeps it out of the picker. Readiness was also wrong. The router answers /health 200 as soon as it listens, while models still download behind it, so the pod went Ready seconds after start and traffic arrived at a server that could not answer — that is how the embedding model got reached at all. A single-model server returns 503 until the weights are resident, which is what the probes needed all along. Splitting fixes both, at the cost of router mode's single endpoint. Consumers now use llama-server for chat and llama-embed for embeddings; both share one PVC as LLAMA_CACHE, so nothing is downloaded twice. The chat model changes to a mixture-of-experts. Llama-3.1-8B was picked only because it was the model benchmarked, which is continuity rather than a quality argument, and it showed: fluent but wrong. Since decode is set by bytes read per token, an MoE reading ~1.85 GB/token beats a dense model reading ~16.5 GB. Measured on k8s-1 with this image: tg128 11.59 tok/s (dense 8B: 10.78) pp512 120.93 t/s pp2048 126.47 t/s pp8192 92.79 t/s pp16384 69.84 t/s Four times the model, slightly faster. The prefill falloff is sublinear, so the Gated Delta Net collapse of llama.cpp#27638 does not occur here and long documents stay viable; context therefore goes to 16k, which is cheap because only 10 of 40 layers use full attention. GLM-5.2/5.3 were evaluated and are impossible on this hardware at 754B parameters (~400 GB), as is GLM-5.3-Flash at 321B with an architecture the pinned build does not implement. Dense Qwen3.8-27B reads 16.5 GB/token and manages 3.3 tok/s. Memory goes to 24Gi/32Gi: Vulkan allocates the iGPU working set from system RAM and it counts against the pod cgroup, measured at 7.54 GiB RSS for 5.2 GiB of weights. --ubatch-size 1024 because 2048 and above fail with ErrorDeviceLost on this Vulkan/Mesa stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'feat(llama-server): local LLM inference on the Intel iGPU' (#721 ) from llama-server into master
2026-09-01 16:25:04 +02:00 · 2026-09-01 12:31:13 +00:00 · 2026-09-01 14:25:21 +02:00 · 2026-09-01 14:24:54 +02:00 · 2026-09-01 14:24:54 +02:00 · 2026-09-01 10:44:44 +00:00
76 changed files with 3006 additions and 3009 deletions
@@ -87,30 +87,6 @@ steps:
          --destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest

-  - name: build-vmtest
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/vmtest
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/vmtest/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
-
  - name: build-tofu
    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/tofu
@@ -171,28 +147,6 @@ steps:
  # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
-  #
-  # Published THREE times, and the third one is not a typo. Its consumers — the
-  # zigbee2mqtt seed-data init container and the trmnl renderer — start while
-  # the cluster is still being rebuilt, and at that moment gitea.renner.uno does
-  # not exist, because Gitea runs inside the cluster being built. A cold-rebuild
-  # rehearsal caught both in ImagePullBackOff.
-  #
-  # So there is a copy on ghcr.io. But the manifests must NOT point at it:
-  # normal operation pulls from the local registry, and an external reference
-  # would make every steady-state pull leave the network. The recovery source
-  # and the everyday source are different things, exactly as they are for Flux's
-  # git source.
-  #
-  # Talos resolves that with machine.registries.mirrors, which takes an ordered
-  # endpoint list — gitea first, ghcr second (bootstrap.sh). Containerd keeps
-  # the repository path when it falls through, so the two registries have to
-  # agree on it: `gitea_admin/alpine-jq` cannot be served by ghcr, where the
-  # namespace is fixed to the account name. Hence the second Gitea destination
-  # under konairius/, which the manifests reference and which matches the ghcr
-  # path exactly.
-  #
-  # The gitea_admin/ copy stays until nothing references it any more.
  - name: build-alpine-jq
    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-jq
@@ -205,47 +159,23 @@ steps:
      GITEA_USER: gitea_admin
      GITEA_TOKEN:
        from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
    commands:
      - mkdir -p /kaniko/.docker
      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
      - TAG=$(date +%Y-%m-%d)
      - |
        /kaniko/executor \
          --context . \
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
-          --destination gitea.renner.uno/konairius/alpine-jq:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-jq:latest \
-          --destination ghcr.io/konairius/alpine-jq:$TAG \
-          --destination ghcr.io/konairius/alpine-jq:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest

  # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
  # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
  # node-side fetch from GitHub that had NO checksum verification and sat on the
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
-  #
-  # This one ALSO publishes to ghcr.io. A cold-rebuild rehearsal proved why: the
-  # DaemonSet could not pull from gitea.renner.uno, because Gitea runs inside the
-  # very cluster being rebuilt. Without the plugins there is no macvlan, so
-  # Multus cannot attach ceph-san, so Rook's network canary never starts, so the
-  # CephCluster sits at "Configuring Ceph Mons" — and OpenBao waits forever for a
-  # PVC that will never bind. Moving the download here fixed the integrity gap
-  # but quietly swapped github.com on the critical path for something strictly
-  # worse: a registry that by definition does not exist during a recovery. The
-  # GHCR copy is public, so a bare node can pull it with no credential and no
-  # Gitea.
-  #
-  # The test that proved this also proved the rule is not specific to this
-  # image. alpine-jq and alpine-openssl mirror for the same reason — see the
-  # note on build-alpine-jq. Anything an app needs while the cluster is still
-  # being built cannot live only in that cluster's own registry.
  - name: build-cni-plugins
    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/cni-plugins
@@ -258,36 +188,23 @@ steps:
      GITEA_USER: gitea_admin
      GITEA_TOKEN:
        from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
    commands:
      - mkdir -p /kaniko/.docker
      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
      - TAG=$(date +%Y-%m-%d)
      - |
        /kaniko/executor \
          --context . \
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
-          --destination gitea.renner.uno/konairius/cni-plugins:$TAG \
-          --destination gitea.renner.uno/konairius/cni-plugins:latest \
-          --destination ghcr.io/konairius/cni-plugins:$TAG \
-          --destination ghcr.io/konairius/cni-plugins:latest
+          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest

  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
  # needing s_client/x509. Named for what it is; first consumer is the paperless
  # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
-  #
-  # Mirrored to ghcr.io for the reason given on build-alpine-jq: it is an init
-  # container, so paperless does not start without it, and during a rebuild
-  # gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
-  # paperless in Init:ErrImagePull on exactly this image.
  - name: build-alpine-openssl
    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
    directory: ci/images/alpine-openssl
@@ -300,25 +217,17 @@ steps:
      GITEA_USER: gitea_admin
      GITEA_TOKEN:
        from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
    commands:
      - mkdir -p /kaniko/.docker
      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
      - TAG=$(date +%Y-%m-%d)
      - |
        /kaniko/executor \
          --context . \
          --dockerfile Containerfile \
          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
-          --destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-openssl:latest \
-          --destination ghcr.io/konairius/alpine-openssl:$TAG \
-          --destination ghcr.io/konairius/alpine-openssl:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest

  # Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
  # run as two services on edge-1. Replaces pip-into-venv on the host, where an
@@ -1,84 +0,0 @@
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Everything `bootstrap.sh vmtest` needs, so the rebuild test can run as a Job in
-# the cluster instead of on a workstation. The workstation has 31G and the test
-# wants ~14.5G for four VMs — a single run left under 1G headroom, and two
-# overlapping runs OOM-killed the session. The nodes have ~94G with 70G free.
-#
-# Separate from ci-k8s rather than built on it: that image pins talosctl v1.12.6,
-# and this test creates a cluster at the version talos/patches/cluster.yaml
-# installs. A talosctl older than the Talos it provisions is exactly the kind of
-# skew the test exists to catch, so it must not be baked in here.
-#
-# KEEP TALOS_VERSION IN STEP with the installer image tag in
-# kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
-# file at run time and passes it to the provisioner; this pin only decides which
-# client talks to it.
-ARG TALOS_VERSION=v1.13.9
-ARG KUBECTL_VERSION=v1.36.2
-ARG FLUX_VERSION=2.9.2
-ARG KOPIA_VERSION=0.23.1
-
-# qemu-system-x86_64 and qemu-img are the provisioner's actual backend, and ovmf
-# supplies the UEFI firmware those VMs boot from — without it the provisioner
-# stops with "the required flash image was not found in any of the expected
-# paths", having already downloaded the ISO and generated the PKI. Note the
-# firmware comes from ovmf specifically: qemu-system-x86_64 ships an edk2 CODE
-# image but no VARS, which is the half the provisioner needs.
-#
-# iptables is shelled out to by the CNI plugins that build the VM network; the
-# provisioner fails with "error accessing iptables" without it. The rest is
-# what the phase shells out to: helm for the Cilium install, yq to read the
-# HelmRelease and the installer image, jq for OpenBao's seal status.
-#
-# Versions come from the digest-pinned base rather than being pinned one by one,
-# the same way ci-ansible does it.
-RUN apk add --no-cache \
-      bash \
-      ca-certificates \
-      curl \
-      git \
-      helm \
-      iptables \
-      jq \
-      openssl \
-      ovmf \
-      qemu-img \
-      qemu-system-x86_64 \
-      yq
-
-RUN curl -sfL -o /usr/local/bin/talosctl \
-      "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
- && chmod +x /usr/local/bin/talosctl
-
-RUN curl -sfL -o /usr/local/bin/kubectl \
-      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
- && curl -sfL -o /tmp/kubectl.sha256 \
-      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl.sha256" \
- && echo "$(cat /tmp/kubectl.sha256)  /usr/local/bin/kubectl" | sha256sum -c - \
- && rm -f /tmp/kubectl.sha256 \
- && chmod +x /usr/local/bin/kubectl
-
-RUN curl -sfL -o /tmp/flux.tar.gz \
-      "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
- && echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3  /tmp/flux.tar.gz" | sha256sum -c - \
- && tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
- && chmod +x /usr/local/bin/flux \
- && rm -f /tmp/flux.tar.gz
-
-# kopia reads the Velero backup repository directly, which is what the OpenBao
-# restore does when there is no Velero yet to do it: the credentials Velero
-# needs live in OpenBao, so OpenBao cannot be restored by Velero. Its absence
-# was found by a rebuild rehearsal reaching the restore for the first time and
-# stopping at "kopia not found".
-#
-# Same checksum-verified download as the tools above; the tarball unpacks into
-# a versioned directory, so the binary is pulled out by name.
-RUN curl -sfL -o /tmp/kopia.tar.gz \
-      "https://github.com/kopia/kopia/releases/download/v${KOPIA_VERSION}/kopia-${KOPIA_VERSION}-linux-x64.tar.gz" \
- && echo "416d0f84a3dbb321a8b2d8f0997b1a0a6e915babe79ee76fa6e4d2bd1e1c5178  /tmp/kopia.tar.gz" | sha256sum -c - \
- && tar -xzf /tmp/kopia.tar.gz -C /tmp \
- && mv "/tmp/kopia-${KOPIA_VERSION}-linux-x64/kopia" /usr/local/bin/kopia \
- && chmod +x /usr/local/bin/kopia \
- && rm -rf /tmp/kopia.tar.gz "/tmp/kopia-${KOPIA_VERSION}-linux-x64"
@@ -119,9 +119,11 @@ against do not exist, and the node ends up with no address on any interface.
 Beyond that it carries:

 - `siderolabs/intel-ucode` — no hypervisor supplies microcode here
- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
-  today; present so wanting hardware transcoding later is a config change
-  rather than another rolling reinstall
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. The Intel GPU device plugin
+  (`infrastructure/intel-gpu-plugin`) turns this into the schedulable resource
+  `gpu.intel.com/i915`, which `apps/llama-server` requests for LLM inference
+  and which any hardware-transcoding workload would use too. Without the
+  extension the plugin finds no `/dev/dri` and advertises nothing
 - `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
  rule auto-authorises any Thunderbolt device attached once those modules load
@@ -15,6 +15,7 @@ resources:
  - alertmanager-gate-egress.yaml
  - prometheus-gate-egress.yaml
  - wordpress-gate-egress.yaml
+  - outpost-servicemonitors.yaml
  - ldap-outpost-cert.yaml
  - ldap-outpost-lb.yaml
  - ldap-outpost-cnp.yaml
@@ -57,14 +57,7 @@ spec:
          # and leaves this one untouched (tags are not deleted), so this keeps
          # resolving — but the image then came from the branch build rather than
          # master. Re-point it at a master-built tag when convenient.
-          #
-          # konairius/, not gitea_admin/: this init container runs during a cold
-          # rebuild, when Gitea does not exist because it runs inside the
-          # cluster being rebuilt. The node then falls back to a ghcr.io copy
-          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
-          # keeps the repository path when it falls through — so the path has to
-          # exist on both registries. Steady state still pulls from Gitea.
-          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
          command:
            - sh
            - -c
@@ -0,0 +1,164 @@
+---
+# The chat endpoint, and the only one behind the ingress.
+#
+# Single-model rather than router mode. Router mode served chat and embeddings
+# from one port, but two things made that wrong here:
+#
+#   1. Its /health answers 200 as soon as the router process listens, while the
+#      models are still downloading behind it. Kubernetes marked the pod Ready
+#      within seconds of start, so traffic arrived at a server that could not
+#      answer. In single-model mode /health returns 503 until the model is
+#      resident, which is what the probes below actually need.
+#   2. Every model on the port is selectable in the built-in WebUI, including
+#      the embedding model. Asking an embedding model to chat returns fluent
+#      nonsense ("capital capital capital ...") rather than an error, and
+#      `--embeddings` does not prevent it. There is no flag to hide a model
+#      from the list — `--tags` is informational and `--ui-config` only sets UI
+#      preferences — so the only way to keep it out of the picker is to keep it
+#      off this endpoint. It runs in embed.yaml instead.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llama-server
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-server
+spec:
+  replicas: 1
+  # A rolling update would load a second copy of the weights onto the same
+  # iGPU and into the same node's RAM before releasing the first.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: llama-server
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: llama-server
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        # Makes the CephFS volume group-writable so the weights can be cached.
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: llama-server
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            # The one place this model is named; llama-server resolves the repo
+            # and downloads it itself, so there is nothing to keep in sync.
+            #
+            # A 35B mixture-of-experts with only ~3B parameters active per
+            # token. Decode on this hardware is bound by how many bytes are read
+            # per token rather than by parameter count, and this model reads
+            # ~1.85 GB/token where a dense 27B reads ~16.5 GB — so it is both
+            # far larger and faster than any dense model that would fit.
+            #
+            # Measured here (llama-bench, k8s-1, this image): tg128 11.59 tok/s,
+            # against 10.78 for a dense 8B. Prefill 120.9 t/s at 512 tokens and
+            # 69.8 at 16384 — a sublinear falloff, so the Gated Delta Net
+            # prefill collapse of llama.cpp#27638 does not occur on this stack.
+            - --hf-repo
+            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
+            # Every layer on the Intel iGPU. Full offload matches CPU throughput
+            # while leaving the node's 14 cores free for the rest of the
+            # cluster; the iGPU is here for that isolation, not for a speed-up.
+            - --n-gpu-layers
+            - "99"
+            # Decode runs on the GPU, so extra threads buy nothing. At 11.59
+            # tok/s this model moves 21.4 GB/s against the 53.9 GB/s a dense
+            # model sustains here, so it is the iGPU's 96 EUs working through
+            # many small per-expert matmuls that sets the pace, not bandwidth.
+            - --threads
+            - "4"
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
+            - --ctx-size
+            - "16384"
+            # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
+            # with ErrorDeviceLost.
+            - --ubatch-size
+            - "1024"
+            # The server otherwise allows every CORS origin and warns about it
+            # at startup. Without this, any page a LAN browser visits could call
+            # this unauthenticated API from the user's machine. Consumers talk
+            # to it server-side; the WebUI is same-origin and unaffected.
+            - --cors-origins
+            - localhost
+          env:
+            # Where --hf-repo caches the weights. Pointing it at the PVC makes
+            # the download a one-time cost, shared with llama-embed, and keeps
+            # the server off a home directory it cannot write as UID 1000.
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          # Vulkan allocates the iGPU's working set from system RAM, and that
+          # counts against the pod's cgroup: a 5.2 GiB model measured 7.54 GiB
+          # RSS. Sized as the 19 GiB of weights plus that overhead, plus the KV
+          # cache — which is unusually cheap here, since only 10 of the 40
+          # layers use full attention.
+          resources:
+            requests:
+              cpu: 500m
+              memory: 24Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "6"
+              memory: 32Gi
+              gpu.intel.com/i915: "1"
+          # /health is 503 while the model loads and 200 once it is resident, so
+          # the pod only goes Ready when it can actually answer. On a cold PVC
+          # that includes pulling ~19 GiB from Hugging Face, hence the long
+          # startup budget (40 min); liveness takes over only after that.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 240
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llama-server
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-server
+spec:
+  selector:
+    app.kubernetes.io/name: llama-server
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
@@ -0,0 +1,119 @@
+---
+# The embedding endpoint, for RAG and semantic search.
+#
+# Deliberately separate from llama-server and deliberately not behind the
+# ingress: an embedding model has no generative training, and when it shares a
+# port with the chat model the built-in WebUI lists it as a choice. Selecting it
+# returns fluent nonsense rather than an error, because `--embeddings` does not
+# actually refuse chat completions. Keeping it on its own ClusterIP-only Service
+# is what makes it unselectable, since llama.cpp has no way to hide a model from
+# the picker.
+#
+# Reachable in-cluster at http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: llama-embed
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: llama-embed
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: llama-embed
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            - --hf-repo
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            # Serves POST /v1/embeddings.
+            - --embeddings
+            - --n-gpu-layers
+            - "99"
+            - --threads
+            - "2"
+            - --ctx-size
+            - "2048"
+            # No browser ever talks to this Service, so the WebUI is dead weight
+            # and one more unauthenticated surface.
+            - --no-webui
+          env:
+            # Same cache as llama-server, so the two share one copy of anything
+            # they have in common and neither re-downloads the other's weights.
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 250m
+              memory: 2Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "2"
+              memory: 4Gi
+              gpu.intel.com/i915: "1"
+          # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
+          # model needs, but still sized for a cold download.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 60
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  selector:
+    app.kubernetes.io/name: llama-embed
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
@@ -0,0 +1,28 @@
+---
+# LAN-only: no `dns.renner.uno/public` label, so the Cloudflare external-dns
+# instance skips it and only the UDM instance publishes the record. The API is
+# unauthenticated, so it must not be reachable from the internet.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: llama-server
+  namespace: ai
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - llm.renner.uno
+      secretName: llama-server-tls
+  rules:
+    - host: llm.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: llama-server
+                port:
+                  number: 8080
@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - pvc.yaml
+  - deployment.yaml
+  - embed.yaml
+  - ingress.yaml
@@ -0,0 +1,21 @@
+---
+# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
+# Prune-disabled so retiring this Kustomization does not throw away several
+# GiB of model files that would have to be pulled from Hugging Face again.
+#
+# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
+# same weights; the models are read-only at inference time.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: llama-server-models
+  namespace: ai
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem
+  resources:
+    requests:
+      storage: 40Gi
@@ -1,46 +0,0 @@
-# Cilium's agent ServiceMonitor, declared here rather than by the chart.
-#
-# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
-# taken field for field off the live object so scraping is unchanged: same name,
-# same namespace, same selector, same 10s interval and node relabeling.
-#
-# It lives here because the chart's template is gated on monitoring.coreos.com/v1
-# existing at install time, and on an empty cluster it does not — those CRDs
-# arrive with kube-prometheus-stack, which reconciles after infrastructure.
-# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
-# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
-#
-# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
-# why; this is the same move for the component where it mattered most.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: cilium-agent
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: cilium-agent
-    app.kubernetes.io/part-of: cilium
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: cilium-agent
-  namespaceSelector:
-    matchNames:
-      - kube-system
-  # The chart's Service keeps rendering (prometheus.enabled), so the target and
-  # its `metrics` port are there regardless of who declares the ServiceMonitor.
-  targetLabels:
-    - k8s-app
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        # Agent metrics are per-node; without this every series lands without a
-        # node label and BPF map pressure cannot be attributed to a machine.
-        - action: replace
-          sourceLabels:
-            - __meta_kubernetes_pod_node_name
-          targetLabel: node
-          replacement: ${1}
@@ -23,20 +23,9 @@ resources:
  # endpoint, which proves reachability, not that anything was written.
  - prometheusrule-cnpg-backup.yaml
  - alloy-servicemonitor.yaml
-  - cilium-servicemonitor.yaml
  - cnpg-podmonitors.yaml
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
-  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
-  # kube-prometheus-stack, which reconciles after infrastructure, so these
-  # cannot live beside the Velero HelmRelease.
-  - velero-podmonitor.yaml
-  - velero-servicemonitor.yaml
-  - authentik-outpost-servicemonitors.yaml
-  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
-  # when it cannot create these, which held the CephCluster in Progressing and
-  # deadlocked the whole apps layer. See the file for the cycle.
-  - rook-servicemonitors.yaml
@@ -1,93 +0,0 @@
-# Rook's two ServiceMonitors, declared here rather than by the operator.
-#
-# Reproduced field for field off the live objects in the production cluster, so
-# scraping is unchanged: same names, same namespace, same selectors, same 10s
-# interval and `cluster` relabeling. The only thing dropped is the CephCluster
-# ownerReference, which Rook set and Flux cannot.
-#
-# They live here because CephCluster.spec.monitoring.enabled is now false, and
-# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
-# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
-# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
-# reconcile.go, v1.20.7):
-#
-#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
-#         // exporter reconciled; its errors are logged and reconcile continues
-#         if cephCluster.Spec.Monitoring.Enabled {
-#             if err := EnableCephExporterServiceMonitor(...); err != nil {
-#                 return errors.Wrap(err, "failed to enable service monitor")
-#             }
-#         }
-#     }
-#
-# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
-# fails with "the server could not find the requested resource (post
-# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
-# aborts, and the CephCluster never leaves status.phase Progressing. That is
-# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
-# closes on itself:
-#
-#     infrastructure-ceph-cluster → CephCluster phase Ready
-#                                 → Rook reconcile settles
-#                                 → monitoring.coreos.com CRDs exist
-#                                 → kube-prometheus-stack (monitoring)
-#                                 → grafana-db
-#                                 → infrastructure-ceph-cluster
-#
-# Nothing outside that loop can break it, so a cold rebuild stopped here with
-# every one of the twenty app Kustomizations behind it.
-#
-# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
-# get, and for the same reason: the metrics stay where they are, only the
-# scrape wiring moves to the layer where its CRDs exist.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-exporter
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-exporter
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: ceph-exporter-http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      # Rook stamps the cluster name so series from several Ceph clusters stay
-      # separable. One cluster here, but dropping it would rename every series.
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster
---
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-mgr
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-mgr
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster
@@ -93,20 +93,7 @@ spec:
          # on every paperless pod start: tools belong in the image, not
          # installed at runtime. Dated tag so a change is a reviewable diff
          # here, matching the ci-ansible / alpine-jq convention.
-          #
-          # konairius/, not gitea_admin/, and that is not cosmetic. This is an
-          # INIT container, so paperless does not start without it — and during
-          # a cold rebuild gitea.renner.uno does not exist, because Gitea runs
-          # inside the cluster being rebuilt. A rehearsal left this pod in
-          # Init:ErrImagePull on exactly this image.
-          #
-          # The node falls back to a ghcr.io copy in that window
-          # (machine.registries.mirrors, applied by bootstrap.sh). Containerd
-          # keeps the repository path when it falls through, so the path has to
-          # exist on both registries — hence konairius/, which matches the ghcr
-          # namespace. Steady state still pulls from Gitea and never leaves the
-          # network.
-          image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
          command:
            - sh
            - -c
@@ -177,14 +177,7 @@ spec:
        # webhook start failing with "expired JWT access token" about
        # thirty minutes after seeding.
        - name: render-schedule
-          # konairius/, not gitea_admin/: this container starts during a cold
-          # rebuild, when Gitea does not exist because it runs inside the
-          # cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff.
-          # The node falls back to a ghcr.io copy then
-          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
-          # keeps the repository path, so it has to exist on both registries.
-          # Steady state still pulls from Gitea.
-          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
          command:
            - sh
            - -c
@@ -27,7 +27,7 @@ spec:
        # Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
        # is the only consumer; a major bump is a separate migration.
        - name: valkey
-          image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
+          image: valkey/valkey:9.1.2-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
          args:
            - valkey-server
            - --save
@@ -1,3 +1,18 @@
+# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
+# isolated from the apps namespace.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    # Image-build steps run a privileged docker/buildx plugin pod (see the
+    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
+    # without this the privileged step pods are rejected at admission. The
+    # namespace is dedicated to ephemeral CI step pods, so scoping the
+    # exemption here keeps it off apps/.
+    pod-security.kubernetes.io/enforce: privileged
+---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
@@ -8,10 +8,10 @@ metadata:
 spec:
  interval: 1m0s
  ref:
-    branch: vmtest-flux
+    branch: master
  secretRef:
    name: flux-system
-  url: https://gitea.renner.uno/gitea_admin/homelab.git
+  url: https://gitea.renner.uno/k.renner/homelab.git
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
@@ -28,19 +28,6 @@ spec:
    - name: infrastructure
    - name: infrastructure-crds
    - name: infrastructure-ceph-cluster
-    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
-    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
-    # kind is unknown when this path is first applied, and a server-side
-    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
-    #
-    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
-    #   no matches for kind "ObjectStore" in version ...
-    #
-    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
-    # woodpecker — already declares it. This was the only one that did not, and
-    # it took speech-to-phrase, zone-configurator and matter-server down with
-    # it, since all three wait on home-assistant.
-    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
@@ -122,20 +122,8 @@ spec:
  path: ./kubernetes/infrastructure/rook-ceph-cluster
  prune: true
  wait: true
-  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
-  # existing cluster could ever hit and no new one could ever escape:
-  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
-  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
-  # defined in this very Kustomization. A cold rebuild stopped here for good,
-  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
-  #
-  # The edge was never needed. infrastructure runs wait: false, so it applies
-  # the rook-ceph namespace and the operator immediately and only its readiness
-  # waits on the healthChecks — everything this path needs exists long before
-  # infrastructure reports Ready. On an empty API server this Kustomization
-  # retries on retryInterval until the operator installs its CRDs, the same
-  # self-healing shape post-crds already relies on; on a live cluster the CRDs
-  # are always present and nothing changes.
+  dependsOn:
+    - name: infrastructure
  # kstatus doesn't know how to read Rook CR readiness, so without these
  # this Kustomization reports Ready as soon as the manifests apply — long
  # before OSDs/mons/MDS are actually serving I/O. Apps depending on this
@@ -0,0 +1,22 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: llama-server
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 1m
+  # On a cold PVC the chat model pulls ~19 GiB of GGUF weights from Hugging
+  # Face and then loads onto the iGPU before /health goes green, so a first
+  # start is far slower than a normal app's.
+  timeout: 60m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/llama-server
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster
@@ -1,115 +0,0 @@
-# The upstream ceph-csi-drivers chart, which is how Rook expects CSI drivers to
-# be provided since v1.20: "CSI drivers are admin-managed via the
-# ceph-csi-operator. Rook no longer deploys CSI."
-#
-# This replaces a static rendering of the chart's ServiceAccounts and RBAC. That
-# workaround (#babed078) was written during a five-day outage in which both
-# ctrlplugin Deployments sat at 0/2 and PVC attach/detach was blocked
-# cluster-wide, and it justified itself with:
-#
-#   "helm-controller v1.5.4 predates disableTakeOwnership, so Helm would refuse
-#    to adopt the operator-managed Driver and OperatorConfig CRs"
-#
-# That is not so. disableTakeOwnership landed in helm-controller v1.2.0, three
-# minors BEFORE the version running here, and it is an opt-OUT: taking ownership
-# of existing resources is the default. Nothing stood in the way of the chart.
-#
-# What the workaround could not do is create the Driver CRs, and a cold-rebuild
-# rehearsal is where that surfaced. Rook's v1.18 -> v1.20 migration had created
-# them once on this cluster, so they looked permanent; on an empty cluster Rook
-# creates the CephConnection and ClientProfile and stops there, no Driver is
-# ever made, the ceph-csi-operator deploys no provisioner, and every PVC waits
-# forever. OpenBao's among them, which is where the rebuild stopped.
-#
-# cephConnections and clientProfiles stay EMPTY here on purpose. Rook owns both:
-# the CephConnection carries the mon Service IPs Rook keeps current, and the
-# ClientProfile points at Secrets Rook creates. Handing either to Helm would
-# freeze a value that is supposed to move. Verified on a fresh cluster — Rook
-# creates exactly those two and nothing else.
-apiVersion: source.toolkit.fluxcd.io/v1
-kind: HelmRepository
-metadata:
-  name: ceph-csi-operator
-  namespace: rook-ceph
-spec:
-  interval: 24h
-  url: https://ceph.github.io/ceph-csi-operator/
---
-apiVersion: helm.toolkit.fluxcd.io/v2
-kind: HelmRelease
-metadata:
-  name: ceph-csi-drivers
-  namespace: rook-ceph
-spec:
-  interval: 30m
-  # Wait for the rook-ceph HelmRelease to finish, not just for the CRDs to
-  # appear. The two are not the same thing, and an earlier version of this file
-  # argued the opposite: "nothing here installs those CRDs, so a dependsOn would
-  # only move the wait". True as far as existence goes — but the csi.ceph.io
-  # CRDs come from the ceph-csi-operator subchart inside the rook-ceph release,
-  # which runs with `crds: CreateReplace` and `retries: 3`, so they are not
-  # created once. They are created again on every attempt, and everything living
-  # under them goes with them.
-  #
-  # A cold rebuild landed in exactly that window:
-  #
-  #   19:24:21  ceph-csi-drivers  install succeeded — both Driver CRs created
-  #   19:25:50  csi.ceph.io CRDs  recreated by a later operator attempt
-  #   19:29:33  rook-ceph-operator install finally succeeded
-  #
-  # The Driver objects were gone, but Helm still had them in its stored release
-  # and would not re-render before the 30m interval. No controller reported
-  # anything: the ceph-csi-operator sat idle with nothing to reconcile, no
-  # provisioner Deployment was ever created, and every PVC on ceph-block stayed
-  # Pending on "waiting for a volume to be created by the external provisioner
-  # rook-ceph.rbd.csi.ceph.com". openbao-0 then never got a node, and the run
-  # failed 15 minutes later reporting that OpenBao would not answer.
-  dependsOn:
-    - name: rook-ceph-operator
-  chart:
-    spec:
-      chart: ceph-csi-drivers
-      version: "~1.0.4"
-      sourceRef:
-        kind: HelmRepository
-        name: ceph-csi-operator
-        namespace: rook-ceph
-  install:
-    # Kept as a backstop. dependsOn removes the race this was covering for, but
-    # a CRD replace outside an install of this release would still empty it, and
-    # a retry is cheaper than a 30-minute wait for the next interval.
-    remediation:
-      retries: 5
-  upgrade:
-    remediation:
-      retries: 3
-  values:
-    # Rook's, not ours — see the header.
-    cephConnections: []
-    clientProfiles: []
-    # Names must match what the StorageClasses already name as their
-    # provisioner (rook-ceph.rbd.csi.ceph.com), which is also what Rook's
-    # migration produced on the live cluster. The chart's default drops the
-    # namespace prefix and would leave every StorageClass pointing at a driver
-    # that does not exist.
-    # clusterName is load-bearing and easy to miss: it becomes --clustername on
-    # the plugin, and ceph-csi hands it to rados_create2 as the Ceph cluster
-    # name. The chart defaults it to "", which librados rejects outright — the
-    # provisioner then fails every request with
-    #   failed to get connection: connecting failed: rados: ret=-22
-    # which reads like a credential or network fault and is neither. The
-    # toolbox keeps working throughout, because it falls back to the default
-    # name "ceph", so the cluster looks healthy while nothing can provision.
-    drivers:
-      rbd:
-        name: rook-ceph.rbd.csi.ceph.com
-        clusterName: rook-ceph
-        enabled: true
-      cephfs:
-        name: rook-ceph.cephfs.csi.ceph.com
-        clusterName: rook-ceph
-        enabled: true
-      nfs:
-        enabled: false
-      nvmeof:
-        enabled: false
@@ -1,6 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-# The ServiceAccounts and RBAC that used to be rendered here by hand now come
-# from the chart that owns them, together with the Driver CRs they exist for.
 resources:
-  - helmrelease.yaml
+  - serviceaccounts.yaml
+  - rbac.yaml
@@ -0,0 +1,442 @@
+# RBAC for the per-driver ServiceAccounts in serviceaccounts.yaml.
+# Rendered from ceph-csi-operator/ceph-csi-drivers chart v1.0.1 templates,
+# with $normalizedDriverName fixed to the operator's hardcoded driver-type
+# names (`rbd`, `cephfs`). Sync with upstream if the chart's RBAC changes.
+# Source: https://github.com/ceph/ceph-csi-operator/tree/v1.0.1/deploy/charts/ceph-csi-drivers/templates
+
+# ---------- rbd-ctrlplugin ----------
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: rbd-ctrlplugin-cr
+rules:
+  - apiGroups: [""]
+    resources: [secrets]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [persistentvolumes]
+    verbs: [get, list, watch, create, delete, patch, update]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims]
+    verbs: [get, list, watch, update]
+  - apiGroups: [storage.k8s.io]
+    resources: [storageclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [events]
+    verbs: [list, watch, create, update, patch]
+  - apiGroups: [storage.k8s.io]
+    resources: [volumeattachments]
+    verbs: [get, list, watch, patch]
+  - apiGroups: [storage.k8s.io]
+    resources: [volumeattachments/status]
+    verbs: [patch]
+  - apiGroups: [""]
+    resources: [nodes]
+    verbs: [get, list, watch]
+  - apiGroups: [storage.k8s.io]
+    resources: [csinodes]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims/status]
+    verbs: [patch]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshots]
+    verbs: [get, list, watch]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotcontents]
+    verbs: [get, list, watch, patch, update]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [""]
+    resources: [configmaps]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts/token]
+    verbs: [create]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotcontents]
+    verbs: [get, list, watch, update, patch]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotcontents]
+    verbs: [get, list, watch, update, patch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [replication.storage.openshift.io]
+    resources: [volumegroupreplicationcontents]
+    verbs: [get, list, watch]
+  - apiGroups: [replication.storage.openshift.io]
+    resources: [volumegroupreplicationclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [authentication.k8s.io]
+    resources: [tokenreviews]
+    verbs: [create]
+  - apiGroups: [authorization.k8s.io]
+    resources: [subjectaccessreviews]
+    verbs: [create]
+  - apiGroups: [cbt.storage.k8s.io]
+    resources: [snapshotmetadataservices]
+    verbs: [get, list]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: rbd-ctrlplugin-crb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: rbd-ctrlplugin-cr
+subjects:
+  - kind: ServiceAccount
+    name: rbd-ctrlplugin-sa
+    namespace: rook-ceph
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: rbd-ctrlplugin-r
+  namespace: rook-ceph
+rules:
+  - apiGroups: [coordination.k8s.io]
+    resources: [leases]
+    verbs: [get, watch, list, delete, update, create]
+  - apiGroups: [csiaddons.openshift.io]
+    resources: [csiaddonsnodes]
+    verbs: [get, watch, list, create, update, delete]
+  - apiGroups: [""]
+    resources: [pods]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [replicasets]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [deployments/finalizers, daemonsets/finalizers]
+    verbs: [update]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: rbd-ctrlplugin-rb
+  namespace: rook-ceph
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: rbd-ctrlplugin-r
+subjects:
+  - kind: ServiceAccount
+    name: rbd-ctrlplugin-sa
+    namespace: rook-ceph
+
+# ---------- rbd-nodeplugin ----------
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: rbd-nodeplugin-cr
+rules:
+  - apiGroups: [""]
+    resources: [secrets]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [persistentvolumes]
+    verbs: [get, list]
+  - apiGroups: [storage.k8s.io]
+    resources: [volumeattachments]
+    verbs: [get, list]
+  - apiGroups: [""]
+    resources: [configmaps]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts/token]
+    verbs: [create]
+  - apiGroups: [""]
+    resources: [nodes]
+    verbs: [get]
+  - apiGroups: [authentication.k8s.io]
+    resources: [tokenreviews]
+    verbs: [create]
+  - apiGroups: [""]
+    resources: [events]
+    verbs: [list, watch, create, update, patch]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims]
+    verbs: [get]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: rbd-nodeplugin-crb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: rbd-nodeplugin-cr
+subjects:
+  - kind: ServiceAccount
+    name: rbd-nodeplugin-sa
+    namespace: rook-ceph
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: rbd-nodeplugin-r
+  namespace: rook-ceph
+rules:
+  - apiGroups: [csiaddons.openshift.io]
+    resources: [csiaddonsnodes]
+    verbs: [get, watch, list, create, update, delete]
+  - apiGroups: [""]
+    resources: [pods]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [replicasets]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [deployments/finalizers, daemonsets/finalizers]
+    verbs: [update]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: rbd-nodeplugin-rb
+  namespace: rook-ceph
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: rbd-nodeplugin-r
+subjects:
+  - kind: ServiceAccount
+    name: rbd-nodeplugin-sa
+    namespace: rook-ceph
+
+# ---------- cephfs-ctrlplugin ----------
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: cephfs-ctrlplugin-cr
+rules:
+  - apiGroups: [""]
+    resources: [secrets]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [configmaps]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [nodes]
+    verbs: [get, list, watch]
+  - apiGroups: [storage.k8s.io]
+    resources: [csinodes]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [persistentvolumes]
+    verbs: [get, list, watch, create, delete, patch, update]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims]
+    verbs: [get, list, watch, patch, update]
+  - apiGroups: [storage.k8s.io]
+    resources: [storageclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [events]
+    verbs: [list, watch, create, update, patch]
+  - apiGroups: [storage.k8s.io]
+    resources: [volumeattachments]
+    verbs: [get, list, watch, patch]
+  - apiGroups: [storage.k8s.io]
+    resources: [volumeattachments/status]
+    verbs: [patch]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims/status]
+    verbs: [patch]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshots]
+    verbs: [get, list]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotcontents]
+    verbs: [get, list, watch, patch, update]
+  - apiGroups: [snapshot.storage.k8s.io]
+    resources: [volumesnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotcontents]
+    verbs: [get, list, watch, update, patch]
+  - apiGroups: [groupsnapshot.storage.k8s.io]
+    resources: [volumegroupsnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotcontents]
+    verbs: [get, list, watch, update, patch]
+  - apiGroups: [groupsnapshot.storage.openshift.io]
+    resources: [volumegroupsnapshotcontents/status]
+    verbs: [update, patch]
+  - apiGroups: [replication.storage.openshift.io]
+    resources: [volumegroupreplicationcontents]
+    verbs: [get, list, watch]
+  - apiGroups: [replication.storage.openshift.io]
+    resources: [volumegroupreplicationclasses]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts/token]
+    verbs: [create]
+  - apiGroups: [authentication.k8s.io]
+    resources: [tokenreviews]
+    verbs: [create]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: cephfs-ctrlplugin-crb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cephfs-ctrlplugin-cr
+subjects:
+  - kind: ServiceAccount
+    name: cephfs-ctrlplugin-sa
+    namespace: rook-ceph
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: cephfs-ctrlplugin-r
+  namespace: rook-ceph
+rules:
+  - apiGroups: [coordination.k8s.io]
+    resources: [leases]
+    verbs: [get, watch, list, delete, update, create]
+  - apiGroups: [csiaddons.openshift.io]
+    resources: [csiaddonsnodes]
+    verbs: [get, watch, list, create, update, delete]
+  - apiGroups: [""]
+    resources: [pods]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [replicasets]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [deployments/finalizers, daemonsets/finalizers]
+    verbs: [update]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: cephfs-ctrlplugin-rb
+  namespace: rook-ceph
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cephfs-ctrlplugin-r
+subjects:
+  - kind: ServiceAccount
+    name: cephfs-ctrlplugin-sa
+    namespace: rook-ceph
+
+# ---------- cephfs-nodeplugin ----------
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: cephfs-nodeplugin-cr
+rules:
+  - apiGroups: [""]
+    resources: [nodes]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [secrets]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [configmaps]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts]
+    verbs: [get]
+  - apiGroups: [""]
+    resources: [serviceaccounts/token]
+    verbs: [create]
+  - apiGroups: [""]
+    resources: [events]
+    verbs: [list, watch, create, update, patch]
+  - apiGroups: [""]
+    resources: [persistentvolumes, persistentvolumeclaims]
+    verbs: [get]
+  - apiGroups: [authentication.k8s.io]
+    resources: [tokenreviews]
+    verbs: [create]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: cephfs-nodeplugin-crb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cephfs-nodeplugin-cr
+subjects:
+  - kind: ServiceAccount
+    name: cephfs-nodeplugin-sa
+    namespace: rook-ceph
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: cephfs-nodeplugin-r
+  namespace: rook-ceph
+rules:
+  - apiGroups: [csiaddons.openshift.io]
+    resources: [csiaddonsnodes]
+    verbs: [get, watch, list, create, update, delete]
+  - apiGroups: [""]
+    resources: [pods]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [replicasets]
+    verbs: [get]
+  - apiGroups: [apps]
+    resources: [deployments/finalizers, daemonsets/finalizers]
+    verbs: [update]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: cephfs-nodeplugin-rb
+  namespace: rook-ceph
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cephfs-nodeplugin-r
+subjects:
+  - kind: ServiceAccount
+    name: cephfs-nodeplugin-sa
+    namespace: rook-ceph
@@ -0,0 +1,35 @@
+# Per-driver ServiceAccounts consumed by the workloads that the
+# ceph-csi-controller-manager generates for each `Driver` CR. Rook v1.20
+# extracted CSI driver management into the upstream ceph-csi-operator
+# (subchart of `rook-ceph`); these SAs are normally provisioned by the
+# separate `ceph-csi-drivers` chart, but installing that chart would also
+# create a parallel set of `Driver` / `OperatorConfig` CRs and Helm cannot
+# adopt the operator-managed ones on helm-controller v1.5.4. We render
+# only the SAs + RBAC the existing operator-managed `Driver` CRs need.
+#
+# Names are fixed by the operator (CSI_SERVICE_ACCOUNT_PREFIX="" on
+# ceph-csi-controller-manager): `<driver-type>-{ctrl,node}plugin-sa`,
+# where driver-type is `rbd` or `cephfs`. Do not rename.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: rbd-ctrlplugin-sa
+  namespace: rook-ceph
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: rbd-nodeplugin-sa
+  namespace: rook-ceph
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cephfs-ctrlplugin-sa
+  namespace: rook-ceph
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cephfs-nodeplugin-sa
+  namespace: rook-ceph
@@ -70,23 +70,9 @@ spec:
      # map only shows up as "no space left on device" in the log while pod
      # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
      # alertable beforehand.
-      #
-      # That reasoning is about the metrics, which this line provides. The
-      # ServiceMonitor is only the scrape wiring and is declared standalone in
-      # apps/monitoring-rules instead — the same treatment velero/helmrelease
-      # already gives its own, and for the same reason: the chart's template is
-      # gated on monitoring.coreos.com/v1 existing at install time.
-      #
-      # Here it is worse than for velero. Cilium's chart does not skip the
-      # template, it aborts rendering — "Service Monitor requires
-      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
-      # healthChecks the infrastructure Kustomization waits on. So on an empty
-      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
-      # blocked monitoring, which is where those CRDs come from. The lowest
-      # layer in the cluster was waiting on the highest.
      enabled: true
      serviceMonitor:
-        enabled: false
+        enabled: true
    bpf:
      # Per-endpoint policy map size: one entry per (identity, direction,
      # protocol, port) allowed. When it fills, endpoint regeneration fails
@@ -54,7 +54,7 @@ spec:
    # required.
    #
    # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
    # so there is no live impact, but re-enabling it will require pointing the
    # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
    tls:
@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - bouncer-key-externalsecret.yaml
  - helmrelease.yaml
@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
  - helmrelease.yaml
@@ -9,26 +9,9 @@ spec:
 ---
 # v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
 # (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
-# filter pins us to the bare three-component release.
-#
-# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
-# upgrade tied to the Rook operator version. That coupling had already come
-# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
-# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
-# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
-# the mons with a client a whole major ahead of them.
-#
-# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
-# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
-# Nothing broke only because the four CSI users predate the cipher change and
-# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
-# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
-# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
-# because a fresh cluster has no legacy keys at all.
-#
-# So the cutover happens now, with the version the client already is. Rook
-# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
-# upstream reports data corruption there when read affinity is enabled.
+# filter pins us to the bare three-component release. Range stays at
+# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
+# upgrade tied to the Rook operator version.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -42,4 +25,4 @@ spec:
    extract: '$ver'
  policy:
    semver:
-      range: ">=20.2.1 <21.0.0"
+      range: "19.x"
@@ -0,0 +1,118 @@
+---
+# Advertises the on-board Intel iGPU as the schedulable resource
+# `gpu.intel.com/i915`, so pods can request it without a /dev/dri hostPath or
+# any privilege escalation — the plugin hands the render node to the container
+# through the kubelet device-plugin API. The same resource covers both GPU
+# compute (LLM inference) and QuickSync video transcoding; it is not
+# compute-specific.
+#
+# All three nodes are MS-01 with an i9-12900H (Alder Lake-P, PCI 8086:46a6,
+# Iris Xe 96 EU) at 0000:00:02.0. Talos carries the driver in the `i915` system
+# extension, already present in the schematic, so no machineconfig change is
+# needed.
+#
+# Vendored from upstream rather than installed via the
+# `intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
+# custom resource, which only works alongside the Intel Device Plugin Operator
+# and its CRDs, cert-manager webhook and controller. With one device type on
+# three identical nodes, the operator manages nothing this DaemonSet does not.
+#
+# Source: intel/intel-device-plugins-for-kubernetes, v0.36.0,
+# deployments/gpu_plugin/base/intel-gpu-plugin.yaml
+# Upgrading means re-syncing this file against that path at the new tag and
+# bumping the image tag below.
+#
+# Namespace is kube-system because the plugin mounts hostPaths (/dev/dri,
+# /sys/class/drm, the kubelet device-plugin socket and the CDI path). The
+# cluster enforces PodSecurity `baseline`, which forbids hostPath volumes, and
+# kube-system is the only namespace in the API server's exemption list.
+#
+# Node Feature Discovery is not deployed, so upstream's `nfd_labeled_nodes`
+# overlay is omitted along with its `intel.feature.node.kubernetes.io/gpu`
+# nodeSelector. Every node carries the same iGPU, leaving nothing to discover;
+# the plugin exits cleanly on a node without /dev/dri.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: intel-gpu-plugin
+  namespace: kube-system
+  labels:
+    app: intel-gpu-plugin
+spec:
+  selector:
+    matchLabels:
+      app: intel-gpu-plugin
+  updateStrategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+  template:
+    metadata:
+      labels:
+        app: intel-gpu-plugin
+    spec:
+      containers:
+        - name: intel-gpu-plugin
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+            - name: HOST_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.hostIP
+          image: intel/intel-gpu-plugin:0.36.0
+          imagePullPolicy: IfNotPresent
+          args:
+            # Slots the single iGPU is handed out as, letting that many
+            # containers share it concurrently. A slot count, not fractional
+            # accounting — nothing partitions the device between sharers, so
+            # they contend for it. Sized for an inference server plus room for
+            # a transcoder and a benchmark pod alongside it.
+            - -shared-dev-num=4
+          securityContext:
+            seLinuxOptions:
+              type: "container_device_plugin_t"
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+            seccompProfile:
+              type: RuntimeDefault
+          resources:
+            requests:
+              memory: "45Mi"
+              cpu: 40m
+            limits:
+              memory: "90Mi"
+              cpu: 100m
+          volumeMounts:
+            - name: devfs
+              mountPath: /dev/dri
+              readOnly: true
+            - name: sysfsdrm
+              mountPath: /sys/class/drm
+              readOnly: true
+            - name: kubeletsockets
+              mountPath: /var/lib/kubelet/device-plugins
+            - name: cdipath
+              mountPath: /var/run/cdi
+      volumes:
+        - name: devfs
+          hostPath:
+            path: /dev/dri
+        - name: sysfsdrm
+          hostPath:
+            path: /sys/class/drm
+        - name: kubeletsockets
+          hostPath:
+            path: /var/lib/kubelet/device-plugins
+        - name: cdipath
+          hostPath:
+            path: /var/run/cdi
+            type: DirectoryOrCreate
+      nodeSelector:
+        kubernetes.io/arch: amd64
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - daemonset.yaml
@@ -10,6 +10,7 @@ resources:
  - cert-manager/
  - snapshot-controller/
  - metrics-server/
+  - intel-gpu-plugin/
  - crowdsec/
  - traefik/
  - rook-ceph/
@@ -30,11 +30,6 @@ spec:
      labels:
        app: cni-plugins-installer
    spec:
-      # Same reasoning as multus-daemonset.yaml. This one installs the macvlan
-      # binary the ceph-san attachment is made of, so a node that cannot
-      # schedule it has no plugin to delegate to — the failure the init
-      # container comment below already describes, reached by a different road.
-      priorityClassName: system-node-critical
      hostNetwork: true
      tolerations:
        - operator: Exists
@@ -49,32 +44,7 @@ spec:
          # at all and put github.com on the critical path of every node rebuild.
          # Dated tag so a version change is a reviewable diff here, matching the
          # ci-ansible / alpine-jq convention.
-          #
-          # konairius/, not gitea_admin/, and the difference is what makes a
-          # rebuild possible. Gitea runs inside this cluster, so during a rebuild
-          # it does not exist: a rehearsal sat here in ImagePullBackOff with "no
-          # route to host", and because the plugins never installed there was no
-          # macvlan, so Multus could not attach ceph-san, so Rook's network
-          # canary never started, so the CephCluster hung at "Configuring Ceph
-          # Mons" and OpenBao waited forever on a PVC that could not bind. The
-          # failure surfaced four layers from its cause.
-          #
-          # This used to point straight at ghcr.io, which fixed the rebuild and
-          # broke the normal case: every steady-state pull then left the network
-          # for an image the local registry already holds. The fallback belongs
-          # to the node, not to the manifest — machine.registries.mirrors names
-          # Gitea first and ghcr.io second (bootstrap.sh), so nothing here has to
-          # know about the recovery path.
-          #
-          # Containerd keeps the repository path when it falls through, so the
-          # path must resolve on both registries; the build pushes one digest to
-          # konairius/ on each, verified equal. Bumping the tag means building
-          # first — an unpublished tag is unpullable on both.
-          #
-          # No imagePullSecret: the ghcr namespace is private and its credential
-          # is in machine.registries too, because a Secret would have to come
-          # from OpenBao, which needs Ceph, which needs these binaries.
-          image: gitea.renner.uno/konairius/cni-plugins:2026-09-04
+          image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
          command:
            - sh
            - -c
@@ -154,40 +154,6 @@ spec:
        app: multus
        name: multus
    spec:
-      # Upstream ships this DaemonSet with no priority class, so it runs at 0
-      # and can preempt nothing. Any event that makes the pod need a fresh
-      # scheduling decision — a rollout, a node restart, an eviction — then has
-      # to fit it onto a node that may since have filled up, and if it does not
-      # fit it simply stays Pending. The node has no multus after that, and
-      # nothing reports it: pods still start, they just never get their
-      # secondary attachment, and the annotation everything waits on never
-      # appears.
-      #
-      # Seen on a cold rebuild: two of four nodes sat at 99% memory requests
-      # when the DaemonSet rolled, both multus pods went Pending, and Rook's
-      # network canary hung in Init:0/1 waiting for a network-status annotation
-      # that could never come. The CephCluster stayed in Progressing, which is
-      # what infrastructure-ceph-cluster gates on, so the whole apps layer
-      # stopped behind a scheduling decision three layers away.
-      #
-      # node-critical, the same class cilium and cni-dhcp-daemon carry, because
-      # that is what this is: without multus a node cannot attach a secondary
-      # network, and only that node is affected. Nothing cluster-wide fails.
-      #
-      # This does outrank kube-apiserver, which Talos runs as a static pod at
-      # system-cluster-critical (2000000000, against 2000001000 here), and on a
-      # control plane that has run out of memory the kubelet will pick the API
-      # server as the pod to evict. That was observed twice, at 6GiB and at
-      # 8GiB.
-      #
-      # It is not a reason to demote this DaemonSet. Kubernetes puts the CNI
-      # above the control plane deliberately — a node without its CNI cannot run
-      # anything at all — and a class name is a statement about what the pod is,
-      # not a dial for picking a number. The eviction says the control plane is
-      # starved, and that is where it has to be fixed: enough allocatable
-      # memory, or kubeReserved/systemReserved so the static pods keep what they
-      # need. Demoting this to cluster-critical would only move the victim.
-      priorityClassName: system-node-critical
      hostNetwork: true
      hostPID: true
      tolerations:
@@ -146,11 +146,6 @@ spec:
        app: whereabouts
        name: whereabouts
    spec:
-      # Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
-      # secondary-network path, so a node without it cannot hand out an address
-      # on ceph-san and the attachment fails just as completely as if multus
-      # itself were missing. Upstream leaves the priority class unset.
-      priorityClassName: system-node-critical
      hostNetwork: true
      serviceAccountName: whereabouts
      # Every node here is a control plane node, so the NoSchedule toleration
@@ -184,30 +184,14 @@ kind: Namespace
 metadata:
  name: woodpecker
 ---
-# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
-# from `apps`. Distinct from `woodpecker`, which holds the service itself.
+# Local LLM inference (kubernetes/apps/llama-server). Declared here, like
+# `woodpecker` above, so the `infrastructure` KS creates it before
+# `infrastructure-crds` applies the namespace's CiliumNetworkPolicies.
 #
-# Here rather than in apps/woodpecker for the reason spelled out above that
-# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
-# ../network-policies/, and infrastructure-crds runs before every app KS. With
-# the namespace declared in the workload KS the order inverted, and a rebuild
-# stopped with
-#
-#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
-#   namespaces "woodpecker-ci" not found
-#
-# which held infrastructure-crds, and with it all twenty app Kustomizations
-# that depend on it. The same reasoning was already written down for
-# `woodpecker` and simply not carried across to its sibling.
-#
-# Image-build steps run a privileged docker/buildx plugin pod (see the server's
-# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
-# label the privileged step pods are rejected at admission. The namespace holds
-# nothing but ephemeral CI step pods, so the exemption stays off apps/.
+# No pod-security labels: the workload needs no hostPath or privilege — the
+# Intel GPU device plugin passes the iGPU in through the device-plugin API —
+# so the cluster's default `baseline` enforcement is enough.
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    pod-security.kubernetes.io/enforce: privileged
+  name: ai
@@ -6,5 +6,6 @@ kind: Kustomization
 # offsite lives in post-crds/velero-schedules.yaml; the staleness alert
 # in apps/monitoring-rules/prometheusrule-netbird-backup.yaml.
 resources:
+  - externalsecret.yaml
  - pvc.yaml
  - cronjob.yaml
@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
  - daemonset.yaml
@@ -0,0 +1,96 @@
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: ai
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
+    # home-assistant-ns.yaml for the rationale (CT-RELATED path not
+    # passing related ICMP empirically; per-namespace shape, mirroring
+    # commit 3c517a0's verified-safe pattern with the 66dbe9f type set).
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    # rook-ceph: the model weights live on a CephFS RWX volume
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": rook-ceph
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+    # host/remote-node: kubelet probes + node-local reach (Ceph mons on host net)
+    # world: image pulls and llama-server's Hugging Face model download
+    - toEntities:
+        - host
+        - remote-node
+        - world
+    # ICMP — egress mirror; see ingress block above.
+    - toEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
@@ -24,4 +24,5 @@ resources:
  - woodpecker-ns.yaml
  - trmnl-ns.yaml
  - wordpress-ns.yaml
+  - ai-ns.yaml
  - allow-cilium-health.yaml
@@ -10,8 +10,33 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret that publishes the rotated token to Flux lives in
-# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
+# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
+# own it) so a sync failure leaves the last-good value in place; Flux's
+# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
+# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
+# provisioned by hand (see policies/rotator-gitea-flux.hcl).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: flux-system-git-auth
+  namespace: flux-system
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: flux-system
+    creationPolicy: Merge
+  data:
+    - secretKey: username
+      remoteRef:
+        key: infrastructure/flux-git
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: infrastructure/flux-git
+        property: token
 ---
 apiVersion: v1
 kind: ServiceAccount
@@ -1,60 +0,0 @@
-# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
-# WITHOUT owning it, so a sync failure leaves the last-good value in place;
-# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
-# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
-# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
-#
-# See creationPolicy below for why that is Orphan and not Merge — the
-# distinction is what makes a recovery from the break-glass source possible.
-#
-# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
-# rotation CronJob: external-secrets.io CRDs are installed by the ESO
-# HelmRelease in the infrastructure Kustomization, so this manifest cannot
-# pass server-side dry-run there on a fresh cluster.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    # Orphan, not Merge: create the Secret if it is absent, but never own it.
-    #
-    # Merge cannot create — it only writes into a Secret that already exists,
-    # on the assumption that `flux bootstrap` made one. That assumption breaks
-    # in exactly the situation this cluster is rebuilt from. The break-glass
-    # bootstrap pulls from the GitHub mirror and keeps its token in
-    # flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
-    # so this ExternalSecret has nothing to merge into — and reports Ready while
-    # producing no Secret at all. The operator then performs the documented
-    # handover back to Gitea and finds the GitRepository pointing at a Secret
-    # that does not exist.
-    #
-    # Owner would create it but attach an ownerReference, so deleting this
-    # object would take Flux's credential with it. Orphan creates and updates
-    # without ownership, which is what the paragraph at the top of this file
-    # actually wanted: a sync failure, or the loss of this object, leaves the
-    # last-good value in place.
-    #
-    # Safe on a live cluster: the Secret holds exactly the two keys written
-    # below (verified against production), so writing it whole is the same
-    # result as merging into it.
-    creationPolicy: Orphan
-    # Explicit rather than relying on the default, because it is the half that
-    # matters here: if the OpenBao entry ever disappears, keep serving the
-    # credential Flux already has rather than leaving it with none.
-    deletionPolicy: Retain
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token
@@ -10,25 +10,4 @@ resources:
  # configured (2026-07-13): a store that can't turn Ready would block this
  # KS's wait gate and freeze the dependsOn chain.
  - openbao-clustersecretstore.yaml
-  # ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
-  # the infrastructure Kustomization, so these cannot pass its server-side
-  # dry-run on a fresh cluster — Flux rejects the whole apply set, including
-  # the ESO HelmRelease that would have installed the CRDs. They sat next to
-  # their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
-  # nothing had ever applied this layer against an empty API server before.
-  - crowdsec-bouncer-key-externalsecret.yaml
-  - external-dns-unifi-externalsecret.yaml
-  - flux-git-externalsecret.yaml
-  - netbird-backup-externalsecret.yaml
-  - netbird-ingress-externalsecret.yaml
-  - velero-externalsecret.yaml
-  - velero-hetzner-creds-externalsecret.yaml
-  # traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
-  # infrastructure. Traefik is deliberately absent from that Kustomization's
-  # healthChecks, so this KS may dry-run before the chart lands; it retries on
-  # retryInterval rather than deadlocking, because nothing here installs the
-  # CRDs it is waiting for.
-  - traefik-crowdsec-middleware.yaml
-  - traefik-rate-limit-middleware.yaml
-  - traefik-security-headers-middleware.yaml
  - ../network-policies/
@@ -5,7 +5,7 @@ metadata:
  namespace: rook-ceph
 spec:
  cephVersion:
-    image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  # Defaults to false. With three OSDs and size 3 there is no spare copy, so
@@ -78,20 +78,7 @@ spec:
    enabled: true
    ssl: false
  monitoring:
-    # false, and this is not a decision about metrics. `metricsDisabled` is the
-    # switch that turns metrics off; this one only decides whether Rook creates
-    # the ServiceMonitors — and creating them is fatal to the reconcile when it
-    # fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
-    # Rook's node-daemon reconcile aborted, the CephCluster never left
-    # status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
-    # waits on exactly that — with grafana-db → monitoring → those same CRDs
-    # behind it, closing the loop.
-    #
-    # The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
-    # only the scrape wiring moved, to apps/monitoring-rules/
-    # rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
-    # Both objects are reproduced there field for field off the live cluster.
-    enabled: false
+    enabled: true
  # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
  # Multus. The attachment is san-network-attachment.yaml.
  #
@@ -144,7 +131,7 @@ spec:
    useAllDevices: false
    # Two forms while the production drives are migrated:
    #
-    #   by-partlabel/r-osd-data  the target. Talos names the drive in the
+    #   by-partlabel/r-ceph-osd  the target. Talos names the drive in the
    #                            machine config (RawVolumeConfig ceph-osd), so
    #                            this file needs to know nothing about models,
    #                            serials or hostnames.
@@ -159,7 +146,7 @@ spec:
    #
    # The second half goes when those three are repartitioned, one node at a
    # time. Until then it is what keeps the running OSDs claimed.
-    devicePathFilter: "^/dev/disk/by-(partlabel/r-osd-data|id/nvme-SAMSUNG_MZQLB960HAJR)"
+    devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
    config:
      osdsPerDevice: "1"
  # Highest built-in class: kubelet evicts these last and the kernel gives
@@ -3,26 +3,33 @@
 # without every Ceph daemon landing on host networking to get there. The
 # alternative and why it was rejected is in cephcluster.yaml.
 #
-# host-device: the SAN NIC is dedicated to Ceph and has exactly one consumer
-# per node — the OSD pod — so it is handed to that pod whole, not virtualised
-# under it. The device keeps its identity (MAC, MTU) and moves back to the
-# host when the pod goes away.
-#
-# `device` names the role, not hardware: every machine renames its own SAN
-# interface to ceph-san at boot (udev rule in its Talos patch). No kernel
-# interface name appears in this file.
-#
 # ISOLATION
 #
-# VLAN 49 carries Ceph-internal traffic and nothing else. No gateway and no
-# routes: there is no router on this segment and there must not be one.
-# whereabouts adds neither unless asked, so nothing here can leak onto or off
-# the SAN.
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
 #
-# The pool is the whole usable /29, .1 through .6 — headroom for a leaked
-# allocation to sit until the reconciler collects it. One address per node is
-# in use; a second OSD on a node would need a second SAN interface, which is a
-# hardware change, not a config one.
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
 #
 # IPAM
 #
@@ -30,18 +37,29 @@
 # a server that must not exist here, and host-local keeps its state per node
 # with no cluster-wide coordination, so two nodes would hand out the same
 # address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
 apiVersion: k8s.cni.cncf.io/v1
 kind: NetworkAttachmentDefinition
 metadata:
  name: ceph-san
  namespace: rook-ceph
 spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "ceph-san",
-      "type": "host-device",
-      "device": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
      "ipam": {
        "type": "whereabouts",
        "range": "10.99.0.0/29",
@@ -38,7 +38,7 @@ spec:
      serviceAccountName: rook-ceph-default
      containers:
        - name: rook-ceph-tools
-          image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
+          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
          command:
            - /bin/bash
            - -c
@@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - helmrelease.yaml
+  - crowdsec-middleware.yaml
+  - security-headers-middleware.yaml
+  - rate-limit-middleware.yaml
@@ -1,9 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
  - snapshot-readonly-storageclass.yaml
  - node-agent-config.yaml
  - helmrelease.yaml
+  - servicemonitor.yaml
+  - podmonitor.yaml
  - restore-test-rbac.yaml
  - restore-test-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/
@@ -1,69 +0,0 @@
-# First machine of the throwaway cluster on Hetzner.
-#
-# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
-# order bootstrap.sh attaches them.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-1
-  nodeLabels:
-    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
-    # rather than MAC: the address follows from the order bootstrap.sh attaches
-    # the networks, while the MAC is assigned by Hetzner at attach time and is
-    # not known when this file is written.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
-# In the generated schema rather than machine.network.hostname: naming the host
-# in both is rejected at apply time. `auto: off` switches off the generated
-# `auto: stable`, which is mutually exclusive with a static name.
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-1
---
-# Every interface is stated, the public one included. Talos stops running its
-# default DHCP operators for the whole machine as soon as one new-style network
-# document exists, so an omitted interface gets no address.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
---
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
---
-# Ceph cluster network, renamed to the role name by the udev rule in the
-# machine section. The node holds no address here; the link exists for the OSD
-# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
---
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    # Two disks: the system disk and the attached volume. Matching on transport
-    # as the qemu patches do selects nothing here — a Hetzner volume arrives
-    # over virtio-scsi, and a selector matching zero disks is skipped silently.
-    match: '!system_disk'
-  # bootstrap.sh reads this size and creates the volume to match.
-  minSize: 50GiB
-  grow: true
@@ -1,50 +0,0 @@
-# Second machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-2
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-2
---
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
---
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
---
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
---
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true
@@ -1,50 +0,0 @@
-# Third machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-3
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
---
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-3
---
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
---
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
---
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
---
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true
@@ -28,13 +28,6 @@ machine:
    # this label instead of listing hostnames, so which boxes carry storage is
    # stated once, here, by the machine that knows.
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
-    # role name at boot, before the network stack opens links — a rule applied
-    # to a RUNNING node cannot rename (Resource busy) and takes effect at the
-    # next reboot.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
@@ -131,15 +124,14 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
-# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
-# section above renames the port to ceph-san at boot, so shared manifests name
-# the role and only this file knows the hardware.
+# wire. OSD-to-OSD replication and nothing else.
 #
-# The node holds NO address here on purpose: the link is handed whole to the
-# OSD pod (host-device attachment). Giving the node an address would put it on
-# a segment reserved for Ceph-internal traffic.
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
+# Giving the node an address would put it on a segment reserved for
+# Ceph-internal traffic.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000
@@ -23,10 +23,6 @@ machine:
    topology.kubernetes.io/zone: saturn
    # Holds a Ceph OSD drive — see k8s-1.yaml.
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
@@ -99,12 +95,11 @@ up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.
-# Renamed to the role name by the udev rule above — see k8s-1.yaml.
 #
-# The node holds NO address here on purpose: the link is handed whole to the
-# OSD pod (host-device attachment).
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000
@@ -19,10 +19,6 @@ machine:
    topology.kubernetes.io/zone: neptune
    # Holds a Ceph OSD drive — see k8s-1.yaml.
    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
@@ -82,9 +78,9 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
-# here on purpose; the link is handed whole to the OSD pod (host-device).
+# here on purpose; the link is only a macvlan parent for the OSD pods.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000
@@ -1,25 +0,0 @@
-# Site patch for the throwaway cluster on Hetzner Cloud.
-#
-# This site exists for one thing the other two cannot test: whether a node that
-# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
-# answers, so containerd never reaches the second endpoint in
-# machine.registries.mirrors and the fallback stays unproven.
-#
-# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
-# frames from it do not cross between servers — measured: same-node ARP
-# resolves, cross-node does not. OSD replication therefore does not run here.
-#
-# What belongs in this file is what the cluster is. Interfaces, disks and names
-# are machine properties and live in the node patches.
-machine:
-  kubelet:
-    nodeIP:
-      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
-      # kubelet can pick the public NIC.
-      validSubnets:
-        - 10.90.0.0/16
-cluster:
-  # Three control planes that also carry workloads, which is the production
-  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
-  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
-  allowSchedulingOnControlPlanes: true
@@ -43,15 +43,3 @@ apiVersion: v1alpha1
 kind: HostnameConfig
 auto: off
 hostname: vm-1
---
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2
@@ -21,7 +21,7 @@ auto: off
 hostname: vm-2
 ---
 # The OSD device, named by the machine that has it. Ceph then consumes
-# /dev/disk/by-partlabel/r-osd-data and needs to know nothing about models,
+# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
 # serials or hostnames — the same CephCluster works here and on the MS-01s.
 #
 # The selector is a CEL expression, so each machine says whatever identifies its
@@ -29,33 +29,9 @@ hostname: vm-2
 # real nodes it will name the Samsung.
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
  diskSelector:
    match: disk.transport == "virtio" && !system_disk
  minSize: 5GiB
  grow: true
---
-# The Ceph cluster network link. Production gives it a second physical port;
-# this machine has one NIC, so the segment is a VLAN on it.
-#
-# A dummy link would be simpler and worthless — no traffic between nodes, so
-# Ceph would replicate over an interface going nowhere and the test would pass
-# having proven nothing. A VLAN rides the provisioner's bridge that every VM is
-# on, so the SAN carries real frames on its own tag.
-#
-# No MTU, unlike production's 9000: the qemu NIC is created with host_mtu=1500.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
---
-# Load-bearing. Declaring any network document makes Talos configure this link
-# explicitly, and the implicit DHCP that gives the node its address stops. A
-# first attempt with only the VLANConfig took vm-4 off the network — NotReady,
-# unreachable over the Talos API. Verified on a second throwaway node that this
-# document keeps the parent addressed.
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2
@@ -15,21 +15,9 @@ hostname: vm-3
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
  diskSelector:
    match: disk.transport == "virtio" && !system_disk
  minSize: 5GiB
  grow: true
---
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2
@@ -19,21 +19,9 @@ hostname: vm-4
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
  diskSelector:
    match: disk.transport == "virtio" && !system_disk
  minSize: 5GiB
  grow: true
---
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2
@@ -1,128 +0,0 @@
-# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
-#
-# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
-# one-shot test, and a Kustomization would recreate it after every completion.
-# It is applied by the vmtest pipeline, or by hand for a single run.
-#
-# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
-# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
-# a single run left under 1G free and two overlapping runs OOM-killed the
-# session. The nodes have ~94G each with about 70G unused.
-#
-# The whole script runs here, flux and OpenBao included, so that what a rebuild
-# rehearses is what a real recovery would execute. The unseal key is not baked
-# in and not stored as a Secret: the operator streams all five secrets into an
-# in-memory emptyDir at the moment the run asks for them, and they die with the
-# Job without ever reaching etcd or a backup of it.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: vmtest
-  labels:
-    # The pod is privileged; say so rather than have the admission policy
-    # reject it for reasons that look unrelated.
-    pod-security.kubernetes.io/enforce: privileged
---
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: vmtest
-  namespace: vmtest
-spec:
-  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
-  ttlSecondsAfterFinished: 86400
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: vmtest
-    spec:
-      restartPolicy: Never
-      # Nothing here should be able to reach the cluster it runs in. The phase
-      # points KUBECONFIG at its own throwaway cluster, and without a mounted
-      # token a stray kubectl cannot fall back to this one.
-      automountServiceAccountToken: false
-      containers:
-        - name: vmtest
-          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
-          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
-          # three builds old, which predated OVMF, and died with "the required
-          # flash image was not found" before the first VM booted. The tag is
-          # carried alongside for readability — containerd resolves the digest.
-          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
-          # Privileged for /dev/kvm and for the bridges and taps the provisioner
-          # builds. It needs no hostNetwork: the provisioner's network lives in
-          # this pod's own namespace, and the script that talks to 10.5.0.x runs
-          # in the same pod.
-          securityContext:
-            privileged: true
-          # A hard ceiling, because the failure mode is known: an unbounded run
-          # is what OOM-killed the workstation. 40G is the VMs (16G control
-          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
-          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
-          # of room for the CNI DaemonSets themselves. The control plane went
-          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
-          # memory and the kubelet evicted kube-apiserver to admit a
-          # node-critical pod — bootstrap.sh carries the full account next to
-          # the sizing flags. The hosts have ~95GiB each, so this fits.
-          resources:
-            requests:
-              cpu: "6"
-              memory: 42Gi
-            limits:
-              memory: 48Gi
-          env:
-            # talosctl caches the image and the CNI bundle under $HOME; point it
-            # at the scratch volume rather than the container layer.
-            - name: HOME
-              value: /work
-            # The VM disks. Short on purpose — the provisioner puts a qemu
-            # monitor socket under it and sun_path caps the path at 108 bytes.
-            - name: VM_STATE
-              value: /work/vm
-            # Which branch to test. master unless a run is checking a change.
-            - name: BRANCH
-              value: master
-            # Secrets are streamed in here at the moment the run needs them,
-            # never baked into the image and never stored as a Secret. They live
-            # in this emptyDir for the length of the Job and go with it — so
-            # they do not reach etcd, or any backup of it.
-            - name: OPENBAO_SECRETS_DIR
-              value: /secrets
-          command: ["/bin/bash", "-ec"]
-          args:
-            - |
-              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
-              # that clones from the cluster it is rehearsing the loss of
-              # cannot start when it matters — and a run already died here on
-              # resolving gitea.renner.uno.
-              git clone --depth 1 --branch "$BRANCH" \
-                https://github.com/konairius/Homelab.git /work/homelab
-              exec /work/homelab/kubernetes/bootstrap.sh vmtest
-          volumeMounts:
-            - name: work
-              mountPath: /work
-            - name: secrets
-              mountPath: /secrets
-            - name: dev-kvm
-              mountPath: /dev/kvm
-      volumes:
-        # The VM disks and the provisioner state. emptyDir on the node's disk
-        # rather than a PVC: it is scratch that dies with the Job, and putting it
-        # on Ceph would have the test's own writes compete with the storage it is
-        # meant to leave alone.
-        - name: work
-          emptyDir:
-            # The VM images are sparse, so this bounds what is actually written,
-            # not the 232GiB the geometry declares. 60Gi was already half spent
-            # by four 10GiB systems disks; the node carries this comfortably with
-            # ~810GB free on the filesystem behind it.
-            sizeLimit: 150Gi
-        # In memory, so the material never touches a node's disk.
-        - name: secrets
-          emptyDir:
-            medium: Memory
-            sizeLimit: 1Mi
-        - name: dev-kvm
-          hostPath:
-            path: /dev/kvm
-            type: CharDevice
@@ -13,18 +13,7 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
-# The default is unchanged and deliberately so: "secrets" resolves to working
-# credentials and this script has always used it. Both "secrets" and "hetzner"
-# exist in ~/.aws/credentials on the workstation.
-#
-# The defect is only the assignment. Forcing the value defeated the env-based
-# path this file's own header offers, and it also overrode a caller who had
-# already selected a profile — which is how a 403 from HeadObject can appear
-# while the credentials the caller intended were perfectly good. That error
-# reads as a bucket permissions problem and sends you looking in the wrong
-# place.
-export AWS_PROFILE="${AWS_PROFILE:-secrets}"
+export AWS_PROFILE="secrets"


 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }
Author	SHA1	Message	Date
claude-botandClaude Opus 5	89e011fd2e	feat(llama-server): split embeddings off, move chat to Qwen3.6-35B-A3B ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Two defects surfaced once the endpoint was live, and both trace back to router mode. The embedding model was selectable in the chat UI. Every model on the port appears in the picker, and choosing the embedding one returns fluent nonsense ("capital capital capital ...") rather than an error, because --embeddings does not actually refuse chat completions. llama.cpp offers no way to hide a model from the list: --tags is informational, --ui-config only sets UI preferences. So the embedding model moves to its own Deployment on a ClusterIP-only Service with --no-webui, which is what keeps it out of the picker. Readiness was also wrong. The router answers /health 200 as soon as it listens, while models still download behind it, so the pod went Ready seconds after start and traffic arrived at a server that could not answer — that is how the embedding model got reached at all. A single-model server returns 503 until the weights are resident, which is what the probes needed all along. Splitting fixes both, at the cost of router mode's single endpoint. Consumers now use llama-server for chat and llama-embed for embeddings; both share one PVC as LLAMA_CACHE, so nothing is downloaded twice. The chat model changes to a mixture-of-experts. Llama-3.1-8B was picked only because it was the model benchmarked, which is continuity rather than a quality argument, and it showed: fluent but wrong. Since decode is set by bytes read per token, an MoE reading ~1.85 GB/token beats a dense model reading ~16.5 GB. Measured on k8s-1 with this image: tg128 11.59 tok/s (dense 8B: 10.78) pp512 120.93 t/s pp2048 126.47 t/s pp8192 92.79 t/s pp16384 69.84 t/s Four times the model, slightly faster. The prefill falloff is sublinear, so the Gated Delta Net collapse of llama.cpp#27638 does not occur here and long documents stay viable; context therefore goes to 16k, which is cheap because only 10 of 40 layers use full attention. GLM-5.2/5.3 were evaluated and are impossible on this hardware at 754B parameters (~400 GB), as is GLM-5.3-Flash at 321B with an architecture the pinned build does not implement. Dense Qwen3.8-27B reads 16.5 GB/token and manages 3.3 tok/s. Memory goes to 24Gi/32Gi: Vulkan allocates the iGPU working set from system RAM and it counts against the pod cgroup, measured at 7.54 GiB RSS for 5.2 GiB of weights. --ubatch-size 1024 because 2048 and above fail with ErrorDeviceLost on this Vulkan/Mesa stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 16:25:04 +02:00
k.renner	593fb6b7d9	Merge pull request 'feat(llama-server): local LLM inference on the Intel iGPU' (#721 ) from llama-server into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#721	2026-09-01 12:31:13 +00:00
claude-botandClaude Opus 5	d4a683a0ad	docs(cold-rebuild): the i915 extension now backs a schedulable GPU resource ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details The runbook said nothing requests /dev/dri and the extension was carried only so hardware transcoding would later be a config change. Both halves are now out of date: infrastructure/intel-gpu-plugin advertises gpu.intel.com/i915 from it, and apps/llama-server requests that resource. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 14:25:21 +02:00
claude-botandClaude Opus 5	1551523764	refactor(llama-server): drop the download script, let llama-server fetch weights The initContainer's inline shell script named each model a second time, so the model existed in two places: models.ini and the download script. Changing a model in the ConfigMap alone left the pod fetching the old file and failing to start, and the script itself was untested, unlinted YAML-embedded shell. llama-server resolves Hugging Face repos natively, and `hf-repo` is honoured inside a preset section: server-models.cpp calls unset_reserved_args(preset, false) for model instances, so LLAMA_ARG_HF_REPO survives into the child; only the router's own base preset has it stripped. Pointing LLAMA_CACHE at the models PVC makes the download a one-time cost as before. Verified against the pinned image rather than inferred: `-hf Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0` reaches "model loaded", and the rendered models.ini yields "Loaded 2 custom model presets" with both `chat` and `embed` registered and --hf-repo present in the spawned child's args. models.ini is now the only place a model is named. Also sets --cors-origins localhost. The server defaults to allowing every origin and logs a warning about it at startup; without this, any page a LAN browser visits could call this unauthenticated API from the user's machine. Every consumer talks to it server-side, and the built-in WebUI is same-origin, so nothing needs cross-origin access. Co-Authored-By: Claude Bot <claude-bot@renner.uno> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 14:24:54 +02:00
claude-botandClaude Opus 5	40362925fa	feat(llama-server): local LLM inference on the Intel iGPU Adds llama.cpp's server in router mode as the cluster's OpenAI-compatible inference endpoint, in a new `ai` namespace, running entirely on the on-board Iris Xe via the `gpu.intel.com/i915` resource. One Deployment serves both a chat model and an embedding model: started with --models-preset and no -m, llama-server supervises each INI section as its own subprocess behind a single port, routed by the request body's "model" field. That gives /v1/chat/completions, /v1/embeddings and /v1/rerank from one pod, with a crash in one model not taking out the other. Backend and build are chosen from measurement rather than documentation. On this hardware (i9-12900H, Iris Xe 96 EU) Vulkan beats the SYCL/oneAPI build by +50% prefill and +36% decode, so the Vulkan image is pinned to b10731 — the exact build benchmarked. Full offload (-ngl 99) matches CPU throughput at ~10.8 tok/s decode and ~137 t/s prefill while leaving the node's 14 cores free for the rest of the cluster; the iGPU is here for that isolation, not speed. The weights are fetched by an initContainer on first start, so no manual seeding step stands between a merge and a working endpoint. They land on a prune-disabled CephFS RWX PVC and are skipped on later starts. Exposed LAN-only at llm.renner.uno: the API is unauthenticated, so the `dns.renner.uno/public` label is deliberately omitted and only the UDM external-dns instance publishes the record. The pod needs no hostPath and no privilege — the device plugin passes the render node in through the device-plugin API — so `ai` takes the cluster's default `baseline` enforcement with no pod-security labels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 14:24:54 +02:00
k.renner	0fa4b8d361	Merge pull request 'feat(intel-gpu-plugin): expose the Intel iGPU as gpu.intel.com/i915' (#720 ) from intel-gpu-device-plugin into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/security-scan Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details ci/woodpecker/cron/e2e Pipeline failed Details Reviewed-on: k.renner/homelab#720	2026-09-01 10:44:44 +00:00
claude-botandClaude Opus 5	c868b8cd9d	feat(intel-gpu-plugin): expose the Intel iGPU as gpu.intel.com/i915 ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Adds the Intel GPU device plugin as a DaemonSet in kube-system, making the on-board Iris Xe of each MS-01 node schedulable as `gpu.intel.com/i915`. Pods can then request the iGPU without a /dev/dri hostPath or privilege escalation — the plugin passes the render node through the kubelet device-plugin API. The resource covers QuickSync transcoding as well as GPU compute. Vendored from upstream v0.36.0 rather than installed from the `intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin` custom resource requiring the Intel Device Plugin Operator, its CRDs and a cert-manager webhook. One device type on three identical nodes gives the operator nothing to manage that the DaemonSet does not. Runs in kube-system because the plugin mounts hostPaths; the cluster enforces PodSecurity `baseline`, which forbids those, and kube-system is the only namespace in the API server's exemption list. NFD is not deployed, so upstream's nfd_labeled_nodes overlay and its nodeSelector are omitted. `-shared-dev-num=4` lets several containers share the single iGPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 12:38:42 +02:00
security-bot	99445753d1	chore(image-automation): bump valkey/valkey:9.1.2-alpine ci/woodpecker/cron/alert-monitor Pipeline was successful Details ci/woodpecker/cron/issue-fix Pipeline was successful Details Updates managed by image-automation-controller.	2026-09-01 02:21:46 +00:00