fix(gitea-mcp): set command so args don't replace the image CMD

docker.gitea.com/gitea-mcp-server has CMD ["/app/gitea-mcp"] and no ENTRYPOINT, so a Pod with only args runs "-t" as the executable: the container failed with 'exec: "-t": executable file not found in $PATH' and crash-looped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge pull request 'feat(gitea-mcp): in-cluster Gitea MCP server on the NetBird proxy' (#1240 ) from feat/gitea-mcp into master
2026-10-11 11:37:11 +02:00 · 2026-10-11 09:30:29 +00:00 · 2026-10-11 11:28:52 +02:00 · 2026-10-11 11:28:44 +02:00 · 2026-10-11 11:25:19 +02:00 · 2026-10-11 11:24:49 +02:00
44 changed files with 1278 additions and 1110 deletions
@@ -101,7 +101,7 @@ steps:
        bash scripts/ci/make-diag-kubeconfig.sh

  - name: alert-triage-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -35,19 +35,19 @@ when:
 steps:
  - name: model-selection-tests
    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    commands:
      - sh scripts/ci/test-model-selection.sh

  - name: promtool-rules-guard-tests
    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    commands:
      - sh scripts/ci/test-promtool-rules.sh

  - name: yaml-citations-guard-tests
    depends_on: []
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    commands:
      - sh scripts/ci/test-yaml-citations.sh

@@ -26,7 +26,7 @@ steps:
    # fetch-models.sh response mapping. No cluster, no secrets, no network — the
    # ci-claude image just provides sh + jq. Keeps the flag-injection regression
    # test from rotting (it exercises the exact injection shapes).
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    commands:
      - sh scripts/ci/test-model-selection.sh

@@ -62,7 +62,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: issue-fix-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -70,7 +70,7 @@ steps:
    # this step is skipped — otherwise we'd fork-bomb PRs.
    when:
      - branch: master
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -30,7 +30,7 @@ when:

 steps:
  - name: update-audit-agent
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
        from_secret: claude_code_oauth_token
@@ -17,7 +17,7 @@ when:

 steps:
  - name: checkout-notes
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    pull: true
    environment:
      WIKI_BOT_TOKEN:
@@ -54,7 +54,7 @@ steps:
      - bash scripts/ci/make-diag-kubeconfig.sh

  - name: distill
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    pull: true
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
@@ -89,14 +89,14 @@ steps:
          -p

  - name: audit
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    pull: true
    commands:
      # Findings are report lines in output/, not a pipeline failure.
      - python3 -B notes/bin/wiki_audit.py --vault "$$PWD/notes"

  - name: consolidate
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    pull: true
    environment:
      CLAUDE_CODE_OAUTH_TOKEN:
@@ -145,7 +145,7 @@ steps:
        exit $$rc

  - name: push-notes
-    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.289
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
    pull: true
    environment:
      WIKI_BOT_TOKEN:
@@ -1,7 +1,10 @@
 #!/usr/bin/env python3
 """netbird-service-sync: reconcile NetBird reverse-proxy services from Ingresses.

-Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
+Source of truth: every Ingress carrying one of the public labels.
+dns.renner.uno/public also takes the host out of the LAN DNS (external-dns-unifi
+skips it); netbird.renner.uno/public publishes on the proxy only and leaves the
+LAN record in place.
 For each host on such an Ingress the NetBird management gets one reverse-proxy
 service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
 subnet resource that covers the cluster's service CIDR. SNI and Host reach
@@ -42,7 +45,7 @@ NETWORK_NAME = os.environ.get("NETWORK_NAME", "k8s public ingress")
 ROUTER_GROUP = os.environ.get("ROUTER_GROUP_NAME", "k8s-ingress-router")
 SOURCE_GROUP = os.environ.get("SOURCE_GROUP_NAME", "All")
 TRAEFIK_NS, TRAEFIK_SVC = os.environ.get("TRAEFIK_SERVICE", "ingress/traefik").split("/", 1)
-PUBLIC_LABEL = os.environ.get("PUBLIC_LABEL", "dns.renner.uno/public")
+PUBLIC_LABELS = os.environ.get("PUBLIC_LABELS", "dns.renner.uno/public,netbird.renner.uno/public").split(",")
 LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "443"))
 INTERVAL = int(os.environ.get("INTERVAL_SECONDS", "60"))
 HTTP_TIMEOUT = 20
@@ -86,10 +89,12 @@ def k8s(path):


 def desired_hosts():
-    sel = urllib.parse.quote(f"{PUBLIC_LABEL}=true")
-    ingresses = k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}")
+    items = []
+    for label in PUBLIC_LABELS:
+        sel = urllib.parse.quote(f"{label}=true")
+        items += k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}").get("items", [])
    hosts = {}
-    for ing in ingresses.get("items", []):
+    for ing in items:
        ref = f'{ing["metadata"]["namespace"]}/{ing["metadata"]["name"]}'
        for rule in ing.get("spec", {}).get("rules", []):
            host = rule.get("host")
@@ -0,0 +1,443 @@
+# Gitea MCP Server Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Deploy the official Gitea MCP server in-cluster at `https://gitea-mcp.renner.uno/mcp`, published on the NetBird proxy, authenticating every request with the caller's own Gitea PAT and rejecting tokenless requests.
+
+**Architecture:** One stateless Deployment in a new `gitea-mcp` namespace, Flux-managed through its own Kustomization. Traefik Ingress with the proxy-only public label; netbird-service-sync publishes it. gitea-mcp runs in `-oauth` mode purely to get the 401-on-missing-token gate; it talks to Gitea via the in-cluster Service.
+
+**Tech Stack:** Kubernetes manifests, Flux (Kustomization, image automation), Cilium CiliumNetworkPolicy, Traefik Ingress, cert-manager.
+
+**Spec:** `docs/superpowers/specs/2026-10-11-gitea-mcp-design.md`
+
+## Global Constraints
+
+- Image `docker.gitea.com/gitea-mcp-server`, initial tag `1.8.1`, ImagePolicy semver `>=1.8.1`, marker `# {"$imagepolicy": "flux-system:gitea-mcp"}`.
+- Args exactly: `-t http -p 8080 -oauth -oauth-public-url https://gitea-mcp.renner.uno -H http://gitea.gitea.svc.cluster.local:3000`. No `-T`/`GITEA_ACCESS_TOKEN`, no `-d`.
+- Host `gitea-mcp.renner.uno`; Ingress label `netbird.renner.uno/public: "true"` (not `dns.renner.uno/public`); cert issuer `letsencrypt-cluster`; TLS secret `gitea-mcp-tls`.
+- Pod label `app.kubernetes.io/name: gitea-mcp`; namespace `gitea-mcp`; Service `gitea-mcp:8080`.
+- No `world` egress from `gitea-mcp`.
+- Manifest comments: one terse factual line at most; rationale goes in commit/PR text.
+- Commits as claude-bot: `git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit …`; each git command its own Bash call.
+
+## Review Focus
+
+1. Tokenless `POST /mcp` from the internet → must be 401, never anonymous Gitea data (Task 4 step 4).
+2. Request with an invalid/revoked PAT → 401 `invalid_token` (Task 4 step 5).
+3. Kubelet probes with `Host: <podIP>` → `/healthz` must stay 200 despite the oauth Host check (Task 3 readiness, Task 4 step 2).
+4. LAN client resolving `gitea-mcp.renner.uno` → must reach Traefik VIP directly, not the proxy (Task 4 step 3).
+5. Gitea refusing gitea-mcp's connection (missing CNP half) → MCP calls fail; authenticated `list_my_repos` must succeed (Task 4 step 6).
+
+---
+
+### Task 1: Namespace and network policies
+
+**Files:**
+- Modify: `kubernetes/infrastructure/namespaces/namespaces.yaml` (append after the `gitea` Namespace)
+- Create: `kubernetes/infrastructure/network-policies/gitea-mcp-ns.yaml`
+- Modify: `kubernetes/infrastructure/network-policies/kustomization.yaml` (add entry after `gitea-ns.yaml`)
+- Modify: `kubernetes/apps/gitea/networkpolicy.yaml` (append CNP)
+
+**Interfaces:**
+- Produces: namespace `gitea-mcp`; ingress to pods labelled `app.kubernetes.io/name: gitea-mcp` on 8080 from Traefik; egress from `gitea-mcp` to `gitea` pod :3000; Gitea-side ingress allowance from namespace `gitea-mcp` on :3000.
+
+- [ ] **Step 1: Add the Namespace** — insert after the `name: gitea` Namespace document in `namespaces.yaml`:
+
+```yaml
+---
+# Gitea MCP server. Declared here so it exists before infrastructure-crds applies its CNPs.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: gitea-mcp
+```
+
+- [ ] **Step 2: Create `gitea-mcp-ns.yaml`**
+
+```yaml
+---
+# Baseline policy for the `gitea-mcp` namespace.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny
+  namespace: gitea-mcp
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea
+            app.kubernetes.io/name: gitea
+      toPorts:
+        - ports:
+            - port: "3000"
+              protocol: TCP
+---
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-mcp-allow-traefik
+  namespace: gitea-mcp
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea-mcp
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+```
+
+- [ ] **Step 3: Register it** — in `network-policies/kustomization.yaml` add `  - gitea-mcp-ns.yaml` directly after `  - gitea-ns.yaml`.
+
+- [ ] **Step 4: Gitea-side allowance** — append to `kubernetes/apps/gitea/networkpolicy.yaml`:
+
+```yaml
+---
+# gitea-mcp reaches the in-cluster Service; egress half is in network-policies/gitea-mcp-ns.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-gitea-mcp
+  namespace: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+            app.kubernetes.io/name: gitea-mcp
+      toPorts:
+        - ports:
+            - port: "3000"
+              protocol: TCP
+```
+
+- [ ] **Step 5: Validate builds**
+
+Run each separately:
+`kubectl kustomize kubernetes/infrastructure/namespaces` → output contains `name: gitea-mcp`
+`kubectl kustomize kubernetes/infrastructure/network-policies` → output contains `gitea-mcp-allow-traefik`
+`kubectl kustomize kubernetes/apps/gitea` → output contains `gitea-allow-gitea-mcp`
+
+- [ ] **Step 6: Server-side dry-run of the Gitea-side CNP** (namespace already exists):
+`kubectl apply --dry-run=server -f kubernetes/apps/gitea/networkpolicy.yaml`
+Expected: every CNP `configured (server dry run)` or `unchanged`, `gitea-allow-gitea-mcp created (server dry run)`.
+
+- [ ] **Step 7: Commit**
+
+`git add kubernetes/infrastructure/namespaces/namespaces.yaml kubernetes/infrastructure/network-policies/gitea-mcp-ns.yaml kubernetes/infrastructure/network-policies/kustomization.yaml kubernetes/apps/gitea/networkpolicy.yaml`
+`git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit -m "feat(gitea-mcp): namespace and network policies" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`
+
+### Task 2: Image automation
+
+**Files:**
+- Create: `kubernetes/infrastructure/image-automation/gitea-mcp.yaml`
+- Modify: `kubernetes/infrastructure/image-automation/kustomization.yaml` (insert `  - gitea-mcp.yaml` alphabetically after `  - gatus.yaml`)
+
+**Interfaces:**
+- Produces: ImagePolicy `flux-system:gitea-mcp`, referenced by the marker in Task 3.
+
+- [ ] **Step 1: Create `gitea-mcp.yaml`**
+
+```yaml
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  image: docker.gitea.com/gitea-mcp-server
+  interval: 1h
+---
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: gitea-mcp
+  policy:
+    semver:
+      range: ">=1.8.1"
+```
+
+- [ ] **Step 2: Register it** in `image-automation/kustomization.yaml` after `  - gatus.yaml`.
+
+- [ ] **Step 3: Validate**
+`kubectl kustomize kubernetes/infrastructure/image-automation` → contains `docker.gitea.com/gitea-mcp-server`
+`kubectl apply --dry-run=server -f kubernetes/infrastructure/image-automation/gitea-mcp.yaml` → both objects `created (server dry run)`.
+
+- [ ] **Step 4: Commit**
+
+`git add kubernetes/infrastructure/image-automation/gitea-mcp.yaml kubernetes/infrastructure/image-automation/kustomization.yaml`
+`git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit -m "feat(gitea-mcp): track docker.gitea.com/gitea-mcp-server via image automation" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`
+
+### Task 3: App manifests and Flux Kustomization
+
+**Files:**
+- Create: `kubernetes/apps/gitea-mcp/deployment.yaml`, `service.yaml`, `ingress.yaml`, `kustomization.yaml`
+- Create: `kubernetes/clusters/homelab/gitea-mcp.yaml`
+
+**Interfaces:**
+- Consumes: namespace + CNPs (Task 1), ImagePolicy `flux-system:gitea-mcp` (Task 2).
+- Produces: `https://gitea-mcp.renner.uno/mcp` and `/healthz`.
+
+- [ ] **Step 1: `deployment.yaml`**
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: gitea-mcp
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: gitea-mcp
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: gitea-mcp
+          image: docker.gitea.com/gitea-mcp-server:1.8.1 # {"$imagepolicy": "flux-system:gitea-mcp"}
+          args:
+            - -t
+            - http
+            - -p
+            - "8080"
+            # Only for the 401 on tokenless requests; clients send their own Gitea PAT.
+            - -oauth
+            - -oauth-public-url
+            - https://gitea-mcp.renner.uno
+            - -H
+            - http://gitea.gitea.svc.cluster.local:3000
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          ports:
+            - containerPort: 8080
+              name: http
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 5
+            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 2
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+      volumes:
+        - name: tmp
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
+```
+
+- [ ] **Step 2: `service.yaml`**
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: gitea-mcp
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
+```
+
+- [ ] **Step 3: `ingress.yaml`**
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+    # Published on the NetBird proxy; LAN DNS stays on the Traefik VIP.
+    netbird.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: gitea-mcp.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea-mcp
+                port:
+                  number: 8080
+  tls:
+    - secretName: gitea-mcp-tls
+      hosts:
+        - gitea-mcp.renner.uno
+```
+
+- [ ] **Step 4: `kustomization.yaml`**
+
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml
+```
+
+- [ ] **Step 5: `kubernetes/clusters/homelab/gitea-mcp.yaml`**
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/gitea-mcp
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+```
+
+- [ ] **Step 6: Validate**
+`kubectl kustomize kubernetes/apps/gitea-mcp` → renders Deployment, Service, Ingress, all in namespace `gitea-mcp`.
+`kubectl apply --dry-run=server -f kubernetes/clusters/homelab/gitea-mcp.yaml` → `created (server dry run)`.
+Confirm `kubernetes/clusters/homelab/` is picked up without a listing file: `ls kubernetes/clusters/homelab/flux-system` and read its `kustomization.yaml`; if it lists files explicitly, add `gitea-mcp.yaml` where `gitea.yaml` is listed.
+
+- [ ] **Step 7: Commit**
+
+`git add kubernetes/apps/gitea-mcp kubernetes/clusters/homelab/gitea-mcp.yaml`
+`git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit -m "feat(gitea-mcp): deploy the Gitea MCP server at gitea-mcp.renner.uno" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`
+
+### Task 4: PR, rollout and live verification
+
+- [ ] **Step 1: Push and open the PR**
+
+`git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push -u origin feat/gitea-mcp`
+`tea pr create --login claude-bot --repo k.renner/homelab --base master --head feat/gitea-mcp --title "feat(gitea-mcp): in-cluster Gitea MCP server on the NetBird proxy" --description "<summary of spec + verification plan; ends with 🤖 Generated with [Claude Code](https://claude.com/claude-code)>"`
+Wait for `ci-manifests-test` green (`woodpecker` CLI). Owner merges; never self-merge.
+
+- [ ] **Step 2: After merge — rollout**
+`flux get kustomization gitea-mcp` → Ready True. `kubectl -n gitea-mcp get pods` → 1/1 Running, 0 restarts.
+`curl -sS -o /dev/stdout -w "\nHTTP %{http_code}\n" https://gitea-mcp.renner.uno/healthz` → `ok`, HTTP 200.
+
+- [ ] **Step 3: DNS paths**
+`getent ahosts gitea-mcp.renner.uno` from the LAN → Traefik VIP (fd99::86 / 10.2.99.86).
+Public answer: query Cloudflare's authoritative NS directly (UDM intercepts `@1.1.1.1`) → NetBird proxy address. NetBird management lists a reverse-proxy service for `gitea-mcp.renner.uno` (netbird-service-sync logs via `logcli`).
+
+- [ ] **Step 4: Tokenless request is rejected**
+
+```
+curl -sS -i https://gitea-mcp.renner.uno/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' --data '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'
+```
+Expected: `HTTP/2 401` with `WWW-Authenticate: Bearer resource_metadata=…`.
+
+- [ ] **Step 5: Invalid token is rejected** — same request plus `-H 'Authorization: Bearer invalid'` → 401 with `error="invalid_token"`.
+
+- [ ] **Step 6: Authenticated calls work** — same `initialize` request with `--oauth2-bearer "$(cat /home/krenner/.config/claude-bot/gitea-token)"` → 200 with `serverInfo`. Then `tools/call` `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"list_my_repos","arguments":{}}}` with the same token → non-empty repo list; `list_my_repos` resolves the caller from the token, so a result proves the PAT was used. If the server demands `Mcp-Protocol-Version`/`Mcp-Method` headers, take the required values from the error and resend.
+
+- [ ] **Step 7: Image policy** — `flux get image policy gitea-mcp` → latest image `docker.gitea.com/gitea-mcp-server:1.8.1` or newer.
+
+- [ ] **Step 8: Wiki** — route through `wiki-ingest` → `wiki-distill`: netbird-service-sync label mechanism (`dns.renner.uno/public` vs `netbird.renner.uno/public`), and gitea-mcp deployment facts (oauth-mode-as-auth-gate, PAT passthrough, `/healthz` outside the Host check).
@@ -0,0 +1,127 @@
+# Gitea MCP server — design
+
+Date: 2026-10-11
+Status: draft, awaiting review
+
+## Goal
+
+Run the official Gitea MCP server (`gitea.com/gitea/gitea-mcp`) in the cluster
+and publish it on the NetBird proxy, so MCP clients anywhere (Claude Code on
+the workstation, Paperclip agents, other machines) can use Gitea through MCP
+with their own Gitea personal access token.
+
+## Decisions
+
+| Topic | Decision |
+|---|---|
+| Hosting | In-cluster HTTP service, own namespace `gitea-mcp`, one stateless replica. |
+| Exposure | Public via NetBird proxy, host `gitea-mcp.renner.uno`, label `netbird.renner.uno/public` (proxy-only; the LAN record on the Traefik VIP stays, so LAN clients don't hairpin). |
+| Client auth | Bearer PAT per request (`Authorization: Bearer <pat>` or `token <pat>`). The server holds no token. Every call runs with exactly the caller's Gitea permissions. |
+| Anonymous access | Rejected with 401. Achieved by running with `-oauth`: in that mode a missing token is refused, a present token is passed through unchanged and checked with one `GET /api/v1/user`. No OAuth app or client secret is needed server-side. |
+| Gitea backend | `http://gitea.gitea.svc.cluster.local:3000` (in-cluster Service, no hairpin via the ingress LB). |
+| Image | `docker.gitea.com/gitea-mcp-server`, starting at `1.8.1`, tracked by Flux image automation (semver). |
+| Read-only mode | Off; write tools exposed. Write access is bounded by each caller's token. |
+
+## Why the `-oauth` flag with PATs
+
+Gitea answers anonymous API calls: `GET /api/v1/repos/search` without
+credentials returns `k.renner/homelab`, `konairius/Wiki` and
+`gitea_admin/_cargo-index`. In plain HTTP mode (no `-oauth`, no server token)
+gitea-mcp forwards tokenless requests as an anonymous Gitea client, so the MCP
+endpoint would serve those repos to anyone. Upstream has no "require token"
+flag outside OAuth mode.
+
+With `-oauth -oauth-public-url https://gitea-mcp.renner.uno` (upstream
+`pkg/oauth/oauth.go`):
+
+- `/mcp` without a token → 401 with an RFC 9728 `WWW-Authenticate` challenge.
+- `/mcp` with a token → `TokenRejected` calls Gitea `GET /user` with it; only a
+  401 there rejects. The token is then used as-is for the API calls, so a plain
+  PAT works. There is no JWT/introspection/audience check.
+- The server only publishes `/.well-known/oauth-protected-resource` metadata;
+  config is the public URL and the Gitea URL, nothing else.
+- `/mcp` additionally checks `Host` against the public URL's host (or
+  loopback) and returns 421 otherwise. `/healthz` is outside both checks.
+
+Costs: one extra `/user` call per MCP request. The metadata document advertises
+the in-cluster Gitea URL as authorization server; this only matters for
+OAuth-flow clients (claude.ai connectors), which are out of scope.
+
+## Architecture
+
+```
+client ──HTTPS, Authorization: Bearer <pat>──▶ gitea-mcp.renner.uno
+  ├─ internet: NetBird proxy (TLS passthrough, PROXY v2) ─▶ Traefik
+  └─ LAN:      Traefik VIP (LAN DNS record)                ─▶ Traefik
+Traefik (TLS termination, letsencrypt-cluster cert) ──HTTP :8080──▶ gitea-mcp pod
+gitea-mcp ──HTTP :3000, same token──▶ gitea.gitea.svc (Gitea API)
+```
+
+Traefik forwards `Host` and the `Mcp-*` headers unchanged by default, which the
+Host check and the MCP transport need. `/mcp` is stateless POST only, so no
+sticky sessions or long-lived streams.
+
+## Components
+
+All paths under `kubernetes/`.
+
+- `infrastructure/namespaces/namespaces.yaml` — Namespace `gitea-mcp`.
+- `clusters/homelab/gitea-mcp.yaml` — Flux Kustomization, path
+  `./kubernetes/apps/gitea-mcp`, `prune: true`, `wait: true`, dependsOn
+  `infrastructure`, `infrastructure-crds`.
+- `apps/gitea-mcp/`
+  - `deployment.yaml` — 1 replica; label `app.kubernetes.io/name: gitea-mcp`;
+    args `-t http -p 8080 -oauth -oauth-public-url https://gitea-mcp.renner.uno
+    -H http://gitea.gitea.svc.cluster.local:3000`; no `GITEA_ACCESS_TOKEN`, no
+    `-d` (debug may log tokens via the SDK); pod/container securityContext as in
+    gatus (non-root 65532, drop ALL, no privilege escalation, RuntimeDefault
+    seccomp) plus `readOnlyRootFilesystem: true` with a disk-backed `emptyDir`
+    (`sizeLimit: 2Gi`) at `/tmp` for the app's own rotated log file; liveness/readiness `httpGet /healthz` on `http`;
+    requests `cpu: 10m, memory: 32Mi`, limit `memory: 128Mi`; image line
+    carries the `$imagepolicy` marker.
+  - `service.yaml` — ClusterIP `gitea-mcp`, port 8080 → `http`.
+  - `ingress.yaml` — host `gitea-mcp.renner.uno`, `ingressClassName: traefik`,
+    `cert-manager.io/cluster-issuer: letsencrypt-cluster`, TLS secret
+    `gitea-mcp-tls`, label `netbird.renner.uno/public: "true"`.
+  - `kustomization.yaml`.
+- `infrastructure/network-policies/gitea-mcp-ns.yaml` (+ entry in that
+  directory's `kustomization.yaml`) — default-deny CNP following
+  `gitea-ns.yaml`: ingress from same namespace, `monitoring`, `host`/
+  `remote-node`, ICMP; ingress from Traefik (`ingress` ns,
+  `app.kubernetes.io/name: traefik`) on 8080; egress to kube-dns and to
+  `gitea` ns `app.kubernetes.io/name: gitea` on 3000. No `world` egress.
+- `apps/gitea/networkpolicy.yaml` — new CNP `gitea-allow-gitea-mcp`, modelled on
+  `gitea-allow-flux-rotator`: ingress to Gitea on 3000 from the `gitea-mcp`
+  namespace.
+- `infrastructure/image-automation/gitea-mcp.yaml` (+ kustomization entry) —
+  ImageRepository `docker.gitea.com/gitea-mcp-server`, ImagePolicy semver
+  `>=1.8.1`.
+
+## Error handling
+
+- Invalid/expired token → Gitea `/user` 401 → gitea-mcp returns 401
+  `invalid_token`.
+- Gitea unreachable → upstream's token check fails open (transport errors are
+  not treated as rejection), but the subsequent API call fails, so the client
+  gets a tool error, never data.
+- Pod down → Traefik 503; Flux `wait` surfaces a failed rollout as
+  `FluxResourceNotReady`.
+
+## Verification (after Flux reconciles)
+
+1. `gitea-mcp` Kustomization Ready; pod Ready.
+2. `https://gitea-mcp.renner.uno/healthz` → 200, from the LAN and through the
+   NetBird proxy (public resolution).
+3. `POST /mcp` `initialize` without a token → 401.
+4. `POST /mcp` with the `claude-bot` PAT: `initialize`, then `tools/call`
+   `list_my_repos` → returns repos.
+5. NetBird management lists a reverse-proxy service for
+   `gitea-mcp.renner.uno`.
+6. ImagePolicy `gitea-mcp` resolves `1.8.1` (or newer).
+
+## Out of scope
+
+- Registering the server in Claude Code / other clients.
+- OAuth browser flow for claude.ai connectors (would need a Gitea OAuth2 app
+  and `-H https://gitea.renner.uno`).
+- Read-only mode, tool filtering (`-O`/`-S`).
@@ -165,26 +165,33 @@ permanent recovery.

 ---

-## Velero restore-test: 80% OSD capacity gate
+## Velero backup verification (backup-verify)

-The monthly Velero restore-test restores into the production Ceph pools. Before
-each schedule's restore it projects the restore's raw size (DataUpload
-`totalBytes` x the pool's replication/EC factor, spread over all OSDs) onto
-current OSD usage from Prometheus. If the fullest OSD would go above **80%**
-(`OSD_MAX_PCT` in `restore-test-cronjob.yaml`), that schedule is not restored:
-the Job log shows it as **SKIP** with the projected %, and the summary lists it
-under "Skipped (backup NOT restore-tested)". A SKIP does not fail the Job, so
-read the summary each month. If the projection cannot be made (Prometheus
-down, unsized DataUpload), the schedule is not restored and counts as FAIL.
-Background: REN-441, REN-442.
+Monthly (1st, 08:00 UTC) CronJob `velero/backup-verify`. It creates a throwaway
+Hetzner Cloud VM in fsn1 (project `homelab-backup-verify`) and runs
+`kopia snapshot verify --verify-files-percent=100` read-only against the latest
+snapshot of every DataUpload of each schedule's newest finished backup. Snapshot
+sizes are checked against `DataUpload.status.progress.totalBytes`. Nothing is
+restored into the cluster. A full run takes about an hour (2026-10-10: 39 PVCs,
+~410 GB, 61 min, cx23).
+
+- Results: `kubectl -n velero logs job/<backup-verify-...>`; `FAIL` lines name the
+  backup, PVC, snapshot and cause.
+- A failed run fires `KubeJobFailed` (velero).
+- Manual run: `kubectl -n velero create job bv-manual --from=cronjob/backup-verify`;
+  limit it with env `ONLY_NAMESPACES=apps,gitea`.
+- Every run first deletes everything labelled `managed-by=backup-verify` in the
+  Hetzner project, so a leftover VM is cleaned up by the next run. Do not start a
+  manual run while another one is running: its sweep deletes the other run's VM.
+- This checks data readability, not a full restore. Disaster-recovery rehearsal is separate.

 ---

 ## CNPG restore-test (verify backups are restorable — non-destructive)

-The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
-PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
-code path). Run this drill periodically to prove the CNPG backups are actually
+The monthly Velero backup verification (`kubernetes/infrastructure/velero/`)
+checks the Velero PVC backups but **does not cover CNPG** (barman-cloud is a
+separate code path). Run this drill periodically to prove the CNPG backups are actually
 restorable, **without touching the live clusters**. It restores into a throwaway
 cluster in the *same* namespace (so it can reference the existing namespaced
 `ObjectStore` and creds) and declares the barman plugin with
@@ -0,0 +1,75 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: gitea-mcp
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: gitea-mcp
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: gitea-mcp
+          image: docker.gitea.com/gitea-mcp-server:1.8.1 # {"$imagepolicy": "flux-system:gitea-mcp"}
+          # The image has no ENTRYPOINT; args alone would replace its CMD.
+          command:
+            - /app/gitea-mcp
+          args:
+            - -t
+            - http
+            - -p
+            - "8080"
+            # Only for the 401 on tokenless requests; clients send their own Gitea PAT.
+            - -oauth
+            - -oauth-public-url
+            - https://gitea-mcp.renner.uno
+            - -H
+            - http://gitea.gitea.svc.cluster.local:3000
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          ports:
+            - containerPort: 8080
+              name: http
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 5
+            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 2
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+      volumes:
+        # Holds the app's own rotated log file (100MB x 11 max); stdout carries the same lines.
+        - name: tmp
+          emptyDir:
+            sizeLimit: 2Gi
@@ -0,0 +1,28 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+    # Published on the NetBird proxy; LAN DNS stays on the Traefik VIP.
+    netbird.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: gitea-mcp.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea-mcp
+                port:
+                  number: 8080
+  tls:
+    - secretName: gitea-mcp-tls
+      hosts:
+        - gitea-mcp.renner.uno
@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml
@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-mcp
+  namespace: gitea-mcp
+  labels:
+    app.kubernetes.io/name: gitea-mcp
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: gitea-mcp
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
@@ -5,6 +5,9 @@ metadata:
  namespace: gitea
  labels:
    app.kubernetes.io/name: gitea
+    # Published on the NetBird proxy; LAN DNS stays on the Traefik VIP so
+    # SSH (gitea-ssh) and node image pulls stay local.
+    netbird.renner.uno/public: "true"
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-cluster
    # Long-running git fetch/push and container-registry blob uploads:
@@ -72,3 +72,23 @@ spec:
        - ports:
            - port: "6443"
              protocol: TCP
+---
+# gitea-mcp reaches the in-cluster Service; egress half is in network-policies/gitea-mcp-ns.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-gitea-mcp
+  namespace: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+            app.kubernetes.io/name: gitea-mcp
+      toPorts:
+        - ports:
+            - port: "3000"
+              protocol: TCP
@@ -95,7 +95,7 @@ spec:
              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.10.0 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.10.1 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -15,9 +15,6 @@ spec:
          # Per-check rather than ceph_health_status, so one permanently-active
          # check cannot mask every other Ceph warning.
          #
-          # AUTH_INSECURE_* is excluded: it cannot clear until the nodes run a
-          # kernel supporting aes256k over the CSI mount path (#527).
-          #
          # POOL_NO_REDUNDANCY is in scope. An unreplicated pool makes
          # `ceph osd ok-to-stop` refuse for every OSD, so it reports blocked
          # maintenance and not just data risk — do not exclude it for a pool
@@ -34,20 +31,14 @@ spec:
          # both episodes in the 30d to 2026-08-23 ran ~26h and cleared
          # untouched, the first ~24h after the last stall.
          #
-          # Both began inside the Velero restore-test sweep — a ~5h sustained
-          # restore that outruns the SLC cache on these QLC NVMe. The sweep
-          # runs monthly, 08:00-14:00Z on the 1st.
-          #
-          # Expected for the hardware, but not excludable:
-          # this is also the earliest signal of a genuinely failing drive, and
-          # all three are at 80-87% endurance used (see cephcluster.yaml).
-          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
+          # Not excludable: it is also the earliest signal of a failing drive.
+          expr: ceph_health_detail == 1
          for: 6h
          labels:
            severity: warning
          annotations:
            summary: "Ceph health check {{ $labels.name }} active for 6h"
-            description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+            description: "Ceph health check {{ $labels.name }} has been active for six hours. BLUESTORE_SLOW_OP_ALERT stays for 24h after the last stall even when latency is back to normal; find the OSD in the detail output and check that drive's SMART data (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."

        - alert: CephPoolLostRedundancy
          # Names the pool, which POOL_NO_REDUNDANCY does not.
@@ -50,11 +50,9 @@ data:
  PAPERLESS_TIKA_ENABLED: "1"
  PAPERLESS_TIKA_ENDPOINT: "http://tika:9998"
  PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://gotenberg:3000"
-  # AI suggestions against the cluster's own LLM (kubernetes/apps/llama-server).
-  # Nothing leaves the cluster: both endpoints are ClusterIP Services, so
-  # document text is never sent to a third party and there is no API key to
-  # rotate — llama-server has no authentication, and the ai namespace's network
-  # policy is what restricts who may reach it.
+  # AI suggestions against Mistral AI's hosted API (OpenAI-compatible). Document
+  # text is sent to Mistral. The key is PAPERLESS_AI_LLM_API_KEY, from the
+  # paperless-secrets Secret (OpenBao apps/paperless, field minstral-token).
  #
  # CAVEAT, and it is a real one: paperless reads these from an
  # ApplicationConfiguration row in the database and falls back to the
@@ -63,20 +61,14 @@ data:
  # here — and leave those fields untouched in the UI.
  PAPERLESS_AI_ENABLED: "true"
  PAPERLESS_AI_LLM_BACKEND: "openai-like"
-  PAPERLESS_AI_LLM_ENDPOINT: "http://llama-server.ai.svc.cluster.local:8080/v1"
-  # Must match --alias in kubernetes/apps/llama-server/deployment.yaml. The
-  # server ignores the field on the wire, but paperless sends it and the value
-  # is what appears in its logs.
-  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
-  # ClusterIP addresses are private, and paperless blocks non-public endpoints
-  # unless this is set. Default is already true; explicit because the whole
-  # integration silently stops working if it ever flips.
+  PAPERLESS_AI_LLM_ENDPOINT: "https://api.mistral.ai/v1"
+  PAPERLESS_AI_LLM_MODEL: "mistral-small-2603"
+  # The embedding endpoint is the in-cluster llama-embed, a ClusterIP address,
+  # and paperless blocks non-public endpoints unless this is set. Default is
+  # already true; explicit because the embedding integration silently stops
+  # working if it ever flips.
  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
-  # Under llama-server's own 16384, leaving room for the response.
  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
-  # The 120s default is too tight at ~11 tok/s: a classification is roughly
-  # 2-4k prompt tokens plus a second round trip for localisation.
-  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "300"
  # Matched against Django's settings.LANGUAGES, so the region is required —
  # "de" alone does not resolve. Without it the model returns German tags but
  # English document types, which was reproducible against this endpoint.
@@ -86,6 +78,9 @@ data:
  # primarily English archives — the better fit for a German corpus.
  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
+  # The embedding key falls back to PAPERLESS_AI_LLM_API_KEY when unset, which
+  # would send the Mistral token to llama-embed. llama-embed has no auth.
+  PAPERLESS_AI_LLM_EMBEDDING_API_KEY: "unused"
  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
  # embedding model invalidates every stored vector, so a swap means the LLM
  # index has to be rebuilt.
@@ -200,6 +195,11 @@ spec:
                secretKeyRef:
                  name: paperless-secrets
                  key: secret-key
+            - name: PAPERLESS_AI_LLM_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: minstral-token
            - name: PAPERLESS_DBUSER
              valueFrom:
                secretKeyRef:
@@ -0,0 +1,18 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/gitea-mcp
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
@@ -4,7 +4,9 @@ metadata:
  name: alertmanager
  namespace: flux-system
 spec:
-  # The notification controller appends `/api/v2/alerts/` itself, so
-  # spec.address is the bare base URL with trailing slash.
+  # The notification controller POSTs to spec.address verbatim (it does
+  # not append a path), so the address must be the full Alertmanager v2
+  # alerts endpoint. The bare base URL is answered with 405 Method Not
+  # Allowed and every Flux notification is dropped.
  type: alertmanager
-  address: http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/
+  address: http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/api/v2/alerts/
@@ -0,0 +1,20 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  image: docker.gitea.com/gitea-mcp-server
+  interval: 1h
+---
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: gitea-mcp
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: gitea-mcp
+  policy:
+    semver:
+      range: ">=1.8.1"
@@ -7,6 +7,7 @@ resources:
  - ceph.yaml
  - cnpg-barman-plugin.yaml
  - gatus.yaml
+  - gitea-mcp.yaml
  - gotenberg.yaml
  - home-assistant.yaml
  - immich.yaml
@@ -198,6 +198,12 @@ kind: Namespace
 metadata:
  name: gitea
 ---
+# Gitea MCP server. Declared here so it exists before infrastructure-crds applies its CNPs.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: gitea-mcp
+---
 # netbird-service-sync, moving out of `openbao` (issue #876). Declared here like
 # `woodpecker` above, so it exists before infrastructure-crds applies its
 # CiliumNetworkPolicies. No pod-security labels: the cluster default applies.
@@ -1,5 +1,6 @@
 # Reconciles NetBird reverse-proxy services from Ingresses that carry
-# dns.renner.uno/public: one TLS-passthrough service per host on :443,
+# dns.renner.uno/public or netbird.renner.uno/public (proxy only, LAN DNS
+# untouched): one TLS-passthrough service per host on :443,
 # targeting Traefik's ClusterIP through a subnet resource over the service
 # CIDR (ci/images/netbird-service-sync/controller.py).
 #
@@ -80,7 +81,7 @@ spec:
          # image directory (branch and master alike) and the registry keeps
          # only the newest manifest per tag, so the digest must come from
          # the MASTER build of the code change.
-          image: gitea.renner.uno/gitea_admin/netbird-service-sync:2026-09-23@sha256:64f6057c508640e1dbc50457dc26e9b2de058a111fd46ca096021d7444b5bbff
+          image: gitea.renner.uno/gitea_admin/netbird-service-sync:2026-10-11@sha256:2e8c5e45cffd515cc93e04dc9fd47403a7910368fc7516e94c78d8f53dcd5272
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
@@ -0,0 +1,82 @@
+---
+# Baseline policy for the `gitea-mcp` namespace.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny
+  namespace: gitea-mcp
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea-mcp
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": gitea
+            app.kubernetes.io/name: gitea
+      toPorts:
+        - ports:
+            - port: "3000"
+              protocol: TCP
+---
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-mcp-allow-traefik
+  namespace: gitea-mcp
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea-mcp
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
@@ -25,6 +25,7 @@ resources:
  - woodpecker-ci-ns.yaml
  - woodpecker-ns.yaml
  - gitea-ns.yaml
+  - gitea-mcp-ns.yaml
  - netbird-service-sync-ns.yaml
  - trmnl-ns.yaml
  - matrix-ns.yaml
@@ -50,17 +50,6 @@ spec:
        - ports:
            - port: "9090"
              protocol: TCP
-    # velero restore-test Job: its capacity gate reads OSD usage from
-    # Prometheus :9090 (restore-test.sh, preflight_capacity). Scoped to the
-    # Job's pod label; the velero namespace has no egress policy.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": velero
-            app.kubernetes.io/name: restore-test
-      toPorts:
-        - ports:
-            - port: "9090"
-              protocol: TCP
    # Kubelet probes + control-plane (PrometheusRule/ServiceMonitor admission
    # webhooks via kube-prometheus-stack-operator are called by kube-apiserver).
    # Also covers external alloy-syslog traffic — ETP=Cluster on the LB SNATs
@@ -6,7 +6,7 @@ metadata:
 spec:
  endpointSelector: {}
  ingress:
-    # Intra-namespace — velero server ↔ node-agent ↔ restore-test job.
+    # Intra-namespace — velero server ↔ node-agent.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": velero
@@ -6,7 +6,7 @@ metadata:
 spec:
  schedule: "0 8 1 * *"
  timeZone: Etc/UTC
-  suspend: true
+  suspend: false
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 3
@@ -23,16 +23,19 @@ START_S=$(date +%s)

 log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; }

-sweep() {
-  local kind id
-  for kind in server firewall ssh-key; do
-    for id in $(hcloud "$kind" list -l "$LABEL" -o noheader -o columns=id); do
-      log "deleting $kind $id"
-      hcloud "$kind" delete "$id" >&2 || log "WARN: could not delete $kind $id"
-    done
+delete_kind() {
+  local id
+  for id in $(hcloud "$1" list -l "$LABEL" -o noheader -o columns=id); do
+    log "deleting $1 $id"
+    hcloud "$1" delete "$id" >&2 || log "WARN: could not delete $1 $id"
  done
 }

+sweep() {
+  local kind
+  for kind in server firewall ssh-key; do delete_kind "$kind"; done
+}
+
 build_plan() {
  local only
  only=$(jq -cn --arg s "$ONLY_NAMESPACES" '$s | split(",") | map(select(. != ""))')
@@ -49,13 +52,20 @@ build_plan() {
  log "plan: $(jq -r 'group_by(.status) | map("\(.[0].status)=\(length)") | join(" ")' "$WORK/plan.json")"
 }

+# cloud-config as JSON (a YAML subset), so jq does the escaping.
 render_user_data() {
-  printf '#cloud-config\nssh_pwauth: false\nssh_deletekeys: true\nssh_genkeytypes: []\nssh_keys:\n  ed25519_private: |\n'
-  sed 's/^/    /' "$WORK/host_key"
-  printf '  ed25519_public: %s\n' "$(cat "$WORK/host_key.pub")"
-  printf 'packages: [jq, curl, ca-certificates]\nruncmd:\n'
-  printf '  - mkdir -p /run/verify && mount -t tmpfs -o size=1500m,mode=0700 tmpfs /run/verify\n'
-  printf '  - [shutdown, -h, "+240"]\n'
+  printf '#cloud-config\n'
+  jq -n --rawfile priv "$WORK/host_key" --arg pub "$(cat "$WORK/host_key.pub")" '{
+    ssh_pwauth: false,
+    ssh_deletekeys: true,
+    ssh_genkeytypes: [],
+    ssh_keys: {ed25519_private: $priv, ed25519_public: $pub},
+    packages: ["jq", "curl", "ca-certificates"],
+    runcmd: [
+      "mkdir -p /run/verify && mount -t tmpfs -o size=1500m,mode=0700 tmpfs /run/verify",
+      ["shutdown", "-h", "+240"]
+    ]
+  }'
 }

 create_vm() {
@@ -75,12 +85,14 @@ create_vm() {
      return 0
    fi
    log "server type $type not available"
+    # A create that timed out may still have left the server behind under $NAME.
+    delete_kind server
  done
  return 1
 }

 wait_ssh() {
-  local v4 v6 hostkey addr deadline
+  local v4 v6 hostkey addr deadline rc
  v6=$(hcloud server ip -6 "$NAME")
  v4=$(hcloud server ip "$NAME")
  hostkey=$(cut -d' ' -f1,2 "$WORK/host_key.pub")
@@ -96,6 +108,14 @@ wait_ssh() {
        VM_ADDR=$addr
        log "ssh ready via $addr"
        return 0
+      else
+        rc=$?
+      fi
+      # 255 is an ssh connection error; anything else is cloud-init itself failing.
+      if [ "$rc" -ne 255 ]; then
+        log "cloud-init failed on $addr (exit $rc):"
+        ssh "${SSH_OPTS[@]}" "root@$addr" 'cloud-init status --long' < /dev/null >&2 || true
+        return 1
      fi
    done
    sleep 10
@@ -112,11 +132,14 @@ secrets() {
 push() { ssh "${SSH_OPTS[@]}" "root@$VM_ADDR" "cat > /run/verify/$2" < "$1"; }

 run_vm() {
-  ssh "${SSH_OPTS[@]}" "root@$VM_ADDR" 'mkdir -p /run/verify' < /dev/null
  jq '[.[] | select(.status == "VERIFY")]' "$WORK/plan.json" > "$WORK/vm-plan.json"
-  push "$WORK/vm-plan.json" plan.json
-  push "$SCRIPT_DIR/check.jq" check.jq
-  push "$SCRIPT_DIR/vm-verify.sh" vm-verify.sh
+  if ! { ssh "${SSH_OPTS[@]}" "root@$VM_ADDR" 'mkdir -p /run/verify' < /dev/null \
+         && push "$WORK/vm-plan.json" plan.json \
+         && push "$SCRIPT_DIR/check.jq" check.jq \
+         && push "$SCRIPT_DIR/vm-verify.sh" vm-verify.sh; }; then
+    log "could not stage files on the VM; every VERIFY entry will report no result"
+    return 0
+  fi
  # Background + wait: bash runs the TERM trap during `wait`, not during a foreground child.
  ( secrets | ssh "${SSH_OPTS[@]}" "root@$VM_ADDR" 'bash /run/verify/vm-verify.sh /run/verify/plan.json' \
      | tee "$WORK/vm.out" >&2 ) &
@@ -6,7 +6,9 @@ case "$verb" in
  create)
    if [ "$kind" = server ]; then
      type=$(printf '%s\n' "$@" | grep -A1 -x -- '--type' | tail -n1)
+      if [ -s "$state" ]; then echo "name is already used (uniqueness_error)" >&2; exit 1; fi
      for bad in ${STUB_UNAVAILABLE_TYPES:-}; do [ "$type" = "$bad" ] && { echo "server type unavailable" >&2; exit 1; }; done
+      if [ "$type" = "${STUB_CREATE_LEAVES_SERVER:-}" ]; then echo "server-id" >> "$state"; echo "action timed out" >&2; exit 1; fi
      echo '{"server":{"id":4711}}'
    fi
    echo "${kind}-id" >> "$state" ;;
@@ -2,9 +2,12 @@
 printf '%s\n' "$*" >> "$STUB_LOG"
 args=" $* "
 case "$args" in
-  *" repository connect "*) [ -z "${STUB_CONNECT_FAIL:-}" ] ;;
+  *" repository connect "*)
+    echo "CHECK=${KOPIA_CHECK_FOR_UPDATES:-unset}" >> "$STUB_LOG"
+    [ -z "${STUB_CONNECT_FAIL:-}" ] ;;
  *" snapshot list "*) cat "$STUB_SNAPSHOTS" ;;
  *" snapshot verify "*)
+    if read -r -t 0.2 _; then echo "STDIN_LEAK" >> "$STUB_LOG"; fi
    for d in ${STUB_BAD_DIRS:-}; do
      case "$args" in *"--directory-id=$d "*) echo "error processing $d: boom" >&2; exit 1 ;; esac
    done ;;
@@ -1,12 +1,19 @@
 #!/usr/bin/env bash
 printf 'ssh %s\n' "$*" >> "$STUB_LOG"
-cat > /dev/null
 case "$*" in
+  *"cloud-init status --long"*)
+    printf 'status: error\nerrors: %s\n' "${STUB_CLOUDINIT_ERROR:-boom}" ;;
  *"cloud-init status --wait"*)
    echo "wait started" >> "$STUB_LOG"
-    sleep "${STUB_WAIT_SLEEP:-0}" ;;
+    sleep "${STUB_WAIT_SLEEP:-0}"
+    [ -z "${STUB_CLOUDINIT_FAIL:-}" ] || exit 1 ;;
+  *"cat > /run/verify/"*)
+    cat > /dev/null
+    [ -z "${STUB_PUSH_FAIL:-}" ] || exit 1 ;;
  *"bash /run/verify/vm-verify.sh"*)
+    cat > "${STUB_STDIN:-/dev/null}"
    echo "run started" >> "$STUB_LOG"
    sleep "${STUB_VM_SLEEP:-0}"
    cat "$STUB_VM_OUT" ;;
+  *) cat > /dev/null ;;
 esac
@@ -32,7 +32,47 @@ grep -q 'PASS=1 FAIL=0 SKIP=0' <<<"$out" && ok "happy summary" || bad "happy sum
 grep -q 'type cx23' "$STUB_LOG" && ok "happy uses cx23" || bad "happy uses cx23"
 grep -q $'^OK\tapps-daily\tapps-daily-2\tapps/gatus-data\tsnapshot=s1\t' <<<"$out" && ok "summary row names backup and snapshot" || bad "summary row: $out"
 grep -qE '^elapsed=[0-9]+s server_type=cx23$' <<<"$out" && ok "summary elapsed and server type" || bad "summary elapsed/type: $out"
-grep -q 'mount -t tmpfs -o size=1500m,mode=0700 tmpfs /run/verify' "$T/work/user-data.yaml" && ok "user-data mounts tmpfs at /run/verify" || bad "user-data tmpfs mount"
+[ "$(head -n1 "$T/work/user-data.yaml")" = "#cloud-config" ] && ok "user-data cloud-config header" || bad "user-data header"
+tail -n +2 "$T/work/user-data.yaml" | jq -e '
+  .ssh_pwauth == false
+  and (.ssh_keys.ed25519_private | startswith("-----BEGIN OPENSSH PRIVATE KEY-----"))
+  and (.ssh_keys.ed25519_public | startswith("ssh-ed25519 "))
+  and (.runcmd | map(tostring) | any(test("mount -t tmpfs -o size=1500m,mode=0700 tmpfs /run/verify")))' > /dev/null \
+  && ok "user-data parses, pins host key, mounts tmpfs" || bad "user-data structure"
+
+# secrets: AWS ini with and without spaces around '=' reaches the VM on stdin.
+want_stdin=$'AWS_ACCESS_KEY_ID=ak\nAWS_SECRET_ACCESS_KEY=sk\nKOPIA_PASSWORD=pw'
+setup
+STUB_STDIN="$T/stdin" bash "$bv/backup-verify.sh" > /dev/null 2> "$T/err"
+[ "$(cat "$T/stdin")" = "$want_stdin" ] && ok "secrets on stdin (no spaces)" || bad "secrets on stdin (no spaces): $(cat "$T/stdin")"
+setup
+printf '[default]\naws_access_key_id = ak\naws_secret_access_key = sk\n' > "$T/secrets/cloud"
+STUB_STDIN="$T/stdin" bash "$bv/backup-verify.sh" > /dev/null 2> "$T/err"
+[ "$(cat "$T/stdin")" = "$want_stdin" ] && ok "secrets on stdin (spaces)" || bad "secrets on stdin (spaces): $(cat "$T/stdin")"
+
+# no-types: every server type unavailable -> exit 1, entries FAIL, nothing left.
+setup
+if out=$(STUB_UNAVAILABLE_TYPES="cx23 cax11" bash "$bv/backup-verify.sh" 2> "$T/err"); then bad "no-types exit 1"; else ok "no-types exit 1"; fi
+grep -q $'^FAIL\t.*no result from VM' <<<"$out" && ok "no-types reports FAIL per entry" || bad "no-types output: $out"
+[ "$(leftovers)" = 0 ] && ok "no-types teardown" || bad "no-types teardown"
+
+# retry: cx23 create leaves a server behind server-side -> removed before cax11.
+setup
+if STUB_CREATE_LEAVES_SERVER=cx23 bash "$bv/backup-verify.sh" > /dev/null 2> "$T/err"; then ok "retry exit 0"; else bad "retry exit 0"; cat "$T/err"; fi
+grep -q 'type cax11' "$STUB_LOG" && ok "retry uses cax11" || bad "retry uses cax11"
+
+# push-fail: staging files on the VM fails -> summary still printed, exit 1.
+setup
+if out=$(STUB_PUSH_FAIL=1 bash "$bv/backup-verify.sh" 2> "$T/err"); then bad "push-fail exit 1"; else ok "push-fail exit 1"; fi
+grep -q 'PASS=0 FAIL=1 SKIP=0' <<<"$out" && ok "push-fail still summarizes" || bad "push-fail output: $out"
+
+# cloud-init fails: reason logged, no wait until SSH_WAIT_S.
+setup
+start=$(date +%s)
+SSH_WAIT_S=60 STUB_CLOUDINIT_FAIL=1 STUB_CLOUDINIT_ERROR=pkg-install-failed bash "$bv/backup-verify.sh" > /dev/null 2> "$T/err"
+took=$(( $(date +%s) - start ))
+grep -q 'pkg-install-failed' "$T/err" && ok "cloud-init failure reason logged" || bad "cloud-init reason missing"
+[ "$took" -lt 10 ] && ok "cloud-init failure fails fast (${took}s)" || bad "cloud-init failure took ${took}s"

 # fallback: cx23 unavailable -> cax11 (Review Focus 5).
 setup
@@ -19,6 +19,8 @@ check "connect is readonly" "$(grep -c -- '--readonly' "$STUB_LOG")" "1"
 check "prefix per namespace" "$(grep -c -- '--prefix=kopia/apps/' "$STUB_LOG")" "1"
 check "full verify" "$(grep -c -- '--verify-files-percent=100' "$STUB_LOG")" "2"
 check "verify parallelism" "$(grep -c -- '--file-parallelism=8' "$STUB_LOG")" "2"
+check "verify reads no loop stdin" "$(grep -c 'STDIN_LEAK' "$STUB_LOG" || true)" "0"
+check "kopia update check off" "$(grep -c '^CHECK=false$' "$STUB_LOG")" "1"

 # 2: connect fails -> every entry FAIL, script still prints DONE.
 : > "$STUB_LOG"
@@ -10,6 +10,7 @@ BUCKET=${BUCKET:-renner-uno-k8s-velero}
 S3_ENDPOINT=${S3_ENDPOINT:-fsn1.your-objectstorage.com}
 S3_REGION=${S3_REGION:-fsn1}
 KOPIA_VERSION=0.16.0
+export KOPIA_CHECK_FOR_UPDATES=false
 declare -A KOPIA_SHA256=(
  [x64]=a29f2cc1a49f985d1bfe09340eda0f7ed7b3c98037704da249b47034f1be1a18
  [arm64]=dc20b1d50d0d6e6d0e429f4f6062088f188100f3c9dab6d868d232169ab413a8
@@ -77,7 +78,7 @@ verify_namespace() {
    root=$(jq -r .rootID <<<"$e")
    bytes=$(jq -r .kopiaBytes <<<"$e")
    if "$KOPIA" --config-file="$cfg" snapshot verify --directory-id="$root" \
-         --verify-files-percent=100 --file-parallelism=8 >&2; then
+         --verify-files-percent=100 --file-parallelism=8 < /dev/null >&2; then
      result OK "$ns" "$pvc" "$snap" "verified $bytes bytes"
    else
      result FAIL "$ns" "$pvc" "$snap" "kopia verify failed (see log)"
@@ -4,28 +4,20 @@ resources:
  - snapshot-readonly-storageclass.yaml
  - node-agent-config.yaml
  - helmrelease.yaml
-  - restore-test-rbac.yaml
-  - restore-test-cronjob.yaml
  - backup-verify-rbac.yaml
  - backup-verify-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/
 # (velero-schedules.yaml) so they reconcile after the Velero HelmRelease
 # finishes installing velero.io CRDs.

-# Bake restore-test.sh into a ConfigMap mounted by the CronJob. Hash
-# suffix is disabled because Kustomize's NameReference transformer does
-# not rewrite configMap references inside the deeply nested CronJob
+# Bake the backup-verify scripts into a ConfigMap mounted by the CronJob.
+# Hash suffix is disabled because Kustomize's NameReference transformer
+# does not rewrite configMap references inside the deeply nested CronJob
 # pod template (spec.jobTemplate.spec.template.spec.volumes), so the
 # CronJob would otherwise dangle pointing at a stale name. CronJobs
 # already read the latest ConfigMap content at each pod start, so the
 # hash buys nothing here.
 configMapGenerator:
-  - name: restore-test-script
-    namespace: velero
-    options:
-      disableNameSuffixHash: true
-    files:
-      - restore-test.sh
  - name: backup-verify-scripts
    namespace: velero
    options:
@@ -1,122 +0,0 @@
---
-# Monthly Velero restore-test CronJob — 08:00 UTC on the 1st.
-#
-# The sweep restores every discovered schedule, immich's ~180 GiB library
-# included. At `size: 2` that is ~360 GiB raw, ~120 GiB per OSD, against a
-# ~65 GB/day weekday baseline on three QLC NVMe at 81-87% endurance used —
-# ~47% of each drive's weekly write budget. Monthly keeps every restore path
-# exercised and cuts that share ~4x.
-#
-# 08:00 — *after* that day's own daily backups have finished (paperless
-# 03:15 through apps 07:15, each normally ~1 min). The previous 02:00 slot
-# was chosen as "well before the next day's backups", which missed that the
-# daily schedules fire on the same day: a sweep starting at 02:00 and
-# running ~5 h sat on top of the entire 03:15-07:15 backup window and
-# starved it of node-agent DataUpload concurrency. Observed effect on
-# 2026-07-19 and 2026-07-26 — audiobookshelf-daily took 2 h 18 m and
-# 2 h 33 m respectively against a ~1 min baseline, with the books
-# DataUpload not starting until 06:18. Both are Sundays, the only day the
-# sweep ran then; no other day shows it.
-#
-# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of the
-# next day's 03:15.
-#
-# Day-of-month rather than a fixed weekday: Kubernetes parses cron with
-# robfig/cron's standard parser, which has no `#` nth-weekday operator, and
-# restricting both day fields (`0 8 1-7 * 0`) ORs them — every Sunday plus
-# the 1st through 7th.
-#
-# Concurrency policy Forbid: the immich restore (~180 GiB, millions of
-# small files) alone can run multiple hours, so the full sweep is sized
-# at up to 6 h. If a run overlaps with the next firing it is silently
-# replaced. Failed runs are kept (history > 0) so the failure context is
-# inspectable in `kubectl get jobs -n velero -l job-name~restore-test`.
-#
-# The script lives at restore-test.sh in this directory and is mounted
-# from a configmap built by configMapGenerator (see kustomization.yaml).
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: restore-test
-  namespace: velero
-spec:
-  schedule: "0 8 1 * *"
-  timeZone: Etc/UTC
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 2
-  failedJobsHistoryLimit: 5
-  startingDeadlineSeconds: 3600
-  jobTemplate:
-    spec:
-      backoffLimit: 0
-      activeDeadlineSeconds: 21600  # 6 h hard cap (immich alone can take ~3 h)
-      ttlSecondsAfterFinished: 3456000  # 40 d — enough to inspect after the next monthly run
-      template:
-        metadata:
-          labels:
-            # Matched by monitoring-ns.yaml: the capacity gate reads OSD
-            # usage from Prometheus :9090.
-            app.kubernetes.io/name: restore-test
-        spec:
-          serviceAccountName: restore-test
-          restartPolicy: Never
-          # On TERM (deadline or deletion) the script cancels the restore
-          # in flight: up to 2 min for DataDownloads to stop, up to 5 min
-          # for the namespace to go, then the PV purge (REN-451).
-          terminationGracePeriodSeconds: 600
-          securityContext:
-            runAsNonRoot: true
-            runAsUser: 65532
-            runAsGroup: 65532
-            seccompProfile:
-              type: RuntimeDefault
-          containers:
-            - name: restore-test
-              image: alpine/k8s:1.37.1@sha256:712dc5385d1c1a3f83b747fa9e40f64d433a97f72ff42f9755ff7dd1f62846e4
-              command: ["bash", "/script/restore-test.sh"]
-              securityContext:
-                allowPrivilegeEscalation: false
-                readOnlyRootFilesystem: true
-                capabilities:
-                  drop: ["ALL"]
-              env:
-                - name: RESTORE_PHASE_TIMEOUT_S
-                  value: "14400"
-                # Skip a restore projected to take the fullest OSD past
-                # this (SKIP, not FAIL). See restore-test.sh.
-                - name: OSD_MAX_PCT
-                  value: "80"
-                # Cancel a running restore when the fullest OSD reaches
-                # this (below Ceph nearfull 85%) and stop the sweep.
-                - name: OSD_ABORT_PCT
-                  value: "83"
-                # Keep equal to activeDeadlineSeconds above: no restore
-                # starts that its timeout would carry past it.
-                - name: JOB_DEADLINE_S
-                  value: "21600"
-                # readOnlyRootFilesystem=true blocks kubectl from writing
-                # discovery cache to ~/.kube. Point HOME at the writable
-                # tmp emptyDir so kubectl works.
-                - name: HOME
-                  value: /tmp
-              volumeMounts:
-                - name: script
-                  mountPath: /script
-                  readOnly: true
-                - name: tmp
-                  mountPath: /tmp
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 64Mi
-                limits:
-                  memory: 256Mi
-          volumes:
-            - name: script
-              configMap:
-                name: restore-test-script
-                defaultMode: 0755
-            - name: tmp
-              emptyDir:
-                medium: Memory
-                sizeLimit: 16Mi
@@ -1,73 +0,0 @@
---
-# RBAC for the monthly restore-test CronJob.
-#
-# Permissions required by restore-test.sh:
-#   - velero.io: read schedules/backups/BSL, create+delete restores,
-#       cancel (patch) the DataDownloads of a restore it aborts
-#   - core: create+delete ephemeral <ns>-restore-test namespaces, the
-#       PVCs Velero restores into them, and the probe pods that mount
-#       those PVCs
-#   - core: pods/exec to run `find` inside the probe pod
-#   - core: pods watch for `kubectl wait --for=condition=ready`
-#   - apps: read DaemonSet status (node-agent readiness check)
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: restore-test
-  namespace: velero
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: restore-test
-rules:
-  - apiGroups: [""]
-    resources: ["namespaces"]
-    verbs: ["get", "list", "create", "delete"]
-  - apiGroups: [""]
-    resources: ["persistentvolumeclaims"]
-    verbs: ["get", "list", "create", "delete"]
-  - apiGroups: [""]
-    resources: ["persistentvolumes"]
-    # patch needed to flip reclaimPolicy from Retain → Delete on test residue
-    # before deleting, so CSI purges the backing CephFS subvolume / RBD image.
-    verbs: ["get", "list", "delete", "patch"]
-  - apiGroups: ["snapshot.storage.k8s.io"]
-    resources: ["volumesnapshots"]
-    verbs: ["get", "list", "create", "delete"]
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list", "watch", "create", "delete"]
-  - apiGroups: [""]
-    resources: ["pods/exec", "pods/log"]
-    verbs: ["get", "create"]
-  - apiGroups: [""]
-    resources: ["events"]
-    verbs: ["get", "list"]
-  - apiGroups: ["apps"]
-    resources: ["daemonsets"]
-    verbs: ["get", "list"]
-  - apiGroups: ["velero.io"]
-    resources: ["schedules", "backups", "backupstoragelocations", "datauploads"]
-    verbs: ["get", "list"]
-  - apiGroups: ["velero.io"]
-    # patch sets spec.cancel to stop a restore that is filling Ceph or
-    # outlives the Job; deleting the Restore does not stop it (REN-451).
-    resources: ["datadownloads"]
-    verbs: ["get", "list", "patch"]
-  - apiGroups: ["velero.io"]
-    resources: ["restores"]
-    verbs: ["get", "list", "watch", "create", "delete"]
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: restore-test
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: restore-test
-subjects:
-  - kind: ServiceAccount
-    name: restore-test
-    namespace: velero
@@ -1,801 +0,0 @@
-#!/usr/bin/env bash
-# Monthly verification that Velero PVC backups are actually restorable.
-#
-# What this exercises (per service):
-#   1. Velero Restore reaches phase=Completed against yesterday's
-#      scheduled Backup (real artifact, not a synthetic one).
-#   2. Each restored PVC is mountable and its file count is within ±10%
-#      (or ±ABS_PARITY_SLACK files, whichever is wider) of the live
-#      source volume, ignoring the transient subtrees listed in
-#      volatile_paths_for().
-#
-# What this does NOT exercise:
-#   - CNPG database restore (barman-cloud, separate code path). CNPG
-#     volumes are never restored, cloned or counted here, even when a
-#     backup holds one (see CNPG_EXCLUDE).
-#   - Application-level boot against the restored data. Add per-service
-#      hooks below if/when needed; for now, file-presence is the bar.
-#
-# Discovery: every velero.io Schedule labelled
-#   app.kubernetes.io/part-of=homelab-backup
-# is picked up automatically, so new services don't need a code change
-# here.
-#
-# Cost: each service is restored sequentially and torn down before the
-# next, so peak storage cost is the size of the largest single service
-# (immich media, immich-library-bulk, ~250 GiB at time of writing). Immich
-# alone can take multiple hours (millions of small files), so the full
-# sweep is sized at up to 6 h via the cronjob's activeDeadlineSeconds.
-#
-# Capacity gate: before each restore, the raw size of everything the run
-# writes for it (restored PVCs plus source-side probe clones) is projected
-# onto the fullest OSD (see preflight_capacity). A schedule whose
-# projected peak is above OSD_MAX_PCT (80%) is not restored and is reported
-# as SKIP, not FAIL. Skips are listed in the summary so an untested backup
-# stays visible.
-#
-# Mid-run guard (REN-451): the gate is a projection made once. While a
-# restore runs, every poll reads the fullest OSD; at OSD_ABORT_PCT, or with
-# no reading from Prometheus for PROM_MISS_MAX polls, the restore is
-# cancelled, torn down and the sweep stops (FAIL). A restore that times
-# out is cancelled the same way, and so is the one in flight when the Job
-# is killed (TERM/EXIT trap). A restore that cannot finish its
-# RESTORE_PHASE_TIMEOUT_S before the Job deadline is not started (SKIP).
-#
-# Exit non-zero if any check fails. Prints a summary at the end.
-
-set -uo pipefail
-
-PASS=0
-FAIL=0
-SKIP=0
-FAILURES=()
-SKIPS=()
-RUN_ID=$(date -u +%s)
-RUN_START=$RUN_ID
-RESTORE_PHASE_TIMEOUT_S=${RESTORE_PHASE_TIMEOUT_S:-14400}   # 4 h — immich media (immich-library-bulk, ~250 GiB, millions of small files) does not finish in 2 h
-# Must equal the Job's activeDeadlineSeconds. A restore is started only if
-# its RESTORE_PHASE_TIMEOUT_S plus CLEANUP_RESERVE_S (parity probes and
-# teardown) ends before it.
-JOB_DEADLINE_S=${JOB_DEADLINE_S:-21600}
-CLEANUP_RESERVE_S=${CLEANUP_RESERVE_S:-900}
-
-# CNPG volumes are out of scope (REN-451). Without this filter a backup
-# that holds a CNPG PVC is restored as a second copy of the database, and
-# the parity probe snapshots and clones the live one next to production.
-# CNPG puts cnpg.io/cluster on every PVC it owns, and Velero keeps PVC
-# labels on restore. Used by the Restore, the capacity gate and the
-# parity loop.
-CNPG_LABEL=cnpg.io/cluster
-CNPG_EXCLUDE="!${CNPG_LABEL}"
-
-log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
-
-check() {
-    local name="$1"; shift
-    printf '  %-72s ' "$name"
-    if "$@" >/tmp/check.out 2>&1; then
-        echo "OK"
-        PASS=$((PASS + 1))
-    else
-        echo "FAIL"
-        FAILURES+=("$name -- $(tail -4 /tmp/check.out | tr '\n' ' ')")
-        FAIL=$((FAIL + 1))
-    fi
-}
-
-skip() {
-    local name="$1" reason="$2"
-    printf '  %-72s SKIP\n' "$name"
-    SKIPS+=("$name -- $reason")
-    SKIP=$((SKIP + 1))
-}
-
-# Capacity gate (REN-441, REN-442). Every restore lands on the production
-# Ceph pools, so a large one pushes the OSDs toward nearfull while it is
-# alive. On 2026-10-01 audiobookshelf-books (87 GB, EC 2+1) took the
-# fullest OSD from 74.1% to 78.6%, and the immich library (~198 GB)
-# projected to ~85-86%, Ceph's nearfull line.
-#
-# The projection: raw bytes of the restore spread evenly over all OSDs,
-# added to every OSD's usage now; the fullest result is the peak. Raw
-# bytes = sum of DataUpload totalBytes x the pool's raw/usable factor,
-# once for the restored PVC and once more for the source-side probe clone
-# (count_files_in_source_pvc). CNPG volumes are left out: they are neither
-# restored nor cloned.
-# Even spread is exact for this cluster (three OSDs, one per host, every
-# pool spans all three); it would understate a skewed layout.
-OSD_MAX_PCT=${OSD_MAX_PCT:-80}
-# Mid-run abort line. Below mon_osd_nearfull_ratio 0.85 so the restore is
-# cancelled before Ceph warns; on 2026-10-01 the immich restore climbed
-# ~5 points an hour, so one 15 s poll overshoots by far less than the gap.
-OSD_ABORT_PCT=${OSD_ABORT_PCT:-83}
-# Prometheus keeps its TSDB on Ceph and stopped scraping at 95% on
-# 2026-10-01. No reading for this many polls in a row (15 s each) counts
-# as full.
-PROM_MISS_MAX=${PROM_MISS_MAX:-4}
-PROMETHEUS_URL=${PROMETHEUS_URL:-http://kube-prometheus-stack-prometheus.monitoring.svc:9090}
-
-# Raw bytes written per usable byte, by StorageClass. Source of truth is
-# kubernetes/infrastructure/rook-ceph-cluster/ (storageclass.yaml,
-# cephfs-ci.yaml): cephfs-bulk is EC 2+1, cephfs-ci-data0 is size 2,
-# replicapool and cephfs-data0 are size 3. Any class not listed counts as
-# 3, the largest factor in the cluster.
-RAW_FACTORS='{"ceph-filesystem-bulk": 1.5, "ceph-filesystem-ci": 2}'
-
-# Raw factor of the probe clone, by the source PVC's StorageClass. Mirrors
-# sc_for_snapshot_probe: ceph-block clones to ceph-block-snapshot-readonly
-# (replicapool) and ceph-filesystem to ceph-filesystem-ephemeral
-# (cephfs-data0), both size 3. Any other class gets no clone. The clones
-# are counted as if all were alive at once: each is deleted with
-# --wait=false and CephFS purges asynchronously, so the sum is the safe
-# bound.
-CLONE_FACTORS='{"ceph-block": 3, "ceph-filesystem": 3}'
-
-# Prometheus instant query; echoes the first sample value.
-prom_scalar() {
-    curl -sf -m 30 -G --data-urlencode "query=$1" "${PROMETHEUS_URL}/api/v1/query" \
-        | jq -er '.data.result[0].value[1]'
-}
-
-# Echo "<raw_bytes> <now_pct> <peak_pct>" for a backup of src_ns, or echo
-# a reason and return non-zero. The CNPG volumes to leave out are the
-# source namespace's CNPG PVCs now; a CNPG PVC that has gone since the
-# backup is still counted, which only errs toward skipping. Usage is read
-# fresh per schedule, after the previous schedule's teardown; deletion lag
-# can only make it read higher, so it errs on the side of skipping.
-preflight_capacity() {
-    local backup="$1" src_ns="$2" cnpg raw now peak
-    cnpg=$(kubectl get pvc -n "$src_ns" -l "$CNPG_LABEL" -o json \
-        | jq -ec '[.items[].metadata.name]') || {
-        echo "cannot list CNPG PVCs in ${src_ns}"
-        return 1
-    }
-    raw=$(kubectl get dataupload -n velero -l "velero.io/backup-name=${backup}" -o json \
-        | jq -er --argjson f "$RAW_FACTORS" --argjson c "$CLONE_FACTORS" --argjson cnpg "$cnpg" '
-            [.items[]
-             | select(.spec.sourcePVC as $p | $cnpg | index([$p]) | not)
-             | (.spec.csiSnapshot.storageClass // "") as $sc
-             | {b: .status.progress.totalBytes, k: (($f[$sc] // 3) + ($c[$sc] // 0))}]
-            | if any(.b == null) then error("no totalBytes") else map(.b * .k) | add // 0 | floor end') || {
-        echo "cannot size backup ${backup} (DataUpload without totalBytes?)"
-        return 1
-    }
-    now=$(prom_scalar 'max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes) * 100') \
-        && peak=$(prom_scalar "max((ceph_osd_stat_bytes_used + ${raw} / scalar(count(ceph_osd_stat_bytes))) / ceph_osd_stat_bytes) * 100") || {
-        echo "Prometheus query failed (${PROMETHEUS_URL})"
-        return 1
-    }
-    # NaN or an empty vector must not read as "below the gate".
-    if ! [[ "$now" =~ ^[0-9]+(\.[0-9]+)?$ && "$peak" =~ ^[0-9]+(\.[0-9]+)?$ ]]; then
-        echo "no usable OSD usage from Prometheus (now=${now} peak=${peak})"
-        return 1
-    fi
-    echo "$raw $now $peak"
-}
-
-# Force-delete Released PVs whose claimRef matches a pattern. The ceph-block
-# and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
-# production PVC deletions can be undone manually. The restore-test has the
-# opposite need: every PVC it creates is throwaway and MUST not survive the
-# iteration. Without this, each run leaks O(100 GiB) of CephFS
-# subvolumes and the cluster eventually goes OSD_FULL (incident 2026-05-17).
-#
-# Why this still exists after ceph-*-ephemeral landed. Three categories of
-# throwaway PVC are created per iteration, and only one is ours to choose a
-# StorageClass for:
-#
-#   1. source-side snapshot-probe PVCs (rt-src-*, this script creates them)
-#      -> FIXED at source: sc_for_snapshot_probe now returns only Delete
-#         classes, so these no longer reach Released at all.
-#   2. destination PVCs in *-restore-test namespaces (Velero creates them)
-#      -> NOT fixable here. Velero restores the PVC spec from the backup,
-#         so the SC is whatever production uses (ceph-block/ceph-filesystem,
-#         both Retain). The only remap knob is the change-storage-class
-#         RestoreItemAction ConfigMap, which Velero applies to *every*
-#         restore in the installation with no per-restore scoping — using it
-#         would silently strip Retain from a real disaster-recovery restore,
-#         which is precisely when the safety net matters. Not worth it to
-#         save a cleanup loop here.
-#   3. Velero data-mover staging PVCs in velero/ (*-rtest-*)
-#      -> NOT fixable here either. These derive from the target PVC, and the
-#         node-agent restorePVC config supports only `ignoreDelayBinding`;
-#         unlike backupPVC there is no storageClass field (Velero v1.18).
-#
-# So categories 2 and 3 keep this function load-bearing. Verified against
-# the Velero v1.18 restore-reference and data-movement-restore-pvc docs,
-# and against a live run: the destination PVCs and velero/*-rtest-* staging
-# PVCs both provision on the Retain classes.
-purge_released_pvs() {
-    local pattern="$1"   # regex applied to .spec.claimRef.namespace + "/" + .spec.claimRef.name
-    local pvs pv
-    pvs=$(kubectl get pv -o json 2>/dev/null \
-        | jq -r --arg p "$pattern" '
-            .items[]
-            | select(.status.phase == "Released")
-            | select(((.spec.claimRef.namespace // "") + "/" + (.spec.claimRef.name // "")) | test($p))
-            | .metadata.name')
-    for pv in $pvs; do
-        kubectl patch pv "$pv" --type=merge \
-            -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' >/dev/null 2>&1 || true
-        kubectl delete pv "$pv" --wait=false --ignore-not-found >/dev/null 2>&1 || true
-    done
-}
-
-cleanup_service() {
-    local dst_ns="$1" restore="$2"
-    [[ -n "$restore" ]] && kubectl delete restore.velero.io "$restore" \
-        -n velero --ignore-not-found --wait=false >/dev/null 2>&1 || true
-    kubectl delete ns "$dst_ns" --ignore-not-found --wait=false >/dev/null 2>&1 || true
-    # Block until namespace is actually gone, otherwise the next service's
-    # pre-cleanup races with leftover terminating pods.
-    local ns_gone=1
-    for _ in $(seq 1 60); do
-        if ! kubectl get ns "$dst_ns" >/dev/null 2>&1; then
-            ns_gone=0; break
-        fi
-        sleep 5
-    done
-    # Reclaim PVs left behind in Released state for this iteration:
-    #   - destination PVCs in dst_ns
-    #   - Velero data-mover backup PVCs in velero/ named *-rtest-${RUN_ID}-*
-    #   - source-side snapshot-probe PVCs in any ns named rt-src-*-${RUN_ID}
-    purge_released_pvs "^(${dst_ns}/.*|velero/.*-rtest-${RUN_ID}-.*|.*/rt-src-.*-${RUN_ID})$"
-    return $ns_gone
-}
-
-# The restore in flight, for cancel_restore and the TERM/EXIT trap. Empty
-# once it is torn down.
-CUR_RESTORE=""
-CUR_DST_NS=""
-ABORT_SWEEP=""
-
-# DataDownloads of a restore that are not finished. Matched by owner
-# reference as well as label: Velero hashes a label value over 63 chars.
-running_datadownloads() {
-    kubectl get datadownload -n velero -o json \
-        | jq -r --arg r "$1" '
-            .items[]
-            | select(.metadata.labels["velero.io/restore-name"] == $r
-                     or any(.metadata.ownerReferences[]?; .kind == "Restore" and .name == $r))
-            | select(.status.phase as $p | ["Completed", "Failed", "Canceled"] | index([$p]) | not)
-            | .metadata.name'
-}
-
-# Stop a restore that may still be writing, then tear it down. Deleting
-# the Restore, or killing this Job, does not stop Velero: on 2026-10-01 the
-# immich DataDownloads kept writing after the Job died at 14:00 until Ceph
-# went OSD_FULL at 15:10. spec.cancel on each DataDownload is the stop.
-cancel_restore() {
-    local restore="$1" dst_ns="$2" dd left=""
-    for dd in $(running_datadownloads "$restore"); do
-        kubectl patch datadownload -n velero "$dd" --type=merge \
-            -p '{"spec":{"cancel":true}}' >/dev/null 2>&1 || true
-    done
-    # Up to 2 min for them to stop, so their staging PVs are Released
-    # before the purge in cleanup_service.
-    for _ in $(seq 1 24); do
-        left=$(running_datadownloads "$restore" 2>/dev/null) || left="?"
-        [[ -z "$left" ]] && break
-        sleep 5
-    done
-    [[ -n "$left" ]] && echo "DataDownloads not stopped after 2 min: ${left//$'\n'/ }"
-    cleanup_service "$dst_ns" "$restore"
-    CUR_RESTORE=""
-    CUR_DST_NS=""
-}
-
-# Wait for the Restore to finish while watching the fullest OSD (see the
-# header: mid-run guard). Returns 0 on Completed. Any abort or timeout
-# cancels and tears the restore down before returning 1; an abort also
-# sets ABORT_SWEEP so no further restore starts.
-wait_for_restore() {
-    local restore="$1" dst_ns="$2" deadline p last="" osd misses=0
-    deadline=$(( $(date +%s) + RESTORE_PHASE_TIMEOUT_S ))
-    while [[ $(date +%s) -lt $deadline ]]; do
-        p=$(kubectl -n velero get restore.velero.io "$restore" \
-            -o jsonpath='{.status.phase}' 2>/dev/null)
-        [[ "$p" = Completed ]] && return 0
-        # Failed, PartiallyFailed, FailedValidation: Velero has stopped.
-        if [[ "$p" == *Failed* ]]; then
-            kubectl -n velero get restore.velero.io "$restore" -o yaml | tail -60
-            return 1
-        fi
-        if [[ "$p" != "$last" ]]; then
-            echo "$(date -u +%H:%M:%S) restore phase: $p"
-            last="$p"
-        fi
-        osd=$(prom_scalar 'max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes) * 100' 2>/dev/null)
-        if [[ "$osd" =~ ^[0-9]+(\.[0-9]+)?$ ]]; then
-            misses=0
-            if awk -v o="$osd" -v m="$OSD_ABORT_PCT" 'BEGIN { exit !(o >= m) }'; then
-                ABORT_SWEEP="fullest OSD $(printf '%.1f' "$osd")% reached ${OSD_ABORT_PCT}% during ${restore}"
-            fi
-        else
-            misses=$((misses + 1))
-            (( misses >= PROM_MISS_MAX )) \
-                && ABORT_SWEEP="no OSD usage from Prometheus for ${misses} polls during ${restore}"
-        fi
-        if [[ -n "$ABORT_SWEEP" ]]; then
-            echo "ABORT: ${ABORT_SWEEP}; cancelling"
-            cancel_restore "$restore" "$dst_ns"
-            return 1
-        fi
-        # Background sleep so a TERM runs the trap at once, not after it.
-        sleep 15 & wait $!
-    done
-    echo "TIMEOUT after ${RESTORE_PHASE_TIMEOUT_S}s; cancelling"
-    kubectl -n velero get restore.velero.io "$restore" -o yaml | tail -60
-    cancel_restore "$restore" "$dst_ns"
-    return 1
-}
-
-# Transient subtrees that must not count toward file-count parity, keyed
-# by "<source-namespace>/<pvc>". Values are whole directories relative to
-# the volume root.
-#
-# Why this exists: parity compares a backup taken up to ~24 h ago against
-# the live volume *now*. Any regenerable cache that churns faster than the
-# ±10% tolerance produces a spurious FAIL even when the backup is
-# byte-for-byte complete. On 2026-07-26 one Audiobookshelf playback session
-# had written 1531 HLS transcode files (~153 MiB) into metadata/streams/
-# after the backup ran, against 808 persistent files — a 3x inflation that
-# failed the run for no real reason. A verification layer that cries wolf
-# gets ignored, which is worse than not having one.
-#
-# Keep this list minimal and justified: every entry is a blind spot. Only
-# add paths whose loss is genuinely a non-event on restore.
-volatile_paths_for() {
-    case "$1" in
-        # streams: HLS transcode segments for in-flight playback sessions.
-        # tmp:     scratch space for downloads/imports.
-        # cache/ is deliberately NOT excluded: it is regenerable, but it is
-        # stable enough (190 files) not to threaten the tolerance, and
-        # keeping it counted preserves real coverage on this volume.
-        audiobookshelf/audiobookshelf-metadata) echo "streams tmp" ;;
-        *) echo "" ;;
-    esac
-}
-
-# Emit the `find` command run inside the probe pod, pruning the given
-# space-separated volume-root-relative paths. Paths are single-quoted for
-# the remote shell; parens are backslash-escaped so the remote sh hands
-# literal ones to find rather than opening a subshell.
-#
-# The -prune form was verified in the probe image itself (alpine:3.21,
-# BusyBox v1.37.0) against the live audiobookshelf-metadata volume:
-# unfiltered 2339, pruned 775 == 2339 - 1531 (streams) - 33 (tmp). A path
-# listed here that does not exist on the volume is a no-op, not an error
-# (exit 0), so a stale entry degrades gracefully rather than zeroing the
-# count. An empty exclusion list reproduces the original command verbatim.
-build_count_cmd() {
-    local excludes="$1" find_expr="" p
-    for p in $excludes; do
-        find_expr="${find_expr:+$find_expr -o }-path '/d/$p'"
-    done
-    if [[ -n "$find_expr" ]]; then
-        echo "find /d -xdev \\( $find_expr \\) -prune -o -type f -print 2>/dev/null | wc -l"
-    else
-        echo "find /d -xdev -type f 2>/dev/null | wc -l"
-    fi
-}
-
-# Mount the named PVC in a transient probe pod and emit the count of
-# regular files, excluding the optional space-separated volatile paths in
-# $4. Echoes "" and returns non-zero on any failure (pod scheduling,
-# mount, exec). Pod is RO-mounted so we never disturb the underlying
-# volume; cleanup is best-effort but unconditional.
-count_files_in_pvc() {
-    local ns="$1" pvc="$2" pod_suffix="$3" excludes="${4:-}"
-    local pod="probe-${pod_suffix:0:50}"
-    cat <<EOF | kubectl apply -f - >/dev/null
-apiVersion: v1
-kind: Pod
-metadata:
-  name: $pod
-  namespace: $ns
-  labels:
-    app.kubernetes.io/managed-by: restore-test
-spec:
-  restartPolicy: Never
-  securityContext:
-    runAsNonRoot: true
-    runAsUser: 65532
-    runAsGroup: 65532
-    fsGroup: 65532
-    seccompProfile: { type: RuntimeDefault }
-  containers:
-    - name: probe
-      image: alpine:3.21
-      command: ["sh", "-c", "sleep 1800"]
-      securityContext:
-        allowPrivilegeEscalation: false
-        capabilities: { drop: ["ALL"] }
-      volumeMounts:
-        - { name: d, mountPath: /d, readOnly: true }
-  volumes:
-    - name: d
-      persistentVolumeClaim:
-        claimName: $pvc
-        readOnly: true
-EOF
-    if ! kubectl wait -n "$ns" --for=condition=ready "pod/$pod" --timeout=300s >/dev/null 2>&1; then
-        kubectl describe -n "$ns" "pod/$pod" 2>&1 | tail -25 >&2
-        kubectl delete -n "$ns" "pod/$pod" --ignore-not-found --wait=false >/dev/null 2>&1 || true
-        return 1
-    fi
-    local n
-    n=$(kubectl exec -n "$ns" "$pod" -- sh -c "$(build_count_cmd "$excludes")" 2>/dev/null)
-    kubectl delete -n "$ns" "pod/$pod" --ignore-not-found --wait=false >/dev/null 2>&1 || true
-    [[ "$n" =~ ^[0-9]+$ ]] || return 1
-    echo "$n"
-}
-
-# Map a source PVC's storage class to (snapshotClass, tmp PVC SC) used
-# for the source-side count probe. ext4 RBD snapshots need the
-# noload-mounting SC; CephFS has no dirty-journal failure mode and only
-# needs a plain CephFS class.
-#
-# BOTH tmp classes must be reclaimPolicy: Delete — these probe PVCs are
-# throwaway and are the one leak category this script fully controls
-# (see purge_released_pvs). ceph-block-snapshot-readonly is already
-# Delete, and its `noload` mountOption is load-bearing (skips ext4
-# journal replay on the RO snapshot mount), so it must NOT be swapped
-# for ceph-block-ephemeral, which lacks that option.
-sc_for_snapshot_probe() {
-    local source_sc="$1"
-    case "$source_sc" in
-        ceph-block)      echo "ceph-block-snapclass ceph-block-snapshot-readonly" ;;
-        ceph-filesystem) echo "ceph-filesystem-snapclass ceph-filesystem-ephemeral" ;;
-        *) return 1 ;;
-    esac
-}
-
-# Snapshot the source PVC (PVC name preserved across restore — same
-# in src and dst), clone to a tmp PVC, count files excluding the optional
-# space-separated volatile paths in $3. Cleans up its own artifacts in
-# src_ns regardless of outcome.
-count_files_in_source_pvc() {
-    local src_ns="$1" pvc="$2" excludes="${3:-}"
-    local source_sc capacity sc_pair snap_class tmp_sc
-    source_sc=$(kubectl get pvc -n "$src_ns" "$pvc" -o jsonpath='{.spec.storageClassName}' 2>/dev/null)
-    capacity=$(kubectl get pvc -n "$src_ns" "$pvc" -o jsonpath='{.spec.resources.requests.storage}' 2>/dev/null)
-    if ! sc_pair=$(sc_for_snapshot_probe "$source_sc"); then
-        echo "unsupported source SC: $source_sc" >&2
-        return 1
-    fi
-    snap_class=${sc_pair% *}
-    tmp_sc=${sc_pair#* }
-
-    local snap="rt-src-${pvc:0:40}-${RUN_ID}"
-    local tmp_pvc="rt-src-${pvc:0:40}-${RUN_ID}"
-
-    # Best-effort cleanup hook — runs on every return path.
-    cleanup_source_probe() {
-        kubectl delete pvc -n "$src_ns" "$tmp_pvc" \
-            --ignore-not-found --wait=false >/dev/null 2>&1 || true
-        kubectl delete volumesnapshot -n "$src_ns" "$snap" \
-            --ignore-not-found --wait=false >/dev/null 2>&1 || true
-    }
-    trap cleanup_source_probe RETURN
-
-    cat <<EOF | kubectl apply -f - >/dev/null
-apiVersion: snapshot.storage.k8s.io/v1
-kind: VolumeSnapshot
-metadata:
-  name: $snap
-  namespace: $src_ns
-  labels:
-    app.kubernetes.io/managed-by: restore-test
-spec:
-  volumeSnapshotClassName: $snap_class
-  source:
-    persistentVolumeClaimName: $pvc
-EOF
-    # Wait ReadyToUse — RBD CSI provisioner is fast (seconds); allow 5 min.
-    local ready=""
-    for _ in $(seq 1 60); do
-        ready=$(kubectl get volumesnapshot -n "$src_ns" "$snap" \
-            -o jsonpath='{.status.readyToUse}' 2>/dev/null)
-        [ "$ready" = "true" ] && break
-        sleep 5
-    done
-    if [ "$ready" != "true" ]; then
-        echo "snapshot not ReadyToUse after 5m" >&2
-        return 1
-    fi
-
-    cat <<EOF | kubectl apply -f - >/dev/null
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: $tmp_pvc
-  namespace: $src_ns
-  labels:
-    app.kubernetes.io/managed-by: restore-test
-spec:
-  accessModes: [ReadOnlyMany]
-  storageClassName: $tmp_sc
-  resources:
-    requests:
-      storage: $capacity
-  dataSource:
-    apiGroup: snapshot.storage.k8s.io
-    kind: VolumeSnapshot
-    name: $snap
-EOF
-
-    count_files_in_pvc "$src_ns" "$tmp_pvc" "src-${pvc:0:40}-${RUN_ID}" "$excludes"
-}
-
-# Restored count must be within max(±10%, ±ABS_PARITY_SLACK files) of the
-# source's current file count. That tolerance absorbs day-of drift since
-# the backup ran without letting "lost half the data" pass silently.
-# Volatile subtrees are pruned from BOTH sides — keyed off the source
-# identity, since the PVC name is preserved across the restore but the
-# destination namespace is not.
-#
-# Why the absolute floor exists. A purely relative window is structurally
-# unable to pass a small volume: ±10% of 9 files is ±0.9, i.e. a budget of
-# ZERO files, and ±10% of 2 files is ±0.2. The 2026-09-01 run failed on
-# exactly that — audiobookshelf/libation-config source=9 restored=8 and
-# trmnl/trmnl-renderer-data source=2 restored=3, both off by one file and
-# in opposite directions, with all 12 restores at phase=Completed and the
-# other 46 checks green. Parity compares a backup taken up to ~24 h ago
-# against the live volume *now*, so a single config write on a tiny volume
-# between those two moments is expected, not a defect in the backup.
-#
-# The floor only governs volumes below ~10*ABS_PARITY_SLACK files; above
-# that the 10% window still binds, so real data loss on the volumes that
-# carry data keeps failing loudly. Two files of slack rather than one
-# because a single logical write on a config volume routinely moves more
-# than one file (an app writing a settings file plus a transient sidecar),
-# and a floor that buys exactly one month is not worth the change.
-#
-# What the floor must NOT swallow, and does not: a non-empty source that
-# restored to an *empty* volume fails unconditionally below, before any
-# tolerance is applied. The old relative-only check caught that case as a
-# side effect (delta = 1.0 > 0.10); it is now explicit so the floor cannot
-# erode it on a volume of ABS_PARITY_SLACK files or fewer.
-ABS_PARITY_SLACK=${ABS_PARITY_SLACK:-2}
-
-verify_pvc_file_count_parity() {
-    local src_ns="$1" dst_ns="$2" pvc="$3"
-    local restored source excludes
-    excludes=$(volatile_paths_for "${src_ns}/${pvc}")
-    restored=$(count_files_in_pvc "$dst_ns" "$pvc" "dst-${pvc:0:40}-${RUN_ID}" "$excludes") || {
-        echo "restored-count probe failed"
-        return 1
-    }
-    source=$(count_files_in_source_pvc "$src_ns" "$pvc" "$excludes") || {
-        echo "source-count probe failed (restored=$restored)"
-        return 1
-    }
-    echo "source=$source restored=$restored${excludes:+ (excluded: $excludes)}"
-    # Both zero is a degenerate but legal state (empty volume).
-    if [[ "$source" -eq 0 && "$restored" -eq 0 ]]; then
-        return 0
-    fi
-    # Empty on either side while the other has data: never tolerable.
-    if [[ "$source" -eq 0 || "$restored" -eq 0 ]]; then
-        return 1
-    fi
-    # Use awk to compute the window without bc.
-    awk -v s="$source" -v r="$restored" -v slack="$ABS_PARITY_SLACK" 'BEGIN {
-        delta = r - s; if (delta < 0) delta = -delta
-        tol = 0.10 * s; if (tol < slack) tol = slack
-        exit !(delta <= tol)
-    }'
-}
-
-echo "=== Velero PVC backup restore verification ==="
-log "run_id=$RUN_ID"
-
-# The Job deadline (or a deleted Job) sends TERM; bash is PID 1 and only
-# acts on it with a trap. Stop the restore in flight before the pod goes:
-# terminationGracePeriodSeconds in the CronJob covers cancel_restore.
-# fd 3 is the pod log, so the trap logs even inside check's redirect.
-exec 3>&1
-on_exit() {
-    [[ -n "$CUR_RESTORE" ]] || return 0
-    log "stopping in-flight restore ${CUR_RESTORE}" >&3
-    cancel_restore "$CUR_RESTORE" "$CUR_DST_NS" >&3 2>&1
-}
-trap on_exit EXIT
-trap 'log "SIGTERM (Job deadline or deletion)" >&3; exit 143' TERM
-
-check "BSL default Available" bash -c \
-    'kubectl -n velero get backupstoragelocation default -o jsonpath="{.status.phase}" | grep -q Available'
-
-check "node-agent DaemonSet has all pods ready" bash -c '
-    desired=$(kubectl -n velero get ds node-agent -o jsonpath="{.status.desiredNumberScheduled}")
-    ready=$(kubectl -n velero get ds node-agent -o jsonpath="{.status.numberReady}")
-    test "$ready" -ge "$desired" -a "$desired" -ge 1
-'
-
-# Sweep residue from *earlier* runs before starting this one.
-#
-# cleanup_service's purge is scoped to ${RUN_ID} for the velero/*-rtest-*
-# and rt-src-* patterns, which is correct while a run is in progress (it
-# must not touch a concurrent run's artifacts) but means a run that dies
-# before its cleanup — activeDeadlineSeconds, OOM, node eviction, a killed
-# Job — strands its PVs forever: no later run's pattern matches a previous
-# RUN_ID. That is the residual leak path behind incident 2026-05-17, and it
-# survives the ceph-*-ephemeral fix because categories 2 and 3 above are on
-# Retain classes regardless.
-#
-# Safe to run unscoped here: concurrencyPolicy Forbid guarantees no other
-# restore-test Job is active, so every Released PV matching these
-# restore-test-only naming patterns is by definition abandoned. Scoped to
-# Released PVs only, so nothing Bound to a live workload is ever touched.
-log "sweeping Released PVs left by earlier runs"
-purge_released_pvs '^(.*-restore-test/.*|velero/.*-rtest-.*|.*/rt-src-.*)$'
-
-mapfile -t SCHEDULES < <(
-    kubectl get schedule.velero.io -n velero \
-        -l app.kubernetes.io/part-of=homelab-backup \
-        -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'
-)
-
-if [[ ${#SCHEDULES[@]} -eq 0 ]]; then
-    echo "FAIL: no Schedules with label app.kubernetes.io/part-of=homelab-backup"
-    exit 1
-fi
-log "discovered ${#SCHEDULES[@]} schedules: ${SCHEDULES[*]}"
-
-for schedule in "${SCHEDULES[@]}"; do
-    src_ns=$(kubectl get schedule.velero.io -n velero "$schedule" \
-        -o jsonpath='{.spec.template.includedNamespaces[0]}')
-    dst_ns="${src_ns}-restore-test"
-
-    echo
-    echo "=== service: ${src_ns} (schedule=${schedule}) ==="
-
-    if [[ -n "$ABORT_SWEEP" ]]; then
-        skip "${src_ns}: restore (sweep stopped)" "$ABORT_SWEEP"
-        continue
-    fi
-
-    # Newest terminal Backup wins. PartiallyFailed is intentionally
-    # included: Velero only restores DataUploads that completed, so
-    # the partial source still gives a valid restore for the PVCs that
-    # did upload — and excluding them would silently let the test miss
-    # weeks where one DataUpload kept failing (e.g. the ext4-noload
-    # window before that fix).
-    #
-    # Sort happens in jq, not kubectl: `kubectl --sort-by` is silently
-    # ignored when output is `-o jsonpath`, so the previous form picked a
-    # non-deterministic backup and could choose a snapshot already pruned
-    # by kopia maintenance ("Unable to load snapshot : snapshot not found").
-    backup=$(kubectl get backup.velero.io -n velero \
-        -l "velero.io/schedule-name=${schedule}" -o json \
-        | jq -r '
-            [.items[]
-             | select(.status.phase == "Completed"
-                   or .status.phase == "PartiallyFailed")]
-            | sort_by(.status.startTimestamp)
-            | last
-            | .metadata.name // empty')
-
-    if [[ -z "$backup" ]]; then
-        check "${src_ns}: a Completed/PartiallyFailed Backup exists" false
-        continue
-    fi
-    log "${src_ns}: using backup ${backup}"
-
-    # No DataUploads = the schedule is a no-op for this service (e.g.
-    # authentik has no PVCs). Skip with PASS. csiVolumeSnapshots* status
-    # fields aren't populated on Velero v1.18, so count DataUpload
-    # children directly instead. Fail loudly if the list itself errors,
-    # so an RBAC regression doesn't silently pass every service.
-    if ! du_out=$(kubectl get dataupload -n velero \
-        -l "velero.io/backup-name=${backup}" -o name 2>&1); then
-        check "${src_ns}: list DataUploads (RBAC?)" bash -c "echo '$du_out'; false"
-        continue
-    fi
-    du_count=$(echo "$du_out" | grep -c . || true)
-    if [[ "${du_count:-0}" -eq 0 ]]; then
-        check "${src_ns}: no-PVC schedule (skip restore)" true
-        continue
-    fi
-
-    # Capacity gate. If the projection cannot be made, do not restore and
-    # count a FAIL: an unsized restore is the risk the gate exists for.
-    if ! cap=$(preflight_capacity "$backup" "$src_ns"); then
-        check "${src_ns}: capacity pre-flight" bash -c "echo '$cap'; false"
-        continue
-    fi
-    read -r raw now peak <<<"$cap"
-    raw_gib=$(awk -v b="$raw" 'BEGIN { printf "%.0f", b / 2^30 }')
-    printf -v now '%.1f' "$now"
-    printf -v peak '%.1f' "$peak"
-    if awk -v p="$peak" -v m="$OSD_MAX_PCT" 'BEGIN { exit !(p > m) }'; then
-        skip "${src_ns}: restore (capacity gate)" \
-            "projected fullest OSD ${peak}% > ${OSD_MAX_PCT}% (now ${now}%, restore ~${raw_gib} GiB raw)"
-        continue
-    fi
-    log "${src_ns}: capacity ok, fullest OSD ${now}% -> projected ${peak}% (~${raw_gib} GiB raw)"
-
-    # Time budget: a restore the Job deadline would cut off is the one
-    # nothing watches once the Job dies.
-    left_s=$(( RUN_START + JOB_DEADLINE_S - $(date +%s) ))
-    if (( RESTORE_PHASE_TIMEOUT_S + CLEANUP_RESERVE_S > left_s )); then
-        skip "${src_ns}: restore (time budget)" \
-            "${left_s}s left before the Job deadline < ${RESTORE_PHASE_TIMEOUT_S}s restore timeout + ${CLEANUP_RESERVE_S}s reserve"
-        continue
-    fi
-
-    # Pre-cleanup in case a prior run left residue.
-    cleanup_service "$dst_ns" "" >/dev/null 2>&1 || true
-
-    restore_name="${schedule}-rtest-${RUN_ID}"
-    CUR_RESTORE=$restore_name
-    CUR_DST_NS=$dst_ns
-    cat <<EOF | kubectl apply -f - >/dev/null
-apiVersion: velero.io/v1
-kind: Restore
-metadata:
-  name: ${restore_name}
-  namespace: velero
-spec:
-  backupName: ${backup}
-  includedNamespaces: [${src_ns}]
-  namespaceMapping:
-    ${src_ns}: ${dst_ns}
-  # Only the data — Flux owns workload manifests.
-  includedResources:
-    - persistentvolumeclaims
-    - persistentvolumes
-  # No CNPG volume, whatever the backup holds (CNPG_EXCLUDE).
-  labelSelector:
-    matchExpressions:
-      - key: ${CNPG_LABEL}
-        operator: DoesNotExist
-EOF
-
-    # A function, not `bash -c`: it must run in this shell so the TERM trap
-    # fires during the wait and ABORT_SWEEP/CUR_RESTORE stick.
-    check "${src_ns}: restore reaches phase=Completed" \
-        wait_for_restore "$restore_name" "$dst_ns"
-    # Cancelled and torn down (abort or timeout): nothing left to probe.
-    [[ -z "$CUR_RESTORE" ]] && continue
-
-    # Probe each restored PVC, never a CNPG one (CNPG_EXCLUDE): its source
-    # probe would snapshot and clone the live database.
-    mapfile -t PVCS < <(
-        kubectl get pvc -n "$dst_ns" -l "$CNPG_EXCLUDE" \
-            -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null
-    )
-    if [[ ${#PVCS[@]} -eq 0 ]]; then
-        check "${src_ns}: at least one PVC was restored" false
-    else
-        for pvc in "${PVCS[@]}"; do
-            check "${src_ns}: PVC ${pvc} file count parity (±10% / ±${ABS_PARITY_SLACK})" \
-                verify_pvc_file_count_parity "$src_ns" "$dst_ns" "$pvc"
-        done
-    fi
-
-    cleanup_service "$dst_ns" "$restore_name"
-    CUR_RESTORE=""
-    CUR_DST_NS=""
-done
-
-echo
-echo "=== Summary ==="
-echo "PASS: $PASS"
-echo "FAIL: $FAIL"
-echo "SKIP: $SKIP"
-# A skipped service has no restore evidence this month. Listed on every
-# run so it is not mistaken for a pass; it does not fail the Job.
-if [[ $SKIP -gt 0 ]]; then
-    echo
-    echo "Skipped (backup NOT restore-tested):"
-    printf '  - %s\n' "${SKIPS[@]}"
-fi
-if [[ $FAIL -gt 0 ]]; then
-    echo
-    echo "Failures:"
-    printf '  - %s\n' "${FAILURES[@]}"
-    exit 1
-fi
-exit 0
@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Leave-one-out nearest-neighbour benchmark for an embedding model.
+
+Embeds a labelled corpus of Paperless documents and, for each document, predicts
+its correspondent and document type from the label of its nearest neighbour
+(cosine similarity, the document itself excluded). Accuracy is the share of
+documents whose predicted label equals the filed one.
+
+Corpus: JSON lines {"id", "correspondent", "doctype", "text"}, exported
+read-only from the paperless database, for example on the CNPG primary:
+
+    select json_build_object('id', d.id, 'correspondent', d.correspondent_id,
+        'doctype', d.document_type_id,
+        'text', left(regexp_replace(d.content, '\\s+', ' ', 'g'), 3000))
+    from documents_document d
+    where d.correspondent_id is not null and d.document_type_id is not null
+      and length(d.content) > 200
+    order by md5(d.id::text || '<seed>') limit 400
+
+Ceiling is the share of documents whose label occurs on at least one other
+document; no nearest-neighbour method can exceed it. Floor is the share of the
+most common label.
+
+Usage:
+    knn-bench.py --corpus corpus.jsonl -e https://api.mistral.ai/v1/embeddings \\
+        -m mistral-embed --key-env MISTRAL_API_KEY --batch 8
+    knn-bench.py --corpus corpus.jsonl -m bge-m3 \\
+        -e http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings \\
+        --via-pod "deploy/llama-server -n ai"
+
+Standard library only. Vectors are cached per model next to the corpus.
+"""
+import argparse
+import json
+import math
+import os
+import re
+import shlex
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from collections import Counter
+
+
+def post(endpoint, payload, key, via_pod):
+    body = json.dumps(payload).encode()
+    if via_pod:
+        cmd = ["kubectl", "exec", "-i", *shlex.split(via_pod), "--", "curl", "-sS", "-m", "300",
+               "-H", "Content-Type: application/json", "-d", "@-", endpoint]
+        proc = subprocess.run(cmd, input=body, capture_output=True, check=False)
+        if proc.returncode != 0:
+            raise RuntimeError(f"kubectl exec failed: {proc.stderr.decode()}")
+        return json.loads(proc.stdout)
+    headers = {"Content-Type": "application/json"}
+    if key:
+        headers["Authorization"] = f"Bearer {key}"
+    req = urllib.request.Request(endpoint, data=body, headers=headers)
+    for attempt in range(6):
+        try:
+            with urllib.request.urlopen(req, timeout=300) as resp:
+                return json.load(resp)
+        except urllib.error.HTTPError as err:
+            if err.code == 429 and attempt < 5:
+                time.sleep(2 ** attempt)
+                continue
+            raise RuntimeError(f"HTTP {err.code}: {err.read().decode()}") from err
+    raise RuntimeError("retries exhausted")
+
+
+def embed_all(docs, args, key):
+    cache = f"{args.corpus}.{args.model}.{args.max_chars}.vecs.json"
+    if os.path.exists(cache):
+        with open(cache) as fh:
+            vecs = json.load(fh)
+        if len(vecs) == len(docs):
+            print(f"vectors loaded from {cache}")
+            return vecs
+    vecs = []
+    shrunk = 0
+    start = time.time()
+    for i in range(0, len(docs), args.batch):
+        chunk = [d["text"] for d in docs[i:i + args.batch]]
+        for _ in range(6):
+            try:
+                out = post(args.endpoint, {"model": args.model, "input": chunk}, key, args.via_pod)
+                break
+            except RuntimeError as err:
+                # Mistral reports "Input id N has T tokens, exceeding max M tokens": cut that text to fit.
+                hit = re.search(r"Input id (\d+) has (\d+) tokens, exceeding max (\d+)", str(err))
+                if not hit:
+                    raise
+                idx, have, limit = (int(g) for g in hit.groups())
+                chunk[idx] = chunk[idx][:int(len(chunk[idx]) * limit / have * 0.95)]
+                shrunk += 1
+        else:
+            raise RuntimeError(f"documents {i}-{i + len(chunk) - 1} still over the token limit")
+        if "data" not in out:
+            raise RuntimeError(f"no embeddings in response for documents {i}-{i + len(chunk) - 1}: {json.dumps(out)[:500]}")
+        data = sorted(out["data"], key=lambda item: item["index"])
+        if len(data) != len(chunk):
+            raise RuntimeError(f"asked for {len(chunk)} vectors, got {len(data)}")
+        vecs.extend(item["embedding"] for item in data)
+        done = len(vecs)
+        if done % 50 < args.batch:
+            print(f"  embedded {done}/{len(docs)} ({time.time() - start:.0f}s)", flush=True)
+    if shrunk:
+        print(f"  {shrunk} over-limit texts were shortened to fit the model's token limit")
+    with open(cache, "w") as fh:
+        json.dump(vecs, fh)
+    return vecs
+
+
+def normalise(vec):
+    norm = math.sqrt(sum(x * x for x in vec)) or 1.0
+    return [x / norm for x in vec]
+
+
+def nearest_neighbours(vecs):
+    n = len(vecs)
+    best = [(-2.0, -1)] * n
+    for i in range(n):
+        vi = vecs[i]
+        for j in range(i + 1, n):
+            s = sum(a * b for a, b in zip(vi, vecs[j]))
+            if s > best[i][0]:
+                best[i] = (s, j)
+            if s > best[j][0]:
+                best[j] = (s, i)
+    return [j for _, j in best]
+
+
+def score(docs, neighbours, field):
+    labels = [d[field] for d in docs]
+    counts = Counter(labels)
+    hits = sum(labels[i] == labels[neighbours[i]] for i in range(len(docs)))
+    ceiling = sum(counts[label] > 1 for label in labels)
+    floor = max(counts.values())
+    n = len(docs)
+    return 100 * hits / n, 100 * ceiling / n, 100 * floor / n, len(counts)
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--corpus", required=True)
+    ap.add_argument("-e", "--endpoint", required=True)
+    ap.add_argument("-m", "--model", required=True)
+    ap.add_argument("--key-env", default="", help="environment variable holding the API key")
+    ap.add_argument("--via-pod", default="", help='run curl in this pod, e.g. "deploy/llama-server -n ai"')
+    ap.add_argument("--batch", type=int, default=1)
+    ap.add_argument("--max-chars", type=int, default=0, help="truncate each text to this many characters (0 = as exported)")
+    args = ap.parse_args()
+
+    key = os.environ.get(args.key_env, "") if args.key_env else ""
+    if args.key_env and not key:
+        sys.exit(f"{args.key_env} is not set")
+
+    with open(args.corpus) as fh:
+        docs = [json.loads(line) for line in fh if line.strip()]
+    if args.max_chars:
+        for d in docs:
+            d["text"] = d["text"][:args.max_chars]
+    print(f"model={args.model} documents={len(docs)} max_chars={args.max_chars} endpoint={args.endpoint}")
+
+    start = time.time()
+    vecs = [normalise(v) for v in embed_all(docs, args, key)]
+    embed_s = time.time() - start
+    neighbours = nearest_neighbours(vecs)
+
+    print(f"dimensions={len(vecs[0])} embed+load={embed_s:.0f}s")
+    for field in ("correspondent", "doctype"):
+        acc, ceiling, floor, classes = score(docs, neighbours, field)
+        print(f"{field:<14} accuracy={acc:5.1f}%  ceiling={ceiling:5.1f}%  floor={floor:5.1f}%  labels={classes}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""Paired comparison of two embedding models scored by knn-bench.py.
+
+Both runs must cover the same documents in the same order. For each label field
+it reports how the per-document outcomes overlap, an exact two-sided McNemar
+test on the documents where exactly one model is right, and how often the two
+models pick the same nearest neighbour.
+
+Usage:
+    knn-compare.py --corpus-a a.jsonl --vecs-a a.vecs.json \\
+                   --corpus-b b.jsonl --vecs-b b.vecs.json
+
+The vectors files are the caches knn-bench.py writes next to its corpus.
+Standard library only.
+"""
+import argparse
+import importlib.util
+import json
+import math
+import os
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+spec = importlib.util.spec_from_file_location("knn_bench", os.path.join(HERE, "knn-bench.py"))
+bench = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(bench)
+
+
+def load(corpus, vecs):
+    with open(corpus) as fh:
+        docs = [json.loads(line) for line in fh if line.strip()]
+    with open(vecs) as fh:
+        vectors = [bench.normalise(v) for v in json.load(fh)]
+    if len(docs) != len(vectors):
+        raise SystemExit(f"{vecs}: {len(vectors)} vectors for {len(docs)} documents")
+    return docs, bench.nearest_neighbours(vectors)
+
+
+def mcnemar_exact(only_a, only_b):
+    """Two-sided exact p-value for a fair coin over the discordant pairs."""
+    n = only_a + only_b
+    if n == 0:
+        return 1.0
+    k = min(only_a, only_b)
+    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
+    return min(1.0, 2 * tail)
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    for side in "ab":
+        ap.add_argument(f"--corpus-{side}", required=True)
+        ap.add_argument(f"--vecs-{side}", required=True)
+    args = ap.parse_args()
+
+    docs_a, nn_a = load(args.corpus_a, args.vecs_a)
+    docs_b, nn_b = load(args.corpus_b, args.vecs_b)
+    if [d["id"] for d in docs_a] != [d["id"] for d in docs_b]:
+        raise SystemExit("the two corpora do not hold the same documents in the same order")
+
+    n = len(docs_a)
+    same_nn = sum(a == b for a, b in zip(nn_a, nn_b))
+    print(f"documents={n}  same nearest neighbour in both models: {same_nn} ({100 * same_nn / n:.1f}%)")
+    for field in ("correspondent", "doctype"):
+        ok_a = [docs_a[i][field] == docs_a[nn_a[i]][field] for i in range(n)]
+        ok_b = [docs_b[i][field] == docs_b[nn_b[i]][field] for i in range(n)]
+        both = sum(a and b for a, b in zip(ok_a, ok_b))
+        only_a = sum(a and not b for a, b in zip(ok_a, ok_b))
+        only_b = sum(b and not a for a, b in zip(ok_a, ok_b))
+        neither = n - both - only_a - only_b
+        p = mcnemar_exact(only_a, only_b)
+        print(f"{field:<14} both right={both}  only A={only_a}  only B={only_b}  both wrong={neither}  "
+              f"McNemar p={p:.3f}")
+
+
+if __name__ == "__main__":
+    main()
Author	SHA1	Message	Date
claude-botandClaude Opus 5.5	879b53cf8b	fix(gitea-mcp): set command so args don't replace the image CMD ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details docker.gitea.com/gitea-mcp-server has CMD ["/app/gitea-mcp"] and no ENTRYPOINT, so a Pod with only args runs "-t" as the executable: the container failed with 'exec: "-t": executable file not found in $PATH' and crash-looped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:37:11 +02:00
k.renner	146178655d	Merge pull request 'feat(gitea-mcp): in-cluster Gitea MCP server on the NetBird proxy' (#1240 ) from feat/gitea-mcp into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1240	2026-10-11 09:30:29 +00:00
claude-botandClaude Opus 5.5	3d20f4e651	docs(gitea-mcp): spec matches the disk-backed /tmp ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:28:52 +02:00
claude-botandClaude Opus 5.5	b5c9aab10a	fix(gitea-mcp): disk-backed /tmp for the app's rotated log file gitea-mcp always writes a lumberjack log (100MB rotation, 10 backups) and falls back to /tmp on a read-only root. A 16Mi in-memory emptyDir would fill and get the pod evicted; the log also counted against the memory limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:28:44 +02:00
claude-botandClaude Opus 5.5	47449250f8	feat(gitea-mcp): deploy the Gitea MCP server at gitea-mcp.renner.uno Runs gitea-mcp in -oauth mode with no server token: tokenless /mcp requests get 401 instead of reaching Gitea anonymously, and a plain PAT in the Authorization header is passed through to Gitea. Published on the NetBird proxy via netbird.renner.uno/public; LAN DNS stays on the Traefik VIP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:25:19 +02:00
claude-botandClaude Opus 5.5	799d35cd56	feat(gitea-mcp): track docker.gitea.com/gitea-mcp-server via image automation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:24:49 +02:00
claude-botandClaude Opus 5.5	b27b5f2d5b	feat(gitea-mcp): namespace and network policies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:24:35 +02:00
claude-botandClaude Opus 5.5	47a48950e8	docs(gitea-mcp): implementation plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:23:09 +02:00
claude-botandClaude Opus 5.5	274b96b8b3	docs(gitea-mcp): design spec for the in-cluster Gitea MCP server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:20:23 +02:00
k.renner	3d7dd87821	Merge pull request 'feat(netbird-service-sync): pin the build with netbird.renner.uno/public support' (#1239 ) from claude/netbird-service-sync-pin into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1239	2026-10-11 08:41:43 +00:00
claude-botandClaude Opus 5.5	b5b499f2f6	feat(netbird-service-sync): pin the build with netbird.renner.uno/public support ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details Master build of #1238 (pipeline 13250). Activates the proxy-only label, so the gitea Ingress is published on the NetBird proxy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 10:40:20 +02:00
k.renner	1335560c94	Merge pull request 'feat(gitea): publish HTTP on the NetBird proxy, keep LAN DNS local' (#1238 ) from claude/gitea-public into master ci/woodpecker/push/build-ci-images Pipeline was successful Details ci/woodpecker/cron/issue-fix Pipeline was successful Details Reviewed-on: k.renner/homelab#1238	2026-10-11 08:39:38 +00:00
claude-botandClaude Opus 5.5	69a5652c39	feat(netbird-service-sync): proxy-only public label netbird.renner.uno/public ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details dns.renner.uno/public publishes a host on the NetBird proxy AND drops its UDM record (external-dns-unifi labelFilter). For gitea that sends LAN SSH to the proxy host's own sshd and node image pulls over the WAN, since the hairpin covers only 80/443 and not the node VLAN. netbird-service-sync now also selects netbird.renner.uno/public, which external-dns-unifi does not filter on, so the host is published on the proxy while the LAN record on the Traefik VIP stays. The gitea Ingress switches to the new label; it takes effect once the master build of this image is pinned in the deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 10:31:47 +02:00
claude-botandClaude Opus 5.5	3a8a41fc16	feat(gitea): expose the HTTP ingress publicly ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details Label the gitea Ingress dns.renner.uno/public so netbird-service-sync publishes git.renner.uno and gitea.renner.uno on the NetBird proxy. The gitea-ssh IngressRouteTCP stays unlabeled and LAN-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 10:25:32 +02:00
k.renner	358ae278d4	Merge pull request 'feat(embed-eval): leave-one-out kNN benchmark for embedding models' (#1237 ) from feat/embed-knn-bench into master ci/woodpecker/cron/ci-claude-refresh Pipeline was successful Details ci/woodpecker/cron/e2e Pipeline failed Details ci/woodpecker/cron/wiki-maintenance Pipeline was successful Details ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1237	2026-10-10 20:00:49 +00:00
k.renner	123759b709	Merge pull request 'fix(velero): backup-verify review minors' (#1235 ) from fix/backup-verify-minors into master Reviewed-on: k.renner/homelab#1235	2026-10-10 20:00:39 +00:00
claude-botandClaude Sonnet 5.5	cb6f6e803a	feat(embed-eval): paired comparison of two kNN benchmark runs ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:35:54 +02:00
claude-botandClaude Sonnet 5.5	775419c2a4	feat(embed-eval): shorten texts the model reports as over its token limit Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:30:13 +02:00
claude-botandClaude Sonnet 5.5	daf55565ee	feat(embed-eval): leave-one-out kNN benchmark for embedding models Embeds a labelled Paperless corpus and scores nearest-neighbour correspondent and document-type accuracy against ceiling and floor. Standard library only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:23:48 +02:00
k.renner	5cb7c191ba	Merge pull request 'feat(paperless): use Mistral AI as the LLM backend' (#1236 ) from feat/paperless-mistral into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1236	2026-10-10 17:00:37 +00:00
claude-botandClaude Sonnet 5.5	3e0a2173be	fix(paperless): use the documented Mistral Small 4 model id ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 18:55:55 +02:00
claude-botandClaude Sonnet 5.5	c789c448de	feat(paperless): use Mistral AI as the LLM backend ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was canceled Details Point PAPERLESS_AI_LLM_* at api.mistral.ai (mistral-small-latest), key from paperless-secrets minstral-token. Embeddings stay on llama-embed; an explicit embedding key stops the Mistral token falling back to it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 18:53:50 +02:00
security-bot	0e02723daf	chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.10.1 ci/woodpecker/cron/alert-monitor Pipeline was successful Details Updates managed by image-automation-controller.	2026-10-10 16:46:18 +00:00
claude-botandClaude Opus 5.5	68ee347844	fix(velero): backup-verify review minors ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details - vm-verify: `kopia snapshot verify` read the loop's stdin, so anything reading stdin would eat the remaining plan entries; it now reads /dev/null. KOPIA_CHECK_FOR_UPDATES=false. - cloud-init failing on the VM no longer waits out SSH_WAIT_S: ssh exit codes other than 255 mean cloud-init itself failed, so the orchestrator logs `cloud-init status --long` and gives up at once. - A failure staging files on the VM is no longer fatal: the run still prints the per-entry summary (no result from VM) and exits 1. - A server create that times out after the server exists no longer blocks the fallback type with a duplicate name: labelled servers are deleted between attempts. - cloud-config is rendered as JSON (a YAML subset) with jq, so the multi-line host key is escaped by jq and the test can parse it. - Tests: secrets parsing (with/without spaces around =), user-data structure, every server type unavailable, retry after a leftover server, push failure, cloud-init failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 18:33:48 +02:00
k.renner	4282f6945b	Merge pull request 'fix(monitoring): CephHealthWarn — drop stale exclusions and history, actionable description' (#1234 ) from fix/ceph-alert-restore-test-window into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1234	2026-10-10 16:02:26 +00:00
claude-botandClaude Opus 5.5	221b350ea2	fix(monitoring): CephHealthWarn covers AUTH_INSECURE again, actionable description ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details - AUTH_INSECURE_* is no longer excluded. #527 is done: every CephX key is aes256k since 2026-09-23 (#931, #934, #935) and all six AUTH_INSECURE_* checks report 0, so a new one is a real regression. - The description now only says what to do: BLUESTORE_SLOW_OP_ALERT outlives the stall by 24h; find the OSD in `ceph health detail` and check that drive's SMART data. No history in the annotation. - Comment: drop the AUTH_INSECURE paragraph and the restore-test note; the "80-87% endurance used" claim was wrong for the OSDs (nvme2 at 2-3% per smartctl_device_percentage_used, the 83-91% drives are the nvme1 EPHEMERAL disks), so it goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 18:00:21 +02:00
claude-botandClaude Opus 5.5	80e18f60d0	fix(monitoring): drop the restore-test window from CephHealthWarn ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was canceled Details The alert told the reader that BLUESTORE_SLOW_OP_ALERT stalls between 08:00-14:00Z on the 1st are expected because of the monthly Velero restore-test sweep. restore-test was removed in #1233 (replaced by backup-verify, which writes nothing to the OSDs), so a stall in that window is no longer explained by a scheduled job and must not be waved through. Expression, for and labels are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 17:55:57 +02:00
k.renner	d4afb61324	Merge pull request 'fix: [alert] FluxResourceNotReady (cluster) (#1217 )' (#1219 ) from alert-fix/issue-1217 into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#1219	2026-10-10 15:50:22 +00:00
k.renner	7a85fd9390	Merge pull request 'fix(ci): pr-review context clone uses konairius/shelly-manager' (#1231 ) from claude/pr-review-latest-cli into master Reviewed-on: k.renner/homelab#1231	2026-10-10 15:50:05 +00:00
k.renner	83c5c9958b	Merge pull request 'chore(ci): bump ci-claude image to 2.1.296' (#1232 ) from ci-image/claude-2.1.296 into master Reviewed-on: k.renner/homelab#1232	2026-10-10 15:49:43 +00:00
k.renner	a7fee5c2fb	Merge pull request 'feat(velero): enable backup-verify, remove restore-test' (#1233 ) from feat/backup-verify-cutover into master Reviewed-on: k.renner/homelab#1233	2026-10-10 15:49:28 +00:00
claude-botandClaude Opus 5.5	f27f124cfb	feat(velero): enable backup-verify, remove restore-test ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details backup-verify passed its live runs on 2026-10-10: - apps only: PASS=2, 61 s, VM torn down - negative (forged snapshot IDs): FAIL=2 "snapshot not in repository", Job Failed, VM torn down - abort (Job deleted after ssh ready): TERM handled in 3 s, server, firewall and key gone within 16 s; next run swept nothing - full: PASS=39 FAIL=0 SKIP=2, 3688 s, cx23, ~410 GB verified, nothing left behind The CronJob is unsuspended. restore-test (CronJob, RBAC, script, ConfigMap) and its Prometheus :9090 ingress rule in monitoring-ns go; Flux prune removes them and the failed restore-test-29847360 Job with its CronJob, which clears the open KubeJobFailed (velero) alert. RECOVERY.md describes backup-verify instead of the OSD capacity gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 17:48:01 +02:00
claude-bot	77bbf65a55	chore(ci): bump ci-claude image to 2.1.296 ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details	2026-10-10 14:34:22 +00:00
issue-fix-bot	76515019f8	fix: [alert] FluxResourceNotReady (cluster) (#1217 ) ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details	2026-10-09 04:42:43 +00:00