fix: [alert] CephPoolLostRedundancy (rook-ceph) (#968)

Reviewed-on: k.renner/homelab#972

docs/superpowers/specs/2026-09-25-paperclip-design.md

@@ -16,7 +16,7 @@ credentials are out of scope (see "Deferred").
 |---|---|
 | Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
 | Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
-| Claude auth | **One shared subscription OAuth token** (`CLAUDE_CODE_OAUTH_TOKEN`, from `claude setup-token`) for all agents. |
+| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`); Paperclip stores and refreshes it. Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
 | Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
 | Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
 
@@ -25,8 +25,8 @@ credentials are out of scope (see "Deferred").
 Upstream `runChildProcess()` spawns every local adapter with
 `{...sanitizeInheritedPaperclipEnv(process.env), ...opts.env}`; the sanitizer strips only
 `PAPERCLIP_*` keys. Every agent therefore sees `DATABASE_URL`, `BETTER_AUTH_SECRET`,
-`CLAUDE_CODE_OAUTH_TOKEN`, the shared filesystem (including every other agent's
-workspace) and every agent-scoped secret it can read from disk. Under B, no bot may hold
+the shared filesystem (including every other agent's workspace and the
+`ai-local-logins` credentials) and every agent-scoped secret it can read from disk. Under B, no bot may hold
 a credential we would not tolerate any other bot using. Per-bot isolation is C's job.
 
 ## Architecture
@@ -79,6 +79,11 @@ All modeled on `kubernetes/apps/paperless/` unless stated.
     claim (`/bootstrap/claim`) requires an already signed-in Better Auth account.
     A follow-up flips it to `true`.
   - `PAPERCLIP_TELEMETRY_DISABLED=1` (read by `resolveTelemetryConfig`).
+  - `PAPERCLIP_API_URL=http://paperclip.paperclip.svc.cluster.local:3100` plus
+    `PAPERCLIP_ALLOWED_HOSTNAMES=paperclip.paperclip.svc.cluster.local`: upstream
+    otherwise hands agents the public URL, and the gate (`intercept_header_auth`)
+    rejects their bearer tokens. The Service name also works for sandbox pods under
+    option C; agent JWTs still authenticate each call.
   - `envFrom: secretRef paperclip-secrets`
 - Reloader: `secret.reloader.stakater.com/reload: paperclip-secrets`.
 
@@ -106,7 +111,6 @@ extract):
 - `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`
 - `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex). Held in OpenBao, not on the PVC, so
   encrypted agent secrets in Postgres survive PVC loss.
-- `CLAUDE_CODE_OAUTH_TOKEN` (shared subscription token)
 
 ### Ingress and SSO
 - Hostname `paperclip.renner.uno`, Traefik `Ingress`, `cert-manager.io/cluster-issuer:
@@ -152,10 +156,15 @@ available since v0.35.0). First use of digest reflection in this repo.
 - Nothing is granted under B that we would not tolerate any agent misusing.
 
 ## Bootstrap (one-time)
-1. Place `apps/paperclip` values in OpenBao (generated secrets + shared OAuth token).
-2. Merge; Flux deploys.
-3. Claim the admin account via the `bootstrap-ceo` invite, add yourself to
-   "Paperclip Users" in Authentik.
+1. Place `apps/paperclip` values in OpenBao (the three generated secrets).
+2. Merge; Flux deploys with sign-up enabled.
+3. Sign up through the gate and claim the instance as admin; then set
+   `PAPERCLIP_AUTH_DISABLE_SIGN_UP=true`.
+4. In onboarding "Connect a model", choose Claude subscription and run the shown login
+   **as the node user** (the entrypoint makes exec sessions root, and root-owned 0600
+   credentials are unreadable to the server):
+   `kubectl exec -it -n paperclip deploy/paperclip -- gosu node sh -c 'export CLAUDE_CONFIG_DIR=<shown path> && mkdir -p "$CLAUDE_CONFIG_DIR" && claude auth login'`
+   Repeat after loss of the PVC or database.
 
 ## Verification
 - Flux Kustomization `paperclip` Ready; CNPG `pclp-pg` healthy; first ScheduledBackup
@@ -163,7 +172,7 @@ available since v0.35.0). First use of digest reflection in this repo.
 - `curl https://paperclip.renner.uno/api/health` through the gate returns healthy after
   Authentik login; unauthenticated request redirects to `auth.renner.uno`.
 - Direct pod access from another namespace is denied (CNP).
-- A test agent (claude-local) completes a trivial task using the shared OAuth token.
+- A test agent (claude-local) completes a trivial task using the shared Claude connection.
 - Flux ImagePolicy reports a `nightly@sha256:` digest.
 
 ## Deferred

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -52,19 +52,23 @@ spec:
         - alert: CephPoolLostRedundancy
           # Names the pool, which POOL_NO_REDUNDANCY does not.
           #
-          # There is no pool-size metric, but bytes_used/stored IS the
-          # replication factor. The >1e6 floor drops pools too small to measure
-          # meaningfully, empty ones included.
+          # There is no pool-size metric, but bytes_used/stored is the
+          # replication factor for replicated pools and (k+m)/k for EC pools.
+          # The EC 2+1 pools (cephfs-bulk, rgw buckets.data) sit at exactly
+          # 1.5 with ~1e-7 accounting drift either side, so the cut-off is
+          # midway between them and an unreplicated pool (1.0) (#968).
+          # The >1e6 floor drops pools too small to measure meaningfully,
+          # empty ones included.
           #
           # No pool is exempt.
           expr: |
-            ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
+            ceph_pool_bytes_used / ceph_pool_stored < 1.25 and ceph_pool_stored > 1e6
           for: 30m
           labels:
             severity: critical
           annotations:
             summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
-            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
+            description: "Pool {{ $labels.pool_id }} has a raw/stored ratio below 1.25 (replicated size 1), meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
 
         - alert: CephHealthError
           # HEALTH_ERR means data availability or redundancy is at risk.

kubernetes/apps/paperclip/paperclip.yaml

@@ -49,9 +49,13 @@ spec:
               value: private
             - name: PAPERCLIP_PUBLIC_URL
               value: https://paperclip.renner.uno
-            # Sign-up stays open until the first admin is claimed; the gate restricts who reaches it.
+            # Agents reach the API via the Service, not through the Authentik gate.
+            - name: PAPERCLIP_API_URL
+              value: http://paperclip.paperclip.svc.cluster.local:3100
+            - name: PAPERCLIP_ALLOWED_HOSTNAMES
+              value: paperclip.paperclip.svc.cluster.local
             - name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
-              value: "false"
+              value: "true"
             - name: PAPERCLIP_TELEMETRY_DISABLED
               value: "1"
           envFrom:

kubernetes/apps/wiki/externalsecrets.yaml

@@ -42,6 +42,10 @@ spec:
       remoteRef:
         key: apps/ai/wiki-api-keys
         property: workstation
+    - secretKey: Auth__ApiKeys__paperclip
+      remoteRef:
+        key: apps/ai/wiki-api-keys
+        property: paperclip
 ---
 # TypeSafe System One key for the judge adapter (Wiki:Judges:Models:typesafe).
 apiVersion: external-secrets.io/v1