feat(paperclip): read-only Prometheus path via apiserver proxy (REN-36)
Owner picked option B on REN-37: reach Prometheus through the apiserver
service proxy under an RBAC Role holding one verb, instead of an L4 Cilium
allow on :9090.
The port cannot be the boundary. The chart leaves `reloadStrategy` unset, so
the operator defaults it to HTTP and appends `--web.enable-lifecycle`, which
puts `POST /-/quit` and `POST /-/reload` on :9090 with no auth in front of
them. An L4 allow cannot distinguish those from `GET /api/v1/query`. The
apiserver can: it maps HTTP method to RBAC verb, so a Role with only `get`
refuses both in the authorizer before the request reaches Prometheus.
ServiceAccount paperclip/paperclip-prometheus-read
Role monitoring/paperclip-prometheus-read
get on services/proxy, resourceNames:
["kube-prometheus-stack-prometheus:9090"]
RoleBinding monitoring/paperclip-prometheus-read -> the SA above
paperclip.yaml serviceAccountName + automountServiceAccountToken: true
Shape copied from monitoring/ci-diagnostics-rbac.yaml, which has run this
same proxy path for the alert-monitor since 2026-07-16. No network-policy
edit: paperclip-egress already allows kube-apiserver on 6443.
BLAST RADIUS: applying this restarts the paperclip pod, and every agent with
it. Nothing in production monitoring restarts — Prometheus, Alertmanager,
Loki and the ingress controllers are untouched.
REVERT: revert this commit. One commit, four files, no state left behind;
Flux prunes the SA, Role and RoleBinding and rolls the pod back to no
mounted token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3a457bd63c
commit
35bcdf1043
@@ -64,7 +64,12 @@ All modeled on `kubernetes/apps/paperless/` unless stated.
|
||||
- Starts as root: upstream `docker-entrypoint.sh` remaps the `node` user, chowns
|
||||
`/paperclip`, then drops to uid 1000 via `gosu`. Caps limited to CHOWN,
|
||||
DAC_OVERRIDE, FOWNER, SETGID, SETUID; `fsGroup: 1000`.
|
||||
- `automountServiceAccountToken: false` (all agents share the pod filesystem).
|
||||
- `serviceAccountName: paperclip-prometheus-read` with
|
||||
`automountServiceAccountToken: true`. All agents share the pod filesystem, so all
|
||||
of them can read that token — it is mounted anyway because the Role behind it
|
||||
grants exactly `get` on `services/proxy` for
|
||||
`kube-prometheus-stack-prometheus:9090` in `monitoring` and nothing else
|
||||
(`prometheus-read-rbac.yaml`, REN-36). It was `false` until then.
|
||||
- Port 3100; `Service paperclip` ClusterIP :3100.
|
||||
- Probes: startup and readiness on `GET /api/health` with `Host: localhost` (the
|
||||
private-hostname guard rejects the pod IP); liveness is a TCP check, since
|
||||
|
||||
@@ -7,6 +7,7 @@ resources:
|
||||
- objectstore.yaml
|
||||
- scheduled-backup.yaml
|
||||
- pvc.yaml
|
||||
- prometheus-read-rbac.yaml
|
||||
- paperclip.yaml
|
||||
- gate-service.yaml
|
||||
- ingress.yaml
|
||||
|
||||
@@ -20,8 +20,13 @@ spec:
|
||||
app.kubernetes.io/name: paperclip
|
||||
spec:
|
||||
enableServiceLinks: false
|
||||
# Every agent shares this filesystem; bots bring their own cluster credentials.
|
||||
automountServiceAccountToken: false
|
||||
# Every agent shares this filesystem, so every agent can read this token.
|
||||
# It is mounted anyway because it is worth exactly one thing: `get` on the
|
||||
# Prometheus Service proxy, for monitoring reads (REN-36). See
|
||||
# prometheus-read-rbac.yaml. Bots still bring their own Gitea and cluster
|
||||
# credentials; this is not a general-purpose cluster credential.
|
||||
serviceAccountName: paperclip-prometheus-read
|
||||
automountServiceAccountToken: true
|
||||
securityContext:
|
||||
fsGroup: 1000
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
# A read-only Prometheus path for the agents in the paperclip pod (REN-36).
|
||||
#
|
||||
# WHY THE APISERVER PROXY AND NOT A NETWORK RULE
|
||||
#
|
||||
# Prometheus runs with `--web.enable-lifecycle` (the chart leaves
|
||||
# `reloadStrategy` unset, so the operator defaults it to HTTP and appends the
|
||||
# flag), and :9090 carries no auth of its own. That puts `POST /-/quit` and
|
||||
# `POST /-/reload` on the same TCP port as `GET /api/v1/query`. An L4 allow on
|
||||
# 9090 cannot tell them apart, so it would hand this pod a kill switch for
|
||||
# cluster monitoring.
|
||||
#
|
||||
# The apiserver can tell them apart: it maps the HTTP method of a service-proxy
|
||||
# request onto an RBAC verb — GET->get, POST->create, PUT->update. A Role
|
||||
# holding only `get` therefore refuses `/-/quit` and `/-/reload` in the
|
||||
# authorizer, before the request is ever forwarded to Prometheus. Read-only by
|
||||
# construction, and `--web.enable-lifecycle` stops mattering.
|
||||
#
|
||||
# Shape copied from `../monitoring/ci-diagnostics-rbac.yaml` (the
|
||||
# `ci-diagnostics-service-proxy` Role), which has run this same path for the
|
||||
# alert-monitor since 2026-07-16.
|
||||
#
|
||||
# ONE VERB, ONE RESOURCE NAME — ON PURPOSE
|
||||
#
|
||||
# Every process in the paperclip pod can read the mounted ServiceAccount token,
|
||||
# so the token is only worth what this Role grants. It grants `get` on one
|
||||
# Service. Widening it is a new card and a new review, not an edit here.
|
||||
#
|
||||
# `resourceNames` is matched verbatim by the authorizer, not as a selector:
|
||||
# `svc`, `svc:9090` and `http:svc:9090` are three different strings. Only the
|
||||
# port-qualified form is granted, so the one URL that works is
|
||||
# /api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/...
|
||||
# A 403 on some other form is the grant working. Do not fix it by adding a
|
||||
# string.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: paperclip-prometheus-read
|
||||
namespace: paperclip
|
||||
labels:
|
||||
app.kubernetes.io/name: paperclip
|
||||
# Named for the only thing it is worth. If a future change gives this account a
|
||||
# second grant, the name is a lie and the review should say so. Whether the
|
||||
# token is mounted is decided on the pod (`paperclip.yaml`), not here.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: paperclip-prometheus-read
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: paperclip
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services/proxy"]
|
||||
verbs: ["get"]
|
||||
resourceNames:
|
||||
- "kube-prometheus-stack-prometheus:9090"
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: paperclip-prometheus-read
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: paperclip
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: paperclip-prometheus-read
|
||||
# The Role and this binding live in `monitoring`; the subject lives in
|
||||
# `paperclip`, so `namespace:` here is load-bearing. Omit it and the binding
|
||||
# silently resolves to a nonexistent `monitoring/paperclip-prometheus-read`,
|
||||
# which fails closed with a 403 indistinguishable from a wrong resourceName.
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: paperclip-prometheus-read
|
||||
namespace: paperclip
|
||||
Reference in New Issue
Block a user