Reviewed-by: argus-bot <svc-argus@renner.uno>
Comments-only. Citation re-verified at head a8c48b54: line 250 of
kubernetes/apps/monitoring-rules/probes-blackbox.yaml is the quoted probe URL,
inside the blackbox-http Probe in namespace monitoring. Both versions of
paperless-ns.yaml parse to identical JSON across all three CiliumNetworkPolicy
documents, so re-applying is a no-op against the running policy.
Revert: git revert -m 1 <this commit>
Reviewed and merged by Argus (REN-105), review 1038 at c40c337e.
Delete-path bound re-derived from v2.2.13: GetByName exact FriendlyName equality against the default template {{.Name}}-{{.Namespace}}; GetAllByName and uptimerobot GetAll have no reachable callers. Ships zero EndpointMonitors, so the first reconcile makes no UptimeRobot call and the 9 existing monitors are untouched.
CI green at c40c337e (ci-manifests-test, ci-scripts-test, pr-review). Nothing applied to a cluster pre-merge; verification was lint/render only.
Open follow-ups for Atlas on REN-6, both comment-only: rbac.yaml misstates the failure mode for the omitted pods/services grant (a denial blocks the cached client in cache sync and wedges the single reconcile worker while the pod stays Ready, rather than costing a log line), and the CRD vendor header cites a post-tag master commit instead of the v2.2.13 tag.
Revert: git revert -m 1 of this merge commit.
`paperless-ns.yaml` cited the blackbox probe target as
`monitoring-rules/probes-blackbox.yaml:250`. From
`kubernetes/infrastructure/network-policies/`, that fragment reads as a
sibling `infrastructure/monitoring-rules/` directory, which does not
exist — `monitoring-rules/` lives only under `kubernetes/apps/`.
Use the full repo-root path, matching the style already used six lines
above for `kubernetes/apps/paperless/network-policy.yaml`. Line 250 and
the probe target itself were correct and are re-verified at this commit:
line 250 is
`- http://paperless.paperless.svc.cluster.local:80/accounts/login/`.
Comments only. Parsing the file before and after with js-yaml yields
identical structures for all three CiliumNetworkPolicy documents, and
stripping comment lines from both versions gives an identical diff. No
`spec` change, no new resource, nothing applied to the cluster.
Follow-up from the REN-110 review of PR #997 (merged as 0995e8d5).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous wording claimed home-assistant-ns.yaml was a second
port-restricted `monitoring` grant and that 17 others were namespace-wide.
Both numbers were wrong. Counted per ingress list item over
kubernetes/infrastructure/network-policies at master:
20 `monitoring` grants across 19 files
(influxdb-ns.yaml carries two — one whole-namespace, one Grafana-pod)
exactly one of them — this rule — carries a `toPorts`
home-assistant-ns.yaml looks port-scoped only because the `toPorts` that
follows its `monitoring` rule is indented under the next list item, the
`evcc` rule. Its `monitoring` grant is namespace-wide.
Comment-only. The parsed CiliumNetworkPolicy is byte-for-byte identical in
structure to master's.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docs-only. One file, docs/superpowers/specs/2026-09-25-paperclip-design.md, +20/-2.
Closes the four documentation items on the REN-47 review bar (1/14, 5, 7, 15) for the
option-B apiserver-proxy grant, and drops the stale Prometheus-only resourceName text
that REN-95 left behind.
BLAST RADIUS: none. Every Flux Kustomization spec.path in this repo is under
./kubernetes/; docs/ is rendered by nothing. Flux applies nothing from this merge and
no workload restarts. No owner-facing surface changes.
REVERT: git revert %(SHA)s -m 1
One commit, one file, no cluster state to unwind. Nothing to re-apply, no token to
rotate, no pod to restart.
Reviewed by argus-bot (review 1031) against manifests, not against the PR body:
Role is one rule, verbs [get], resources [services/proxy], both resourceNames match
prometheus-read-rbac.yaml on master; chart float >=72.0.0 confirmed in
helmrelease-prometheus.yaml. Authored by atlas-bot, merged by argus-bot.
Implements the owner's decision recorded on REN-52 (2026-09-25, option B),
which was never shipped: one ClusterRoleBinding, roleRef the built-in `view`
ClusterRole, subject the ServiceAccount REN-36 created. No new
ServiceAccount, no new ClusterRole, no write verb, no Secrets.
Kept in its own file rather than appended to prometheus-read-rbac.yaml so it
reverts alone, and because that file's header argues for staying narrow —
which this grant is not. Both of that file's now-false claims are corrected
in place: the token is no longer worth only what its Role grants, and the
`paperclip-prometheus-read` name no longer describes it (REN-52 ruled the
name stays, since `serviceAccountName` is in the pod template and renaming it
would restart the Deployment and evict running agents).
No extra Flux CRD ClusterRole: `flux-view-flux-system` already carries
`rbac.authorization.k8s.io/aggregate-to-view: "true"` with get/list/watch on
`*` across all six Flux API groups, so `view` picks them up by aggregation.
REN-52 pre-approved adding one only if that were absent.
Additive RBAC on an existing ServiceAccount: no Pod restart, no remount, no
traffic path changed. RBAC is evaluated per request, so it takes effect on
the already-mounted token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stakater IngressMonitorController in its own namespace, with zero
EndpointMonitor resources, so it creates nothing on arrival.
- Vendored EndpointMonitor CRD from chart 2.2.13, image pinned to v2.2.13
and deliberately kept out of Flux image automation.
- Namespaced Role only (no ClusterRole): WATCH_NAMESPACE scopes the
manager's cache to uptime-monitor. Verbs derived from what v2.2.13
actually calls; every omission is justified inline in rbac.yaml.
- The ExternalSecret now renders IMC's config.yaml, because IMC reads its
whole configuration from one Secret key rather than an env var. Config
stays in git; only the key is injected.
- Per-namespace CiliumNetworkPolicy, copied from the netbird-service-sync
neighbour: kubelet probes in, DNS + kube-apiserver + world:443 out.
- The Flux Kustomization keeps the healthCheckExprs gate and the
`wait: true` correction that #993 already merged, byte for byte. What
this commit adds to that file is only what is about REN-6: the
EndpointMonitor-vs-annotation mechanism in the header, and why
Deployment health is not a proxy for a current key now that the
controller shares the Kustomization with the ExternalSecret.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Argus's review bar for the option-B proxy grant (REN-47) asks for three notes on
the hand-over page. None of them was ever written down, and the page had also
gone stale: it still named only the Prometheus resourceName after REN-95 added
Alertmanager.
Added, next to `automountServiceAccountToken: true` where the grant is already
described:
- Both granted resourceNames, not just Prometheus (bar item 1/14).
- The Role scopes the HTTP *method*, not the endpoint — `get` refuses
`POST /-/reload` in the authorizer, but every GET endpoint of both services is
reachable, including `/api/v1/status/config` (bar item 5). Verified live this
run: the six `credentials` fields in that config render as `<secret>`.
- Callers must force GET; Prometheus accepts POST on `/api/v1/query` and that
form 403s under this Role (bar item 7).
- The drift defence: a 403 means check the Service name against `resourceNames`
first, because the chart floats at `>=72.0.0` and a rename breaks this path
silently (bar item 15).
BLAST RADIUS: none. One markdown file under docs/, not rendered by any
Kustomization, so Flux applies nothing and no workload restarts.
REVERT: revert this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a PodMonitor for the six Flux controllers in flux-system. Scrape only —
the not-Ready alert rules that were in the first revision of this branch are
removed, because they cannot work at the versions we run.
Why the rules are gone rather than merely deferred: `gotk_reconcile_condition`
and `gotk_suspend_status` are still declared by fluxcd/pkg/runtime v0.110.3
(runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml
(source/kustomize v1.9.5, helm v1.6.4, notification v1.9.4, image-reflector and
image-automation v1.2.5) no controller calls RecordReadiness, RecordCondition
or RecordSuspend — zero call sites across all six tagged source trees, only
RecordDuration. An unpopulated GaugeVec exports no series, so those rules would
have evaluated against nothing for ever and the `absent()` dead-man's-switch
would have mailed admin@renner.uno every 12h from 30m after merge. Flux moved
per-object readiness to kube-state-metrics custom-resource-state
(`gotk_resource_info{ready=...}`), which is REN-102 — a prerequisite for the
alert, not a follow-up to it. Caught in review by argus-bot on #992.
What the scrape still buys on its own: `gotk_reconcile_duration_seconds` per
kind/name/namespace, controller-runtime's reconcile/error/workqueue series,
`up{job="monitoring/flux-controllers"}` under the chart's default TargetDown
rule, and the target REN-102 needs in place anyway.
PodMonitor and not ServiceMonitor is forced: four of the six controllers have
no Service, and the two that do publish the artifact/webhook port, not :8080.
No network policy change is needed and that was checked, not assumed —
gotk-components.yaml ships `allow-scraping` admitting every namespace to :8080
and flux-system has no CiliumNetworkPolicy. Nothing in the Flux install is
touched; Prometheus adds a read-only target.
Verified offline only: `kustomize build kubernetes/apps/monitoring-rules` clean
and the PodMonitor renders; all six Deployments carry both selector labels and
the `http-prom` container port. Not applied to the cluster — no kubeconfig in
this account's grants.
Refs REN-97, REN-102.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The healthCheckExprs block merged in #993 treated any `Ready=False` on the
ExternalSecret as Failed. The `flux-failures` Alert forwards every Kustomization
error event to Alertmanager and Flux health checks have no `for:` window, so a
single unlucky poll mailed the owner about something that was about to fix
itself.
REN-106 asked for `reason == 'SecretSyncedError'`, matching wordpress.yaml.
Checked against the ESO the cluster runs (v2.11.0) that is the wrong reason:
`SecretSyncedError` is the reason ESO sets when it returns an error and keeps
retrying — an OpenBao blip, a token renewal and a not-yet-populated key all
produce it. `SecretOwnedByOther` and `SecretImmutable` are the two reasons ESO
returns nil on, i.e. where it has stopped retrying.
So `failed` now names those two, and `SecretSyncedError` falls through to
in-progress and fails the Kustomization at the existing `timeout: 5m`, which
becomes the missing `for:` window. Detection of a genuinely stuck sync moves
from immediate to ~5m; a transient blip no longer reaches the owner at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. No manifest field changes, so `kustomize build` output is
byte-identical before and after.
Two gaps in what REN-103's merged scrape says about itself, both raised in
review of PR #994/#996:
* The metrics endpoint is plain HTTP with no authentication. That is chart
2.11.0's default (`metrics.listen.secure.enabled` and
`metrics.listen.auth.enabled` both false, values.yaml:365-369) and this
repo overrides neither, so what actually bounds the exposure is the
CiliumNetworkPolicy, not the endpoint. A reader adding `tlsConfig` to the
PodMonitor would have broken the scrape; say so in the file.
* The network policy comment implied the grant reaches the controller. It
does not: the CNP's `endpointSelector` is {}, so monitoring→:8080 is
admitted for all three ESO deployments, which all serve metrics on that
port. Only the controller is scraped, because the PodMonitor's selector is
exact on the unsuffixed name. Reachable-but-unvisited is the accurate
description and the one a future reader needs.
Also corrected two counts in the same comment. The port-scoping is not unique
in this directory — home-assistant-ns.yaml does it too, so 17 of the other 18
`monitoring` grants are namespace-wide, not all 18. And there is no single
"health port": the controller's is 8082, the webhook's and cert-controller's
are 8081 (values.yaml:501-504, 564-570, 882-885).
Revert: `git revert` this commit. It restores the previous comment text and
touches no applied field.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comments only. No `spec` change, no pod template change, no rendered
manifest change.
Three comments told the reader the ingress side of paperless and navidrome
is locked to the Authentik gate outposts. It is not. `default-deny-ingress`
for both namespaces uses `endpointSelector: {}`, so it selects the app pods
too, and Cilium unions the allow rules of every CNP selecting an endpoint.
The effective ingress set is therefore the outpost PLUS whole namespaces:
`database` and `monitoring` for paperless, and `monitoring` plus the
lidarr-gate outpost for navidrome. Because both apps trust
X-Authentik-Username unconditionally, any pod in those namespaces can claim
any username.
- kubernetes/apps/paperless/network-policy.yaml: replace "Lock the ingress
side to the gate outpost only" with what is actually enforced, citing the
rules in paperless-ns.yaml.
- kubernetes/infrastructure/network-policies/paperless-ns.yaml: note that
the database and monitoring allows below do broaden the trust boundary in
exactly the way the traefik note above warns about, and why they are kept
(CNPG, and the blackbox probe of the paperless login page).
- kubernetes/apps/navidrome/deployment.yaml: ND_EXTAUTH_TRUSTEDSOURCES is
the whole pod CIDR and is not a narrowing; record why it stays that way
(the outpost pod IP is not stable) and list what the CNP really admits.
Verification: `kubectl kustomize` (v1.31.0 / kustomize v5.4.2) over all 68
kustomization roots in the repo renders byte-identical before and after;
every added and removed line in the diff is a comment line.
Revert: `git revert <this commit>`; restores master 3d6c9940 exactly.
Refs REN-108, REN-28.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Make the lidarr and tika ImagePolicy sort keys a total order, and undo the
automated ls42 -> ls11 downgrade that the defective lidarr key caused.
Reviewed and merged by argus-bot; authored by atlas-bot (four-eyes).
Pod roll of deploy/lidarr in ns navidrome signed off by Zeus on REN-91.
Revert: policy only `git revert e50ccaec`; pin only `git revert 430475a7`;
everything `git revert 430475a7 e50ccaec` (restores master ff4ddaa6 exactly).
Refs REN-91.
Nothing in this cluster reported a failed ExternalSecret sync:
`externalsecret_status_condition` held 0 series, ESO's metrics port was never
scraped, and monitoring-rules had no ESO alert. A key that silently stops
syncing was invisible to Prometheus and to Alertmanager.
This is the scrape only. The alert rule is deliberately left out: its `for:`
window has to be tuned against real data, and that data does not exist until
this has been live for a while. Shipping a guessed window to the owner's
channel is the failure mode REN-100 warned about.
A PodMonitor rather than the chart's `serviceMonitor.enabled`, for two reasons
checked against chart 2.11.0:
* There is no metrics Service to select. `metrics.service.enabled` defaults
to false and the HelmRelease overrides nothing, so templates/service.yaml
renders nothing for the controller. A hand-written ServiceMonitor would
match no Service.
* `serviceMonitor.enabled: true` would work but triggers a Helm upgrade of
ESO, re-rendering its Deployment and re-applying its CRDs
(`crds: CreateReplace`). ESO is what all 54 ExternalSecret manifests in
this repo depend on. A PodMonitor gets the same series and touches the
release not at all.
Unlike the Flux PodMonitor, this needs a network policy change: the cluster is
default-deny and external-secrets-ns.yaml admitted only intra-namespace
traffic, host/remote-node and ICMP. Without the `monitoring` grant the scrape
is dropped and the metric stays absent, so the PodMonitor alone would have been
decoration. The grant is port-restricted to 8080, tighter than the equivalent
rule in velero-ns.yaml, and is additive — it removes nothing, so ESO's existing
reachability to OpenBao and the apiserver is unchanged.
Revert: `git revert <sha>`. Removes the PodMonitor and the ingress grant
together, returning to the current no-ESO-metrics state. No ESO object, no
ExternalSecret and no Secret is touched in either direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment in this file claimed `wait: true` was how a key that failed to
sync became visible. It is not. Flux ships no status reader for
`ExternalSecret`, so kstatus falls back to its generic rule, which inspects
only `status.observedGeneration` and conditions of type `Reconciling` /
`Stalled`. ESO sets none of those — a failed sync is `Ready=False,
reason=SecretSyncedError` — so kstatus returns Current and the Kustomization
goes Ready with no Secret behind it.
Corroborated in-repo rather than from docs: `llama-server` is also `wait: true`
and carries `ai/wiki-embed-basic-auth`, which sat in ESO retry backoff through
~10 failures on 2026-09-20 and needed a manual `force-sync` while its
Kustomization stayed Ready.
Adds a `healthCheckExprs` entry for `external-secrets.io/v1 ExternalSecret`,
the same mechanism already used here for CNPG `Cluster`, MariaDB and the Ceph
CRs, and corrects the comment.
Both expressions are guarded with `has(status.conditions)`, copying
wordpress.yaml. Unguarded, CEL errors rather than evaluating while `status` is
still empty — the state a freshly applied ExternalSecret is in, which is
precisely the case this check exists for.
Blast radius is this Kustomization alone: nothing `dependsOn` uptime-monitor,
and its only object today is the ExternalSecret itself. No existing workload,
ingress controller, DNS record or other Kustomization is touched.
Revert: `git revert <sha>`, or drop the `healthCheckExprs` block. Reverting
restores the previous behaviour (Ready regardless of sync state) and cannot
affect any other Kustomization.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Appends kube-prometheus-stack-alertmanager:9093 to the resourceNames of
Role paperclip-prometheus-read in namespace monitoring. Additive, get-only,
no pod template change.
Reviewed and merged by argus-bot; authored by atlas-bot (four-eyes).
Negative baseline reproduced live via SelfSubjectAccessReview before merge.
Makes the UptimeRobot API key available in the new `uptime-monitor` namespace via ESO. No controller, no monitors, no workload — the key only. First step of REN-2, closes REN-5.
Reviewed by argus-bot against a fresh clone at 25cabaed, not against the PR body. Verified: remoteRef key is mount-relative (store sets provider.vault.path: homelab; `grep -rn "key: homelab/"` = 0 hits); eso-read.hcl already grants homelab/data/*; allowlist is 20 entries, sorted, one occurrence; uptime-monitor is absent from apps/kustomization.yaml and clusters/homelab has no kustomization.yaml, so no double-application; both dependsOn targets exist in flux-system; no duplicate namespace name across the 31 declared; no key material in the diff.
Blast radius is one leaf Kustomization that nothing dependsOn -- deliberately NOT added to infrastructure/post-crds, where an unpopulated OpenBao path would stall the 19 Kustomizations that dependsOn infrastructure-crds.
Merge commit on purpose so the documented revert stays valid: git revert 25cabaed. prune: true then removes the Kustomization, ExternalSecret, Secret and namespace.
Non-blocking finding handed to REN-6: the comment claiming `wait: true` makes a failed ExternalSecret sync visible is wrong -- kstatus reports an ESO ExternalSecret Current regardless of Ready=False, which is why the `ai` ExternalSecret under the wait:true llama-server KS could sit in backoff through ~10 failures on 2026-09-20. Fix is a healthCheckExprs block, as already used for CNPG and Ceph CRs here.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reverts the image change in b858b4aa. ls11 is 3.1.0.4875 as built on
2025-11-19; ls42 is the same upstream build rebuilt on 2026-09-23. The
31 rebuilds in between are LinuxServer security rebuilds of the base
image, so the cluster is currently running ten months of known-missing
base-image patches.
Kept as its own commit so it can be reverted independently of the policy
change, and done here rather than left to the controller: with the fixed
policy merged, image-automation would compute ls42 and push the same two
lines to master itself within ~30 min. Doing it in the PR makes the pod
roll a reviewed change that lands at a known time instead of an
unattended bot commit -- which is the failure mode this card is about.
Blast radius: one rolling restart of deploy/lidarr in ns navidrome, same
shape as the 60s outage b858b4aa caused. Lidarr is a read-only
missing-album view; nothing else consumes it. No other workload moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lidarr extracted only <build> from <maj>.<min>.<patch>.<build>-ls<n>, so
all 32 published ls rebuilds of build 4875 shared one sort key and the
winner was an arbitrary tie-break. It fired: b858b4aa auto-committed
ls42 -> ls11 to master and rolled the production pod.
numerical cannot order two fields and alphabetical over a concatenation
mis-sorts on width (ls9 vs ls11), so the key becomes a synthetic semver
<build>.<ls>.0 under a semver policy -- width-safe, and equivalent to
(build, ls) lexicographic. The Lidarr major moves into filterTags
because the range now operates on the synthetic key; a 4.x bump needs a
deliberate edit, per the tight-range rule in README.md.
tika has the same defect: extracting <maj>.<min>.<patch> from a
4-component tag discards the image revision. The README called this
theoretical; it is not -- 2.5.0.1/2.5.0.2, 2.6.0.0/2.6.0.1,
2.7.0.0/2.7.0.1 and 2.9.2.0/2.9.2.1 all collide. Every 3.x release so
far is .0, so the revision is pinned in the filter the way
protonmail-bridge.yaml pins its -1 suffix. No-op today; if upstream
ships a 3.x .1 the policy stops bumping instead of picking wrongly.
Verified against the real published tag lists rather than by inspection:
over 1000 lidarr tags the new filter matches 34, produces 34 distinct
keys, has zero publish-order inversions across the visible history, and
selects 3.1.0.4875-ls42. tika reduces to one tag per key and still
selects 3.3.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Appends one `resourceNames` entry to the existing `paperclip-prometheus-read`
Role so the read-only apiserver service-proxy path also covers Alertmanager:
kube-prometheus-stack-alertmanager:9093
This is the widening that the file's own header comment said required a new
card and a new review — REN-94/REN-95 are that card, and Hermes confirmed the
target against the live cluster before this was written.
Nothing else about the grant changes. Same single rule, same `verbs: ["get"]`,
same `resources: ["services/proxy"]`, same ServiceAccount, same RoleBinding.
No new rule, no cluster-scoped kind, no second verb. `get`-only still means the
authorizer refuses POST/PUT to either target, so Alertmanager's silence and
`/-/reload` write paths stay denied for the same structural reason Prometheus's
`/-/quit` does.
No NetworkPolicy edit: `network-policy.yaml` already allows egress to entity
`kube-apiserver` on 6443, and the Alertmanager leg of the proxy hop originates
at the apiserver, not at the paperclip pod.
Unlike REN-36 this touches a Role only, not the pod template, so Flux applies
it without restarting the paperclip Deployment and without evicting any agent.
Comment updates are included because the header asserted "ONE VERB, ONE
RESOURCE NAME" and named only the Prometheus proxy URL; both would have become
false next to a second name.
Revert: revert this one commit. The grant returns to Prometheus-only and no
workload restarts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Makes the key at OpenBao `infrastructure/uptime-robot`, field `api_key`,
available as a Secret in the new `uptime-monitor` namespace and nowhere
else. No new operator: this uses the External Secrets Operator and the
`openbao` ClusterSecretStore the cluster already runs.
Nothing is monitored by this commit. The namespace holds only the key;
the IngressMonitorController, its ServiceAccount and its RBAC arrive
separately (REN-6).
* `data` + `property` instead of `dataFrom: extract`, so only the one
field lands in the cluster rather than every field at the KV path.
* Adds `uptime-monitor` to the ClusterSecretStore `conditions.namespaces`
allowlist, sorted. Without this ESO reports "not allowed to reference
ClusterSecretStore" and the Secret silently never syncs.
* A DEDICATED Flux Kustomization rather than adding the ExternalSecret to
infrastructure/post-crds. That path runs `wait: true` with no explicit
healthChecks and 19 other Kustomizations dependOn it, so an
ExternalSecret whose OpenBao path is not yet populated would never turn
Ready and would stall most of the cluster. Here the blast radius of a
missing key is this Kustomization alone, which nothing depends on.
* No OpenBao policy change: eso-read already grants `homelab/data/*`.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stops a random critical false page on the storage layer. The rule inferred a replication factor from bytes_used/stored, which for an ec:2+1 pool is coding overhead of exactly 1.5 -- the rule's own threshold -- so pool 15 cephfs-bulk paged for 199 minutes on 2026-09-25 with the cluster HEALTH_OK.
Reviewed by argus-bot against live Prometheus using the expression parsed from e3c5c4c0. Fire-window replay: old expr 231 samples on pool 15, new expr 0. Positive control at threshold <100: old selects pools {1,3,5,6,13,15}, new selects {1,3,5,6,13} -- exactly one pool removed and it is the EC one. pool_id survives the join so the annotations still render.
Merge commit on purpose so the documented revert stays valid: git revert e3c5c4c0.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
CephPoolLostRedundancy infers a replication factor from
ceph_pool_bytes_used/ceph_pool_stored and pages critical below 1.5. For an
erasure-coded pool that ratio is not replication, it is coding overhead:
(k+m)/k. Both EC pools in this cluster are ec:2+1, so their correct,
healthy value is exactly 3/2 = 1.5 — the rule's own threshold.
cephfs-bulk (pool 15) reads 480042086400/320028049408 = 1.5000000384. The
margin is 12 KiB in 480 GB, so the comparison is decided by the last bits
of a float division and lands on either side at random. It paged critical
for 199 minutes on 2026-09-25 (12:37-15:56 UTC, two emails) while the
cluster was HEALTH_OK with zero degraded PGs and every OSD up, then cleared
untouched. migration.rgw.buckets.data (pool 12) is the same ec:2+1 shape.
The rule's premise is also wrong for EC: its description claimed a single
OSD loss destroys data, but 2+1 carries a parity shard and tolerates one
OSD loss, the same as replica-2. Summary and description now say
"replicated" and drop that claim.
Gated on the exporter's own `type` label rather than moving the threshold.
Nudging 1.5 to 1.49 would keep a knife-edge and would also narrow the real
replica-2 -> replica-1 signal (1.0) this rule exists to catch.
NO COVERAGE GAP: a genuine EC redundancy loss raises POOL_NO_REDUNDANCY,
which is a live series in ceph_health_detail and is already deliberately in
scope for CephHealthWarn. That reads Ceph's own verdict instead of
inferring one from bytes, which is the right instrument — an EC pool that
loses a shard does not move its stored/used ratio at all.
VERIFIED against live Prometheus (read-only apiserver proxy, REN-36), using
the expression as serialized from this file:
* 7d replay, old expr: pool 15 matches 49 samples (defect reproduced)
* 7d replay, new expr: 0 matches
* fire-window replay (12:00-16:30 UTC): old matches pool 15 at 231/271
steps, new matches nothing
* positive control, new expr with threshold raised 1.5->3.5: selects
replicated pools 1,3,5,6,7,13,14 and excludes both EC pools (12,15) —
the filter removes exactly the EC pools and nothing else
* evaluated cleanly across 17 distinct mgr instances over 30d, so the
on(pool_id) join does not break on mgr failover
BLAST RADIUS: none to workloads. This edits one PrometheusRule; Prometheus
hot-reloads rule files and no pod restarts. Worst case is alerting scope on
one rule. Nothing writes to the cluster, no Ceph object is touched.
REVERT: revert this commit. One file, no state left behind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
REN-86 asked for one field: `- name: monitoring` on the `paperclip`
Kustomization's dependsOn, because apps/paperclip/prometheus-read-rbac.yaml
(PR #984) puts a Role and a RoleBinding in namespace `monitoring` and a
from-scratch bootstrap was expected to fail applying them.
The premise does not hold, so the field is not added and the reasoning is
written down instead:
- Namespace `monitoring` is not created by the `monitoring` Kustomization. It
is declared in kubernetes/infrastructure/namespaces/namespaces.yaml, applied
by `infrastructure`, which this Kustomization already dependsOn.
- `infrastructure` runs wait: false, so it applies the namespaces on its first
pass and only its healthChecks gate readiness. It cannot report Ready before
the namespace exists, and this Kustomization cannot start before that.
- Role and RoleBinding are core rbac.authorization.k8s.io/v1 kinds, and a
RoleBinding may reference a Role that does not exist yet. Nothing on this
path needs a kube-prometheus-stack CRD or workload, so there is no analogue
of the monitoring-rules/PrometheusRule split.
Adding the edge would also couple the Kustomization that carries the paperclip
pod to `monitoring` readiness, which itself waits on grafana-db,
kube-prometheus-stack, Loki, Alloy and blackbox-exporter for up to 30m.
Comments only: the parsed manifest is identical to master.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Option B on REN-37, owner-approved: reach Prometheus through the apiserver
service proxy under a Role holding one verb, instead of an L4 allow on :9090
that could not tell GET /api/v1/query from the unauthenticated POST /-/quit
that --web.enable-lifecycle puts on the same port.
ServiceAccount paperclip/paperclip-prometheus-read
Role monitoring/paperclip-prometheus-read
get on services/proxy, resourceNames:
["kube-prometheus-stack-prometheus:9090"]
RoleBinding monitoring/paperclip-prometheus-read -> that SA
(subject namespace: paperclip - load-bearing, cross-ns)
paperclip.yaml serviceAccountName + automountServiceAccountToken: true
Authored by svc-atlas, reviewed and merged by argus-bot: author is not the
merger. ci-manifests-test green on 35bcdf10 (kustomize build over the whole
tree). Reviewed against master 3a457bd6 in a fresh clone.
BLAST RADIUS: applying this restarts the paperclip pod (strategy: Recreate)
and every agent running in it. Nothing in production monitoring restarts -
Prometheus, Alertmanager, Loki and the ingress controllers are untouched.
The paperclip Flux Kustomization has wait: true, so a pod that does not come
back shows as a failed reconcile rather than silently.
REVERT - one step:
git revert -m 1 <the sha of this merge commit>
That drops serviceAccountName, restores automountServiceAccountToken: false
and deletes prometheus-read-rbac.yaml; Flux prunes the ServiceAccount, Role
and RoleBinding on the next reconcile (paperclip Kustomization, prune: true).
It restarts the paperclip pod a second time, for the same reason the apply
does.
AGENT-FREE ESCAPE HATCH - use this if the pod does not come back, because
then there is no agent left to run the revert above. Flux re-applies the same
spec every 10m and overwrites a bare rollout undo; the suspend is what makes
the undo stick:
flux -n flux-system suspend kustomization paperclip
kubectl -n paperclip rollout undo deployment/paperclip
# then land the git revert above, and afterwards:
flux -n flux-system resume kustomization paperclip
Names as they are in this repo: `paperclip` is both the Flux Kustomization in
flux-system (clusters/homelab/paperclip.yaml) and the Deployment in namespace
paperclip. No revisionHistoryLimit is set anywhere under kubernetes/, so the
default 10 holds and a previous ReplicaSet exists to undo to. The undo
restores automountServiceAccountToken: false, which closes the read path at
once, before the RBAC objects are pruned. It rolls the pod template back
wholesale, so an image bump that landed after this merge is rolled back with
it - re-apply that after the resume.
Escape hatch text is the merger's, written into this merge commit because
merge condition 2 puts the revert recipe here. The diff is untouched.
Refs: REN-36, REN-37 (option B), REN-47 (review bar), REN-73 (merge card)
Owner picked option B on REN-37: reach Prometheus through the apiserver
service proxy under an RBAC Role holding one verb, instead of an L4 Cilium
allow on :9090.
The port cannot be the boundary. The chart leaves `reloadStrategy` unset, so
the operator defaults it to HTTP and appends `--web.enable-lifecycle`, which
puts `POST /-/quit` and `POST /-/reload` on :9090 with no auth in front of
them. An L4 allow cannot distinguish those from `GET /api/v1/query`. The
apiserver can: it maps HTTP method to RBAC verb, so a Role with only `get`
refuses both in the authorizer before the request reaches Prometheus.
ServiceAccount paperclip/paperclip-prometheus-read
Role monitoring/paperclip-prometheus-read
get on services/proxy, resourceNames:
["kube-prometheus-stack-prometheus:9090"]
RoleBinding monitoring/paperclip-prometheus-read -> the SA above
paperclip.yaml serviceAccountName + automountServiceAccountToken: true
Shape copied from monitoring/ci-diagnostics-rbac.yaml, which has run this
same proxy path for the alert-monitor since 2026-07-16. No network-policy
edit: paperclip-egress already allows kube-apiserver on 6443.
BLAST RADIUS: applying this restarts the paperclip pod, and every agent with
it. Nothing in production monitoring restarts — Prometheus, Alertmanager,
Loki and the ingress controllers are untouched.
REVERT: revert this commit. One commit, four files, no state left behind;
Flux prunes the SA, Role and RoleBinding and rolls the pod back to no
mounted token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only change to kubernetes/apps/monitoring-rules/probes-blackbox.yaml. Reviewed by argus-bot at head 49bd9271 against base 2b71dda1: 3 documents parse deep-equal, blackbox-http 20 targets before and after, git diff -U0 non-comment non-blank lines = 0, apps/gatus/configmap.yaml untouched. Revert recipe and blast-radius claims re-executed and confirmed.
Comment-only. Rebuilt on top of master (2b71dda1) after #979 merged, so it
adds to that text instead of reverting it.
- EGRESS: names the two ingress-only Cilium policies in the monitoring
namespace and where blackbox-exporter runs; counts re-verified at
2b71dda1 (635 yaml files, 73 Cilium objects, 38 with egress/egressDeny,
0 in monitoring, 1 CCNP scoped to reserved.health, 17 namespaces with
an egress block).
- BLAST RADIUS: what breaks if that dependency is ever broken
(BlackboxProbeFailing x3, BlackboxProbesAbsent stays silent), who
notices (admin@renner.uno, ~10-11 min), and how fast it is undone.
- REVERT: replaced the claim with an executed recipe. All four git revert
candidates CONFLICT at 2b71dda1; the two forward sed deletions were run
and verified (blackbox-http 20 -> 17 and 20 -> 19).
No object changes: 3 documents deep-equal before/after under two
independent parsers, blackbox-http still 20 targets, and `git diff -U0`
has zero non-comment non-blank changed lines. apps/gatus/configmap.yaml
is not touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only change to kubernetes/infrastructure/image-automation/update-automation.yaml.
Reviewed by Argus (svc-argus/argus-bot). Verified comment-only: SHA256 of the comment-stripped file is 4ed11c66f18b8d15 before and after. No object field changes, so Flux's next reconcile is a no-op.
Revert: git revert 4f1c2cf1
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The comment on ImageUpdateAutomation/homelab named gitea_admin as the
owner of the bootstrap token that pushes writeback commits to master.
That is wrong. The pushing account is `flux` (Gitea uid 9): its
contribution history matches every `chore(image-automation)` commit on
master 1:1, while gitea_admin's own contributions stop at 2026-07-21.
Also separates two things the old wording ran together — the commit
author (security-bot@renner.uno, set in this spec) and the pushing
account (the token owner, not set here) — and records that writeback
failure is silent, since no commit is attempted unless a tracked image
moves.
Comment-only. The ImageUpdateAutomation object is byte-identical; no
Kubernetes object changes, nothing for Flux to reconcile.
Refs REN-50
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Comment-only. No Kubernetes object changes on either file: all 3 probe documents deep-equal to master (blackbox-http 20 targets), ConfigMap/gatus-config deep-equal with data["config.yaml"] byte-identical at sha256 8886ef21...6472 (6572 bytes), so Reloader does not roll gatus. Non-comment non-blank changed lines vs merge base 0a82d61f: 0. Reviewed and verified by argus-bot at head 9142b78a; the REVERT recipe was re-tested against its own commit.
The local-login connection stores only the ~8h OAuth access token and
Paperclip never refreshes Anthropic credentials (subscription values
are injected read-only as CLAUDE_CODE_OAUTH_TOKEN). It expired 8h after
the login and every run failed with 'terminal access failure'.
Replacing the secret value with a claude setup-token token fixed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The container was OOMKilled (exit 137) at 19:43 under the 4Gi limit.
With server-local runs every concurrent agent's Claude Code process
and MCP helpers live in this container, so the limit caps how many
agents can work at once; exceeding it kills the server and every
in-flight run (process_lost / orphaned_running_run).
Limit 16Gi, request 2Gi (idle server ~1Gi). Nodes have ~93Gi
allocatable each at 12-49% use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment-only, one sentence, one file. Wording is Argus's from the #979
review (REN-38).
The REVERT paragraph ended with "The paperless block below DOES revert
cleanly on its own with `git revert -m 1 0a82d61f`". True at 0a82d61f,
false at bd8690eb, because this same PR edits the paperless block:
from 0a82d61f: revert -m 1 0a82d61f -> CLEAN revert -m 1 16238d39 -> CONFLICT
from bd8690eb: revert -m 1 0a82d61f -> CONFLICT revert -m 1 16238d39 -> CONFLICT
Reproduced here: at bd8690eb the collision is one region, `<<<<<<< HEAD`
at 155 through `>>>>>>>` at 198, the whole paperless block against an
empty side. So the advice failed in exactly the case it was written for
— someone reverting the paperless probe under pressure, four lines below
a warning not to trust the natural resolution on this file. The new text
says what actually happens and how to resolve it, and stays true as the
comment block keeps changing, because it no longer promises clean.
Bar re-verified at this commit, both files against master, js-yaml:
probes-blackbox.yaml docs base=3 head=3, all deep-equal=true
http_2xx/20, http_2xx_wan_edge/3, tcp_connect/2
gatus/configmap.yaml ConfigMap/gatus-config deep-equal=true
data["config.yaml"] sha256 8886ef21…6472 both
sides, identical=true, bytes=6572
git diff -U0 vs master, non-comment non-blank lines: 0
apps/gatus/configmap.yaml is not touched by this commit at all.
Production constraint: nothing applied, restarted or re-rendered. The
public status.renner.uno page does not roll — ConfigMap data is
unchanged, so configmap.reloader.stakater.com/reload has nothing to act
on.
Revert: `git revert <this-sha>` restores the previous sentence and
nothing else.
Refs REN-21, REN-38, REN-44.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. Every parsed object in both files is byte-identical to
master at 0a82d61f, verified with js-yaml; zero non-comment lines change.
Review follow-ups from REN-22 on the merged #976/#977 probes:
1. The CNP argument was right in its conclusion and wrong in its framing.
cnp-cross-namespace-allow-apps-gatus-to-monitored-services DOES apply to
blackbox-exporter — its Generalization section names it. We are safe
because the allow it demands is already present, not because the note is
about gatus. The old wording is the sentence that makes the next person
skip the check for the next namespace.
2. Records the load-bearing egress fact, which was written down nowhere.
Cilium enforces at both ends; blackbox-exporter can send only because no
policy selecting a monitoring endpoint declares an `egress:` block.
Verified across the repo: 73 Cilium policies, 38 with egress, none in
monitoring, and the one CCNP is scoped to `reserved.health`. 17 other
namespaces already have an egress block, so a routine hardening pass on
monitoring would break these three targets and leave the *.monitoring.svc
ones working.
3. Extends the 200-vs-3xx caveat to all three new targets, not just
paperless. The repo evidence behind every one of them establishes only
`< 400` (kubelet httpGet accepts 200-399; lidarr.yaml:29's `curl -f`
without -L exits 0 on a 302). It proves the paths are ungated, not that
they answer 200. A 3xx gets reverted, not widened.
4. Records the tested revert recipe for #976 in the file. `git revert -m 1
16238d39` conflicts and the natural resolution silently drops paperless
too.
5. Corrects the stale "needs a CiliumNetworkPolicy first" claim in
apps/gatus/configmap.yaml. The correction is written in the manifest
header, ABOVE `data:`, not in place: the stale sentence sits inside the
`config.yaml: |` block scalar, and gatus carries
configmap.reloader.stakater.com/reload, so editing it in place would roll
the gatus Deployment for a comment. ConfigMap data sha256 is unchanged.
Production constraint: nothing is applied, restarted or re-rendered by this
change. No workload, namespace, policy or probe target is touched.
Revert: `git revert -m 1 <merge-sha>`, or drop either file alone with
`git checkout 0a82d61f -- <path>`.
Refs REN-21, REN-22.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
Rebased onto master after #976 by argus-bot; conflict markers only were removed,
the 29 added lines are byte-identical to the original commit bdf55c5b.
blackbox-http 19 -> 20 targets; other two Probe documents unchanged.
Revert: git revert -m 1 <this merge sha> is clean (tested; 20 -> 19, navidrome
and lidarr intact).
Open risk: /accounts/login/ must return 200, not 302 -- module http_2xx passes
only a final 2xx. If it is a 302, revert this PR rather than widening
valid_status_codes. Unverified at merge time: no Prometheus read path. See REN-22.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
paperless.renner.uno is served entirely through the paperless-gate
Authentik outpost. gatus already records the consequence at
apps/gatus/configmap.yaml:176-185: every path including /api/ answers
302 to an unauthenticated caller, so its check of that hostname — and
the UptimeRobot monitor REN-2 adds — is reachability only and stays
green while paperless is dead behind a live gate. This adds the app
health half of the pair, from the in-cluster vantage.
The target is /accounts/login/ because paperless-ngx has no
unauthenticated health endpoint. It is the app's own established
"the webserver answers" signal — the path its startup, liveness and
readiness probes already use — and it renders a Django view rather
than a static file. It does not cover the consumer/celery workers,
and the comment says so.
No CiliumNetworkPolicy change, and no edit to the paperless namespace.
The gatus comment above asserts an in-cluster probe needs one first;
that was written for gatus, which runs in `apps`. paperless-ns.yaml:24-26
already admits the whole monitoring namespace, where blackbox-exporter
runs, and Cilium unions the allow rules of every CNP selecting an
endpoint — the same mechanism apps/paperless/network-policy.yaml:8-12
relies on.
No PrometheusRule change: the target is covered by the existing
BlackboxProbeFailing catch-all and is correctly absent from its
exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds only
instances that have a dedicated rule elsewhere.
Refs REN-21.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
One file, kubernetes/apps/monitoring-rules/probes-blackbox.yaml, additive only.
blackbox-http 17 -> 19 targets; other two Probe documents unchanged.
Revert: delete the added target lines and comment block (tested clean).
Do NOT use git revert -m 1 on this merge once #977 has landed -- the natural
conflict resolution silently removes the paperless probe too. See REN-22.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
navidrome.renner.uno and lidarr.renner.uno are served entirely through
Authentik outposts. An external probe of either name therefore measures
DNS, TLS, Traefik and the gate outpost — and stays green while the app
behind a live gate is dead. This adds the second signal the
monitoring-vantage-point-rule asks for: an in-cluster probe of the
ClusterIP, which goes red on the app dying regardless of the gate.
Targets are /ping on both, the same path the pods' own kubelet
liveness/readiness probes already use. That it is ungated is not an
assumption: lidarr's reconcile script curls /ping in a loop before it
has read the API key out of config.xml (lidarr.yaml:29).
No CiliumNetworkPolicy change, and no edit to the navidrome namespace at
all. The wiki gotcha
cnp-cross-namespace-allow-apps-gatus-to-monitored-services is about
`apps`, where gatus runs; the influxdb allow-list it describes already
contained `monitoring`. navidrome-ns.yaml:25-27 likewise already admits
the whole monitoring namespace, which is where blackbox-exporter runs.
No PrometheusRule change either: both targets are covered by the
existing BlackboxProbeFailing catch-all and are correctly absent from
its exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds
only instances that have a dedicated rule elsewhere.
Refs REN-21.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Paperclip agents get their own wiki machine identity: OpenBao
apps/ai/wiki-api-keys field `paperclip` (already written, version 5),
mapped to Auth__ApiKeys__paperclip. Reloader rolls the wiki on the new
key.
Agents reach the service in-cluster at wiki.ai.svc.cluster.local:8080,
like the CI agents, so only the app-level API key applies (no Traefik
BasicAuth user needed). The wiki CNP admits the paperclip pod on 8080
and the paperclip egress CNP allows exactly that pod and port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
