Both pilot EndpointMonitors were created with an empty alert_contacts
list, so they notified nobody. Set uptimeRobotConfig.alertContacts to the
two contacts every hand-made monitor in the account already uses
(7880965 e-mail, 8026823 type 13, threshold 0, recurrence 0), in the
order the API returns them so IMC's string comparison stays stable.
Per the owner's ruling on REN-138 no test alert is fired; delivery is
owner-verified. Revert this commit to return to empty contacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backupPVC in node-agent-config is keyed by source storage class.
ceph-filesystem-bulk had no entry, so its data-mover took the full
CephFS subvolume clone path and sat Pending on "unbound immediate
PersistentVolumeClaims" (4 min audiobookshelf, 22 min immich on
2026-09-26, paging KubePodNotReady). Growth with data would hit the
data-mover prepare timeout and fail the backup.
- node-agent-config: ceph-filesystem-bulk -> readOnly: true with
storageClass ceph-filesystem-bulk-snapshot-readonly
- new StorageClass ceph-filesystem-bulk-snapshot-readonly:
backingSnapshot "true" (shallow RO volume, no clone), reclaimPolicy
Delete
- nodeAgent.podLabels revision label so node-agent restarts and loads
the config (node-agent only, not the server)
Refs REN-177.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents outside the pod (Claude Code on the workstation, via
`paperclipai agent local-cli` / agent API keys) only make outbound REST
calls to /api/* with Authorization: Bearer. The Authentik gate on
paperclip.renner.uno rejects those (intercept_header_auth), so they get a
separate machine door, following the wiki-index pattern:
- paperclip-api.renner.uno (no dns.renner.uno/public label: LAN-only)
routes /api to the paperclip Service, bypassing the gate. Paperclip's
bearer agent/board keys are the authentication.
- /api/auth on that host is refused (Traefik ipAllowList matching no
address -> 403), so Better Auth password sign-in/sign-up stays behind
the gate; upstream serves all password login under /api/auth.
- The host is added to PAPERCLIP_ALLOWED_HOSTNAMES (private-hostname
guard would 403 it) and the CNP admits Traefik on 3100.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments that pointed at other YAML as `file.yaml:N` went stale on any
edit above the cited line; three drift fixes had been merged by hand
(REN-181). The cited block now carries `# anchor: <name>` and the
comment cites `path.yaml#<name>`.
- Sweep: all 32 line-number citations in YAML comments converted.
Several were already stale at head and now point at the right block
(see PR). Vendored files (Flux gotk-components, IMC CRD) get no
anchor; those two citations name the object/schema path instead.
- Guard: scripts/ci/yaml-citations.sh in ci-manifests-test.yml; fails
on a line-number citation or a path#name whose file/anchor is gone.
Self-test in ci-scripts-test.yml.
- CONTRIBUTING.md: the convention in three lines.
Comment-only under kubernetes/: every file parses to identical objects
before and after, so Flux applies nothing and no ConfigMap data changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in this repo checked an alert expression. ci-manifests-test.yml proved
every PrometheusRule assembled into valid YAML and stopped there — no check that
an expression parses, that a label join matches anything, or that a `for` window
behaves the way its comment claims.
On REN-129 the CNPG surge rule was written `for: 2m` and reviewed at that value.
Against Iris's seven measured nights it fires on six of them: two owner emails a
night, 04:16 and 04:22 local, for a WAL backlog that clears itself in under ten
minutes. The arithmetic that caught it was hand-written from scratch twice, in
two runs, and thrown away both times because there was nowhere in the repo to
commit it. It is now a negative control.
Adds:
* scripts/ci/promtool-rules.sh — discovers every PrometheusRule (17 today),
strips `spec:` into a temporary mirrored tree, runs `promtool check rules` on
each, then `promtool test rules` over the committed suites. Discovery floors
on both, as in the sibling steps.
* a `promtool` step in ci-manifests-test.yml, `depends_on: []` like the other
four, on prom/prometheus:v3.15.0. The 3.x pin is load-bearing: the cluster
runs kube-prometheus-stack >= 72.0.0, and alert `for` evaluation is
version-sensitive, so a 2.x promtool would check a different evaluator than
production. The script asserts the major version it is handed.
* kubernetes/apps/monitoring-rules/tests/ — the REN-129 suite (14 cases over
the seven measured nights, an adversarial envelope built from the extremes of
the measured fill/drain ranges, two stall shapes, a threshold boundary, a
failover, a multi-instance case and the epoch-zero guard), its generator, and
six negative controls in mutations.txt.
* a mutation phase. Each control breaks one value the rules depend on and the
suites must fail; a control whose sed matches nothing is a failure, not a
skip.
* scripts/ci/test-promtool-rules.sh, run from ci-scripts-test.yml — 17 cases
asserting the script refuses to report success when it has checked nothing.
The mutation phase is not belt-and-braces, it is the point. Two failure modes
bit while writing this and neither changed promtool's exit code: a `rule_files`
entry that did not resolve, where promtool WARNs, loads zero rules, and passes
every `exp_alerts: []` assertion against an empty rule set; and traces generated
against the wrong clock window, where all seven nights were flat zero. Both were
green. A suite that passes proves the rules match the assertions, not that the
assertions reach the rules.
Production: nothing applied to the cluster. No manifest changed, so the
monitoring-rules kustomization builds identically — its `resources:` list is
explicit, and the new tests/ directory is not in it and has no kustomization.yaml
of its own, so Flux does not see it. Worst case is a red step on PRs touching
kubernetes/**; enable_status_check is false on master, so it cannot block a merge
on its own.
Verified with real promtool 3.15.0 against the real tree: 17 manifests parsed
(56 rules), the suite green, all 6 mutations caught. Run again under BusyBox
1.35.0 sh/awk/sed/find/xargs, the toolchain prom/prometheus actually ships.
Guard tests 17/17 under both bash and BusyBox sh. shellcheck 0.11.0 clean at
every severity. woodpecker-cli 3.18.0 lint clean on all 17 pipeline files.
Not run in Woodpecker itself — this PR is its first real pipeline execution.
Revert: revert this commit. It adds files and one step; nothing else reads them.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ceph's mon_osd_nearfull_ratio is 0.85 and nothing reported the approach to
it, so the first signal would have been Ceph flagging nearfull — already the
incident. With failureDomain host, size 3 and three OSDs there is no
rebalance to hide behind: all three cross together.
Two warning rules, both observability only:
- CephOSDNearfullApproaching: fullest OSD sustained >= 75% for 30m. 0.75 is
0.85 minus the largest measured bulk burst (+10.24 pp in 25 min, 09-26
05:18-05:43Z), so it marks the point where one more such burst reaches
nearfull. 30m is the shortest wait that outlives a self-reclaiming burst
(that burst held >= 75% for 19.0 min) and short enough to fire before 0.85
at the fastest observed sustained climb (+11.44 pp/h).
- CephReplicapoolHeadroomLow: replicapool max_avail < 160 GiB for 30m.
160 = 89 GiB held above the 0.85 line + 38 GiB ha-pg has not yet consumed
of its approved 32Gi + 35 GiB observed replicapool churn.
Replayed against the 14d of real series to 2026-09-26 08:00Z: both fire zero
times. Worst 30m-sustained OSD floor 71.632%; lowest 30m-sustained
replicapool ceiling 208.97 GiB. Both expressions evaluated against live
Prometheus at the 09-26 05:40Z peak do select (3 OSDs at 81.87%,
replicapool at 117.41 GiB), so the 30m `for` is what keeps that excursion
quiet, not a join that never matches.
Thresholds are unmerged pending the owner's yes on REN-163.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Criterion 3 passed: the removal test (#1018, merged 101c11f4) deleted monitor
804092841 and the controller logged `Monitor Removed` at 03:57:58Z. This commit
is the deliberate revert of that test, which Zeus sequenced to run only after
the cap reading below. The restored object is byte-identical at the object level
to the version that provably created 804092841.
It also corrects a wrong claim this file asserted. The header stated that a
monitor-count cap was EXCLUDED as the cause of the `www.renner.uno` 403, on the
reasoning that a cap would arrive as HTTP 200 + `stat: fail`. The removal test
refuted that:
03:26:27-03:56:27 account 10/10, seven consecutive `Status Code: 403`
03:57:58 804092841 deleted, account 9
04:01:27 `Monitor Added: www.renner.uno-uptime-monitor`, id
804092931, account back to 10
One freed slot is the only variable. UptimeRobot returns HTTP 403 when the
account monitor limit is reached, so "account is full" is the first hypothesis
for an IMC 403, not a discarded one. Capacity is now settled on fact for
REN-158; REN-8's 18-monitor page against a cap of 10 stays blocked on it.
Expect this restored create to be refused with 403 until the tier is raised —
`www` took the freed slot. Left in git per the capacity ruling so the retry loop
creates it unaided once capacity exists. `audiobooks.renner.uno` stays covered
by its 60s Gatus check throughout, so there is no coverage gap.
Also recorded: the real delete log string is `Removing monitor … from provider`
/ `Monitor Removed`, not the `Deleting Monitor` this file predicted; and the
status page is in all-monitors mode, proven in both directions (9->10 on create,
10->9 on prune) with no API key, which settles REN-9's monitor-list mode.
Verified: js-yaml loadAll gives 2 EndpointMonitors in `uptime-monitor`; the only
functional changed line is the restored object, every other changed line is a
comment; restored spec compared equal to `f20fe8bf^`. Not applied — Flux applies
from master and I cannot merge my own work.
Revert: revert this commit. Flux prunes the object and, because its monitor does
not exist, nothing in UptimeRobot changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#992's header comment says RecordDuration "IS called, by all six controllers",
established by grepping the tagged upstream source trees. Now that the scrape is
merged the claim is measurable, and five of six emit the family: 170 series from
kustomize (33), source (43), image-reflector (68), helm (25) and image-automation
(1), and zero from notification-controller.
Five is permanent, not today-only. notification-controller's three reconcilers
are registered but never walked -- controller_runtime_reconcile_total for
controller=alert/provider/receiver reads 0 over 56 h of uptime with 2 Alerts and
1 Provider live in the apiserver -- so the family is absent from its /metrics
entirely and creating a Flux Alert or Provider does not produce the sixth series.
Comment-only. Every one of the 17 changed lines is a pure comment line and the
comment-stripped file is byte-identical to master (same sha256), so the object
Flux applies is unchanged and no pod rolls.
Refs REN-127, REN-102, REN-97
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
probes-blackbox.yaml:209 cited apps/gatus/configmap.yaml:176-185 as the
place gatus documents the paperless blind spot. That was true at cb9e73bd,
where the sentence was written. A later +29-line insert into configmap.yaml
pushed the block down; at master those lines are now Web Service 9
(music.renner.uno), which carries no prose at all.
The same 10-line Paperless blind-spot comment now runs 205-214.
Comment-only. No spec, no pod template, no new resource.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed and merged by argus-bot (REN-145). Approved at head 088db00e.
Independently verified: the eight GVKs against the single served CRD version in Flux v2.9.5; every CRD field path the config reads; object census = 149 and the reconcile intervals behind the 2h/1h thresholds; the rendered chart keeps --resources and omits --custom-resource-state-only; the whole rendered ClusterRole uses only list+watch with no wildcard; the real kube-state-metrics v2.20.0 binary accepts the rendered config while a malformed control is rejected at startup; the kustomize CI gate green on the simulated merge result, which CI itself did not test.
Own promtool fixtures: 9 alert tests plus a 13.5-day recording-rule simulation, all passing. Warm-up guard empty at 1201 samples and populated at 1297; the pruning case yields 0 under the shipped comparator against 1.1376 under the withdrawn ratio; tag counting survives a kube-state-metrics pod roll.
The missing staleness alert is an accepted, measured reduction; follow-ups tracked in REN-143.
Revert: git revert -m 1 <this merge commit>. Both Kustomizations are prune: true, so nothing is orphaned.
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
Verified against the PR head 17c1dece:
- helmrelease.yaml diff is comments only; parsed YAML of HEAD^ and HEAD is byte-identical (1683 bytes each). No values change, so no helm upgrade and no cilium agent restart.
- ServiceMonitor selector {managed-by: homelab, name: cilium-agent} matches only the new Service; endpoint port `metrics` exists on it; targetLabels k8s-app=cilium present.
- Service selector k8s-app: cilium, headless, distinct name from the chart's cilium-agent Service - no double-scrape or ownership collision if a chart flag is flipped later.
- Revert is one step: the monitoring-rules Flux Kustomization has prune: true, so git revert removes the Service.
- ci-manifests-test and ci-scripts-test green; advisory review-bot: no blocking issues.
REN-156
Approved by argus-bot at head c1b9b259 after re-review (head moved from dd6720f2).
Ceph capacity gate cleared at 03:49Z: replicapool max_avail 168.8 GiB usable, per-OSD 76.1% -> 78.8% worst case against nearfull 0.85, HEALTH_OK.
Four eyes: authored by atlas-bot, merged by argus-bot.
PVCs do not shrink - manifest reverts with git revert <merge-sha> -m 1, volumes do not.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Approved at 7aff5250 (review 1086). CNPGWalArchiveBacklog for: 30m -> 15m; new CNPGWalArchiveBacklogSurge > 64 for 8m; both exprs wrapped in max by (namespace, pod) so a failover does not restart the for timer.
Verified with promtool 3.5.0: 12-case test rules suite over Iris's seven measured nights plus an adversarial envelope case, all passing, with five negative controls each failing. Revert tested: git revert restores the file byte-for-byte.
Closes REN-129.
Deletes the `audiobooks.renner.uno` EndpointMonitor so that the other half of
REN-7 can be proven: an EndpointMonitor creates an UptimeRobot monitor, and
deleting it removes that monitor again. "A monitor that cannot be removed is a
leak." The create side is already shown — monitor 804092841, created
2026-09-26 03:26:27, controller log `Monitor Added`.
What to observe after this merges, keyless and from either side:
* `https://uptime.renner.uno/api/getMonitorList/UgMXmP1iRV` goes from
`psp.totalMonitors: 10` to `9`, and id 804092841 is absent.
* controller log: `Deleting Monitor: audiobooks.renner.uno-uptime-monitor`.
Precondition, checked before writing this: v2.2.13 carries no finalizer, so a
delete while the controller is down is never reconciled and orphans the monitor
permanently. Pod ingressmonitorcontroller-5d577cc58-pgqwg was Ready=True with
0 restarts. Whoever merges this must re-check that the pod is Ready first.
Also records, in the file, why the second pilot object still has no monitor:
UptimeRobot answers its create with HTTP 403 and IMC retries every 300s
forever. 403 is the non-200 branch of `Add()`, so it is not the provider's
validation path — that returns 200 with `stat: fail` — which rules out the
duplicate-URL, monitor-cap and bad-parameter guesses. Tracked on REN-155; the
object is left in place as the reproducer.
Blast radius: removes one EndpointMonitor from `uptime-monitor` and one
UptimeRobot HTTP check every 300s. No Ingress, workload, ingress controller,
DNS record or RBAC rule is touched, and `www.renner.uno` is unchanged.
Revert: revert this commit. Flux re-applies the object and the controller
re-creates the monitor under the same friendly name — a new id, since the name
and not the id is what binds object to monitor. That round trip is also the
idempotent-reconcile check and is worth doing deliberately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sizing comment blamed the 10-minute archiving stall on Velero repository
maintenance. REN-136 disproves that over seven nights:
- sustained archiver drain is ~14 segments/min, flat whether Kopia is
running or not, so Kopia does not slow the drain;
- Kopia's nearest batch drifts across 02:03-02:24Z while the burst stays
pinned to 02:13Z, and a cause that drifts 20 minutes cannot produce an
effect fixed to the minute.
The real driver is a nightly producer burst: 56-119 segments queued at
02:13-02:15 UTC on all seven nights, against a steady-state queue of 0.
Comment now states that, and adds REN-136's stalled-night framing (6.36 GiB
free survives one stalled night, ~18.6 GiB carries about four).
Comment text only. `spec.storage.size: 32Gi` is unchanged and still the sole
semantic change on this branch — verified by parsing origin/master and HEAD
with js-yaml: with spec.storage.size masked the two parse to identical
objects.
Raised by Argus in review of #1014 as a non-blocking finding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed at head 3b96b3ab, test-merged clean into master dd2256ea (tree af52adab).
Verified: one file +73/-0; 0 YAML parse failures repo-wide; jobName blackbox-negative-control unique; all four ports re-derived from the paperless Service/Cluster objects; tcp_connect module declared; paperless-ns.yaml still admits monitoring namespace-wide so the controls read 1 today.
Pages nobody: all 16 PrometheusRules enumerated, the only non-job-pinned probe_success rule (HetznerBackupEndpointDown) is instance-pinned to the Hetzner URL; the chart's up-based TargetDown cannot fire because blackbox answers 200 with probe_success 0 and the 5s module timeout stays inside the 10s scrapeTimeout; all 5 blackbox-overview.json panels are pinned to job=blackbox-http; no Grafana-managed alert references probe_success.
Monitoring-only, no traffic-path change. Revert with: git revert -m 1 <this-sha> on a branch, through a PR.
REN-28 §6.4 gate NOT yet taken - reviewer has no Prometheus credential either. PR 2b must not open until the probe_success == 1 reading exists.
The cilium chart renders templates/cilium-agent/service.yaml only when
prometheus.enabled AND (prometheus.serviceMonitor.enabled OR
prometheus.metricsService). infrastructure/cilium/helmrelease.yaml sets the
first and neither of the second, so no cilium-agent Service existed. The
standalone ServiceMonitor in apps/monitoring-rules selected a Service that was
never there: scrape pool serviceMonitor/kube-system/cilium-agent/0 sat at 0
active / 46 dropped targets and cilium_* had zero series.
Consequence: CiliumBPFMapPressureHigh and CiliumBPFMapFull read
cilium_bpf_map_pressure, so both evaluated an empty vector and could never
fire — the exact failure the helmrelease comment says the metrics were turned
on to prevent. cilium_policy_verdict_total and cilium_drop_count_total were
equally absent, which is why REN-140 needed a purpose-built prober to observe
a network-policy deny.
Declares a sibling metrics Service next to the ServiceMonitor instead of
setting prometheus.metricsService in the HelmRelease. The flag is the
chart-native fix and renders exactly this one object, but it is a values
change: helm-controller would upgrade the CNI release and three-way-merge
against live objects, and the configmap.reloader annotation turns any
resulting cilium-config change into a rolling restart of every agent. A
Service applied by the apps Kustomization cannot do that.
Naming follows authentik-outpost-servicemonitors.yaml: distinct name plus
app.kubernetes.io/managed-by: homelab, with the ServiceMonitor selector pinned
to it, so turning either chart flag on later neither collides on ownership nor
double-scrapes the same endpoints.
The helmrelease.yaml change is comment only — verified by rendering the chart
against the values block before and after: byte-identical, so Flux performs no
helm upgrade.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed and merged by argus-bot (REN-149). Author atlas-bot; four-eyes split intact.
Verified live: no alerting rule can match job="blackbox-ren28-negative" (24 probe_/up/scrape_ rules enumerated; tcp_connect module timeout 5s < scrapeTimeout 10s keeps up==1); pod IPs 10.244.4.166/10.244.2.220 current; probeSelector strict so the release label is load-bearing and is set; monitoring has no egress enforcement.
Temporary. Delete probes-ren28-negative.yaml and its kustomization line once the REN-28 "after" reading is recorded.
Revert: git revert -m 1 <this sha>
REN-28 item 3a. Narrows the navidrome baseline CNP's whole-monitoring-namespace, every-port ingress admit to the blackbox-exporter pod on :4533 and :8686, and repoints two comments elsewhere in the tree that cited the replaced rule.
Reviewed by argus-bot at head 1c1a637c in a private detached worktree. ci-manifests-test and ci-scripts-test green at that head. Blocking review 1065 (a false ICMP justification) resolved by deletion in 1c1a637c.
Revert: git revert -m 1 <this sha>, through a PR.
ha-pg-1 queues WAL for archiving every night: flat 0 until 02:12Z, peak
02:14-02:15Z, drained by 02:22Z, peaks of 56-119 segments on the seven
nights measured 2026-09-19 to 2026-09-26. Point-in-time recovery for
home-assistant is degraded for the ~10 minutes each burst lasts. Nothing
fires, because CNPGWalArchiveBacklog's `for` was 30m.
The cause is a write burst, not IO contention. Drain was a flat 13.0-14.9
segments/min on all seven nights, including the night whose Kopia
repository-maintenance batch started only after the burst had drained;
that batch drifts across 02:03-02:24Z while the peak stays pinned to
02:15Z. Fill is what varies, 18.7-39.7 segments/min. 02:12Z is 04:12
local, when Home Assistant's recorder purge runs. An earlier revision of
this commit named Velero/Kopia as the cause; the 7-night data retracts
that, and the annotations no longer send anyone to the velero namespace.
Three changes, all alert-rule text. Nothing restarts, no storage and no
traffic path is touched.
- CNPGWalArchiveBacklog `for`: 30m -> 15m. This does not fire on the
nightly burst at either value, and should not: every burst in the week
drained itself, and the longest contiguous excursion above 5 was 10
minutes. 15m is for the archiver that has actually stopped, which is
the failure that fills the PVC and stops Postgres.
- New CNPGWalArchiveBacklogSurge, >64 segments (1 GiB) for 8m, warning.
Covers the shape the 15m rule cannot see: a working archiver outrun by
production. 8m because the measured envelope spends at most 5
consecutive minutes above 64 on any night, so 8m clears the worst
observed night by 3 minutes while still catching 1 GiB that is not
draining -- from 119, a healthy archiver is back under 64 inside 5
minutes.
- Both exprs wrapped in `max by (namespace, pod)`. `role` flips on
failover, a changed label is a new series, and a new series restarts
the `for` timer. On the raw series a backlog that persists through a
failover at minute 11 is silent at 16m and 20m and only pages by 25m.
Verified with promtool 3.6.0 against the measured traces, not applied to
the cluster: `check rules` passes, and unit tests cover the ordinary
night, the worst night, the quiet night, a stalled archiver, and a role
flip mid-backlog. Negative controls confirm the tests have teeth -- at
`for: 2m` the surge rule fires at 02:17Z with $value 99 on the ordinary
night, and without `max by` the role-flip case fails.
Revert: `git revert <this commit>` restores both rules exactly. No state
outside Prometheus's rule evaluation is affected either way.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds one Probe, `blackbox-negative-control-paperless`, with its own jobName
`blackbox-negative-control` and four tcp_connect targets in the `paperless`
namespace. The targets exist to be REFUSED: each must read probe_success == 1
at this commit and 0 once REN-28 PR 2b narrows the whole-namespace `monitoring`
admit in paperless-ns.yaml. That flip is the negative test, and it then stays
on as a standing regression alarm.
This is monitoring-only. No traffic-path change, no restart, no new grant, no
policy edit. blackbox-exporter already runs in `monitoring`, whose baseline CNP
is ingress-only, and default-deny.yaml is scoped `namespace: apps`, so the
paperless ingress policy is the only variable that can move these controls.
Deliberately silent: both blackbox alerts are scoped job="blackbox-http", so a
distinct jobName is outside them and four permanently-failing targets page
nobody after 2b. The inverted alert CNPNegativeControlReachable ships in PR 2b,
not here — between 2a and 2b these controls are supposed to read 1.
Appended at end of file so existing line-number citations (including
paperless-ns.yaml's reference to probes-blackbox.yaml:250) do not shift.
Refs REN-28 §5.6, §6.4. Revert: drop this one Probe document.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `fromEntities: cluster` + `icmps` rule added earlier in this branch was
justified by a false premise. Its comment claimed the new `toPorts` was "the
first layer 4 ingress rule in this namespace" — at f177685d the ingress already
ended in an `icmps:` rule, which is itself a layer 4 rule. So the allow was
neither required by this change nor new with it.
It also did not match its own header ("ICMP from the host and the other nodes"
vs `fromEntities: cluster`), and for pods it was redundant: IPv4 type 3 and
IPv6 types 1 and 2 are already in the namespace-wide ICMP rule below it.
ha-pg-allow-cnpg-operator does not transfer as precedent: it is pod-scoped and
carries no `host`/`remote-node` allow, so `cluster` is the only thing admitting
the host there. Four peer baselines with exactly this shape — monitoring,
home-assistant, ai, external-secrets — carry no entity-scoped ICMP allow, and
monitoring is where blackbox-exporter and Prometheus themselves run.
The branch is now exactly the approved REN-28 §4.1 scope: the monitoring
namespace admit is narrowed to the blackbox-exporter pod on 4533/8686, and
nothing else in navidrome-ns.yaml changes.
Revert: `git revert` this commit restores the allow; reverting the whole PR
restores the namespace-wide monitoring admit.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Two of the four REN-28 §7 negative tests need a prober inside `monitoring`
that is not the workload under test. `pods/exec` in `monitoring` was declined
as a wider grant than the change it verifies, so this uses the prober the
cluster already runs: blackbox-exporter, driven by a Probe object, read back
through the existing read-only REN-36 Prometheus grant.
Adds one Probe, jobName `blackbox-ren28-negative`, module tcp_connect, four
targets that are reachable from `monitoring` TODAY and that the REN-28
paperless PR must close:
10.244.4.166:9187 / 10.244.2.220:9187 pprl-pg metrics port -> §7 test 4
pprl-pg-r.paperless.svc:5432 same, stable name
gotenberg.paperless.svc:3000 another paperless-ns workload -> §7 test 2
This is the "reachable before" half. It can only be taken while the current
policy is still in force, which is why it goes in ahead of the paperless PR.
No alert can match it: every rule in prometheusrule-blackbox.yaml is scoped to
job="blackbox-http", and TargetDown cannot fire because a Probe's scrape target
is blackbox-exporter itself, which answers 200 with probe_success 0 on a failed
probe.
No workload is touched, no restart, no policy change, no secret. Revert is
deleting the file and its kustomization line.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One semantic line: spec.storage.size 20Gi -> 32Gi. The sizing comment above it
is rewritten in the same commit because it justified 20Gi on table growth
("leaves room for autovacuum bloat and index churn"), which is no longer the
reason for the number.
The reason is WAL headroom during an archiving stall. 20Gi left 6.63 GiB free;
when WAL archiving stalled for 10 minutes under Velero repository maintenance
(REN-129) the un-archived backlog hit 1.84 GiB, 28% of that free space. Postgres
cannot recycle a WAL segment until the archiver ships it, so a longer stall
walks the volume toward full and a full PGDATA stops the recorder. 32Gi takes
free space to ~18.6 GiB and the same backlog to 10% of headroom.
Owner approved option A on REN-135 (apply now, do not gate on the capacity
read). Options B (archiver maxParallel test) and C (throttle Velero) were
declined, so ha-pg-objectstore.yaml and Velero are untouched.
Verified by parsing the file before and after: with spec.storage.size masked,
the two documents are byte-identical as parsed objects. instances: 2,
storageClass: ceph-block, all 8 postgresql.parameters and resources are
unchanged.
NOT verified here: `ceph df` for the replicapool pool. This runtime has no
kubectl and no Ceph read path — see the PR body.
Revert: `git revert <merge-sha> -m 1` restores the manifest, but the PVC
expansion itself is NOT reversible; PVCs do not shrink.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Per-object Flux readiness cannot come from the controllers: gotk_reconcile_
condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and
populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273
gotk_* series and none of them are about readiness. This turns on
kube-state-metrics custom-resource-state over eight Flux kinds instead, and
builds the standing not-Ready alerts on it.
The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships
as recording rules only: no threshold is derivable from the history that
exists today, and the measurements are in the rule file rather than left
implicit.
- kube-state-metrics customResourceState + rbac.extraRules (list, watch only)
- prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h),
FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m),
and four recording rules for the staleness signal
- image-automation README: the "What watches this" section said both cases
were uncovered
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed and merged by argus-bot (four-eyes: authored by atlas-bot).
Review 1063 on REN-137. Claims 1-3 re-derived from IMC v2.2.13 source and the
vendored CRD; host choice checked against the live status-page feed (9 baseline
monitors, neither target present). Revert: revert this merge commit; prune=true
plus enableMonitorDeletion=true returns the account to the 9-monitor baseline.
Recorded finding (non-blocking): UpTimeMonitorService.Add() returns nothing and
logs every failure, so a rejected newMonitor call is invisible in Kubernetes and
in Flux. A negative status-page result must not be read as "explicit-list mode"
for REN-9 without a controller-log read.
Implements the owner's decision on REN-52 (2026-09-25, option B), tracked by REN-111: one ClusterRoleBinding, built-in `view` ClusterRole, subject the paperclip/paperclip-prometheus-read ServiceAccount from REN-36.
Reviewed at head 230756c3 in a private worktree. Verified: exactly one object, no Secrets, no write verb, no new ServiceAccount; subject resolves against prometheus-read-rbac.yaml:48-52; name paperclip-cluster-view is unique in the repo. Flux CRD coverage confirmed in git — flux-view-flux-system carries aggregate-to-view: "true" (gotk-components.yaml:240) over all six toolkit groups, so the REN-52 pre-approved extra ClusterRole is correctly not added.
Revertibility confirmed rather than assumed: clusters/homelab/paperclip.yaml:14 is prune: true over ./kubernetes/apps/paperclip and the object has no prune: disabled annotation.
Revert: `git revert <this-merge-sha> -m 1`. Flux deletes the binding within the Kustomization's 10m interval and the token falls back to its REN-36 scope (get services/proxy on two named Services). No workload is touched either way.
Not verified here, owed on REN-111: the post-merge SelfSubjectAccessReview from inside the pod, proving the grant took effect and that view-aggregation actually covers the Flux CRDs in the live cluster.
Reviewed and merged by argus-bot (REN-134). CI green on 97b7f79b: ci-manifests-test, ci-scripts-test, pr-review.
One semantic line: OpenBaoUnavailable for: 1m -> 5m. Verified against the repo that 1m was the tightest probe window in the cluster (next tightest 5m), that probe and rule evaluation both run at 30s, and that severity=critical routes to the email receiver rather than a pager. Revert: git revert 97b7f79b.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
navidrome and lidarr both take identity from X-Authentik-Username
unconditionally (navidrome/deployment.yaml), so any pod that can open a TCP
connection to their listeners can claim any username. The baseline CNP
admitted the WHOLE monitoring namespace on EVERY port, which put every
current and future monitoring workload inside that boundary. Only two flows
actually cross it, and both are blackbox probes.
Exposure delta, monitoring -> navidrome namespace:
before: every pod in `monitoring`, all 4 workloads in `navidrome`, every port
after: blackbox-exporter only, :4533 (navidrome) and :8686 (lidarr)
Both selectors and both ports are measured on the live cluster, not inferred:
* `app.kubernetes.io/name: prometheus-blackbox-exporter` resolves to pod
blackbox-exporter-prometheus-blackbox-exporter-57844f6f8d-6gqz8 in
`monitoring` (Prometheus service-discovery pod labels, read through the
REN-36 read-only proxy grant). Chart prometheus-blackbox-exporter-11.19.1.
* `probe_success` is 1 for both
http://navidrome.navidrome.svc.cluster.local:80/ping and
http://lidarr.navidrome.svc.cluster.local:80/ping. Those Services select
`app.kubernetes.io/name: navidrome` / `lidarr` and would have no endpoints
if the pods did not carry the labels, so the probe path is proven end to
end, prober pod to app pod.
* Service :80 -> containerPort 4533 (deployment.yaml) and 8686
(lidarr.yaml). Cilium matches the POD port after service translation, so
`port: "80"` would read clean in review and break both probes.
ICMP rule added, and it is NOT cosmetic. The toPorts above is the first
layer 4 INGRESS rule in this namespace -- navidrome-ns.yaml is the only CNP
selecting these pods, and its ingress was L3-only until now. Cilium drops
ICMP to an endpoint selected by any layer 4 policy unless the traffic is
CT-related, and the existing ICMP allow matches pods only
(io.kubernetes.pod.namespace Exists), which `host` and `remote-node` do not
carry. The kubelet sources the HTTP probes from the host netns, so its PMTUD
path is host-sourced: without the new rule, Destination Unreachable / Packet
Too Big toward these pods are dropped silently and the symptom is a stalled
transfer, not a policy denial. Same rule, same three types, same reason as
ha-pg-allow-cnpg-operator in home-assistant-ns.yaml. This is an ADDITION to
the approved plan text, flagged here because it is mine, not the plan's.
Two citations elsewhere pointed at the rule being replaced and would have
become false -- the defect REN-28 is about, in reverse. Both now cite what a
rule admits rather than a line number, so they cannot go stale again:
* navidrome/deployment.yaml -- the union list of what really reaches the pod
* monitoring-rules/probes-blackbox.yaml -- the justification for why the
REN-21 probes needed no policy change
NOT touched: `host`/`remote-node` (deleting it CrashLoops both apps -- the
kubelet probes come from there), the in-namespace allow, the Traefik and both
gate-outpost allows, the existing ICMP rules, every egress rule, and
ND_EXTAUTH_TRUSTEDSOURCES (item 2 of REN-28, declined and documented in-tree).
No workload restart: a CNP change is picked up by Cilium in seconds.
Revert: `git revert -m 1 <merge-sha>` on master, through a PR like any other
change. Pure CNP + comment change -- no data migration, no restart, no
rollout; Flux reconciles and Cilium re-applies in seconds. Emergency path if
Flux is impaired:
kubectl apply -f <(git show <merge-sha>^:kubernetes/infrastructure/network-policies/navidrome-ns.yaml)
Detection that a revert is needed, inside one 30s scrape interval:
`probe_success == 0` on either navidrome target (alerting already exists in
prometheusrule-blackbox.yaml), or either pod leaving Ready.
REN-28 item 3a, approved by the owner. Paperless (item 3b) is a separate PR
and does not open until >=24h of green probes here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first two of the survey's 14 page rows, and the first objects in this
repo that cause an UptimeRobot write. Both hosts are ungated, both serve a
real application response rather than a gate, and neither has a monitor in
the account today — so the pilot neither duplicates an existing monitor nor
puts anything on the public page that the owner has not already approved.
Uses `spec.url`, not `spec.urlFrom.ingressRef` as REN-7 was written: an
EndpointMonitor can only resolve an ingressRef in its OWN namespace
(pkg/kube/util/url.go:71), `IngressURLSource` has no namespace field, and
every app Ingress lives elsewhere. Reaching them would mean RBAC and a
widened WATCH_NAMESPACE in each production app namespace. Full reasoning in
the file header.
Deliberately omits `uptimeRobotConfig.statusPages`, so that whether these two
monitors appear on uptime.renner.uno on their own answers REN-9's open
question about the page's monitor-list mode.
Also records two source-verified properties that change how this must be
operated, both in comments: a Ready Kustomization cannot prove the monitor
was created (EndpointMonitorStatus is an empty struct, so kstatus returns
Current unconditionally), and v2.2.13 uses no finalizer, so deleting an
EndpointMonitor while the controller is down orphans its monitor.
Verified: kustomize build over all 68 kustomizations, 68 ok / 0 fail; both
rendered EndpointMonitors validated with kubeconform -strict against the
schema extracted from the vendored CRD, with a negative control confirming
the check rejects a bad enum. Lint/render only — NOT applied; Flux applies
from master and these objects do not exist in the cluster yet.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OpenBaoUnavailable went pending at 01:07 UTC on a single 5s HTTPS probe
timeout while OpenBao was healthy, one evaluation short of paging a
critical. At 1m it was the tightest probe-based window in the cluster by
5x, and isolated single-sample probe blips are ambient on the
blackbox-http fleet at roughly one per two hours (three on three
unrelated targets in six hours).
5m matches WoodpeckerDown in the same group. Nothing real is lost: the
rule's own description says existing Secrets keep working and the damage
surfaces at the next ExternalSecret refresh, and OpenBaoQuorumLost
(for: 2m) still covers the fast-moving failure.
expr and severity are unchanged; this only moves the for: window and
replaces the now-wrong "two consecutive failed probes" comment.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. The narrowing in 16c0c0a7 is unchanged; this records what the
two expressions do with the shape neither of them mentions — an ExternalSecret
with no `Ready` condition.
Review of REN-106 asked for the third state to be made explicit, on the
strength of a metrics census showing `ai/wiki-index-basic-auth` with only a
`Deleted=False` condition. That series is real; the object behind it is not.
ESO v2.11.0 builds the `Deleted` condition on a throwaway object in the
`IsNotFound` branch of Reconcile, purely to retire the object's metric series,
and never persists it — so the series outlives the object until the ESO pod
restarts. That ExternalSecret was retired from git in 419cbed3, its target
Secret is gone from `ai`, and its sync counter is frozen while its six live
siblings advance. On a live object ESO writes conditions of type `Ready` only.
What that leaves is three shapes, all decided: no conditions yet (in-progress,
bounded by `timeout: 5m`), `Ready=True` (current), `Ready=False` (failed for
the two non-retrying reasons, in-progress otherwise). Flux evaluates
inProgress -> failed -> current and falls back to in-progress, so an explicit
`inProgress` expression would be a no-op on all three; the reasoning is written
down instead of encoded twice.
Also noted: `Ready=True` covers reasons `SecretDeleted` and `SecretMissing`,
where ESO reports healthy with no Secret behind it. Unreachable under this
object's default Retain/Owner policies, and a trap for whoever changes them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed as argus-bot (review 1056) at head 2821629e. Comment-only, reproduced independently in a private worktree: `git diff -U0` against merge-base 64ee49ab has no added or removed line that is not a whole-line `#` comment, and with comment lines stripped both files are md5-identical to master (crd f753baa9c9684002ad01d309e98755ba, rbac 092ddad9c003209d4bd27721c04048b1). Non-comment bytes identical means the parsed YAML is identical, so the applied objects cannot change: no apply, no RBAC change, no restart of the ingressmonitorcontroller pod.
Provenance re-derived from upstream, not taken from the PR body: 2deeba18 is the v2.2.13 tag commit (2026-05-06), aaffd8a8 is a 2-parent master merge commit (2026-09-25), the two chart CRDs are byte-identical, and the vendored body matches upstream at the tag (md5 f753baa9c9684002ad01d309e98755ba over `apiVersion:` to EOF) — so the new header is a check a human can actually run. All seven cmd/main.go + ingress-wrapper.go + url.go line citations in rbac.yaml verified at the tag.
Render check is CI's ci-manifests-test (green at 2821629e, with ci-scripts-test and pr-review; combined state success) — no kustomize in my sandbox either, so I am not claiming a local render.
Revert: `git revert <this-merge-sha> -m 1`. Nothing in the cluster to unwind.
Closes the two follow-ups recorded on the #995 merge (REN-6).
Reviewed and merged by argus-bot (REN-124). Comments only: js-yaml loadAll base vs head parsed-identical for all three files; gatus data["config.yaml"] byte- and hash-identical at 6572 bytes, so the gatus Deployment does not roll. All 7 repointed ranges resolved against the target files at 317ddbe9. Revert: git revert 317ddbe9.
Both found by Argus in the #995 review and handed to REN-6 as follow-ups.
Comments only — every added and removed line starts with `#`, and both files
are md5-identical to master once comment lines are stripped.
rbac.yaml misstated the failure mode of the omitted pods/services grant. It
claimed a denial "costs a log line, not a reconcile". The function's own error
branches do log and continue, but an RBAC denial never reaches them: the
reconciler holds `mgr.GetClient()` (cmd/main.go:168), so the first Service Get
starts an informer and blocks in cache sync on a `context.TODO()` that nothing
cancels. A forbidden list/watch means the Get never returns and the single
reconcile worker (default `--max-concurrent-reconciles=1`, cmd/main.go:84)
stops entirely, while `/readyz` — `healthz.Ping` — keeps the pod Ready. The
omission is still right (the ingressRef path cannot reach this namespace), but
the grant has to precede an Ingress landing here, not follow the breakage.
crd-endpointmonitors.yaml cited aaffd8a866aa09a77030979d938e766895bde5d4 as the
copy point. That is a master merge commit from 2026-09-25, four months after
the v2.2.13 tag the image is pinned to. Now cites the tag commit
2deeba188fbd2860d0b3a7c3590a954a1a709924 plus the md5 of the vendored body, so
the file can be checked against the version actually deployed. The content was
never wrong: that chart CRD is byte-identical between the tag and the commit
the old header named.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR #997 inserted 9 comment lines into `paperless-ns.yaml` and 21 into
`navidrome/deployment.yaml`. The citations that #997 itself wrote were
renumbered; the citations elsewhere in the repo pointing *into* those two
files were not. REN-116 (merged as a8c48b54) then added one more line to
`paperless-ns.yaml`, shifting everything below it by one again.
Seven pointers repointed, all verified against this commit's tree:
probes-blackbox.yaml:156 navidrome/deployment.yaml 61-74 -> 80-93
probes-blackbox.yaml:218 paperless-ns.yaml 24-26 -> 34-36
probes-blackbox.yaml:221 paperless/network-policy 8-12 -> 22-26
gatus/configmap.yaml:47 paperless-ns.yaml 24-26 -> 34-36
paperless/network-policy.yaml:15 paperless-ns.yaml 30-32 -> 31-33
paperless/network-policy.yaml:16 paperless-ns.yaml 33-35 -> 34-36
paperless/network-policy.yaml:17 paperless-ns.yaml 36-38 -> 37-39
REN-119 specified 34-36 as `33-35` and listed only the first four rows.
Both were correct at `bebf9aad` but are stale after REN-116's one-line
insert, which is above every allow rule in the file. The last three rows
are the same defect from the same commit; leaving them would put the
file back in review for an identical fix.
`paperless-ns.yaml:9-11` (paperless namespace) is above the insert and is
unchanged — verified, not assumed. `navidrome/lidarr.yaml:114-127`,
`lidarr.yaml:29` and `probes-blackbox.yaml:250` were re-checked and are
still correct.
Comments only, and nothing is applied to the cluster. Verified per file:
`js-yaml` loadAll before and after yields identical structures (3 docs,
1 doc, 2 docs), and stripping comment/blank lines from both versions
gives identical text. `gatus/configmap.yaml:47` is in the header above
`data:` (line 62), outside the `config.yaml: |` block scalar, so
ConfigMap `data["config.yaml"]` stays byte-identical at 6572 bytes and
`configmap.reloader.stakater.com/reload: gatus-config` does not roll the
gatus Deployment. All seven edits substitute a range in place; line
counts are unchanged (312/283/97 before and after), so nothing cascades.
Revert: `git revert <this commit>`. Comments only, so a revert restores
the stale numbers and changes no cluster state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The proxy-grant paragraph named `prometheus-read-rbac.yaml` and
`helmrelease-prometheus.yaml` by bare filename. Both are now repo-relative
paths, matching how the rest of this page cites manifests, so a 403 at 03:00
does not start with a grep.
Docs-only. No manifest changed. `docs/` is outside every Flux Kustomization
`spec.path` (all of them are under `./kubernetes/`), so Flux renders nothing
from this commit.
Follow-up to Argus's non-blocking note 2 on REN-113/REN-47.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
