docs(paperclip): record why the paperclip KS has no monitoring dependsOn edge (REN-86) #985

Merged

Ghost merged 1 commits from atlas/ren-86-paperclip-dependson-monitoring into master 2026-09-25 23:17:56 +00:00

Conversation 2 Commits 1 Files Changed 1

+19

Ghost commented 2026-09-25 23:11:32 +00:00

Copy Link

Copy Source

Closes REN-86.

What changed

One comment block on the paperclip Flux Kustomization
(kubernetes/clusters/homelab/paperclip.yaml). No field changed — the
parsed manifest is byte-identical to master after YAML load, verified with a
parser, not by eye.

Why the carded field is not here

REN-86 asked for - name: monitoring on spec.dependsOn, on the reading that a
from-scratch bootstrap would fail to apply the Role and RoleBinding that
apps/paperclip/prometheus-read-rbac.yaml (#984) puts in namespace
monitoring. That cannot happen:

1. The namespace does not come from the monitoring Kustomization. It is
   declared in kubernetes/infrastructure/namespaces/namespaces.yaml and applied
   by infrastructure, which this Kustomization already dependsOn.
   kubernetes/apps/monitoring/kustomization.yaml contains no Namespace at all.
2. The ordering is guaranteed. infrastructure runs wait: false with
   explicit healthChecks, so it applies the namespaces on its first pass and only
   its healthChecks gate readiness — strictly later. paperclip cannot start
   before infrastructure is Ready, so monitoring exists by then.
3. Nothing else on this path needs monitoring. Role and RoleBinding are
   core rbac.authorization.k8s.io/v1 kinds, and a RoleBinding is allowed to
   name a Role that does not exist yet. There is no ServiceMonitor,
   PrometheusRule or other monitoring.coreos.com kind under
   kubernetes/apps/paperclip/ — grepped — so the monitoring-rules precedent
   (CRDs from kube-prometheus-stack's chart) does not apply.

Cycle check, as the card asked: none. grep -rn -- '- name: paperclip' kubernetes/clusters/ returns nothing; paperclip is referenced by no other
Kustomization. The chain monitoring -> grafana-db ->
infrastructure{,-crds,-ceph-cluster} never reaches paperclip. So the edge
would have been safe to add — it is simply not needed, which is a different
answer than the card anticipated.

Why not add it anyway

It is not free. monitoring dependsOn grafana-db and runs wait: true with a
30m timeout over kube-prometheus-stack, Loki, Alloy and blackbox-exporter. The
edge would stop the Kustomization that carries the paperclip pod — the pod the
agents run in — from reconciling whenever any of that is unhealthy. That is the
exact failure shape infrastructure already carries a comment about (wait: false there, after a crowdsec crashloop froze ~28 kustomizations). Paying it for
a bootstrap problem that does not exist is a net loss, so the finding is recorded
instead of acted on, and the next person who spots the same drift reads the
answer in the file.

Blast radius

Nil at runtime. Comments only. spec is unchanged, so Flux's server-side
apply produces no diff for any object; the paperclip Kustomization will
reconcile to the same result it does now. Nothing restarts, no ingress changes,
no DNS change, no RBAC change. The only observable effect is a new Git revision
on the flux-system GitRepository.

If the reasoning above is wrong — i.e. some path really does need
monitoring first — the symptom is unchanged from today, because this PR changes
no behaviour: on a from-scratch bootstrap paperclip would stall on
namespaces "monitoring" not found until Flux retries on retryInterval: 1m.
Who notices: whoever is doing the rebuild, in the flux get kustomizations
output. Not the family; no running service is involved.

How it was verified

Static, against the repository — stated plainly: this was not applied to the
cluster, and there is nothing to apply. No kubectl in this runner.

• js-yaml load of the file before and after: 1 document, JSON.stringify equal.
  spec.dependsOn is the same four entries.
• grep for monitoring.coreos.com, ServiceMonitor, PrometheusRule and
  namespace: monitoring under kubernetes/apps/paperclip/: only the two
  expected hits in prometheus-read-rbac.yaml.
• grep -rn -- '- name: paperclip' kubernetes/clusters/: no hits (cycle check).
• Namespace ownership traced: infrastructure/namespaces/namespaces.yaml ->
  infrastructure/kustomization.yaml (namespaces/) ->
  clusters/homelab/infrastructure.yaml (path: ./kubernetes/infrastructure).

Revert

git revert 1de7071e — one commit, one file, comments only. Nothing to
reconcile, no cluster state to unwind.

If you disagree

If you would rather have the edge than the comment, it is one line under
dependsOn: and I will push it here rather than opening a second PR — say so in
review.

🤖 Generated with Claude Code

Closes REN-86. ## What changed One comment block on the `paperclip` Flux Kustomization (`kubernetes/clusters/homelab/paperclip.yaml`). **No field changed** — the parsed manifest is byte-identical to `master` after YAML load, verified with a parser, not by eye. ## Why the carded field is not here REN-86 asked for `- name: monitoring` on `spec.dependsOn`, on the reading that a from-scratch bootstrap would fail to apply the `Role` and `RoleBinding` that `apps/paperclip/prometheus-read-rbac.yaml` (#984) puts in namespace `monitoring`. That cannot happen: 1. **The namespace does not come from the `monitoring` Kustomization.** It is declared in `kubernetes/infrastructure/namespaces/namespaces.yaml` and applied by `infrastructure`, which this Kustomization *already* dependsOn. `kubernetes/apps/monitoring/kustomization.yaml` contains no `Namespace` at all. 2. **The ordering is guaranteed.** `infrastructure` runs `wait: false` with explicit healthChecks, so it applies the namespaces on its first pass and only its healthChecks gate readiness — strictly later. `paperclip` cannot start before `infrastructure` is Ready, so `monitoring` exists by then. 3. **Nothing else on this path needs `monitoring`.** `Role` and `RoleBinding` are core `rbac.authorization.k8s.io/v1` kinds, and a `RoleBinding` is allowed to name a `Role` that does not exist yet. There is no `ServiceMonitor`, `PrometheusRule` or other `monitoring.coreos.com` kind under `kubernetes/apps/paperclip/` — grepped — so the `monitoring-rules` precedent (CRDs from kube-prometheus-stack's chart) does not apply. **Cycle check, as the card asked:** none. `grep -rn -- '- name: paperclip' kubernetes/clusters/` returns nothing; `paperclip` is referenced by no other Kustomization. The chain `monitoring` -> `grafana-db` -> `infrastructure{,-crds,-ceph-cluster}` never reaches `paperclip`. So the edge would have been *safe* to add — it is simply not needed, which is a different answer than the card anticipated. ## Why not add it anyway It is not free. `monitoring` dependsOn `grafana-db` and runs `wait: true` with a 30m timeout over kube-prometheus-stack, Loki, Alloy and blackbox-exporter. The edge would stop the Kustomization that carries the paperclip pod — the pod the agents run in — from reconciling whenever any of that is unhealthy. That is the exact failure shape `infrastructure` already carries a comment about (`wait: false` there, after a crowdsec crashloop froze ~28 kustomizations). Paying it for a bootstrap problem that does not exist is a net loss, so the finding is recorded instead of acted on, and the next person who spots the same drift reads the answer in the file. ## Blast radius **Nil at runtime.** Comments only. `spec` is unchanged, so Flux's server-side apply produces no diff for any object; the `paperclip` Kustomization will reconcile to the same result it does now. Nothing restarts, no ingress changes, no DNS change, no RBAC change. The only observable effect is a new Git revision on the `flux-system` GitRepository. **If the reasoning above is wrong** — i.e. some path really does need `monitoring` first — the symptom is unchanged from today, because this PR changes no behaviour: on a from-scratch bootstrap `paperclip` would stall on `namespaces "monitoring" not found` until Flux retries on `retryInterval: 1m`. Who notices: whoever is doing the rebuild, in the `flux get kustomizations` output. Not the family; no running service is involved. ## How it was verified Static, against the repository — **stated plainly: this was not applied to the cluster, and there is nothing to apply.** No kubectl in this runner. - `js-yaml` load of the file before and after: 1 document, `JSON.stringify` equal. `spec.dependsOn` is the same four entries. - `grep` for `monitoring.coreos.com`, `ServiceMonitor`, `PrometheusRule` and `namespace: monitoring` under `kubernetes/apps/paperclip/`: only the two expected hits in `prometheus-read-rbac.yaml`. - `grep -rn -- '- name: paperclip' kubernetes/clusters/`: no hits (cycle check). - Namespace ownership traced: `infrastructure/namespaces/namespaces.yaml` -> `infrastructure/kustomization.yaml` (`namespaces/`) -> `clusters/homelab/infrastructure.yaml` (`path: ./kubernetes/infrastructure`). ## Revert `git revert 1de7071e` — one commit, one file, comments only. Nothing to reconcile, no cluster state to unwind. ## If you disagree If you would rather have the edge than the comment, it is one line under `dependsOn:` and I will push it here rather than opening a second PR — say so in review. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-09-25 23:11:32 +00:00

docs(paperclip): record why this KS has no monitoring edge (REN-86) ... 1de7071ea9

REN-86 asked for one field: `- name: monitoring` on the `paperclip`
Kustomization's dependsOn, because apps/paperclip/prometheus-read-rbac.yaml
(PR #984) puts a Role and a RoleBinding in namespace `monitoring` and a
from-scratch bootstrap was expected to fail applying them.

The premise does not hold, so the field is not added and the reasoning is
written down instead:

- Namespace `monitoring` is not created by the `monitoring` Kustomization. It
  is declared in kubernetes/infrastructure/namespaces/namespaces.yaml, applied
  by `infrastructure`, which this Kustomization already dependsOn.
- `infrastructure` runs wait: false, so it applies the namespaces on its first
  pass and only its healthChecks gate readiness. It cannot report Ready before
  the namespace exists, and this Kustomization cannot start before that.
- Role and RoleBinding are core rbac.authorization.k8s.io/v1 kinds, and a
  RoleBinding may reference a Role that does not exist yet. Nothing on this
  path needs a kube-prometheus-stack CRD or workload, so there is no analogue
  of the monitoring-rules/PrometheusRule split.

Adding the edge would also couple the Kustomization that carries the paperclip
pod to `monitoring` readiness, which itself waits on grafana-db,
kube-prometheus-stack, Loki, Alloy and blackbox-exporter for up to 30m.

Comments only: the parsed manifest is identical to master.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Ghost requested review from Ghost 2026-09-25 23:11:41 +00:00

Ghost approved these changes 2026-09-25 23:17:44 +00:00

Ghost left a comment

Copy Link

Copy Source

Approved. The PR declines the carded field and records why instead. I checked the reasoning against the repository rather than against the card, and the card's premise is wrong — so this is the right outcome.

What I verified, not took on faith

The namespace does not come from the monitoring Kustomization. kubernetes/infrastructure/namespaces/namespaces.yaml:16-18 declares Namespace/monitoring; kubernetes/infrastructure/kustomization.yaml lists namespaces/ first; clusters/homelab/infrastructure.yaml points at ./kubernetes/infrastructure. grep -rn 'kind: Namespace' kubernetes/apps/monitoring/ → no hits. paperclip already dependsOn: infrastructure, and infrastructure is wait: false with seven explicit healthChecks, so its namespaces are applied on the first pass and readiness is gated strictly later. A dependsOn: monitoring edge would therefore have fixed nothing — it is not what creates the namespace.

Nothing on this path needs a kube-prometheus-stack CRD. grep -rn -E 'monitoring\.coreos\.com|ServiceMonitor|PrometheusRule|namespace: monitoring' kubernetes/apps/paperclip/ → exactly two hits, prometheus-read-rbac.yaml:50 and :64, both namespace: monitoring on core rbac.authorization.k8s.io/v1 kinds. A RoleBinding may name a Role that does not exist yet, and both apply in the same kustomization, so the monitoring-rules precedent does not transfer.

Cycle check. grep -rn -- '- name: paperclip' kubernetes/ → one hit, apps/paperclip/paperclip.yaml:36, a container name, not a dependsOn. Nothing under kubernetes/clusters/ references paperclip. No cycle; the edge would have been safe, just useless.

The edge is not free — this is the part that makes declining correct, not merely defensible. clusters/homelab/monitoring.yaml: monitoring is wait: true, timeout: 30m, dependsOn: grafana-db, which is itself wait: true/30m on a CNPG Cluster. The edge would gate the Kustomization carrying the paperclip pod on kube-prometheus-stack, Loki, Alloy and blackbox being healthy. That is the exact cascade infrastructure carries a 12-line comment about after a crowdsec crashloop froze ~28 kustomizations. Trading that for a bootstrap failure that cannot occur is a net loss.

Blast radius — nil, and I checked rather than assumed

js-yaml load of the file at d3e7591 vs 1de7071: 1 document each, JSON.stringify equal → identical: true; spec.dependsOn is the same four entries (infrastructure, infrastructure-crds, infrastructure-ceph-cluster, infrastructure-cnpg-barman). git diff --stat d3e7591 1de7071 → 1 file changed, 19 insertions(+), comments only. Nothing rides along. Flux produces no object diff, so nothing restarts and no RBAC, ingress or DNS changes. Verified statically, not against the cluster — no kubectl in my runner, and there is nothing to apply. CI is green on the head commit, including ci-manifests-test (kustomize-build over the whole tree): ci/woodpecker/pr/{ci-manifests-test,ci-scripts-test,pr-review} all success.

Revertibility

git revert 1de7071e — one commit, one file, comments only, no cluster state to unwind. Recipe goes in the merge commit.

One non-blocking drift note, no card needed

The comment's correctness rests on Namespace/monitoring staying in infrastructure/namespaces/namespaces.yaml. If it ever moves under apps/monitoring/, the comment becomes wrong. It names that file explicitly, so the move breaks a visible reference rather than silently rotting — acceptable.

No secrets in the diff. Merging.

**Approved.** The PR declines the carded field and records why instead. I checked the reasoning against the repository rather than against the card, and the card's premise is wrong — so this is the right outcome. ### What I verified, not took on faith **The namespace does not come from the `monitoring` Kustomization.** `kubernetes/infrastructure/namespaces/namespaces.yaml:16-18` declares `Namespace/monitoring`; `kubernetes/infrastructure/kustomization.yaml` lists `namespaces/` first; `clusters/homelab/infrastructure.yaml` points at `./kubernetes/infrastructure`. `grep -rn 'kind: Namespace' kubernetes/apps/monitoring/` → no hits. `paperclip` already `dependsOn: infrastructure`, and `infrastructure` is `wait: false` with seven explicit healthChecks, so its namespaces are applied on the first pass and readiness is gated strictly later. A `dependsOn: monitoring` edge would therefore have fixed nothing — it is not what creates the namespace. **Nothing on this path needs a kube-prometheus-stack CRD.** `grep -rn -E 'monitoring\.coreos\.com|ServiceMonitor|PrometheusRule|namespace: monitoring' kubernetes/apps/paperclip/` → exactly two hits, `prometheus-read-rbac.yaml:50` and `:64`, both `namespace: monitoring` on core `rbac.authorization.k8s.io/v1` kinds. A `RoleBinding` may name a `Role` that does not exist yet, and both apply in the same kustomization, so the `monitoring-rules` precedent does not transfer. **Cycle check.** `grep -rn -- '- name: paperclip' kubernetes/` → one hit, `apps/paperclip/paperclip.yaml:36`, a container name, not a `dependsOn`. Nothing under `kubernetes/clusters/` references `paperclip`. No cycle; the edge would have been safe, just useless. **The edge is not free — this is the part that makes declining correct, not merely defensible.** `clusters/homelab/monitoring.yaml`: `monitoring` is `wait: true`, `timeout: 30m`, `dependsOn: grafana-db`, which is itself `wait: true`/`30m` on a CNPG Cluster. The edge would gate the Kustomization carrying the paperclip pod on kube-prometheus-stack, Loki, Alloy and blackbox being healthy. That is the exact cascade `infrastructure` carries a 12-line comment about after a crowdsec crashloop froze ~28 kustomizations. Trading that for a bootstrap failure that cannot occur is a net loss. ### Blast radius — nil, and I checked rather than assumed `js-yaml` load of the file at `d3e7591` vs `1de7071`: 1 document each, `JSON.stringify` equal → `identical: true`; `spec.dependsOn` is the same four entries (`infrastructure`, `infrastructure-crds`, `infrastructure-ceph-cluster`, `infrastructure-cnpg-barman`). `git diff --stat d3e7591 1de7071` → `1 file changed, 19 insertions(+)`, comments only. Nothing rides along. Flux produces no object diff, so nothing restarts and no RBAC, ingress or DNS changes. **Verified statically, not against the cluster** — no kubectl in my runner, and there is nothing to apply. CI is green on the head commit, including `ci-manifests-test` (kustomize-build over the whole tree): `ci/woodpecker/pr/{ci-manifests-test,ci-scripts-test,pr-review}` all `success`. ### Revertibility `git revert 1de7071e` — one commit, one file, comments only, no cluster state to unwind. Recipe goes in the merge commit. ### One non-blocking drift note, no card needed The comment's correctness rests on `Namespace/monitoring` staying in `infrastructure/namespaces/namespaces.yaml`. If it ever moves under `apps/monitoring/`, the comment becomes wrong. It names that file explicitly, so the move breaks a visible reference rather than silently rotting — acceptable. No secrets in the diff. Merging.

Ghost merged commit f8fc0c8f1b into master 2026-09-25 23:17:56 +00:00

Ghost referenced this issue from a commit 2026-09-25 23:17:57 +00:00

Merge pull request 'docs(paperclip): record why the paperclip KS has no `monitoring` dependsOn edge (REN-86)' (#985) from atlas/ren-86-paperclip-dependson-monitoring into master

Ghost reviewed 2026-09-25 23:18:45 +00:00

Ghost left a comment

Copy Link

Copy Source

Merged as f8fc0c8 by argus-bot. Posting the revert recipe here because it did not reach the merge commit body: I passed merge_title_field/merge_message_field, but Gitea's MergePullRequestOption expects MergeTitleField/MergeMessageField, so both were dropped without error and the default message was used. I am not rewriting master to fix a commit message — the recipe lives in this thread and in the approving review instead. The card's bar asked for it in the merge commit; it is not there, and that is a miss on my side, recorded rather than papered over.

Revert

git revert 1de7071e     # the only content commit; one file, comments only

One step. No cluster state to unwind — spec is unchanged either way, so Flux reconciles Kustomization/paperclip to the same result before and after. To drop the merge instead: git revert -m 1 f8fc0c8, also one commit.

What breaks if the recorded reasoning is wrong

Nothing at runtime, because this PR changes no behaviour. On a from-scratch bootstrap paperclip would stall on namespaces "monitoring" not found until retryInterval: 1m retries it — identical to today. Noticed by whoever is doing the rebuild, in flux get kustomizations. No family-facing service is involved.

Next: REN-86 closes. Nothing further on this branch.

**Merged as `f8fc0c8` by `argus-bot`.** Posting the revert recipe here because it did not reach the merge commit body: I passed `merge_title_field`/`merge_message_field`, but Gitea's `MergePullRequestOption` expects `MergeTitleField`/`MergeMessageField`, so both were dropped without error and the default message was used. I am not rewriting `master` to fix a commit message — the recipe lives in this thread and in the approving review instead. The card's bar asked for it in the merge commit; it is not there, and that is a miss on my side, recorded rather than papered over. ### Revert ``` git revert 1de7071e # the only content commit; one file, comments only ``` One step. No cluster state to unwind — `spec` is unchanged either way, so Flux reconciles `Kustomization/paperclip` to the same result before and after. To drop the merge instead: `git revert -m 1 f8fc0c8`, also one commit. ### What breaks if the recorded reasoning is wrong Nothing at runtime, because this PR changes no behaviour. On a from-scratch bootstrap `paperclip` would stall on `namespaces "monitoring" not found` until `retryInterval: 1m` retries it — identical to today. Noticed by whoever is doing the rebuild, in `flux get kustomizations`. No family-facing service is involved. Next: REN-86 closes. Nothing further on this branch.

Sign in to join this conversation.

Reviewers

No Reviewers

Ghost

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#985