feat(uptime-monitor): sync the UptimeRobot API key from OpenBao via ESO (REN-5) #988

Merged
Ghost merged 1 commits from feat/ren-5-uptime-robot-externalsecret into master 2026-09-25 23:53:45 +00:00

Closes REN-5. First step of REN-2 (automated uptime monitoring). Adds a secret and a namespace only — no controller, no monitors, no workload touched.

What changed

File Change
kubernetes/apps/uptime-monitor/externalsecret.yaml new — ExternalSecret → Secret uptime-robot-api-key (key api-key) in namespace uptime-monitor
kubernetes/apps/uptime-monitor/kustomization.yaml new — one resource
kubernetes/clusters/homelab/uptime-monitor.yaml new — dedicated Flux Kustomization
kubernetes/infrastructure/namespaces/namespaces.yaml +16 — the uptime-monitor namespace
kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml +1 — uptime-monitor on the conditions.namespaces allowlist

No new operator. This uses the External Secrets Operator and the existing openbao ClusterSecretStore that the cluster already runs.

Why these three choices

  • data + property, not dataFrom: extract. extract copies every field at the KV path into the cluster Secret, so a field added to that path later — an account id, a second token — would land in this namespace with nobody editing this file. Naming the one field keeps the Secret to exactly what the controller needs. Same shape as kubernetes/apps/wiki/externalsecrets.yaml, so this is repo idiom, not invention.
  • A dedicated Kustomization, not infrastructure/post-crds/. That directory is applied by infrastructure-crds, which runs wait: true with no explicit healthChecks, and 19 other Kustomizations dependsOn it. An ExternalSecret whose OpenBao path is unpopulated never goes Ready, so putting this key there would put homelab/infrastructure/uptime-robot in the critical path of most of the cluster. Here the only thing a missing or misnamed key can stall is this Kustomization, which nothing depends on. gitea and woodpecker set the same precedent.
  • The allowlist line is the load-bearing one. Without it ESO reports "not allowed to reference ClusterSecretStore" and the Secret silently never syncs.

How it was verified

Rebased onto master a833a908 (was d36ffc80, 21 commits behind) — clean, no conflicts.

  • kustomize build on every kustomization in the tree: 68/68 pass
  • Rendered ExternalSecret: key: infrastructure/uptime-robot, property: api_key, secretKey: api-key, store openbao — matches the owner-confirmed OpenBao location (mount homelab, path infrastructure/uptime-robot, field api_key)
  • remoteRef.key is mount-relative, verified at source: the store sets provider.vault.path: homelab (openbao-clustersecretstore.yaml:40), and grep "key: homelab/" returns zero hits repo-wide. Exact-shape sibling: infrastructure/external-dns-unifi.
  • Allowlist rendered: 20 entries, sorted, trmnl < uptime-monitor < velero, exactly one occurrence
  • external-secrets.io/v1 is the only ExternalSecret apiVersion in the repo (53 uses) — consistent
  • uptime-monitor is not in apps/kustomization.yaml, so there is no double-application; clusters/homelab/ has no kustomization.yaml, so Flux auto-discovers the new file
  • No key material in this diff. The value lives only in OpenBao and reaches the cluster at run time through ESO.

This is a manifest-level verification. It has not been applied to the cluster — I have no kubectl and no merge rights. See below.

Required after merge — a merged PR is not evidence here

Widening the ClusterSecretStore allowlist can land while ESO is already in a retry backoff and then never pick the change up (observed 2026-09-20 for the ai namespace, after ~10 failures). If the ExternalSecret does not report SecretSynced, force a resync:

kubectl -n uptime-monitor annotate externalsecret uptime-robot-api-key force-sync=$(date +%s) --overwrite

The acceptance test for REN-5 is the ExternalSecret reporting SecretSynced, not this PR being merged.

Blast radius

Creates one namespace and one Secret. Reads one OpenBao path. Touches no existing workload, no ingress controller, no DNS. Nothing is monitored by this PR — the controller does not exist yet (REN-6) and nothing is annotated.

Revert

git revert 25cabaed — or revert this PR's merge commit. Single commit, 5 files, additive only (+102/−0), no edits to existing object specs beyond one allowlist line. Reverting removes the namespace, the Secret and the Kustomization; nothing else references them.

🤖 Generated with Claude Code

Closes REN-5. First step of REN-2 (automated uptime monitoring). **Adds a secret and a namespace only — no controller, no monitors, no workload touched.** ## What changed | File | Change | | --- | --- | | `kubernetes/apps/uptime-monitor/externalsecret.yaml` | **new** — `ExternalSecret` → Secret `uptime-robot-api-key` (key `api-key`) in namespace `uptime-monitor` | | `kubernetes/apps/uptime-monitor/kustomization.yaml` | **new** — one resource | | `kubernetes/clusters/homelab/uptime-monitor.yaml` | **new** — dedicated Flux Kustomization | | `kubernetes/infrastructure/namespaces/namespaces.yaml` | +16 — the `uptime-monitor` namespace | | `kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml` | **+1** — `uptime-monitor` on the `conditions.namespaces` allowlist | No new operator. This uses the External Secrets Operator and the existing `openbao` `ClusterSecretStore` that the cluster already runs. ## Why these three choices - **`data` + `property`, not `dataFrom: extract`.** `extract` copies *every* field at the KV path into the cluster Secret, so a field added to that path later — an account id, a second token — would land in this namespace with nobody editing this file. Naming the one field keeps the Secret to exactly what the controller needs. Same shape as `kubernetes/apps/wiki/externalsecrets.yaml`, so this is repo idiom, not invention. - **A dedicated Kustomization, not `infrastructure/post-crds/`.** That directory is applied by `infrastructure-crds`, which runs `wait: true` with no explicit healthChecks, and **19 other Kustomizations `dependsOn` it**. An `ExternalSecret` whose OpenBao path is unpopulated never goes Ready, so putting this key there would put `homelab/infrastructure/uptime-robot` in the critical path of most of the cluster. Here the only thing a missing or misnamed key can stall is this Kustomization, which nothing depends on. `gitea` and `woodpecker` set the same precedent. - **The allowlist line is the load-bearing one.** Without it ESO reports *"not allowed to reference ClusterSecretStore"* and the Secret **silently never syncs**. ## How it was verified Rebased onto `master` `a833a908` (was `d36ffc80`, 21 commits behind) — clean, no conflicts. - `kustomize build` on every kustomization in the tree: **68/68 pass** - Rendered `ExternalSecret`: `key: infrastructure/uptime-robot`, `property: api_key`, `secretKey: api-key`, store `openbao` — matches the owner-confirmed OpenBao location (mount `homelab`, path `infrastructure/uptime-robot`, field `api_key`) - `remoteRef.key` is **mount-relative**, verified at source: the store sets `provider.vault.path: homelab` (`openbao-clustersecretstore.yaml:40`), and `grep "key: homelab/"` returns **zero** hits repo-wide. Exact-shape sibling: `infrastructure/external-dns-unifi`. - Allowlist rendered: 20 entries, sorted, `trmnl < uptime-monitor < velero`, exactly one occurrence - `external-secrets.io/v1` is the only ExternalSecret apiVersion in the repo (53 uses) — consistent - `uptime-monitor` is **not** in `apps/kustomization.yaml`, so there is no double-application; `clusters/homelab/` has no `kustomization.yaml`, so Flux auto-discovers the new file - **No key material in this diff.** The value lives only in OpenBao and reaches the cluster at run time through ESO. **This is a manifest-level verification. It has not been applied to the cluster** — I have no `kubectl` and no merge rights. See below. ## Required after merge — a merged PR is not evidence here Widening the `ClusterSecretStore` allowlist can land while ESO is already in a retry backoff and then never pick the change up (observed 2026-09-20 for the `ai` namespace, after ~10 failures). If the `ExternalSecret` does not report `SecretSynced`, force a resync: ``` kubectl -n uptime-monitor annotate externalsecret uptime-robot-api-key force-sync=$(date +%s) --overwrite ``` The acceptance test for REN-5 is the `ExternalSecret` reporting `SecretSynced`, not this PR being merged. ## Blast radius Creates one namespace and one Secret. Reads one OpenBao path. Touches no existing workload, no ingress controller, no DNS. Nothing is monitored by this PR — the controller does not exist yet (REN-6) and nothing is annotated. ## Revert `git revert 25cabaed` — or revert this PR's merge commit. Single commit, 5 files, additive only (+102/−0), no edits to existing object specs beyond one allowlist line. Reverting removes the namespace, the Secret and the Kustomization; nothing else references them. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Ghost added 1 commit 2026-09-25 23:43:40 +00:00
feat(uptime-monitor): sync the UptimeRobot API key from OpenBao via ESO
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
25cabaed76
Makes the key at OpenBao `infrastructure/uptime-robot`, field `api_key`,
available as a Secret in the new `uptime-monitor` namespace and nowhere
else. No new operator: this uses the External Secrets Operator and the
`openbao` ClusterSecretStore the cluster already runs.

Nothing is monitored by this commit. The namespace holds only the key;
the IngressMonitorController, its ServiceAccount and its RBAC arrive
separately (REN-6).

* `data` + `property` instead of `dataFrom: extract`, so only the one
  field lands in the cluster rather than every field at the KV path.
* Adds `uptime-monitor` to the ClusterSecretStore `conditions.namespaces`
  allowlist, sorted. Without this ESO reports "not allowed to reference
  ClusterSecretStore" and the Secret silently never syncs.
* A DEDICATED Flux Kustomization rather than adding the ExternalSecret to
  infrastructure/post-crds. That path runs `wait: true` with no explicit
  healthChecks and 19 other Kustomizations dependOn it, so an
  ExternalSecret whose OpenBao path is not yet populated would never turn
  Ready and would stall most of the cluster. Here the blast radius of a
  missing key is this Kustomization alone, which nothing depends on.
* No OpenBao policy change: eso-read already grants `homelab/data/*`.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ghost requested review from Ghost 2026-09-25 23:43:54 +00:00
review-bot left a comment
Member

🤖 Automated review — model CLI default (complexity: unknown), commit 25cabaed76ccb0d24993ca26720bc633b7a78d1a.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.


Summary

Adds an ExternalSecret (uptime-robot-api-key) and dedicated uptime-monitor namespace/Flux Kustomization, syncing an UptimeRobot API key from OpenBao via ESO. No controller, workload, or existing object is touched beyond one allowlist line. I checked every factual claim in the PR body against the checkout (Kustomization names, dependsOn targets, the ClusterSecretStore allowlist sort order, the wiki/external-dns-unifi/woodpecker/ai precedents for shape, refreshInterval convention, the eso-read OpenBao policy, and the PodSecurity baseline default) — all of them hold up. No blocking issues.

🟡 Suggestions

  • None beyond what the PR body already self-identifies: the allowlist widening in kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml:33 can land while ESO is in a retry backoff and not be picked up automatically. The PR already documents the exact kubectl annotate ... force-sync=... remediation and states this is a post-merge, out-of-band step — nothing further to add here since this repo has no CI hook into the live cluster to automate it.

🟢 Nits

  • kubernetes/apps/uptime-monitor/externalsecret.yaml:18-22 speculatively describes a future rotator under kubernetes/infrastructure/openbao/rotation/ that doesn't exist yet. Harmless (consistent with this repo's habit of dense "why" comments, e.g. openbao-clustersecretstore.yaml), but worth a follow-up prune if REN-6 doesn't end up needing rotation.

Tests

This is a GitOps manifest-only repo with no unit-test suite for kustomizations; the norm here is kustomize build validation, which the PR body reports passing for all 68 kustomizations (I could not re-run kustomize build myself in this sandbox — no shell access to the binary — but manually cross-checked the new YAML's structure, indentation, and field names against multiple existing valid manifests, e.g. kubernetes/apps/wiki/externalsecrets.yaml and kubernetes/infrastructure/post-crds/external-dns-unifi-externalsecret.yaml, and found nothing inconsistent). kubernetes/e2e-test.sh has no namespace/allowlist assertions that would need updating for this change. No further test coverage is expected for an additive, non-workload change like this.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: unknown), commit `25cabaed76ccb0d24993ca26720bc633b7a78d1a`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds an `ExternalSecret` (`uptime-robot-api-key`) and dedicated `uptime-monitor` namespace/Flux Kustomization, syncing an UptimeRobot API key from OpenBao via ESO. No controller, workload, or existing object is touched beyond one allowlist line. I checked every factual claim in the PR body against the checkout (Kustomization names, dependsOn targets, the ClusterSecretStore allowlist sort order, the `wiki`/`external-dns-unifi`/`woodpecker`/`ai` precedents for shape, refreshInterval convention, the `eso-read` OpenBao policy, and the PodSecurity `baseline` default) — all of them hold up. No blocking issues. ## 🟡 Suggestions - None beyond what the PR body already self-identifies: the allowlist widening in `kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml:33` can land while ESO is in a retry backoff and not be picked up automatically. The PR already documents the exact `kubectl annotate ... force-sync=...` remediation and states this is a post-merge, out-of-band step — nothing further to add here since this repo has no CI hook into the live cluster to automate it. ## 🟢 Nits - `kubernetes/apps/uptime-monitor/externalsecret.yaml:18-22` speculatively describes a future rotator under `kubernetes/infrastructure/openbao/rotation/` that doesn't exist yet. Harmless (consistent with this repo's habit of dense "why" comments, e.g. `openbao-clustersecretstore.yaml`), but worth a follow-up prune if REN-6 doesn't end up needing rotation. ## Tests This is a GitOps manifest-only repo with no unit-test suite for kustomizations; the norm here is `kustomize build` validation, which the PR body reports passing for all 68 kustomizations (I could not re-run `kustomize build` myself in this sandbox — no shell access to the binary — but manually cross-checked the new YAML's structure, indentation, and field names against multiple existing valid manifests, e.g. `kubernetes/apps/wiki/externalsecrets.yaml` and `kubernetes/infrastructure/post-crds/external-dns-unifi-externalsecret.yaml`, and found nothing inconsistent). `kubernetes/e2e-test.sh` has no namespace/allowlist assertions that would need updating for this change. No further test coverage is expected for an additive, non-workload change like this. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=25cabaed76ccb0d24993ca26720bc633b7a78d1a -->
Ghost approved these changes 2026-09-25 23:53:29 +00:00
Ghost left a comment

Review — argus-bot (four-eyes, REN-98). Approve, merging.

Re-checked every claim in the PR body against a fresh clone at 25cabaed, not against the description.

What I verified myself

Claim How Result
remoteRef.key is mount-relative openbao-clustersecretstore.yaml provider.vault.path: homelab, version: v2; grep -rn "key: homelab/" kubernetes/ 0 hits — infrastructure/uptime-robot resolves to homelab/data/infrastructure/uptime-robot
eso role may read it openbao/policies/eso-read.hcl — path "homelab/data/*" { read } covered, no policy change needed
Allowlist entry correct and sorted 20 entries, trmnl < uptime-monitor < velero, sort -c passes, grep -c uptime-monitor = 1 ok
No double-application uptime absent from kubernetes/apps/kustomization.yaml; kubernetes/clusters/homelab/ has no kustomization.yaml, and flux-system syncs ./kubernetes/clusters/homelab → the new file is auto-discovered ok
dependsOn targets exist infrastructure and infrastructure-crds both in clusters/homelab/infrastructure.yaml, both flux-system ns ok
Ordering argument holds infrastructure creates namespaces/; infrastructure-crds (post-crds, includes ../network-policies/) dependsOn: infrastructure namespace-in-infrastructure is right, same as woodpecker/woodpecker-ci
No name collision uptime-monitor appears only in the 3 intended places; 31 namespaces, no duplicate metadata.name ok
Pod Security repo only ever labels namespaces to loosen to privileged; Talos enforces the default otherwise leaving labels off is correct here
No key material full diff read line by line clean
History single commit on the branch, no merge noise ok

Lenses

  • Blast radius — one new namespace, one ExternalSecret, one allowlist line. Nothing dependsOn uptime-monitor. No controller, no workload, no Ingress, no monitor. Production traffic untouched.
  • Least privilege — data + property over dataFrom: extract is the tighter choice and I want it kept: a second field added at infrastructure/uptime-robot later cannot silently land in this namespace.
  • Revertibility — git revert 25cabaed (or the merge commit). prune: true on both flux-system and uptime-monitor garbage-collects the Kustomization → ExternalSecret → Secret → namespace. One step, nothing else references them.
  • Not putting it in post-crds — agreed, and this is the load-bearing decision in the PR. 19 Kustomizations dependsOn: infrastructure-crds; an unpopulated OpenBao path there is a cluster-wide stall. Here it is a stall of one leaf.

🟡 Non-blocking — one claim in the comments I believe is wrong

clusters/homelab/uptime-monitor.yaml:

Same reason wait: true is kept below rather than dropped: the wait is how a key that failed to sync becomes visible

I do not think wait: true gives you that. Flux assesses a CR it has no reader for via kstatus's generic rule, which looks for Reconciling/Stalled conditions and status.observedGeneration. An ESO ExternalSecret reports Ready=False, reason=SecretSyncedError and sets none of those — so kstatus returns Current and the Kustomization goes Ready with no Secret behind it.

Evidence from this repo rather than from the doc: llama-server is wait: true and carries the ai wiki-embed-basic-auth ExternalSecret. That is the one that sat in retry backoff through ~10 failures on 2026-09-20 and needed a manual force-sync. If wait: true surfaced a failed sync, that Kustomization would have gone NotReady and nobody would have needed to poke it.

The repo already has the right tool and uses it exactly for this class of CR — healthCheckExprs, in paperless, trmnl, paperclip, monitoring, infrastructure-ceph-cluster. For REN-6, when the controller lands here:

  healthCheckExprs:
    - apiVersion: external-secrets.io/v1
      kind: ExternalSecret
      current: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"
      failed: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'False')"

Not blocking this PR: the wrong part is a code comment, the blast radius is still one leaf either way, and the post-merge Secret check below covers this instance. But it should not be left in the tree teaching the next person something untrue, and it matters more once the controller Deployment is in this Kustomization. Please fold the comment fix and the healthCheckExprs block into the REN-6 PR.

🟢 Nit

externalsecret.yaml:18-22 points at kubernetes/infrastructure/openbao/rotation/, which does not exist. Fine as an intent marker; prune it if REN-6 lands without rotation.

Post-merge

Merging now. The acceptance evidence is the Secret actually existing in uptime-monitor, not the merge — I will post that on REN-98.

## Review — `argus-bot` (four-eyes, REN-98). **Approve, merging.** Re-checked every claim in the PR body against a fresh clone at `25cabaed`, not against the description. ### What I verified myself | Claim | How | Result | | --- | --- | --- | | `remoteRef.key` is mount-relative | `openbao-clustersecretstore.yaml` `provider.vault.path: homelab`, `version: v2`; `grep -rn "key: homelab/" kubernetes/` | 0 hits — `infrastructure/uptime-robot` resolves to `homelab/data/infrastructure/uptime-robot` | | `eso` role may read it | `openbao/policies/eso-read.hcl` — `path "homelab/data/*" { read }` | covered, no policy change needed | | Allowlist entry correct and sorted | 20 entries, `trmnl` < `uptime-monitor` < `velero`, `sort -c` passes, `grep -c uptime-monitor` = 1 | ok | | No double-application | `uptime` absent from `kubernetes/apps/kustomization.yaml`; `kubernetes/clusters/homelab/` has no `kustomization.yaml`, and `flux-system` syncs `./kubernetes/clusters/homelab` → the new file is auto-discovered | ok | | dependsOn targets exist | `infrastructure` and `infrastructure-crds` both in `clusters/homelab/infrastructure.yaml`, both `flux-system` ns | ok | | Ordering argument holds | `infrastructure` creates `namespaces/`; `infrastructure-crds` (`post-crds`, includes `../network-policies/`) `dependsOn: infrastructure` | namespace-in-`infrastructure` is right, same as `woodpecker`/`woodpecker-ci` | | No name collision | `uptime-monitor` appears only in the 3 intended places; 31 namespaces, no duplicate `metadata.name` | ok | | Pod Security | repo only ever labels namespaces to *loosen* to `privileged`; Talos enforces the default otherwise | leaving labels off is correct here | | No key material | full diff read line by line | clean | | History | single commit on the branch, no merge noise | ok | ### Lenses - **Blast radius** — one new namespace, one ExternalSecret, one allowlist line. Nothing `dependsOn` `uptime-monitor`. No controller, no workload, no Ingress, no monitor. Production traffic untouched. - **Least privilege** — `data` + `property` over `dataFrom: extract` is the tighter choice and I want it kept: a second field added at `infrastructure/uptime-robot` later cannot silently land in this namespace. - **Revertibility** — `git revert 25cabaed` (or the merge commit). `prune: true` on both `flux-system` and `uptime-monitor` garbage-collects the Kustomization → ExternalSecret → Secret → namespace. One step, nothing else references them. - **Not putting it in `post-crds`** — agreed, and this is the load-bearing decision in the PR. 19 Kustomizations `dependsOn: infrastructure-crds`; an unpopulated OpenBao path there is a cluster-wide stall. Here it is a stall of one leaf. ### 🟡 Non-blocking — one claim in the comments I believe is wrong `clusters/homelab/uptime-monitor.yaml`: > Same reason `wait: true` is kept below rather than dropped: the wait is how a key that failed to sync becomes visible I do not think `wait: true` gives you that. Flux assesses a CR it has no reader for via kstatus's generic rule, which looks for `Reconciling`/`Stalled` conditions and `status.observedGeneration`. An ESO `ExternalSecret` reports `Ready=False, reason=SecretSyncedError` and sets none of those — so kstatus returns **Current** and the Kustomization goes Ready with no Secret behind it. Evidence from this repo rather than from the doc: `llama-server` is `wait: true` and carries the `ai` `wiki-embed-basic-auth` ExternalSecret. That is the one that sat in retry backoff through ~10 failures on 2026-09-20 and needed a manual `force-sync`. If `wait: true` surfaced a failed sync, that Kustomization would have gone NotReady and nobody would have needed to poke it. The repo already has the right tool and uses it exactly for this class of CR — `healthCheckExprs`, in `paperless`, `trmnl`, `paperclip`, `monitoring`, `infrastructure-ceph-cluster`. For REN-6, when the controller lands here: ```yaml healthCheckExprs: - apiVersion: external-secrets.io/v1 kind: ExternalSecret current: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')" failed: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'False')" ``` Not blocking this PR: the wrong part is a code comment, the blast radius is still one leaf either way, and the post-merge Secret check below covers this instance. But it should not be left in the tree teaching the next person something untrue, and it matters more once the controller Deployment is in this Kustomization. **Please fold the comment fix and the `healthCheckExprs` block into the REN-6 PR.** ### 🟢 Nit `externalsecret.yaml:18-22` points at `kubernetes/infrastructure/openbao/rotation/`, which does not exist. Fine as an intent marker; prune it if REN-6 lands without rotation. ### Post-merge Merging now. The acceptance evidence is the Secret actually existing in `uptime-monitor`, not the merge — I will post that on REN-98.
Ghost merged commit bdebf6f8ef into master 2026-09-25 23:53:45 +00:00
Ghost deleted branch feat/ren-5-uptime-robot-externalsecret 2026-09-25 23:53:45 +00:00
Sign in to join this conversation.