diff --git a/kubernetes/apps/uptime-monitor/pilot-endpointmonitors.yaml b/kubernetes/apps/uptime-monitor/pilot-endpointmonitors.yaml index d9435487..f4781f1c 100644 --- a/kubernetes/apps/uptime-monitor/pilot-endpointmonitors.yaml +++ b/kubernetes/apps/uptime-monitor/pilot-endpointmonitors.yaml @@ -1,32 +1,45 @@ --- # ============================================================================ -# REN-7 CRITERION 3 — THE REMOVAL TEST. THIS COMMIT DELETES THE `audiobooks` -# ENDPOINTMONITOR ON PURPOSE. IT IS NOT A CLEANUP AND NOT AN ABANDONMENT. +# REN-7 CRITERION 3 PASSED. THIS COMMIT PUTS THE `audiobooks` ENDPOINTMONITOR +# BACK — IT IS THE DELIBERATE REVERT OF THE REMOVAL TEST, NOT A NEW FEATURE. # -# The card requires proof in both directions: an EndpointMonitor creates a -# monitor, and DELETING the EndpointMonitor removes it again. "A monitor that -# cannot be removed is a leak." Create is proven — `audiobooks.renner.uno- -# uptime-monitor`, UptimeRobot id 804092841, created 2026-09-26 03:26:27, -# controller log `Monitor Added`. Removing the object here is the other half. +# The card requires proof in both directions, and both are now on record: # -# What must be observed after this merges: the public feed -# `https://uptime.renner.uno/api/getMonitorList/UgMXmP1iRV` goes from -# `psp.totalMonitors: 10` to `9`, and id 804092841 is absent. The controller -# log shows `Deleting Monitor: audiobooks.renner.uno-uptime-monitor`. +# CREATE 2026-09-26T03:26:27Z `Monitor Added: audiobooks.renner.uno- +# uptime-monitor`, UptimeRobot id 804092841, feed +# `psp.totalMonitors` 9 → 10. +# DELETE 2026-09-26T03:57:58Z object pruned by Flux after #1018 merged as +# 101c11f4; controller log `Removing monitor audiobooks.renner.uno- +# uptime-monitor from provider: UptimeRobot` then `Monitor Removed`; +# feed 10 → 9 with id 804092841 absent, confirmed independently. # -# PRECONDITION, CHECKED BEFORE THIS COMMIT WAS WRITTEN: v2.2.13 has no -# finalizer, so a delete while the controller is down orphans the monitor -# forever (see the caveat at the end of this header). Pod -# `ingressmonitorcontroller-5d577cc58-pgqwg` was `Ready=True`, `restarts=0`, -# up since 2026-09-26T00:43:45Z when this was authored. Anyone merging this -# must re-check that the pod is Ready first; if it is not, hold the merge. +# THE LOG STRING IS `Removing monitor … from provider` / `Monitor Removed`. +# An earlier revision of this header predicted `Deleting Monitor: …`, which +# appears nowhere in v2.2.13. Anyone grepping the predicted string finds +# nothing and wrongly concludes the delete failed. Grep `Monitor Removed`. # -# RE-CREATING IT IS THE REVERT: revert this commit and Flux re-applies the -# object, and the controller creates the monitor again under the same name. -# That round trip is also the `idempotent reconcile` check, so the revert is -# worth doing deliberately rather than only if something goes wrong. Expect a -# NEW UptimeRobot id — the friendly name is what binds object to monitor, not -# the id. +# Re-creating the object is idempotent by design: the friendly name binds +# object to monitor, not the id, so the new monitor carries a NEW id. +# +# PRECONDITION RE-CHECKED BEFORE THIS COMMIT WAS WRITTEN, and it is the one the +# no-finalizer caveat below demands: pod `ingressmonitorcontroller-5d577cc58- +# pgqwg` read from the Kubernetes API at 2026-09-26T08:01Z — `Ready=True`, +# `ContainersReady=True`, `restarts=0`, image v2.2.13, up since +# 2026-09-26T00:43:45Z, the same pod that served the whole create/delete +# window. That rule binds DELETES, so it does not gate this commit, but the +# reviewer should re-read it anyway before merging. +# +# EXPECT THIS CREATE TO BE REFUSED, AND THAT IS NOT A FAULT. The account cap is +# 10 monitors and it is full again (see the capacity section below). Restoring +# the object is what Zeus directed after the cap reading, on the reasoning that +# the retry loop then creates the monitor by itself the moment the tier is +# raised — no further PR needed. Until then this object logs one refused POST +# every 300s and nothing else: nothing pages, nothing degrades, and +# `audiobooks.renner.uno` stays covered by its 60s Gatus check +# (apps/gatus/configmap.yaml:163) throughout. +# +# REVERT: revert this commit. Flux prunes the object again and, because the +# monitor does not exist to begin with, nothing in UptimeRobot changes. # ============================================================================ # # REN-7 — the pilot batch. Two of the 14 monitors from the survey's §A0 page @@ -42,7 +55,9 @@ # appearing on it is authorised rather than a disclosure. That was a deliberate # selection criterion, not a coincidence — see the status-page note below. # -# Neither host has a monitor in the account today. The nine that exist are +# Neither host had a monitor in the account when the pilot was chosen; today +# `www.renner.uno-uptime-monitor` (id 804092931) exists because this pilot +# created it. The nine pre-existing ones are # `renner.uno`, `home.`, `immich.`, `auth.`, `docs.renner.uno` and the four # third-party ones; creating an EndpointMonitor for any of those would produce # a SECOND monitor beside the hand-made one rather than adopt it, because IMC @@ -111,19 +126,30 @@ # proof the alert path works. # # --- -# NO `statusPages` — DELIBERATELY, AS AN EXPERIMENT (Zeus, on REN-7) +# NO `statusPages` — AND THE EXPERIMENT THAT JUSTIFIED LEAVING IT UNSET IS NOW +# ANSWERED: THE PAGE IS IN ALL-MONITORS MODE. KEEP THE FIELD UNSET. # # IMC only adds a monitor to a public status page when # `uptimeRobotConfig.statusPages` is set; with the field absent # `handleStatusPagesConfig` returns without calling the API at all -# (uptime-monitor.go:318-329). Leaving it unset therefore asks the page itself -# the open REN-9 question: after these two monitors exist, do they appear on -# the `uptime.renner.uno` feed on their own (the page is in an all-monitors -# mode) or not (each monitor must be added explicitly)? +# (uptime-monitor.go:318-329). It was left unset to ask the page itself REN-9's +# question, and the pilot answered it in BOTH directions with no API key: # -# Baseline for that comparison, read keyless at 2026-09-26 from +# 9 → 10 on create, monitor appeared on the feed unbidden +# 10 → 9 on prune, monitor disappeared from it unbidden +# +# So `uptime.renner.uno` includes every monitor in the account automatically and +# nothing needs adding to it. Caveat for REN-9, and it is the fragile part: +# all-monitors mode is a setting in the UptimeRobot dashboard, NOT in git. If +# anyone switches that page to a fixed list, this stops being true silently and +# no manifest, review or alert here would catch it. +# +# Baseline that comparison was made against, read keyless at 2026-09-26 from # `https://uptime.renner.uno/api/getMonitorList/UgMXmP1iRV`: 9 monitors, ids -# 801710132/37/43/45/47/52/97 and 801714845/79. +# 801710132/37/43/45/47/52/97 and 801714845/79. Parser notes for whoever reads +# this feed next: there is no top-level `monitors` key — the list is at `data` +# (and again at `psp.monitors`), the count is `psp.totalMonitors`, and an id is +# `monitorId`, not `id`. # # --- # BLAST RADIUS @@ -167,29 +193,45 @@ # through the API rather than hoping a later reconcile finds it. # # --- -# STATE OF THE OBJECT BELOW: ITS CREATE IS BEING REJECTED, AND IT RETRIES -# FOREVER. TRACKED ON REN-155 — LEFT IN PLACE DELIBERATELY. +# CAPACITY: THE ACCOUNT CAP IS EXACTLY 10 MONITORS, PROVEN BY EXPERIMENT AND +# WITHOUT AN API KEY. THIS SUPERSEDES THIS FILE'S EARLIER READING OF THE 403. # -# `www.renner.uno-uptime-monitor` does not exist in UptimeRobot. Controller log, -# every 300s since 2026-09-26T03:26:27Z: +# An earlier revision of this header asserted the opposite — that "a duplicate +# URL, a monitor-count cap and a bad parameter would all arrive as 200 + +# `stat: fail`, not as 403", and therefore that the cap was EXCLUDED. That +# inference was wrong, and the correction is the useful finding here: # -# Creating Monitor: www.renner.uno-uptime-monitor -# AddMonitor Request failed. Status Code: 403 +# UPTIMEROBOT RETURNS HTTP 403 WHEN THE ACCOUNT MONITOR LIMIT IS REACHED. +# It does not use the 200 + `stat: fail` validation path for this. Anyone +# reasoning about an IMC 403 should treat "account is full" as the FIRST +# hypothesis, not a discarded one. # -# That is the `else` branch of `Add()` (uptime-monitor.go:187) — a NON-200, and -# therefore NOT the provider's own validation path, which returns HTTP 200 with -# `{"stat":"fail", ...}` and is logged at :174 with the message text. So the -# common guesses are excluded: a duplicate URL, a monitor-count cap and a bad -# parameter would all arrive as 200 + `stat: fail`, not as 403. The 403 body is -# discarded by that branch, so the provider's reason is not recoverable from the -# log; settling it needs the API key or the account dashboard. +# The proof is a natural experiment the removal test handed us for free, read +# from the controller log of the single pod that spans the whole window: # -# It is left in git rather than removed because it is harmless and it is the -# reproducer: `Add()` is fire-and-forget (`handleCreate` returns nil regardless, -# endpointmonitor_created.go:12-31), so there is no error, no status and no -# alert — only that log line, at INFO. Nothing pages, nothing degrades, and the -# object is the only remaining evidence. Anyone tempted to "clean it up" should -# close REN-155 first. +# 03:26:27 → 03:56:27 account at 10/10. `Creating Monitor: +# www.renner.uno-uptime-monitor` then `AddMonitor +# Request failed. Status Code: 403`, SEVEN consecutive +# 300s cycles. On cycles 2-7 that create is the only +# POST in the cycle, so it is not a burst/rate effect. +# 03:57:58 removal test deletes monitor 804092841 → account 9. +# 04:01:27 next ordinary reconcile, nothing else changed: +# `Monitor Added: www.renner.uno-uptime-monitor`, +# new id 804092931, feed back to 10. +# +# The only variable between the last 403 and the success is one free slot. A +# duplicate-URL or bad-parameter cause cannot explain a create that starts +# succeeding when an UNRELATED monitor is deleted. Capacity is settled on fact; +# the tier decision is REN-158's, and REN-8's §A0 page needs 18 monitors +# against this cap of 10, so no rollout batch may proceed until it is resolved. +# +# Why the object below stayed in git through those seven refusals: `Add()` is +# fire-and-forget (`handleCreate` returns nil regardless, +# endpointmonitor_created.go:12-31) and the failure is logged at INFO, not +# ERROR — so there is no error, no status and no alert, and the retrying object +# was the only evidence that the create was being refused at all. That property +# is worth remembering for REN-8: a rollout that exceeds the cap fails SILENTLY, +# monitor by monitor, with nothing to notice it but this one INFO line. apiVersion: endpointmonitor.stakater.com/v1alpha1 kind: EndpointMonitor metadata: @@ -212,15 +254,27 @@ spec: # matches the interval of the nine monitors already in the account. Written # out rather than left implicit so the value is reviewable here. interval: 300 -# The `audiobooks.renner.uno` EndpointMonitor was here. It is removed by this -# commit as REN-7's criterion-3 removal test — see the header block at the top -# of this file for what that proves, what to observe, and how to put it back. -# Its spec, for the revert: +--- +# Restored by this commit. This object already proved criterion 3 end to end: +# created as id 804092841 at 03:26:27Z, removed as `Monitor Removed` at +# 03:57:58Z when it was pruned from git. See the header for that evidence. # -# url: https://audiobooks.renner.uno/healthcheck # Gatus check, -# # apps/gatus/configmap.yaml:134 -# uptimeRobotConfig: { monitorType: http, interval: 300 } -# -# The path belongs in `spec.url` and not in `spec.healthEndpoint`, because -# `healthEndpoint` is only consulted on the ref-discovery path and is explicitly -# ignored when `spec.url` is set (pkg/kube/util/url.go:31-33). +# Its create will now be refused with HTTP 403 until the account tier is raised +# (REN-158), because `www.renner.uno` took the slot this object's deletion +# freed. That is expected and documented, not a regression to chase. +apiVersion: endpointmonitor.stakater.com/v1alpha1 +kind: EndpointMonitor +metadata: + name: audiobooks.renner.uno + namespace: uptime-monitor +spec: + # The path belongs in `spec.url` and NOT in `spec.healthEndpoint`, because + # `healthEndpoint` is only consulted on the ref-discovery path and is + # explicitly ignored when `spec.url` is set (pkg/kube/util/url.go:31-33). + # `/healthcheck` is ungated, so the check reaches application health rather + # than mere reachability — it is the same endpoint the Gatus check uses + # (apps/gatus/configmap.yaml:163). + url: https://audiobooks.renner.uno/healthcheck + uptimeRobotConfig: + monitorType: http + interval: 300