konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 5 Packages Projects Releases 1 Wiki Activity

fix(woodpecker): revive the rotation alert rules after the namespace move #512

Merged
k.renner merged 1 commits from fix-rotator-woodpecker-url into master 2026-08-19 04:53:20 +00:00
Conversation 2 Commits 1 Files Changed 3
+19 -19
claude-bot commented 2026-08-19 04:42:49 +00:00
Member
Copy Link
Copy Source

Rebased. The issue-fix pipeline filed #510 and landed the same WOODPECKER_URL fix in #511 while this was open — that half is dropped here rather than duplicated. What remains is the part nothing else covered.

The whole alert group was silent

All four rules in prometheusrule-woodpecker-rotation.yaml selected series that do not exist:

rule selected problem
WoodpeckerServerNotReady kube_statefulset_..._ready{ns=apps} server is a Deployment, in woodpecker
WoodpeckerAgentNotReady {ns=apps} agent moved to woodpecker
rotate-Job failed {ns=apps} rotation runs from openbao
rotate-CronJob stale {ns=apps} same

WoodpeckerServerNotReady is a critical "all CI is down" alert. It queried kube_statefulset_status_replicas_ready, but the server became a Deployment when the sqlite→CNPG migration dropped its PVC — so that metric has zero series and the alert could never have fired. That one predates my namespace work entirely.

The two woodpecker-secret-rotate rules have been pointing at apps since the rotation moved to openbao, which the woodpecker kustomization already documents.

Only WoodpeckerAgentNotReady is mine.

Verified against live Prometheus

Each corrected expression was actually run:

kube_deployment_status_replicas_ready{ns=woodpecker,deployment=woodpecker-server}      -> ready=1
kube_deployment_status_replicas_available{ns=woodpecker,deployment=woodpecker-agent}   -> available=1
kube_cronjob_status_last_successful_time{ns=openbao,cronjob=woodpecker-secret-rotate}  -> 1 series

The originals returned nothing — which is why they were silent rather than noisy, and why nobody noticed.

Two stale claims in the same file

  • It pointed at kubernetes/apps/woodpecker/rotate-agent-secret.yaml, which no longer exists.
  • It asserted the CronJob/Job series "stay empty until the rotation feature is merged". That rotation has since landed, in openbao — which is exactly what the last two rules were failing to look at.

Also

Finishes the netpol comments #511 left mid-update, which still credited woodpecker-allow-rotators to apps/*.

gitea.apps.svc:3000 in review-bot.yaml is deliberately not touched — gitea really is still in apps.

How the original miss happened

Recording it because the failure mode isn't "forgot to grep": my repo-wide sweep ran as grep -rn ... --include=*.yaml, which zsh rejected with no matches found: --include=*.yaml. The command never executed, and its empty output read as a clean result. A shell error that prints nothing is indistinguishable from a search that found nothing unless you check the exit status.

**Rebased.** The issue-fix pipeline filed #510 and landed the same `WOODPECKER_URL` fix in #511 while this was open — that half is dropped here rather than duplicated. What remains is the part nothing else covered. ## The whole alert group was silent All four rules in `prometheusrule-woodpecker-rotation.yaml` selected series that do not exist: | rule | selected | problem | |---|---|---| | `WoodpeckerServerNotReady` | `kube_statefulset_..._ready{ns=apps}` | server is a **Deployment**, in `woodpecker` | | `WoodpeckerAgentNotReady` | `{ns=apps}` | agent moved to `woodpecker` | | rotate-Job failed | `{ns=apps}` | rotation runs from **openbao** | | rotate-CronJob stale | `{ns=apps}` | same | `WoodpeckerServerNotReady` is a **`critical` "all CI is down"** alert. It queried `kube_statefulset_status_replicas_ready`, but the server became a Deployment when the sqlite→CNPG migration dropped its PVC — so that metric has **zero series** and the alert could never have fired. That one predates my namespace work entirely. The two `woodpecker-secret-rotate` rules have been pointing at `apps` since the rotation moved to `openbao`, which the woodpecker kustomization already documents. Only `WoodpeckerAgentNotReady` is mine. ## Verified against live Prometheus Each corrected expression was actually run: ``` kube_deployment_status_replicas_ready{ns=woodpecker,deployment=woodpecker-server} -> ready=1 kube_deployment_status_replicas_available{ns=woodpecker,deployment=woodpecker-agent} -> available=1 kube_cronjob_status_last_successful_time{ns=openbao,cronjob=woodpecker-secret-rotate} -> 1 series ``` The originals returned nothing — which is why they were silent rather than noisy, and why nobody noticed. ## Two stale claims in the same file - It pointed at `kubernetes/apps/woodpecker/rotate-agent-secret.yaml`, which no longer exists. - It asserted the CronJob/Job series "stay empty until the rotation feature is merged". That rotation has since landed, in `openbao` — which is exactly what the last two rules were failing to look at. ## Also Finishes the netpol comments #511 left mid-update, which still credited `woodpecker-allow-rotators` to `apps/*`. `gitea.apps.svc:3000` in `review-bot.yaml` is deliberately **not** touched — gitea really is still in `apps`. ## How the original miss happened Recording it because the failure mode isn't "forgot to grep": my repo-wide sweep ran as `grep -rn ... --include=*.yaml`, which zsh rejected with `no matches found: --include=*.yaml`. The command never executed, and its empty output read as a clean result. A shell error that prints nothing is indistinguishable from a search that found nothing unless you check the exit status.
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 847986bc5a2fb78e2b5b6081a6bc92e11fe3d526.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Repoints four stale namespace="apps" PrometheusRule selectors and two rotator WOODPECKER_URL env vars at the post-migration woodpecker/openbao namespaces. I checked every changed expression and URL against the live checkout (Deployment/Service definitions, the woodpecker-allow-rotators CiliumNetworkPolicy, and the openbao/rotation CronJobs) and all of them are correct: woodpecker-server/woodpecker-agent are real Deployments in namespace woodpecker (kubernetes/apps/woodpecker/server.yaml:1-5), the woodpecker-secret-rotate CronJob lives in openbao (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:30-37), and the netpol ingress rule that the two rotators depend on already selects openbao-namespace pods reaching the woodpecker namespace on :8000 (kubernetes/apps/woodpecker/networkpolicy.yaml:16-39). No blocking issues found.

🟡 Suggestions

  • kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:12 — same stale-namespace class of bug this PR is fixing survives here: the comment reads "ESO syncs the woodpecker-secret Secret (ns apps, 10m refresh)" but the Secret has moved to the woodpecker namespace along with the server/agent (per kubernetes/apps/woodpecker/kustomization.yaml:16-23, MIGRATION-namespace.md stage 2/3). Not touched by this diff and not functionally broken (it's a comment, not code), but since this PR is explicitly a sweep for leftover apps-namespace references in the rotation directory, this one instance was missed. Worth a one-line fix while it's fresh.

🟢 Nits

  • kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml:62-63 (unchanged context, pre-existing) — the WoodpeckerSecretRotationJobFailed description still points at kubernetes/infrastructure/openbao/policies/rotator.hcl; the actual file is rotator-woodpecker.hcl (confirmed via ls kubernetes/infrastructure/openbao/policies/). Pre-existing, not introduced by this PR, and outside its stated scope — flagging only because it's adjacent to the lines being edited.

Tests

This is a monitoring-rules/YAML-only change (PrometheusRule expressions, annotation text, and env var values) with no test suite in this repo for that kind of config — consistent with how the rest of kubernetes/apps/monitoring-rules is maintained (no CI validation beyond kustomize build/flux). The PR body documents that every corrected PromQL expression was hand-verified against live Prometheus (table in the description), which is the appropriate substitute for automated tests here and is more rigorous than most changes in this repo get. No further test coverage is expected or needed.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `847986bc5a2fb78e2b5b6081a6bc92e11fe3d526`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Repoints four stale `namespace="apps"` PrometheusRule selectors and two rotator `WOODPECKER_URL` env vars at the post-migration `woodpecker`/`openbao` namespaces. I checked every changed expression and URL against the live checkout (Deployment/Service definitions, the `woodpecker-allow-rotators` CiliumNetworkPolicy, and the `openbao/rotation` CronJobs) and all of them are correct: `woodpecker-server`/`woodpecker-agent` are real `Deployment`s in namespace `woodpecker` (kubernetes/apps/woodpecker/server.yaml:1-5), the `woodpecker-secret-rotate` CronJob lives in `openbao` (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:30-37), and the netpol ingress rule that the two rotators depend on already selects `openbao`-namespace pods reaching the `woodpecker` namespace on :8000 (kubernetes/apps/woodpecker/networkpolicy.yaml:16-39). No blocking issues found. ## 🟡 Suggestions - **kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:12** — same stale-namespace class of bug this PR is fixing survives here: the comment reads "ESO syncs the woodpecker-secret Secret (**ns apps**, 10m refresh)" but the Secret has moved to the `woodpecker` namespace along with the server/agent (per kubernetes/apps/woodpecker/kustomization.yaml:16-23, MIGRATION-namespace.md stage 2/3). Not touched by this diff and not functionally broken (it's a comment, not code), but since this PR is explicitly a sweep for leftover `apps`-namespace references in the rotation directory, this one instance was missed. Worth a one-line fix while it's fresh. ## 🟢 Nits - **kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml:62-63** (unchanged context, pre-existing) — the `WoodpeckerSecretRotationJobFailed` description still points at `kubernetes/infrastructure/openbao/policies/rotator.hcl`; the actual file is `rotator-woodpecker.hcl` (confirmed via `ls kubernetes/infrastructure/openbao/policies/`). Pre-existing, not introduced by this PR, and outside its stated scope — flagging only because it's adjacent to the lines being edited. ## Tests This is a monitoring-rules/YAML-only change (PrometheusRule expressions, annotation text, and env var values) with no test suite in this repo for that kind of config — consistent with how the rest of `kubernetes/apps/monitoring-rules` is maintained (no CI validation beyond kustomize build/flux). The PR body documents that every corrected PromQL expression was hand-verified against live Prometheus (table in the description), which is the appropriate substitute for automated tests here and is more rigorous than most changes in this repo get. No further test coverage is expected or needed. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=847986bc5a2fb78e2b5b6081a6bc92e11fe3d526 -->
claude-bot force-pushed fix-rotator-woodpecker-url from 847986bc5a to 5d542af431 2026-08-19 04:50:50 +00:00 Compare
claude-bot changed title from fix(woodpecker): repoint the rotators and alert rules at the new namespace to fix(woodpecker): revive the rotation alert rules after the namespace move 2026-08-19 04:51:58 +00:00
k.renner merged commit 38eb6b6503 into master 2026-08-19 04:53:20 +00:00
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 5d542af4317be45803d5b4d773559cf3b8d85e1f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR repoints the woodpecker-rotation.rules PrometheusRule (kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml) at the post-migration woodpecker namespace and the Deployment-based server, and updates stale comments in the two rotator manifests (kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml, review-bot.yaml). I verified the new alert expressions against the live manifests: woodpecker-server/woodpecker-agent are indeed Deployments in namespace woodpecker (kubernetes/apps/woodpecker/server.yaml:4-5, agent.yaml:4-5), and the woodpecker-secret-rotate CronJob is indeed in openbao (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:33-34). No blocking issues in the diff itself.

One thing worth flagging to the author: the PR description's main narrative — "the rotators still call the old Service... http://woodpecker.apps.svc.cluster.local:8000" — does not match what's on disk or in this diff. Both rotator manifests already contained http://woodpecker.woodpecker.svc.cluster.local:8000 at this PR's merge-base (before this PR's changes), and a repo-wide grep for woodpecker.apps.svc in the current checkout returns zero matches. The diff to claude-oauth.yaml and review-bot.yaml only touches explanatory comments (netpol namespace labels), not any functional URL/env value — there was no rotator-URL bug left to fix by the time this PR was opened. This isn't a defect in the PR (the comment updates are accurate and harmless), but the description overstates what the diff does and may be describing a problem that was already resolved by an earlier commit; worth a title/body correction so future readers don't think the CronJob URL was broken until this PR landed.

🟡 Suggestions

  • PR description vs. diff mismatch (not a code issue, but worth fixing before merge): the body's "Broken now — caused by the migration" section describes rotators failing DNS lookups against woodpecker.apps.svc.cluster.local:8000, but that literal string doesn't appear anywhere in the repository, and the two rotator files this PR touches (kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml:94, review-bot.yaml:151) already had the correct woodpecker.woodpecker.svc.cluster.local:8000 env value pre-PR — this diff only edits nearby comments. Consider updating the PR description to reflect that only the comments needed fixing here, or double-check whether the rotator-URL fix landed in a separate already-merged commit and reference it.

Tests

No tests apply — this is a PrometheusRule/comment-only change in a GitOps manifest repo, consistent with how the rest of the repo handles alerting rule changes (no test suite for these). The PR body says the corrected PromQL expressions were run against live Prometheus and returned the expected series/values, which is the appropriate verification method here and I have no way to independently re-run those queries; the results as reported look plausible given the manifests on disk (Deployment kind, woodpecker namespace, openbao-namespaced CronJob all check out).

Recommendation

Looks reasonable — no blocking issues; comments above are optional (though I'd fix the PR description before merging so the history doesn't misstate what this diff fixed).

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `5d542af4317be45803d5b4d773559cf3b8d85e1f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR repoints the `woodpecker-rotation.rules` PrometheusRule (`kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml`) at the post-migration `woodpecker` namespace and the Deployment-based server, and updates stale comments in the two rotator manifests (`kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml`, `review-bot.yaml`). I verified the new alert expressions against the live manifests: `woodpecker-server`/`woodpecker-agent` are indeed Deployments in namespace `woodpecker` (`kubernetes/apps/woodpecker/server.yaml:4-5`, `agent.yaml:4-5`), and the `woodpecker-secret-rotate` CronJob is indeed in `openbao` (`kubernetes/infrastructure/openbao/rotation/woodpecker.yaml:33-34`). No blocking issues in the diff itself. One thing worth flagging to the author: the PR description's main narrative — "the rotators still call the old Service... `http://woodpecker.apps.svc.cluster.local:8000`" — does not match what's on disk or in this diff. Both rotator manifests already contained `http://woodpecker.woodpecker.svc.cluster.local:8000` at this PR's merge-base (before this PR's changes), and a repo-wide grep for `woodpecker.apps.svc` in the current checkout returns zero matches. The diff to `claude-oauth.yaml` and `review-bot.yaml` only touches explanatory comments (netpol namespace labels), not any functional URL/env value — there was no rotator-URL bug left to fix by the time this PR was opened. This isn't a defect in the PR (the comment updates are accurate and harmless), but the description overstates what the diff does and may be describing a problem that was already resolved by an earlier commit; worth a title/body correction so future readers don't think the CronJob URL was broken until this PR landed. ## 🟡 Suggestions - **PR description vs. diff mismatch** (not a code issue, but worth fixing before merge): the body's "Broken now — caused by the migration" section describes rotators failing DNS lookups against `woodpecker.apps.svc.cluster.local:8000`, but that literal string doesn't appear anywhere in the repository, and the two rotator files this PR touches (`kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml:94`, `review-bot.yaml:151`) already had the correct `woodpecker.woodpecker.svc.cluster.local:8000` env value pre-PR — this diff only edits nearby comments. Consider updating the PR description to reflect that only the comments needed fixing here, or double-check whether the rotator-URL fix landed in a separate already-merged commit and reference it. ## Tests No tests apply — this is a PrometheusRule/comment-only change in a GitOps manifest repo, consistent with how the rest of the repo handles alerting rule changes (no test suite for these). The PR body says the corrected PromQL expressions were run against live Prometheus and returned the expected series/values, which is the appropriate verification method here and I have no way to independently re-run those queries; the results as reported look plausible given the manifests on disk (Deployment kind, `woodpecker` namespace, `openbao`-namespaced CronJob all check out). ## Recommendation Looks reasonable — no blocking issues; comments above are optional (though I'd fix the PR description before merging so the history doesn't misstate what this diff fixed). <!-- pr-review:sha=5d542af4317be45803d5b4d773559cf3b8d85e1f -->
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#512