The FluxResourceNotReady alerts in #1217 were a blip and have already resolved. They were caused by the Gitea migration crashloop (Flux sources from Gitea) and the openproject wait-for-db init container that PR #1211 fixed. Both are tracked in their own issues, so they need no change here. This PR fixes the side finding instead: Flux notifications are never delivered. The alertmanager Provider's spec.address points at the bare Alertmanager root (:9093/). The Flux notification-controller POSTs to spec.address exactly as written, and Alertmanager answers a POST to / with 405 Method Not Allowed. As a result, the flux-failures and flux-image-automation Alerts send nothing. The change points the address at the v2 alerts endpoint (/api/v2/alerts/, the form shown in the Flux Provider docs) and corrects the comment, which wrongly said the controller adds that path itself.
Changes
kubernetes/infrastructure/flux-notifications/provider.yaml: changed spec.address to .../9093/api/v2/alerts/ and rewrote the incorrect comment.
Verification
Checked live this run:
./.ci-diag/kubectl get kustomization,gitrepository -n flux-system: every Kustomization and the flux-system GitRepository are Ready=True at master@sha1:4e97809. The original alert really has cleared.
Loki query for {namespace="flux-system",pod=~"notification-controller.*"} |= "Method Not Allowed" (2026-10-08 20:00–24:00 UTC) returned lines like this one: failed to send notification ... "alert":{"name":"flux-failures",...,"providerName":"alertmanager"},"error":"postMessage failed: ... status code 405: Method Not Allowed" (22:44:35Z, Kustomization openproject)
./.ci-diag/kubectl get provider -n flux-system alertmanager -o jsonpath={.spec} shows the live spec matches the repo: address: http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/.
./.ci-diag/kubectl get alertmanager -n monitoring -o jsonpath=...routePrefix returns /, so the API is served at /api/v2/alerts with no extra prefix.
The notification-controller image is v1.9.4.
How the reviewer can confirm after Flux reconciles infrastructure:
kubectl get provider -n flux-system alertmanager -o jsonpath={.spec.address} should end in /api/v2/alerts/.
On the next error-severity Flux event (or an image-automation bump commit), kubectl logs -n flux-system deploy/notification-controller should show no postMessage failed ... 405 lines. The Alertmanager UI (https://alerts.renner.uno) should show an alert labelled from source: flux.
Notes
This change was not tested live. I couldn't POST to Alertmanager from the read-only diagnostics account. The fix rests on two things: the observed 405, and Flux's documented Provider format for type: alertmanager, whose example address ends in /api/v2/alerts/.
Once this lands, Flux failures will actually reach Alertmanager and its email route. Expect notifications that were silently dropped before, for example during planned maintenance windows like the one described in kubernetes/apps/woodpecker/MIGRATION-namespace.md.
The issue body was accurate. I confirmed both the outage state and the 405 claim.
Per the issue, #1217 can be closed once the linked Gitea (KubeJobFailed) and openproject (KubePodNotReady) issues are handled. This PR covers the notification part.
## What this PR does
The `FluxResourceNotReady` alerts in #1217 were a blip and have already resolved. They were caused by the Gitea migration crashloop (Flux sources from Gitea) and the openproject `wait-for-db` init container that PR #1211 fixed. Both are tracked in their own issues, so they need no change here. This PR fixes the side finding instead: **Flux notifications are never delivered.** The `alertmanager` Provider's `spec.address` points at the bare Alertmanager root (`:9093/`). The Flux notification-controller POSTs to `spec.address` exactly as written, and Alertmanager answers a POST to `/` with `405 Method Not Allowed`. As a result, the `flux-failures` and `flux-image-automation` Alerts send nothing. The change points the address at the v2 alerts endpoint (`/api/v2/alerts/`, the form shown in the Flux Provider docs) and corrects the comment, which wrongly said the controller adds that path itself.
## Changes
- `kubernetes/infrastructure/flux-notifications/provider.yaml`: changed `spec.address` to `.../9093/api/v2/alerts/` and rewrote the incorrect comment.
## Verification
Checked live this run:
- `./.ci-diag/kubectl get kustomization,gitrepository -n flux-system`: every Kustomization and the `flux-system` GitRepository are `Ready=True` at `master@sha1:4e97809`. The original alert really has cleared.
- Loki query for `{namespace="flux-system",pod=~"notification-controller.*"} |= "Method Not Allowed"` (2026-10-08 20:00–24:00 UTC) returned lines like this one:
`failed to send notification ... "alert":{"name":"flux-failures",...,"providerName":"alertmanager"},"error":"postMessage failed: ... status code 405: Method Not Allowed"` (22:44:35Z, Kustomization `openproject`)
- `./.ci-diag/kubectl get provider -n flux-system alertmanager -o jsonpath={.spec}` shows the live spec matches the repo: `address: http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/`.
- `./.ci-diag/kubectl get alertmanager -n monitoring -o jsonpath=...routePrefix` returns `/`, so the API is served at `/api/v2/alerts` with no extra prefix.
- The notification-controller image is `v1.9.4`.
How the reviewer can confirm after Flux reconciles `infrastructure`:
1. `kubectl get provider -n flux-system alertmanager -o jsonpath={.spec.address}` should end in `/api/v2/alerts/`.
2. On the next error-severity Flux event (or an image-automation bump commit), `kubectl logs -n flux-system deploy/notification-controller` should show no `postMessage failed ... 405` lines. The Alertmanager UI (https://alerts.renner.uno) should show an alert labelled from `source: flux`.
## Notes
- This change was not tested live. I couldn't POST to Alertmanager from the read-only diagnostics account. The fix rests on two things: the observed 405, and Flux's documented Provider format for `type: alertmanager`, whose example address ends in `/api/v2/alerts/`.
- Once this lands, Flux failures will actually reach Alertmanager and its email route. Expect notifications that were silently dropped before, for example during planned maintenance windows like the one described in `kubernetes/apps/woodpecker/MIGRATION-namespace.md`.
- The issue body was accurate. I confirmed both the outage state and the 405 claim.
- Per the issue, #1217 can be closed once the linked Gitea (KubeJobFailed) and openproject (KubePodNotReady) issues are handled. This PR covers the notification part.
Closes #1217
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What this PR does
The
FluxResourceNotReadyalerts in #1217 were a blip and have already resolved. They were caused by the Gitea migration crashloop (Flux sources from Gitea) and the openprojectwait-for-dbinit container that PR #1211 fixed. Both are tracked in their own issues, so they need no change here. This PR fixes the side finding instead: Flux notifications are never delivered. ThealertmanagerProvider'sspec.addresspoints at the bare Alertmanager root (:9093/). The Flux notification-controller POSTs tospec.addressexactly as written, and Alertmanager answers a POST to/with405 Method Not Allowed. As a result, theflux-failuresandflux-image-automationAlerts send nothing. The change points the address at the v2 alerts endpoint (/api/v2/alerts/, the form shown in the Flux Provider docs) and corrects the comment, which wrongly said the controller adds that path itself.Changes
kubernetes/infrastructure/flux-notifications/provider.yaml: changedspec.addressto.../9093/api/v2/alerts/and rewrote the incorrect comment.Verification
Checked live this run:
./.ci-diag/kubectl get kustomization,gitrepository -n flux-system: every Kustomization and theflux-systemGitRepository areReady=Trueatmaster@sha1:4e97809. The original alert really has cleared.{namespace="flux-system",pod=~"notification-controller.*"} |= "Method Not Allowed"(2026-10-08 20:00–24:00 UTC) returned lines like this one:failed to send notification ... "alert":{"name":"flux-failures",...,"providerName":"alertmanager"},"error":"postMessage failed: ... status code 405: Method Not Allowed"(22:44:35Z, Kustomizationopenproject)./.ci-diag/kubectl get provider -n flux-system alertmanager -o jsonpath={.spec}shows the live spec matches the repo:address: http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/../.ci-diag/kubectl get alertmanager -n monitoring -o jsonpath=...routePrefixreturns/, so the API is served at/api/v2/alertswith no extra prefix.v1.9.4.How the reviewer can confirm after Flux reconciles
infrastructure:kubectl get provider -n flux-system alertmanager -o jsonpath={.spec.address}should end in/api/v2/alerts/.kubectl logs -n flux-system deploy/notification-controllershould show nopostMessage failed ... 405lines. The Alertmanager UI (https://alerts.renner.uno) should show an alert labelled fromsource: flux.Notes
type: alertmanager, whose example address ends in/api/v2/alerts/.kubernetes/apps/woodpecker/MIGRATION-namespace.md.Closes #1217