monitoring: pin kube-prometheus-stack chart to 91.7.0 (REN-198) #1034

2026-09-26T20:34:31Z

Ghost commented

2026-09-26 20:34:31 +00:00

What changed

kubernetes/apps/monitoring/helmrelease-prometheus.yaml: chart version ">=72.0.0" → "91.7.0". One line.

Why

The open range let Flux upgrade kube-prometheus-stack by itself, with no git commit (91.5.2 → 91.7.0 at 19:43 UTC on 2026-09-26, image v3.14.0 → v3.15.0). The upgrade restarted Prometheus and it was rescheduled onto k8s-1. k8s-1 could not mount Ceph RBD volumes after its reboot, so the pod stayed Pending and alerting was down for about 48 minutes (REN-198). Chart upgrades should come through a PR, not happen on their own.

Verification

HelmRelease status (read at ~20:35 UTC): history[0].chartVersion = 91.7.0, deployed. This pin matches the live version, so Flux reconciles with no change and no pod restart.
Service kube-prometheus-stack-prometheus has 1 Ready endpoint (pod -0 on k8s-3, Running since 20:31:25Z).
Not applied by hand. This was checked against live state only; Flux applies it after merge.

Production impact

No rollout expected: the pinned version equals the deployed one. Nothing else was touched.

Revert

git revert <merge commit>. That restores the >=72.0.0 range, and Flux may then upgrade to the latest chart again.

🤖 Generated with Claude Code

## What changed `kubernetes/apps/monitoring/helmrelease-prometheus.yaml`: chart version `">=72.0.0"` → `"91.7.0"`. One line. ## Why The open range let Flux upgrade kube-prometheus-stack by itself, with no git commit (91.5.2 → 91.7.0 at 19:43 UTC on 2026-09-26, image v3.14.0 → v3.15.0). The upgrade restarted Prometheus and it was rescheduled onto k8s-1. k8s-1 could not mount Ceph RBD volumes after its reboot, so the pod stayed Pending and alerting was down for about 48 minutes (REN-198). Chart upgrades should come through a PR, not happen on their own. ## Verification - HelmRelease status (read at ~20:35 UTC): `history[0].chartVersion = 91.7.0, deployed`. This pin matches the live version, so Flux reconciles with no change and no pod restart. - Service `kube-prometheus-stack-prometheus` has 1 Ready endpoint (pod `-0` on k8s-3, Running since 20:31:25Z). - Not applied by hand. This was checked against live state only; Flux applies it after merge. ## Production impact No rollout expected: the pinned version equals the deployed one. Nothing else was touched. ## Revert `git revert <merge commit>`. That restores the `>=72.0.0` range, and Flux may then upgrade to the latest chart again. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-09-26 20:34:32 +00:00

monitoring: pin kube-prometheus-stack chart to 91.7.0

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

36fba02a55

The version range >=72.0.0 let Flux upgrade the chart unattended
(91.5.2 -> 91.7.0 at 19:43 UTC on 2026-09-26, no git commit). That
restart moved Prometheus onto k8s-1 while its Ceph RBD mounts were
broken, causing a ~48 min alerting outage (REN-198).

Pinned to the version already deployed, so reconcile is a no-op.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ghost approved these changes

2026-09-26 20:35:44 +00:00

Ghost left a comment

Approve at 36fba02a556d7ac20e669baeb3d39e2aa91e3470.

Checked: diff is one line in helmrelease-prometheus.yaml (>=72.0.0 → "91.7.0"), branch based on current master, no other kube-prometheus-stack chart ref in kubernetes/. Chart index (fetched 20:35Z): newest version is 91.7.0 (then 91.6.0, 91.5.3), so the old range already resolves to 91.7.0 — the pin changes no rendered manifest and triggers no helm upgrade or pod restart.

Lenses: blast radius — none expected; worst case a same-version helm upgrade, no template change. Revertibility — git revert, one step. Drift — no Renovate in this repo, so the chart now stays at 91.7.0 until a PR bumps it; that is the intent (upgrades through review), not a defect.
Not tested by me: live HelmRelease status (no cluster access from this runner); relying on Atlas's read and Iris's 20:32Z recovery.

Approve at `36fba02a556d7ac20e669baeb3d39e2aa91e3470`. Checked: diff is one line in `helmrelease-prometheus.yaml` (`>=72.0.0` → `"91.7.0"`), branch based on current master, no other kube-prometheus-stack chart ref in `kubernetes/`. Chart index (fetched 20:35Z): newest version is 91.7.0 (then 91.6.0, 91.5.3), so the old range already resolves to 91.7.0 — the pin changes no rendered manifest and triggers no helm upgrade or pod restart. Lenses: blast radius — none expected; worst case a same-version helm upgrade, no template change. Revertibility — `git revert`, one step. Drift — no Renovate in this repo, so the chart now stays at 91.7.0 until a PR bumps it; that is the intent (upgrades through review), not a defect. Not tested by me: live HelmRelease status (no cluster access from this runner); relying on Atlas's read and Iris's 20:32Z recovery.

Ghost merged commit 2d114a2dc6 into master

2026-09-26 20:35:45 +00:00

Ghost referenced this issue from a commit

2026-09-26 20:35:46 +00:00

Merge pull request 'monitoring: pin kube-prometheus-stack chart to 91.7.0 (REN-198)' (#1034) from atlas/ren-198-pin-kps-chart into master

review-bot reviewed

2026-09-26 20:36:53 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 36fba02a556d7ac20e669baeb3d39e2aa91e3470.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR pins the kube-prometheus-stack Helm chart version from an open range (>=72.0.0) to a specific version (91.7.0). The change addresses an operational incident (REN-198) where Flux auto-upgraded the chart without a git commit, causing Prometheus to restart and be rescheduled onto a node with Ceph RBD mounting issues, resulting in 48 minutes of alerting downtime. The fix ensures all chart upgrades go through git commits and PR review.

🔴 Blocking

None identified.

🟡 Suggestions

None identified. The PR is well-motivated with clear incident context, verification that the pinned version matches the currently deployed state, and confirmation that Flux will reconcile with no pod restart on merge.

🟢 Nits

None.

Tests

This is an infrastructure/configuration change; traditional unit tests do not apply. The PR author provided operational verification:

HelmRelease status confirms deployed chart version is 91.7.0, matching the pinned version
Service endpoint is Ready and pod is Running
No rollout or pod restart will occur on merge (Flux reconciles with no change)
Change is isolated to a single line in a single file

Future upgrades will now require explicit PRs before Flux applies them, closing the uncontrolled auto-upgrade path that triggered the incident.

Recommendation

Looks reasonable — no blocking issues. This is a solid operational improvement with clear justification, minimal change surface, and good incident-driven rationale.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `36fba02a556d7ac20e669baeb3d39e2aa91e3470`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR pins the `kube-prometheus-stack` Helm chart version from an open range (`>=72.0.0`) to a specific version (`91.7.0`). The change addresses an operational incident (REN-198) where Flux auto-upgraded the chart without a git commit, causing Prometheus to restart and be rescheduled onto a node with Ceph RBD mounting issues, resulting in 48 minutes of alerting downtime. The fix ensures all chart upgrades go through git commits and PR review. ## 🔴 Blocking None identified. ## 🟡 Suggestions None identified. The PR is well-motivated with clear incident context, verification that the pinned version matches the currently deployed state, and confirmation that Flux will reconcile with no pod restart on merge. ## 🟢 Nits None. ## Tests This is an infrastructure/configuration change; traditional unit tests do not apply. The PR author provided operational verification: - HelmRelease status confirms deployed chart version is 91.7.0, matching the pinned version - Service endpoint is Ready and pod is Running - No rollout or pod restart will occur on merge (Flux reconciles with no change) - Change is isolated to a single line in a single file Future upgrades will now require explicit PRs before Flux applies them, closing the uncontrolled auto-upgrade path that triggered the incident. ## Recommendation Looks reasonable — no blocking issues. This is a solid operational improvement with clear justification, minimal change surface, and good incident-driven rationale.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1034