main-926ead0 includes ical-proxy PR #7 (REN-230): the upstream response
body is capped at 2 MiB, so an oversized feed gets 413 instead of being
buffered whole into a 256Mi pod. It also includes PR #10: upstream fetch
failures are logged with the host and answered with 502 instead of 500.
Refs: REN-294
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the #1046 merge (pipeline 11306) the build-agent-runtime step got no
pod: the full rebuild of build-ci-images.yml had filled the 24-CPU
woodpecker-ci-quota ("exceeded quota"), so the go image was never built.
Move the step to .woodpecker/build-agent-runtime.yml. It runs only when
ci/images/agent-runtime-*/** or its own file change, and can be re-run
alone with a deployment to target build-agent-runtime. Step content is
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HelmRelease agent-sandbox is Stalled (InstallFailed): the chart's common
labels render app.kubernetes.io/version from image.tag (contains the
digest, >63 chars) and helm.sh/chart with Flux's +<rev> suffix. Remove
both labels in a kustomize postRenderer. Selector and pod labels are
unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first push of this branch started all 13 build steps: the push trigger
had no branch filter. 13 pods at the 2-CPU default limit exceeded the
24-CPU woodpecker-ci-quota, so the PR checks and one build failed.
- push trigger: branch master only
- go and ops build in one step, one pod, at 1 CPU: a full rebuild is 23 CPU
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two pod images for the Paperclip Kubernetes sandbox (REN-284 step 2):
the stock agent-runtime-claude, pinned by digest, plus
- agent-runtime-go: Go 1.27.1, golangci-lint v2.14.0
- agent-runtime-ops: kubectl v1.36.5, flux v2.9.5, kustomize v5.8.1, helm 4.3.0
Tools are copied from their upstream images pinned by tag and digest, so
the existing Renovate dockerfile manager bumps them. Pushed to
konairius/agent-runtime-{go,ops}/agent-runtime-claude:v1 because the plugin
keeps that name and swaps only the registry prefix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- infrastructure/agent-sandbox: Flux GitRepository (tag v1.0.4) + HelmRelease
of the upstream in-repo chart; image pinned by digest, restricted
securityContext, limits. CRDs via crds: CreateReplace.
- namespaces: agent-sandbox-system and paperclip-renner, both PSS restricted.
- network-policies: CNP for agent-sandbox-system (probes in, apiserver out).
- apps/paperclip/sandbox-rbac.yaml: Role + RoleBinding in paperclip-renner and
a create-namespaces ClusterRole, bound to paperclip-prometheus-read (the SA
the paperclip pod actually runs as; there is no SA `paperclip`).
- renovate.json: group the git tag and the image pin into one PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
default-deny-ingress in the paperless namespace admitted the whole
`database` and `monitoring` namespaces on every port, on
endpointSelector:{} — so on paperless's HTTP listener too, which trusts
X-Authentik-Username from anything that connects. Both admits are
removed and the three load-bearing flows are kept as per-workload
policies:
pprl-pg-allow-cnpg-operator database/cloudnative-pg -> pprl-pg :8000
pprl-pg-allow-prometheus monitoring/prometheus -> pprl-pg :9187
paperless-allow-blackbox monitoring/blackbox -> paperless :8000
No ICMP rule on them (REN-144 option (a)); the baseline ICMP and the
host/remote-node L3 admit are untouched.
Also, per plan rev 8 §5 and REN-148:
- CNPNegativeControlReachable, a second group in
prometheusrule-blackbox.yaml, scoped to job=blackbox-negative-control.
- Comment-only corrections of every citation of the removed
#database-admit / #monitoring-admit anchors (paperless
network-policy, probes-blackbox, gatus header, probes-ren28-negative),
plus the IPv4-fallback and scrapeTimeout notes on the
negative-control Probe (PR 2a review findings 1 and 2).
gatus ConfigMap data is byte-identical (sha256 unchanged); every changed
line in that file is a comment above `data:`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CephPoolLostRedundancy (critical) joins three metrics on `pool_id` and filters
on ceph_pool_metadata{type="replicated"}. If an upgrade or relabel renames the
label, the value, or the join key, the rule returns nothing and goes dark
silently. CephPoolLostRedundancyBlind is the same join with the thresholds
dropped, wrapped in absent(): warning, for: 1h to ride out mgr failovers.
Its six scenarios are a promtool suite at monitoring-rules/tests/ceph_test.yaml,
with five negative controls in mutations.txt, run by the existing `promtool`
step (#1026). The CI half of the original #989 is dropped: #1026 covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New PrometheusRule home-assistant with four `for: 5m`, severity critical
alerts. Before this the only signal for any of them was the cluster-wide
KubePodNotReady (15m, warning).
- HomeAssistantDown: HA scrape target down or absent.
- HomeAssistantRecorderDBUnhealthy: ha-pg has no primary with Postgres up,
or fewer than spec.instances (2) instances with Postgres up.
- MosquittoDown: no Ready broker pod (bootstrap Job pods excluded).
- Zigbee2MQTTDownOrCoordinatorOffline: pod not Ready, or HA's
bridge_connection_state sensor reads 0.
Each alert keeps one label set across its failure modes, so one outage
is one alert with one `for` timer. Unit suite and five negative controls
added; scripts/ci/promtool-rules.sh passes locally with promtool 3.15.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on REN-240 (protection rule 5):
- README.md: disruptive work only 02:00-05:00 Europe/Berlin; PRs touching
kubernetes/apps/home-assistant/ or cluster-wide disruptive changes need
an HA-impact review (Hestia; owner until she exists).
- kubernetes/RUNBOOK-home-assistant-image-rollback.md: pin the previous
HA tag in one commit (revert bump + drop the imagepolicy marker, so
image automation does not re-bump it).
Docs only. No manifest changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Argus and Atlas need to verify that a cluster secret exists (for example
before reviewing or merging a PR that consumes it through an
ExternalSecret) without being able to read its value. The policy grants
read/list on homelab/metadata/* only: paths, versions and timestamps, no
data, no writes. Whole tree, per owner decision.
Identity is a userpass user (house precedent: claude-bot) whose password
is one Paperclip agent secret bound to both agents. Applied once by hand
with an admin token, like the other policies in this directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump the pinned ical-proxy image from main-4783abe to main-73b5d97.
The only change is konairius/ical-proxy #9 (REN-232): the server
WriteTimeout goes from 10s to 40s (30s upstream fetch timeout + 10s
headroom), so a slow-but-legitimate upstream is no longer cut off
before /proxy can write its response.
Refs: REN-239, REN-232
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump the pinned ical-proxy image from main-db90a59 to main-4783abe.
The new build adds two hardening changes from konairius/ical-proxy:
- #6: refuse upstreams and redirect hops that resolve to private,
loopback, link-local or CGNAT addresses. Public redirects are still
followed (up to 5).
- #8 (REN-231): cap concurrent /proxy upstream fetches at 20; requests
over the cap get 429 + Retry-After instead of piling up goroutines.
Refs: REN-238, REN-231
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ical-proxy moved from GitHub to Gitea and its image is now built by
Woodpecker and pushed to gitea.renner.uno/konairius/ical-proxy. The
ghcr.io image gets no new builds.
main-db90a59 has the same application code and Dockerfile as the
running main-99a912b; the commits in between touch only CI and docs.
The package pulls anonymously, so no imagePullSecret is added.
Revert: git revert this commit; Flux rolls the pods back to
ghcr.io/konairius/ical-proxy:main-99a912b.
Refs: REN-201
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #1035: image-automation/ has 33 ImageRepositories, the
disable list had 32. Without this, Renovate and Flux would both bump
ghcr.io/matter-js/python-matter-server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- apps/renovate: CronJob (daily 04:00, routine PRs Monday only via the
shared preset's schedule), runner config, renovate-bot token ExternalSecret.
- clusters/homelab/renovate.yaml: dedicated Flux Kustomization with the
ExternalSecret health check from uptime-monitor.yaml.
- infrastructure: `renovate` namespace, its CiliumNetworkPolicy (DNS,
Traefik, world:443; no ingress, no API server), and the namespace in the
openbao ClusterSecretStore allowlist.
- renovate.json: extends local>k.renner/renovate-config. Managers flux,
kubernetes, woodpecker, dockerfile. Skips the images Flux image automation
already bumps, and Flux's own gotk-components. Digest pins and
Cilium/Rook-Ceph minors get their own PRs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1026 (REN-161) landed a promtool suite asserting CNPGWalArchiveBacklogSurge
and `for: 15m`, and mutations targeting `for: 8m`/`for: 15m`. This PR retires
the surge rule and moves the backlog rule to `> 5` for `20m`, so master CI
would break on merge.
- Generator: drop every Surge assertion; firing cases move to t=25m.
- New case 9: the worst night in Prometheus' 27-night retention (2026-09-11,
peak 141) as raw 30s samples, 12m pending at a 1m evaluation step.
- mutations.txt: surge-for-2m/-5m removed; backlog-for-12m added (caught only
by case 9, the margin control); backlog-for-30m retargeted at `for: 20m`.
- README and the woodpecker step comment updated to match.
Verified: scripts/ci/promtool-rules.sh with promtool 3.15.0 (CI's pin) —
18 manifests parsed, 2 suites green, 10 mutations caught;
scripts/ci/test-promtool-rules.sh passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retune under Zeus's ruling on REN-152: the instrument is persistence, not
depth. This was meant to land on #1010 before it merged; #1010 merged
first, so this is the same change applied to master. Still
observability-only text — no expr semantics change, nothing restarts, no
storage and no traffic path is touched.
- CNPGWalArchiveBacklog `for`: 15m -> 20m.
- CNPGWalArchiveBacklogSurge (>64 for 8m) removed.
Why the depth rule goes. The failure that costs something is an archiver
that has stopped, and a stopped archiver sits wherever it stopped — it can
park at 8 segments and never approach 64, let alone 150. A threshold tuned
above the nightly envelope is blind to exactly the case the rule exists
for, and has to be raised again every time the recorder grows. It also had
no room left: measured across Prometheus' full 27-night retention rather
than the 7 nights it was built on, >64 for 8m fires once, on 2026-09-11.
Why 20m and not 15m. Same 27-night replay, 30s resolution, all 12 CNPG
instances. Only ha-pg-1 ever exceeds 5; every other instance peaks at 2.
Longest contiguous excursion above 5 is 13.0 minutes, on 2026-09-11 at a
peak of 141. Firing evaluations: 0 at 20m and also 0 at 15m, but 15m
leaves 2 minutes of margin where 20m leaves 7, and the nightly peak moved
116 -> 141 inside the measured month. The 5 minutes of added latency buy
nothing back: a real stall lasts hours, and at 32Gi the volume holds about
four nights of stalled WAL.
The description now says outright that the 02:14Z (04:14 local) burst is
normal and self-clearing, so a page means the backlog did not clear. The
drain-rate and headroom pointers (wal.maxParallel, spec.storage.size,
REN-135) move up from the deleted rule rather than being lost.
Verified with promtool 3.5.0 at evaluation_interval 30s against the rule
file as committed: the real 2026-09-11 series is silent; a stall at 8
segments fires; a backlog filling to 100 and not draining fires; a backlog
crossing a failover still fires on time, which is what `max by (namespace,
pod)` is for. Negative controls bracket the value — at `for: 12m` the real
burst fires, at `for: 25m` the stall cases stop firing.
Revert: `git revert` this commit. That restores exactly what is on master
today, >5 for 15m plus the Surge rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The version range >=72.0.0 let Flux upgrade the chart unattended
(91.5.2 -> 91.7.0 at 19:43 UTC on 2026-09-26, no git commit). That
restart moved Prometheus onto k8s-1 while its Ceph RBD mounts were
broken, causing a ~48 min alerting outage (REN-198).
Pinned to the version already deployed, so reconcile is a no-op.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
promtool-rules.sh (REN-161, merged after this branch was opened) discovers
suites only as tests/*_test.yaml. At its old path the ESO suite would have
been skipped without a word. Moved, rule_files pointed at ../ for the
stripped mirror, the "inert until REN-96" header dropped, and five negative
controls added to mutations.txt: for 5m, sum-not-max, by-namespace,
no == 1, any condition. All five are caught.
promtool 3.15.0 (the CI pin): 18 manifests parsed, 2 suites green,
11 mutations caught. Rule file unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment-only. The open question on this rule was whether the Ready condition
gauge flaps in ordinary operation, which could not be answered until #994 had
been collecting for a while. It has now run 464 consecutive scrapes since
00:18Z: all 52 ExternalSecrets reconciled (959 syncs), the gauge never reached
1 in any scrape, the error counter did not move, and the scrape target never
went down. Recorded in the rule file with the queries that produced it.
Also drops a reference to prometheusrule-ceph-test.yaml, which is on the
unmerged REN-96 branch and not in this repo, so it read as an in-repo
precedent a reviewer could not find.
Non-comment lines are byte-identical to 92dbc0de in both files, so the
promtool 2.53.2 check-rules and test-rules runs recorded on the PR still hold.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Comment-only. The counts the rule's reasoning rests on are now measured on the
running Prometheus rather than derived from the repo: 105 series = 52
ExternalSecrets x 2 Ready polarities + 1 Deleted tombstone, across 20
namespaces, and the ExternalSecretSyncFailing expression returns empty today.
The last point is the one a reviewer wants: the rule arrives silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only, again: `diff` of all non-comment lines against bbaf0c6a is empty.
Answers the grouping objection raised in review. Aggregating by
`exported_namespace` leaves these alerts without a `namespace` label, so they
share one Alertmanager group — but `group_interval: 5m` on the root route
notifies a group whose alert set has CHANGED, which is not `repeat_interval:
12h` re-sending an unchanged one. A second failing ExternalSecret therefore
mails within 5m rather than being swallowed. Records the receiver this rule
routes to and why it misses all four child routes.
Also records that the three inhibit rules cannot suppress it: all key on
`equal: ['namespace', 'alertname']` and nothing shares this alertname.
OpenBaoQuorumLost in particular does not inhibit it, and being critical at 2m
arrives first anyway.
Refs REN-103
Comment-only. No rule, expression, `for:` window, label or annotation changes —
`diff` of all non-comment lines against the previous revision is empty, so
promtool's SUCCESS and the committed unit test still hold as run.
Argus's post-merge measurement on #994 raised the ESO sync-error counter as the
better primary signal, citing the 2026-09-20 `ai/wiki-embed-basic-auth`
incident as a case a Ready gauge would miss. Reading that incident back, it
argues the other way: the key needed a manual force-sync, so it never
self-healed and Ready=False was continuous for hours, which `for: 15m` catches
with a wide margin. It was a stuck condition, not a flap.
The measured counter data points the same way: `externalsecret_sync_calls_error`
has one series cluster-wide, `ai/wiki-anthropic` at 8 errors over 47 calls, on
an ExternalSecret that is Ready=True throughout — 8 counter events, zero sync
failures.
Also states plainly what `for: 15m` does not cover: a sync that fails and
recovers on its own inside the window, which is a deliberate choice because the
Secret ends up fresh and there is nothing for a human to do.
Refs REN-103
Merge B of REN-103. #994 made the series exist; this is the rule built on
them. Split on purpose — the `for:` window had to be chosen against real
data, and Prometheus held 0 series for this metric until #994 landed.
Two corrections to what REN-100 specified, both found only once there was
data or source to read:
* `exported_namespace`, not `namespace`. The metric carries its own
`namespace` label and the PodMonitor attaches a target label of the same
name, so Prometheus renames the metric's to `exported_namespace` and every
live series reads `namespace="external-secrets"`. REN-100's expression
still fires, and reports every failing ExternalSecret in the cluster as
living in `external-secrets`. Found by Argus against the live data.
* `max`, not `sum`. Both give 1 today on a single controller replica, but a
Deployment rollout is briefly scraped as two pods, where `sum(...) == 1`
evaluates to 2 and silently stops matching.
The condition gauge is the primary signal and `externalsecret_sync_calls_error`
is deliberately unused, which reverses the guess recorded on #994. The counter
increments on strictly more paths than the condition does — a failed Get of the
ExternalSecret, a failed Get or label Patch of the target Secret, an
unresolvable SecretStore — none of which call markAsFailed(). It is the noisier
signal, not the quieter one. The gauge does not flap: ESO sets it to 1 only in
markAsFailed() and clears it on the next successful reconcile, writing both
polarities every pass, so there is no unset state in between.
`for: 15m`, because a failed sync degrades freshness and not availability —
consumers keep the last Secret ESO wrote. Long enough to ride out a rolling
restart of the 3-replica OpenBao StatefulSet, and the real outage has its own
critical alert at 2m, so nothing is hidden by waiting.
Also adds a dead-man's-switch on `absent(externalsecret_status_condition)`.
Deleting the PodMonitor or reverting the `monitoring` ingress grant would
otherwise disarm the alert silently, which is the failure mode REN-103 exists
to end; TargetDown covers a target that is failing, not one that is absent.
Verified with promtool 2.53.2: `check rules` passes, and the committed
`-test.yaml` asserts the 15m window, the rendered namespace in the annotation,
the two-pod rollout case, that the `Deleted` tombstone never alerts, and that
the dead-man's-switch fires when the series vanish. Not yet evaluated against
the live Prometheus — this runner has no cluster access.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review #1114: 18Gi gave ~3.5 days of archiver outage, below the ~4.0 days
the shared 32Gi gives today. 21Gi gives ~4.1. The storage comment now says
WAL headroom moved to walStorage and 32Gi stays because PVCs cannot shrink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Argus's review of PR #1032: the name exists in neither pkg/ nor the metrics
docs for 1.20.2, and the agent skips unknown --metrics names silently. Zeus
dropped it from scope. metricsService, the Service delete and the new
selector stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner-approved option B on REN-176. Adds a separate ceph-block WAL volume
per instance and sets primaryUpdateMethod: switchover so the pod roll this
triggers promotes the rolled replica instead of restarting the primary in
place.
Permanent: the CNPG 1.30.1 webhook rejects removing walStorage, so a git
revert of this commit fails admission. Merge only in an owner-named window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prometheus.metrics: +cilium_policy_verdict_total (opt-in; per-direction
policy verdict attribution on top of cilium_drop_count_total).
- prometheus.metricsService: true, so the chart renders the headless
`cilium-agent` metrics Service.
- Drop the hand-declared `cilium-agent-metrics` Service from
cilium-servicemonitor.yaml and point the ServiceMonitor at the chart's
Service, so the endpoints are scraped once, not twice.
This is a values change: helm-controller upgrades the CNI release, the
cilium-config `metrics` key changes, and the reloader annotation rolls the
agents node by node. Merge only inside the window Zeus set on REN-166.
Revert: git revert this commit; Flux restores the old Service and selector
and rolls the agents back the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
