konairius/homelab
10
0
Fork 0
Code Issues 10 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

Restore test 3/9: Flux dependency graph and health #1299

Closed
claude-bot wants to merge 3 commits from split/03-dependency-graph into split/02-site-facts
pull from: split/03-dependency-graph
merge into: :split/02-site-facts
Conversation 2 Commits 3 Files Changed 18
+142 -79
claude-bot commented 2026-10-17 09:52:00 +00:00
Member
Copy Link
Copy Source

Part 3 of 9 (stacked on part 2). The Flux dependency graph says what each Kustomization actually needs, and Ready means its workloads run.

  • infrastructure is Ready only once the OpenBao StatefulSet is (so the secret store comes up after unseal).
  • Edges: Kustomizations whose images are in Gitea's registry depend on gitea (authentik, paperless, home-assistant, review-bot, wiki); gitea-mcp, woodpecker and renovate depend on gitea because they work against it; evcc and trmnl on home-assistant; home-assistant and monitoring on influxdb.
  • wait: true for home-assistant, llama-server, wiki, wordpress (with wait: false and no healthChecks Flux checks nothing; wordpress' healthCheckExprs never ran). otbr is suspended (not in use). home-assistant timeout 30m instead of 4h.
  • authentik: the chart's ServiceMonitor renders without a CRD check, so it moves to monitoring-rules. The hand-written radius metrics Service/ServiceMonitor are dropped: authentik 2026.8.3's operator creates both (on a cold cluster the hand-written clusterIP: None copy failed and blocked all of monitoring-rules).
  • scripts/flux-graph.sh draws the graph from the manifests (scripts/flux-graph.sh graph.svg).

Rollout: none beyond the merge. Note: with wait: true, a Kustomization whose pods cannot run now reports not-Ready instead of green.

🤖 Generated with Claude Code

Part 3 of 9 (stacked on part 2). The Flux dependency graph says what each Kustomization actually needs, and Ready means its workloads run. - `infrastructure` is Ready only once the OpenBao StatefulSet is (so the secret store comes up after unseal). - Edges: Kustomizations whose images are in Gitea's registry depend on `gitea` (authentik, paperless, home-assistant, review-bot, wiki); gitea-mcp, woodpecker and renovate depend on `gitea` because they work against it; evcc and trmnl on `home-assistant`; home-assistant and monitoring on `influxdb`. - `wait: true` for home-assistant, llama-server, wiki, wordpress (with `wait: false` and no healthChecks Flux checks nothing; wordpress' healthCheckExprs never ran). otbr is suspended (not in use). home-assistant timeout 30m instead of 4h. - authentik: the chart's ServiceMonitor renders without a CRD check, so it moves to monitoring-rules. The hand-written radius metrics Service/ServiceMonitor are dropped: authentik 2026.8.3's operator creates both (on a cold cluster the hand-written `clusterIP: None` copy failed and blocked all of monitoring-rules). - `scripts/flux-graph.sh` draws the graph from the manifests (`scripts/flux-graph.sh graph.svg`). **Rollout:** none beyond the merge. Note: with `wait: true`, a Kustomization whose pods cannot run now reports not-Ready instead of green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 3 commits 2026-10-17 09:52:00 +00:00
fix(flux): each Kustomization depends on what its workloads need, and waits for them 665d0b12e9 
On a cold cluster (restore rehearsal runs 22-28) apps started before the
things they need and sat in ImagePullBackOff or CreateContainerConfigError,
while their Kustomizations reported Ready or held kustomize-controller
workers. The dependency graph now states those needs:

- infrastructure health-checks the openbao StatefulSet, not only the
  HelmRelease (Ready as soon as the chart is installed, sealed or not). Its
  readiness probe is `bao status`, which fails while sealed, so 3/3 ready
  means every replica is unsealed; infrastructure-crds no longer applies the
  ClusterSecretStore against a sealed OpenBao, which External Secrets then
  backs off for minutes.
- gitea edges: apps whose images live only in Gitea's registry (authentik,
  home-assistant, paperless, review-bot, wiki) and apps that are front ends
  to Gitea (gitea-mcp, woodpecker, renovate). Images also published to
  ghcr.io have a fallback in the nodes' registry mirror and get no edge
  (apps' ical-proxy, trmnl's alpine-jq).
- monitoring depends on influxdb: Grafana mounts grafana-influxdb-token,
  which a Job in the influxdb app writes.
- evcc depends on home-assistant: mqtt-account-controller there delivers
  mqtt-evcc-client-tls, and mosquitto is its broker.
- trmnl depends on home-assistant (the renderer reads HA's API);
  home-assistant depends on influxdb (its InfluxDB integration writes there).
- home-assistant and wordpress wait for their workloads. With wait: false
  and no healthChecks, Ready means only that the manifests applied, and
  healthCheckExprs are never evaluated (kustomize-controller's checkHealth
  returns first). otbr is suspended: it is not in use.
- home-assistant's timeout is 30m, not 4h. The 4h was sized for a seed
  initContainer that waited up to 3h for an HAOS backup; the slowest part
  left is ha-pg recovering from barman. With wait: true the 4h would also
  hold a kustomize-controller worker that long on a failed cold start.

No cycles: gitea depends only on the infrastructure layers; influxdb does
not use monitoring's CRDs; home-assistant does not reach evcc or trmnl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(authentik): its ServiceMonitor lives in monitoring-rules; the radius one is the operator's 1aa30386b6 
The authentik chart renders server/servicemonitor.yaml whenever
server.metrics.serviceMonitor.enabled is set, without checking
.Capabilities for monitoring.coreos.com (verified in chart 2026.8.3). The
authentik Kustomization does not depend on monitoring, so on an empty
cluster the Helm install fails until kube-prometheus-stack has installed the
CRDs. The chart's ServiceMonitor is switched off and declared standalone in
monitoring-rules, as cilium and velero already do. The selector matches the
live Service authentik-server-metrics (port metrics, 9300); the new name
differs from the chart's authentik-server, so the two never contend for one
object while production rolls over.

authentik-outpost-servicemonitors.yaml hand-wrote a headless Service
ak-outpost-local-radius-metrics and a ServiceMonitor for it, because the
radius outpost controller used not to expose its metrics port. authentik
2026.8.3 does: the operator creates that Service (http-metrics 9300) and its
own ServiceMonitor. On a cold cluster the operator created the Service first,
with a ClusterIP; clusterIP is immutable, so the dry-run of the hand-written
clusterIP: None copy failed and took the whole monitoring-rules
Kustomization with it (restore test run 29). The file goes; the operator
owns both. No alert rule refers to the radius job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(scripts): flux-graph.sh draws the Kustomization dependency graph
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot reviewed: comment
Details
075fc61d83 
Reads every kustomize.toolkit.fluxcd.io/v1 Kustomization in
kubernetes/clusters/homelab and emits Graphviz DOT, one edge per dependsOn
entry, pointing from the dependency to its dependent. Each node carries its
wait mode and timeout, drawn as Flux actually checks it: a Kustomization
with healthChecks and no wait is health-checked (as infrastructure is), one
with neither has no check at all, whatever its healthCheckExprs say.
Infrastructure is green, the Velero restore chain blue, wait: false dashed,
suspended Kustomizations grey.

With no argument or a .dot path it writes the full graph; with a .svg or .png
path it renders the transitive reduction (tred). Generated rather than
checked in, so it cannot drift from the manifests. Works with either yq
(mikefarah's or the Python wrapper): yq only converts YAML to JSON, jq builds
the DOT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model claude-opus-5-5, commit 075fc61d8336371f01a9d4a3e2a5f7a212225169.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This PR rewires the Flux dependency graph:

  • New edges: to gitea, influxdb and home-assistant.
  • Health checks: infrastructure now waits on the openbao StatefulSet; home-assistant and wordpress get wait: true; otbr is suspended.
  • authentik metrics: the chart's ServiceMonitor moves to monitoring-rules, and the hand-written radius metrics Service/ServiceMonitor is deleted.
  • New script: scripts/flux-graph.sh draws the graph.

I found no defect I could point at.

  • No cycles: gitea and influxdb depend only on the infrastructure layers.
  • The new openbao check counts: openbao runs updateStrategyType: RollingUpdate (kubernetes/infrastructure/openbao/helmrelease.yaml:57), so the StatefulSet check really tracks readiness.
  • No cold-start trap: bootstrap.sh calls bring_up_openbao before its "Waiting for infrastructure..." loop (kubernetes/bootstrap.sh:1944-1946), so the stricter infrastructure check doesn't block the restore.
  • Base and conflicts: mergeable is true. The base split/02-site-facts is still at the PR's base sha d6a9575. None of these changes is on master yet.

⚪ Unverified

  • authentik server ServiceMonitor selector (kubernetes/apps/monitoring-rules/authentik-servicemonitor.yaml:15-21): it assumes the chart's metrics Service carries app.kubernetes.io/component: server-metrics and a port named metrics. The commit message says this was checked against chart 2026.8.3 and the live Service. I can't read the chart, so I couldn't confirm it. If it's wrong, authentik server metrics stop being scraped and nothing reports it. The CNP already lets monitoring in (infrastructure/network-policies/authentik-ns.yaml).
  • Radius outpost metrics after deleting authentik-outpost-servicemonitors.yaml: the deleted file's own comment said the radius controller has no 9300 port, and that the operator's ServiceMonitor matches no Service. The PR says authentik 2026.8.3 fixed both. That is a claim about the external operator that I couldn't check. If it's wrong, radius metrics disappear silently. The commit says no alert rule uses that job.

🟡 Suggestions

  • kubernetes/clusters/homelab/home-assistant.yaml:18 (wait: true): home-assistant is now Ready only if every workload in it is Ready. That covers HA, mosquitto, zigbee2mqtt, esphome, shelly-manager, mdns-relay, mqtt-account-controller, ha-pg, and mosquitto-dynsec-test (whose own header says it is due for removal). zone-configurator, speech-to-phrase, and now evcc and trmnl all wait behind it. This is the cascade infrastructure.yaml:15-24 warns about, on a smaller scale. Before merging, check that every workload in the home-assistant namespace is Ready in production. Consider dropping mosquitto-dynsec-test first. The old "USB may CrashLoop" reason for wait: false no longer applies: deployment.yaml has no hostPath or USB any more, so removing that comment is right.
  • kubernetes/clusters/homelab/authentik.yaml:27 (and the other new gitea edges): these make gitea (wait: true, 15m timeout) a gate for eight Kustomizations, including the IdP. The edge is needed for a cold start. On a running cluster, though, a stuck gitea reconcile would hold back authentik, home-assistant, paperless and woodpecker changes. A short note in gitea.yaml saying how many Kustomizations depend on it would help whoever debugs a stuck reconcile later.
  • kubernetes/clusters/homelab/trmnl.yaml:24: trmnl only reads HA's API at runtime. With HA now wait: true, any HA-side workload that isn't Ready also blocks trmnl. Keep the edge only if trmnl really can't start without HA.

🟢 Nits

  • PR body: it says wait: true was added for llama-server and wiki. Both already had it (llama-server.yaml, wiki.yaml); the diff only changes home-assistant and wordpress, which matches the commit message.

Tests

The changes are manifest wiring, backed by the restore-test runs cited in the commits; I found nothing that tests the dependency graph itself. scripts/flux-graph.sh has no test, which fits the repo: ci-scripts-test.yml covers only the CI helper scripts, not operator tools like this one.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-opus-5-5`, commit `075fc61d8336371f01a9d4a3e2a5f7a212225169`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR rewires the Flux dependency graph: - **New edges:** to `gitea`, `influxdb` and `home-assistant`. - **Health checks:** `infrastructure` now waits on the openbao StatefulSet; `home-assistant` and `wordpress` get `wait: true`; otbr is suspended. - **authentik metrics:** the chart's ServiceMonitor moves to monitoring-rules, and the hand-written radius metrics Service/ServiceMonitor is deleted. - **New script:** `scripts/flux-graph.sh` draws the graph. I found no defect I could point at. - **No cycles:** `gitea` and `influxdb` depend only on the infrastructure layers. - **The new openbao check counts:** openbao runs `updateStrategyType: RollingUpdate` (`kubernetes/infrastructure/openbao/helmrelease.yaml:57`), so the StatefulSet check really tracks readiness. - **No cold-start trap:** `bootstrap.sh` calls `bring_up_openbao` before its "Waiting for infrastructure..." loop (`kubernetes/bootstrap.sh:1944-1946`), so the stricter `infrastructure` check doesn't block the restore. - **Base and conflicts:** `mergeable` is true. The base `split/02-site-facts` is still at the PR's base sha `d6a9575`. None of these changes is on master yet. ## ⚪ Unverified - **authentik server ServiceMonitor selector** (`kubernetes/apps/monitoring-rules/authentik-servicemonitor.yaml:15-21`): it assumes the chart's metrics Service carries `app.kubernetes.io/component: server-metrics` and a port named `metrics`. The commit message says this was checked against chart 2026.8.3 and the live Service. I can't read the chart, so I couldn't confirm it. If it's wrong, authentik server metrics stop being scraped and nothing reports it. The CNP already lets `monitoring` in (`infrastructure/network-policies/authentik-ns.yaml`). - **Radius outpost metrics after deleting `authentik-outpost-servicemonitors.yaml`:** the deleted file's own comment said the radius controller has no 9300 port, and that the operator's ServiceMonitor matches no Service. The PR says authentik 2026.8.3 fixed both. That is a claim about the external operator that I couldn't check. If it's wrong, radius metrics disappear silently. The commit says no alert rule uses that job. ## 🟡 Suggestions - **`kubernetes/clusters/homelab/home-assistant.yaml:18` (`wait: true`):** home-assistant is now Ready only if every workload in it is Ready. That covers HA, mosquitto, zigbee2mqtt, esphome, shelly-manager, mdns-relay, mqtt-account-controller, ha-pg, and `mosquitto-dynsec-test` (whose own header says it is due for removal). zone-configurator, speech-to-phrase, and now evcc and trmnl all wait behind it. This is the cascade `infrastructure.yaml:15-24` warns about, on a smaller scale. Before merging, check that every workload in the `home-assistant` namespace is Ready in production. Consider dropping `mosquitto-dynsec-test` first. The old "USB may CrashLoop" reason for `wait: false` no longer applies: `deployment.yaml` has no hostPath or USB any more, so removing that comment is right. - **`kubernetes/clusters/homelab/authentik.yaml:27` (and the other new `gitea` edges):** these make `gitea` (`wait: true`, 15m timeout) a gate for eight Kustomizations, including the IdP. The edge is needed for a cold start. On a running cluster, though, a stuck `gitea` reconcile would hold back authentik, home-assistant, paperless and woodpecker changes. A short note in `gitea.yaml` saying how many Kustomizations depend on it would help whoever debugs a stuck reconcile later. - **`kubernetes/clusters/homelab/trmnl.yaml:24`:** trmnl only reads HA's API at runtime. With HA now `wait: true`, any HA-side workload that isn't Ready also blocks trmnl. Keep the edge only if trmnl really can't start without HA. ## 🟢 Nits - **PR body:** it says `wait: true` was added for llama-server and wiki. Both already had it (`llama-server.yaml`, `wiki.yaml`); the diff only changes home-assistant and wordpress, which matches the commit message. ## Tests The changes are manifest wiring, backed by the restore-test runs cited in the commits; I found nothing that tests the dependency graph itself. `scripts/flux-graph.sh` has no test, which fits the repo: `ci-scripts-test.yml` covers only the CI helper scripts, not operator tools like this one. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=075fc61d8336371f01a9d4a3e2a5f7a212225169 -->
claude-bot commented 2026-10-17 20:25:41 +00:00
Author
Member
Copy Link
Copy Source

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).
claude-bot closed this pull request 2026-10-17 20:25:42 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1299