konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Restore test 2/9: site facts in the Talos site patches #1298

Closed
claude-bot wants to merge 1 commits from split/02-site-facts into split/01-small-fixes
pull from: split/02-site-facts
merge into: :split/01-small-fixes
Conversation 2 Commits 1 Files Changed 9
+241 -44
claude-bot commented 2026-10-17 09:51:59 +00:00
Member
Copy Link
Copy Source

Part 2 of 9 (stacked on part 1). Facts that differ per site (homelab, qemu, hcloud) move into the Talos site patches, and what reads them follows.

  • site-homelab.yaml: IPv4 first in nodeIP.validSubnets (as the live nodes run), etcd advertisedSubnets, net.renner.uno/pod-table=47 label, and two inline manifests:
    • cilium-site ConfigMap: native routing, autoDirectNodeRoutes, directRoutingDevice: pod0, native-routing CIDRs. The Cilium HelmRelease reads it via valuesFrom; helm template renders production identically to before.
    • k8s-gateway-site ConfigMap: k8s-gateway's upstream resolvers; read via valuesFrom.
  • ResolverConfig per site: nodes ask k8s-gateway (10.96.0.11) first, so containerd resolves in-cluster names (gitea.renner.uno) inside the cluster from the first pod on.
  • k8s-1..3.yaml: node labels hardware.renner.uno/intel-gpu and net.renner.uno/iot-vlan (used by part 6).
  • site-qemu.yaml: the same facts for the qemu rehearsal site.
  • bootstrap.sh: installs k8s-gateway right after the CNI; merges the site's cilium-site values for the bootstrap Cilium install; creates Cilium's CNI subdirectory and installs Multus on a cold cluster.

Rollout (before merge): apply site-homelab.yaml + k8s-N.yaml node by node (drain, wait for Ceph HEALTH_OK). Then check the labels on the nodes and that kube-system/cilium-site and kube-system/k8s-gateway-site exist (Talos creates inline manifests only on a config change; talosctl upgrade-k8s if not). The HelmReleases fail on a missing valuesFrom ConfigMap, so the ConfigMaps must exist before this merges.

🤖 Generated with Claude Code

Part 2 of 9 (stacked on part 1). Facts that differ per site (homelab, qemu, hcloud) move into the Talos site patches, and what reads them follows. - `site-homelab.yaml`: IPv4 first in `nodeIP.validSubnets` (as the live nodes run), etcd `advertisedSubnets`, `net.renner.uno/pod-table=47` label, and two inline manifests: - `cilium-site` ConfigMap: native routing, `autoDirectNodeRoutes`, `directRoutingDevice: pod0`, native-routing CIDRs. The Cilium HelmRelease reads it via `valuesFrom`; `helm template` renders production identically to before. - `k8s-gateway-site` ConfigMap: k8s-gateway's upstream resolvers; read via `valuesFrom`. - ResolverConfig per site: nodes ask k8s-gateway (10.96.0.11) first, so containerd resolves in-cluster names (gitea.renner.uno) inside the cluster from the first pod on. - `k8s-1..3.yaml`: node labels `hardware.renner.uno/intel-gpu` and `net.renner.uno/iot-vlan` (used by part 6). - `site-qemu.yaml`: the same facts for the qemu rehearsal site. - bootstrap.sh: installs k8s-gateway right after the CNI; merges the site's `cilium-site` values for the bootstrap Cilium install; creates Cilium's CNI subdirectory and installs Multus on a cold cluster. **Rollout (before merge):** apply `site-homelab.yaml` + `k8s-N.yaml` node by node (drain, wait for Ceph HEALTH_OK). Then check the labels on the nodes and that `kube-system/cilium-site` and `kube-system/k8s-gateway-site` exist (Talos creates inline manifests only on a config change; `talosctl upgrade-k8s` if not). The HelmReleases fail on a missing `valuesFrom` ConfigMap, so the ConfigMaps must exist before this merges. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 1 commit 2026-10-17 09:51:59 +00:00
feat(talos): site facts live in the site patches, and what reads them follows
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
review-bot reviewed: comment
Details
d6a9575fcd 
What differs between sites moves out of cluster-wide config into the Talos
site patches (site-homelab.yaml, site-qemu.yaml), and the consumers read it
from there:

- nodeIP.validSubnets lists IPv4 first, as the live nodes run. Talos keeps
  the order (siderolabs/net FilterIPs) and the first entry becomes the
  primary node IP; with fd02:46::/64 first, kube-apiserver exits with
  'service IP family "10.96.0.0/12" must match public address family'.
- etcd.advertisedSubnets (VLAN 46, a fact of the home network) moves
  unchanged from cluster.yaml to site-homelab.yaml; the rendered production
  config is byte-identical.
- Pod routing (routingMode, autoDirectNodeRoutes, directRoutingDevice) and
  the native-routing CIDRs are a cilium-site ConfigMap per site patch, read
  by the Cilium HelmRelease via valuesFrom. bootstrap.sh merges it the same
  way: site values first, HelmRelease values on top. Production renders
  unchanged (helm template, TLS secrets excluded).
- Each site patch states its node resolver list (ResolverConfig: k8s-gateway
  first, then the site's resolver) and k8s-gateway's upstream as a
  k8s-gateway-site ConfigMap, read by the k8s-gateway HelmRelease via
  valuesFrom. bootstrap.sh installs k8s-gateway right after the nodes are
  Ready, from the HelmRelease under the site values, and Flux adopts it:
  until k8s-gateway answers, every lookup waits out the node's 2 s read
  timeout and pods resolve nothing, Flux included.
- bootstrap.sh creates Cilium's cni.confPath subdirectory with a
  hostNetwork pod per node (/etc/cni is read-only to the kubelet on Talos)
  and applies Multus right after Cilium; without it the kubelet reports
  "cni plugin not initialized" and nothing schedules on a cold cluster.
- site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 (table 47
  exists only where the site routes pod traffic into it); k8s-1..3 label
  hardware.renner.uno/intel-gpu and net.renner.uno/iot-vlan, the hardware
  each machine has. The workloads select on these labels in later changes.

Rollout order for production: the cilium-site and k8s-gateway-site
ConfigMaps must exist before the HelmReleases reconcile (valuesFrom is not
optional). Talos creates inline manifests on a config change but never
updates them; changes go through talosctl upgrade-k8s. The node labels go
on with talosctl patch, no reboot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model claude-opus-5-5, commit d6a9575fcd644146916eb09c787ce8b2bbc043da.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This PR moves per-site settings (Cilium routing, k8s-gateway's upstream resolvers, etcd's advertised subnet, node IP order) out of the shared config and into the Talos site patches, which the HelmReleases now read through valuesFrom. bootstrap.sh follows: it now creates Cilium's CNI subdirectory, installs Multus after Cilium, and installs k8s-gateway once the nodes are Ready. The homelab and qemu parts look consistent, and the merge order matches Flux's (site values first, HelmRelease values on top). It is mergeable onto split/01-small-fixes, which hasn't moved, and no newer master commit touches these files. The hcloud site wasn't brought along: bootstrap.sh hcloudtest now stops at the new check, and only part 9/9 fixes that. One yq flag and one k8s-gateway chart key need confirming.

⚪ Unverified

  • k8s-gateway chart keys (kubernetes/talos/patches/site-homelab.yaml:78-83, kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:95): the forwarder now reads /etc/resolv.conf, so production DNS depends on the chart honouring deployment.dnsPolicy / deployment.dnsConfig.
    • If the chart ignores them, the pod falls back to ClusterFirst. Its upstream becomes kube-dns, and the main CoreDNS forwards renner.uno back to 10.96.0.11 (helmrelease.yaml:100-104). The loop plugin would then stop k8s-gateway.
    • I couldn't read the chart (it is external, oci://codeberg.org/k8s-gateway/charts/k8s-gateway). The commit message mentions a helm template check for Cilium only. Please render k8s-gateway with the site values and confirm dnsPolicy: None and the nameservers appear on the Deployment before rollout.
  • yq -y (kubernetes/bootstrap.sh:1459): -y exists only in the Python yq (kislyuk). Every other yq call in the script uses no flag or -r, which both flavours accept. Line 1321 extracts the same kind of values without -y and helm accepts the JSON.
    • The vmtest image installs Alpine's yq package (ci/images/vmtest/Containerfile). I couldn't confirm which flavour that is.
    • If it is mikefarah's yq, line 1459 fails under set -e. Dropping -y works with both.
  • Cilium --wait before Multus (kubernetes/bootstrap.sh:1393-1410): Cilium writes its conflist to /etc/cni/net.d/multus/, which the runtime doesn't read. So nodes stay NotReady until Multus runs, and Multus is only applied after helm upgrade --install cilium … --wait.
    • hubble.relay.enabled: true (cilium/helmrelease.yaml:164-166) adds a pod-network Deployment that can't schedule before then.
    • Whether --wait blocks on it depends on the helm version's readiness rule and the relay's maxUnavailable, neither of which I can see. Part 9 keeps the same order, so it may have passed in practice. A vmtest run on this commit would settle it.

🟡 Suggestions

  • hcloud site gets no site values (kubernetes/talos/patches/site-hcloud.yaml, kubernetes/bootstrap.sh:1327-1331, 1460-1464): the PR body lists hcloud as a site, but site-hcloud.yaml is unchanged.
    • hcloudtest sets SITE=hcloud (bootstrap.sh:793), so the talos phase now fails at the new cilium-site check. With Python yq it fails earlier, at .cluster.inlineManifests[] on null, without the friendly message.
    • split/09-restore-test-harness adds the hcloud cilium-site (tunnel/VXLAN), k8s-gateway-site and etcd.advertisedSubnets. Until that lands, every commit in the stack breaks hcloudtest.
    • Either move that block into this PR or say in the body that hcloud is broken until 9/9. This is not blocking because hcloud probably didn't work on the base either: cluster.yaml pinned etcd to 10.2.46.0/24, which the hcloud nodes don't have. I didn't verify that.
  • Re-runs of the cni-confdir step (kubernetes/bootstrap.sh:1355-1387): if an earlier run left a cni-confdir-$node pod in Failed, kubectl apply leaves it unchanged and the wait times out after 5 minutes. Deleting any leftover pod (--ignore-not-found) before applying would make the step idempotent, like the rest of the phase.

🟢 Nits

  • kubernetes/infrastructure/cilium/helmrelease.yaml:36 still says "THIS IS THE ONLY PLACE CILIUM IS CONFIGURED". Part of the configuration now lives in the site patches' cilium-site.
  • bootstrap.sh:1370 pulls docker.io/library/busybox:1.37 straight from Docker Hub during a cold rebuild. Docker Hub's anonymous rate limit is the risk a pull-through cache was added for in the past (7a52345). It's minor, since it's one small image per node.

Tests

There are no automated tests for bootstrap.sh beyond shellcheck. bootstrap.sh vmtest is the repo's harness for this path. The PR says Cilium's helm template output is unchanged for production, but doesn't say a vmtest or hcloud run was done on this commit. A vmtest run would show three things: the yq -y question, the Cilium --wait/Multus order, and k8s-gateway coming up with dnsPolicy: Default. A helm template of k8s-gateway with the homelab site values would show whether the chart honours the DNS keys.

Recommendation

Looks reasonable — no blocking issues; comments above are optional. Before rollout, please confirm the k8s-gateway DNS keys render and fix or document the hcloud gap.

🤖 **Automated review** — model `claude-opus-5-5`, commit `d6a9575fcd644146916eb09c787ce8b2bbc043da`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR moves per-site settings (Cilium routing, k8s-gateway's upstream resolvers, etcd's advertised subnet, node IP order) out of the shared config and into the Talos site patches, which the HelmReleases now read through `valuesFrom`. `bootstrap.sh` follows: it now creates Cilium's CNI subdirectory, installs Multus after Cilium, and installs k8s-gateway once the nodes are Ready. The homelab and qemu parts look consistent, and the merge order matches Flux's (site values first, HelmRelease values on top). It is mergeable onto `split/01-small-fixes`, which hasn't moved, and no newer master commit touches these files. The hcloud site wasn't brought along: `bootstrap.sh hcloudtest` now stops at the new check, and only part 9/9 fixes that. One `yq` flag and one k8s-gateway chart key need confirming. ## ⚪ Unverified - **k8s-gateway chart keys** (`kubernetes/talos/patches/site-homelab.yaml:78-83`, `kubernetes/infrastructure/k8s-gateway/helmrelease.yaml:95`): the forwarder now reads `/etc/resolv.conf`, so production DNS depends on the chart honouring `deployment.dnsPolicy` / `deployment.dnsConfig`. - If the chart ignores them, the pod falls back to `ClusterFirst`. Its upstream becomes kube-dns, and the main CoreDNS forwards renner.uno back to 10.96.0.11 (`helmrelease.yaml:100-104`). The `loop` plugin would then stop k8s-gateway. - I couldn't read the chart (it is external, `oci://codeberg.org/k8s-gateway/charts/k8s-gateway`). The commit message mentions a `helm template` check for Cilium only. Please render k8s-gateway with the site values and confirm `dnsPolicy: None` and the nameservers appear on the Deployment before rollout. - **`yq -y`** (`kubernetes/bootstrap.sh:1459`): `-y` exists only in the Python `yq` (kislyuk). Every other `yq` call in the script uses no flag or `-r`, which both flavours accept. Line 1321 extracts the same kind of values without `-y` and helm accepts the JSON. - The vmtest image installs Alpine's `yq` package (`ci/images/vmtest/Containerfile`). I couldn't confirm which flavour that is. - If it is mikefarah's yq, line 1459 fails under `set -e`. Dropping `-y` works with both. - **Cilium `--wait` before Multus** (`kubernetes/bootstrap.sh:1393-1410`): Cilium writes its conflist to `/etc/cni/net.d/multus/`, which the runtime doesn't read. So nodes stay NotReady until Multus runs, and Multus is only applied after `helm upgrade --install cilium … --wait`. - `hubble.relay.enabled: true` (`cilium/helmrelease.yaml:164-166`) adds a pod-network Deployment that can't schedule before then. - Whether `--wait` blocks on it depends on the helm version's readiness rule and the relay's `maxUnavailable`, neither of which I can see. Part 9 keeps the same order, so it may have passed in practice. A `vmtest` run on this commit would settle it. ## 🟡 Suggestions - **hcloud site gets no site values** (`kubernetes/talos/patches/site-hcloud.yaml`, `kubernetes/bootstrap.sh:1327-1331, 1460-1464`): the PR body lists hcloud as a site, but `site-hcloud.yaml` is unchanged. - `hcloudtest` sets `SITE=hcloud` (`bootstrap.sh:793`), so the talos phase now fails at the new `cilium-site` check. With Python yq it fails earlier, at `.cluster.inlineManifests[]` on null, without the friendly message. - `split/09-restore-test-harness` adds the hcloud `cilium-site` (tunnel/VXLAN), `k8s-gateway-site` and `etcd.advertisedSubnets`. Until that lands, every commit in the stack breaks `hcloudtest`. - Either move that block into this PR or say in the body that hcloud is broken until 9/9. This is not blocking because hcloud probably didn't work on the base either: `cluster.yaml` pinned etcd to 10.2.46.0/24, which the hcloud nodes don't have. I didn't verify that. - **Re-runs of the cni-confdir step** (`kubernetes/bootstrap.sh:1355-1387`): if an earlier run left a `cni-confdir-$node` pod in `Failed`, `kubectl apply` leaves it unchanged and the wait times out after 5 minutes. Deleting any leftover pod (`--ignore-not-found`) before applying would make the step idempotent, like the rest of the phase. ## 🟢 Nits - `kubernetes/infrastructure/cilium/helmrelease.yaml:36` still says "THIS IS THE ONLY PLACE CILIUM IS CONFIGURED". Part of the configuration now lives in the site patches' `cilium-site`. - `bootstrap.sh:1370` pulls `docker.io/library/busybox:1.37` straight from Docker Hub during a cold rebuild. Docker Hub's anonymous rate limit is the risk a pull-through cache was added for in the past (7a52345). It's minor, since it's one small image per node. ## Tests There are no automated tests for `bootstrap.sh` beyond shellcheck. `bootstrap.sh vmtest` is the repo's harness for this path. The PR says Cilium's `helm template` output is unchanged for production, but doesn't say a vmtest or hcloud run was done on this commit. A `vmtest` run would show three things: the `yq -y` question, the Cilium `--wait`/Multus order, and k8s-gateway coming up with `dnsPolicy: Default`. A `helm template` of k8s-gateway with the homelab site values would show whether the chart honours the DNS keys. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. Before rollout, please confirm the k8s-gateway DNS keys render and fix or document the hcloud gap. <!-- pr-review:sha=d6a9575fcd644146916eb09c787ce8b2bbc043da -->
claude-bot commented 2026-10-17 20:25:39 +00:00
Author
Member
Copy Link
Copy Source

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).
claude-bot closed this pull request 2026-10-17 20:25:40 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1298