Files
homelab/kubernetes/bootstrap.sh
claude-botandClaude Opus 5 9cc20251fd
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
fix(edge): retire the legacy compose project before any role starts a container
Review on #474 found two blocking defects, both in exactly the place the
--check verification is blind: command and shell tasks are skipped, not
simulated, under check mode. So "verified with --check --diff" covered
everything except the two things that would actually have broken.

1. Ordering: a real run would have aborted.

   The legacy-project teardown lived in power-meter-sml, which the play runs
   last. zwave-gateway runs before it and does `docker compose up -d` for a
   container named zwave-js -- while the old shared project still owns that
   name. Docker enforces container names globally, not per project, so the
   command fails with "Conflict: container name already in use". That is not
   the missing-USB-device string the task tolerates, so the task fails and
   the play aborts before ever reaching the cleanup.

   Moved to pre_tasks, which is its correct home anyway: it is a migration
   step for both services, and putting it inside either one meant whichever
   ran first hit the conflict.

   The play now also declares edge_base_dir and edge_legacy_compose in its
   own vars rather than borrowing them from a role's defaults, since
   pre_tasks must not depend on role default resolution. And it removes the
   now-orphaned zwavejs.env left at the old shared project root.

2. Dangling handler.

   The libeatmydata task still notified "Restart edge compose services", a
   handler that was deleted with the old role. Masked today only because
   `creates:` makes the task report ok -- but the role's own comment says to
   delete that .so when the Alpine base bumps, and the next time it does,
   Ansible would fatally error with "handler not found". Now notifies
   "Restart zwave-js".

   Swept every notify in the new roles against every defined handler; that
   was the only unresolved one.

Also from review: kubernetes/bootstrap.sh referenced the deleted role.
Repointed to ansible/docs/edge-host.md.

Re-verified: ok=66 changed=25. The teardown moved from the power-meter-sml
group into pre_tasks and gained the orphaned-env removal, so the count goes
5 + 5 + 14 -> 5 pre-existing + 6 migration + 14 header repointing. Nothing
else moved.

The limit is worth stating plainly: this re-run still cannot prove the
teardown works, for the same reason it missed the bug. The first real apply
is the test.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 11:47:39 +02:00

397 lines
17 KiB
Bash
Executable File
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env bash
set -euo pipefail
# K8s Cluster Bootstrap — split into phases so CI can export access as early as possible.
#
# Usage:
# ./bootstrap.sh # run all phases locally
# ./bootstrap.sh talos # Talos apply + K8s bootstrap + kubeconfig (writes access-staging/)
# ./bootstrap.sh flux # reads access-staging/, installs flux, waits for infrastructure
# ./bootstrap.sh wait-apps # waits for apps/authentik/paperless to reconcile
#
# Flux git source (break-glass): the in-cluster Gitea that Flux normally pulls
# from is itself Flux-managed, so a cold rebuild cannot pull from it until the
# cluster already exists. Set FLUX_SOURCE=github to bootstrap Flux from the
# external GitHub mirror instead — the only way out of that loop:
# FLUX_SOURCE=github GITHUB_TOKEN=<pat> ./bootstrap.sh flux
# The mirror is kept current server-side by Gitea's native push-mirror. Full
# cold-rebuild procedure: kubernetes/RUNBOOK-cold-rebuild.md.
#
# Requires: talosctl, kubectl, flux, sops age key at ~/.config/sops/age/keys.txt
TALOS_DIR="$(cd "$(dirname "$0")/talos" && pwd)"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
ACCESS_STAGING="$REPO_ROOT/kubernetes/access-staging"
NODES=(k8s-1 k8s-2 k8s-3)
# No dedicated workers since the k8s-edge VM was retired (2026-07-02) —
# hardware-bound services live on the bare-metal edge-1 Arch host
# (ansible/docs/edge-host.md).
WORKERS=()
ALL_NODES=("${NODES[@]}" "${WORKERS[@]}")
DOMAIN="srv.renner.uno"
DNS_SERVER="10.2.40.1" # UDM (old PVE DNS VIP 10.2.40.53 decommissioned)
# Cluster API endpoint — Talos VIP (10.2.40.60) elected via etcd, fronted by
# UniFi DNS A record k8s.renner.uno. Defined in talos/patches/k8s-{1,2,3}.yaml.
K8S_API_HOST="k8s.renner.uno"
GITEA_HOST="gitea.renner.uno"
GITEA_OWNER="gitea_admin"
GITEA_REPO="homelab"
# Flux git source: "gitea" (default, in-cluster) or "github" (break-glass mirror
# for a cold rebuild — see the usage note above and RUNBOOK-cold-rebuild.md).
FLUX_SOURCE="${FLUX_SOURCE:-gitea}"
# GitHub break-glass mirror (github.com/konairius/Homelab), kept current by
# Gitea's native push-mirror. Overridable in case the mirror moves.
GITHUB_OWNER="${GITHUB_OWNER:-konairius}"
GITHUB_REPO="${GITHUB_REPO:-Homelab}"
FLUX_PATH="kubernetes/clusters/homelab"
SOPS_KEY="$HOME/.config/sops/age/keys.txt"
PHASE="${1:-all}"
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
is_tty() { [[ -t 1 ]]; }
step() {
local _ignored=$1
local label="$2"
if is_tty; then
printf "${BLUE}==>${NC} %s\n" "$label"
else
printf "==> %s\n" "$label"
fi
}
fail() { echo -e "\n${RED}FAILED:${NC} $1"; exit 1; }
ok() { echo -e "${GREEN}OK${NC}"; }
resolve_node_ip() {
local fqdn=$1
dig +short "$fqdn" A @"$DNS_SERVER" | head -1
}
phase_runs() {
[[ "$PHASE" == "all" || "$PHASE" == "$1" ]]
}
if phase_runs talos; then
echo -e "${YELLOW}=== Phase: Talos bootstrap ===${NC}"
command -v talosctl >/dev/null || fail "talosctl not found"
command -v kubectl >/dev/null || fail "kubectl not found"
[[ -f "$TALOS_DIR/patches/cluster.yaml" ]] || fail "cluster.yaml patch not found"
[[ -f "$TALOS_DIR/patches/cilium-cni.yaml" ]] || fail "cilium-cni.yaml patch not found"
[[ -f "$TALOS_DIR/patches/registry-mirror.yaml" ]] || fail "registry-mirror.yaml patch not found"
step 0 "Generating Talos configs..."
TALOS_GEN_DIR=$(mktemp -d)
talosctl gen config homelab "https://${K8S_API_HOST}:6443" \
--output "$TALOS_GEN_DIR" \
--force
# Strip the auto-generated HostnameConfig docs (conflicts with machine.network.hostname in per-node patches)
for f in "$TALOS_GEN_DIR"/controlplane.yaml "$TALOS_GEN_DIR"/worker.yaml; do
awk '
BEGIN { doc="" }
/^---[ \t]*$/ {
if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
doc=$0 "\n"; next
}
{ doc = doc $0 "\n" }
END {
if (doc !~ /\nkind: HostnameConfig\n/ && doc !~ /^kind: HostnameConfig\n/) printf "%s", doc
}
' "$f" > "$f.tmp" && mv "$f.tmp" "$f"
done
mkdir -p ~/.talos
cp "$TALOS_GEN_DIR/talosconfig" ~/.talos/config
step 0 "Applying Talos configs to nodes..."
apply_node_config() {
local node=$1 base_config=$2
local fqdn="${node}.${DOMAIN}"
echo " $fqdn:"
for i in $(seq 1 60); do
node_ip=$(resolve_node_ip "$fqdn")
if [[ -z "$node_ip" ]]; then
echo " attempt $i/60: DNS returned no IP"
sleep 10
continue
fi
if err=$(talosctl apply-config --insecure -n "$node_ip" \
--file "$base_config" \
--config-patch "@$TALOS_DIR/patches/cluster.yaml" \
--config-patch "@$TALOS_DIR/patches/cilium-cni.yaml" \
--config-patch "@$TALOS_DIR/patches/registry-mirror.yaml" \
--config-patch "@$TALOS_DIR/patches/${node}.yaml" 2>&1); then
echo " applied via $node_ip ${GREEN}(OK)${NC}"
return 0
fi
echo " attempt $i/60: $node_ip - ${err//$'\n'/ | }"
sleep 10
done
fail "Timeout waiting for $fqdn"
}
for node in "${NODES[@]}"; do
apply_node_config "$node" "$TALOS_GEN_DIR/controlplane.yaml"
done
for node in "${WORKERS[@]}"; do
apply_node_config "$node" "$TALOS_GEN_DIR/worker.yaml"
done
echo -n "Configuring talosctl... "
talosctl config endpoint "${NODES[@]/%/.${DOMAIN}}"
talosctl config node "${NODES[0]}.${DOMAIN}"
ok
step 0 "Waiting for Talos cluster formation..."
for i in $(seq 1 30); do
members=$(talosctl get members 2>/dev/null || true)
missing=0
for node in "${ALL_NODES[@]}"; do
grep -q "$node" <<<"$members" || missing=1
done
[[ $missing -eq 0 ]] && break
sleep 5
[[ $i -eq 30 ]] && fail "Talos cluster didn't form"
done
echo -n "Bootstrapping Kubernetes... "
talosctl bootstrap 2>/dev/null || true
ok
echo -n "Getting kubeconfig... "
for i in $(seq 1 30); do
if talosctl kubeconfig ~/.kube/config --force 2>/dev/null; then
sed -i "s|server:.*|server: https://${K8S_API_HOST}:6443|" ~/.kube/config
break
fi
sleep 5
[[ $i -eq 30 ]] && fail "Couldn't get kubeconfig"
done
ok
step 0 "Waiting for K8s nodes..."
expected=${#ALL_NODES[@]}
for i in $(seq 1 60); do
ready=$(kubectl get nodes --no-headers 2>/dev/null | grep -c "Ready" || true)
if [[ "$ready" -eq "$expected" ]]; then
break
fi
sleep 5
[[ $i -eq 60 ]] && fail "Not all nodes ready (got $ready/$expected)"
done
# Stash configs for subsequent phases / export-access step
mkdir -p "$ACCESS_STAGING"
cp ~/.talos/config "$ACCESS_STAGING/talosconfig"
cp ~/.kube/config "$ACCESS_STAGING/kubeconfig"
echo -e "${GREEN}Talos phase complete.${NC} Access files in $ACCESS_STAGING/"
fi
# Helper: load staged access files into home dir (for later CI steps)
load_staged_access() {
[[ -f "$ACCESS_STAGING/talosconfig" && -f "$ACCESS_STAGING/kubeconfig" ]] \
|| fail "access-staging/ missing talosconfig or kubeconfig (run talos phase first)"
mkdir -p ~/.talos ~/.kube
cp "$ACCESS_STAGING/talosconfig" ~/.talos/config
cp "$ACCESS_STAGING/kubeconfig" ~/.kube/config
}
if phase_runs flux; then
echo -e "${YELLOW}=== Phase: FluxCD bootstrap ===${NC}"
command -v kubectl >/dev/null || fail "kubectl not found"
command -v flux >/dev/null || fail "flux not found"
[[ -f "$SOPS_KEY" ]] || fail "SOPS age key not found at $SOPS_KEY"
case "$FLUX_SOURCE" in
gitea) [[ -n "${GITEA_TOKEN:-}" ]] || fail "GITEA_TOKEN env var not set" ;;
github) [[ -n "${GITHUB_TOKEN:-}" ]] || fail "GITHUB_TOKEN env var not set (break-glass mirror bootstrap)" ;;
*) fail "Unknown FLUX_SOURCE '$FLUX_SOURCE' (want gitea|github)" ;;
esac
[[ "$PHASE" == "flux" ]] && load_staged_access
echo -n "Creating SOPS age secret... "
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
kubectl create secret generic sops-age -n flux-system \
--from-file=age.agekey="$SOPS_KEY" \
--dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
ok
if [[ "$FLUX_SOURCE" == "github" ]]; then
step 0 "Bootstrapping FluxCD from the GitHub break-glass mirror..."
echo -e "${YELLOW}BREAK-GLASS: Flux source is github.com/${GITHUB_OWNER}/${GITHUB_REPO}, NOT in-cluster Gitea.${NC}"
echo -e "${YELLOW}Switch the source back to Gitea once it is serving again (see RUNBOOK-cold-rebuild.md).${NC}"
# GITHUB_TOKEN is read from the environment by `flux bootstrap github`.
flux bootstrap github \
--owner="$GITHUB_OWNER" \
--repository="$GITHUB_REPO" \
--branch=master \
--path="$FLUX_PATH" \
--components-extra=image-reflector-controller,image-automation-controller \
--personal \
--token-auth 2>&1 | tail -5
else
step 0 "Bootstrapping FluxCD from in-cluster Gitea..."
flux bootstrap gitea \
--hostname="$GITEA_HOST" \
--owner="$GITEA_OWNER" \
--repository="$GITEA_REPO" \
--branch=master \
--path="$FLUX_PATH" \
--components-extra=image-reflector-controller,image-automation-controller \
--personal \
--token-auth 2>&1 | tail -5
fi
step 0 "Waiting for infrastructure..."
for i in $(seq 1 60); do
ready=$(flux get kustomization infrastructure --no-header 2>/dev/null | grep -c "True" || true)
if [[ "$ready" -ge 1 ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: infrastructure not ready yet, continuing...${NC}"
done
step 0 "Waiting for Ceph cluster..."
for i in $(seq 1 60); do
health=$(kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
if [[ "$health" == "Ready" ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: Ceph not ready yet, continuing...${NC}"
done
step 0 "Waiting for Ceph Object Store..."
for i in $(seq 1 60); do
phase=$(kubectl get cephobjectstore -n rook-ceph migration -o jsonpath='{.status.phase}' 2>/dev/null || true)
if [[ "$phase" == "Ready" ]]; then
break
fi
sleep 10
[[ $i -eq 60 ]] && echo -e "${YELLOW}Warning: Object Store not ready yet, continuing...${NC}"
done
fi
if phase_runs wait-apps; then
echo -e "${YELLOW}=== Phase: Wait for apps ===${NC}"
command -v kubectl >/dev/null || fail "kubectl not found"
command -v flux >/dev/null || fail "flux not found"
[[ "$PHASE" == "wait-apps" ]] && load_staged_access
# Wait for a Flux Kustomization to become Ready=True. Prints a status line
# whenever the Ready/Message pair changes, plus a heartbeat every 60 s so a
# silent-but-progressing reconcile is still visible in CI output. Returns
# 0 on Ready, 1 on timeout (caller decides whether to fail or continue).
wait_kustomization() {
local name="$1" label="$2" max_iters="$3"
local interval=10 elapsed=0 prev_status=""
local timeout_min=$(( max_iters * interval / 60 ))
step 0 "Waiting for ${label} (timeout ${timeout_min}m)..."
for _ in $(seq 1 "$max_iters"); do
local line ready msg status
line=$(flux get kustomization "$name" --no-header 2>/dev/null || true)
# `flux get kustomization --no-header` columns:
# NAME REVISION SUSPENDED READY MESSAGE...
ready=$(echo "$line" | awk '{print $4}')
msg=$(echo "$line" | awk '{for(i=5;i<=NF;i++) printf "%s ",$i}' | sed 's/[[:space:]]*$//')
if [[ "$ready" == "True" ]]; then
printf " [%4ds] Ready ✓\n" "$elapsed"
return 0
fi
status="${ready:-?}: ${msg:-(no status yet)}"
# Print on first iteration, on state change, or as a 60-s heartbeat.
# 240 chars is enough to see the full Flux stalled-resource message.
if [[ "$prev_status" != "$status" ]] || (( elapsed > 0 && elapsed % 60 == 0 )); then
printf " [%4ds] %s\n" "$elapsed" "${status:0:240}"
prev_status="$status"
fi
sleep "$interval"
elapsed=$(( elapsed + interval ))
done
printf " [%4ds] TIMEOUT after %sm — full status follows\n" "$elapsed" "$timeout_min"
# Dump the full untruncated status + Events for the offending resources
# so the post-mortem doesn't need a second cluster trip.
flux get kustomization "$name" 2>&1 | sed 's/^/ /' || true
kubectl get events --sort-by=.lastTimestamp -A 2>/dev/null \
| grep -i -E "warning|error" | tail -20 | sed 's/^/ /' || true
echo -e "${YELLOW}Warning: ${label} not Ready, continuing anyway${NC}"
return 1
}
# Each app KS now owns its CNPG database (+ any data PVCs) directly — the
# per-app `seed-<app>` KSes that used to bootstrap them were folded into the
# app KSes when the seed layer was retired. Each wait below gates on the app
# KS, whose readiness includes its CNPG cluster coming up (initdb on a fresh
# cluster; a cold-rebuild data restore is the separate RECOVERY.md runbook).
# Timeouts stay generous to cover first-time cluster bootstrap on fresh Ceph.
# apps waits on infrastructure-ceph-cluster (Ceph mon/mgr/OSD bring-up
# is genuinely 5–10 min on a fresh OSD-wiped cluster).
wait_kustomization apps "apps" 90
wait_kustomization influxdb "InfluxDB (HR install + token Jobs)" 600
wait_kustomization woodpecker "Woodpecker CI (incl. wp-pg CNPG backend)" 600
wait_kustomization authentik "Authentik (incl. ak-pg CNPG backend)" 600
wait_kustomization monitoring "kube-prometheus-stack (Loki + Prom + AM + Grafana)" 180
wait_kustomization music-assistant "Music Assistant (incl. data PVC)" 300
wait_kustomization paperless "Paperless (incl. pprl-pg CNPG + data PVCs)" 600
wait_kustomization home-assistant "Home Assistant (HAOS backup → native restore)" 180
wait_kustomization audiobookshelf "Audiobookshelf (incl. library/metadata PVCs)" 900
wait_kustomization immich "Immich (incl. im-pg CNPG + library PVC)" 1800
# Wait for cert-manager to finish issuing every Certificate. Kustomization
# Ready only means the Ingress + Certificate manifests are applied; cert-
# manager then runs an async ACME DNS-01 challenge per Certificate (~30s–2m
# via Cloudflare). e2e's TLS check fails on the ingress-nginx default fake
# cert if any Certificate is still Issuing when the test runs.
wait_certificates() {
local max_iters=90 interval=10 elapsed=0 prev_status=""
local timeout_min=$(( max_iters * interval / 60 ))
step 0 "Waiting for all cert-manager Certificates Ready (timeout ${timeout_min}m)..."
for _ in $(seq 1 "$max_iters"); do
local total ready not_ready_names status
total=$(kubectl get certificate -A --no-headers 2>/dev/null | wc -l | tr -d ' ')
ready=$(kubectl get certificate -A --no-headers 2>/dev/null | awk '$3=="True"' | wc -l | tr -d ' ')
if [[ "$total" -gt 0 && "$total" == "$ready" ]]; then
printf " [%4ds] All %s Certificates Ready ✓\n" "$elapsed" "$total"
return 0
fi
not_ready_names=$(kubectl get certificate -A --no-headers 2>/dev/null \
| awk '$3!="True" {print $1"/"$2}' | head -5 | tr '\n' ' ')
status="${ready:-0}/${total:-?} Ready, waiting: ${not_ready_names:-(none)}"
if [[ "$prev_status" != "$status" ]] || (( elapsed > 0 && elapsed % 60 == 0 )); then
printf " [%4ds] %s\n" "$elapsed" "${status:0:140}"
prev_status="$status"
fi
sleep "$interval"
elapsed=$(( elapsed + interval ))
done
printf " [%4ds] TIMEOUT after %sm\n" "$elapsed" "$timeout_min"
echo -e "${YELLOW}Warning: not all Certificates Ready, continuing anyway${NC}"
kubectl get certificate -A --no-headers 2>/dev/null | awk '$3!="True"' | head -10
return 1
}
wait_certificates
echo ""
echo -e "${YELLOW}=== Verification ===${NC}"
echo -n "Nodes: "
kubectl get nodes --no-headers 2>/dev/null | grep -c Ready | tr -d '\n'
echo "/${#ALL_NODES[@]}"
echo -n "HelmReleases: "
flux get helmrelease --all-namespaces --no-header 2>/dev/null | grep -c "True" | tr -d '\n'
echo " ready"
echo -n "Ceph: "
kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph.health}' 2>/dev/null || echo "pending"
echo ""
echo -n "Ingress (local): "
curl -sk --connect-timeout 5 --resolve status.renner.uno:443:10.2.99.86 https://status.renner.uno -o /dev/null -w "%{http_code}" 2>/dev/null || echo "N/A"
echo ""
fi